Recently-ish, Critical Role has introduced a concept that I had never heard of before: passive Insight. Now, it makes sense to me conceptually, as well as being useful for that table and their trigger-happy Insight checks, but I had never heard of it.

Where did this concept originate? Is it from a previous edition, or is it one of Matthew Mercer's homebrew rules?

The Player's Handbook has a section for "Passive Checks" for all ability scores

These rules are found on page 175, entitled "Passive Checks":

Passive Checks

A passive check is a special kind of ability check that doesn't involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.

Here's how to determine a character's total for a passive check:

10 + all modifiers that normally apply to the check

If the character has advantage on the check, add 5. For disadvantage, subtract 5. The game refers to a passive check total as a score.

For example, if a 1st-level character has a Wisdom of 15 and proficiency in Perception, he or she has a passive Wisdom (Perception) score of 14.

The rules on hiding in the "Dexterity" section below rely on passive checks, as do the exploration rules in chapter 8.

In the player's handbook itself, the only explicit use of passive checks is on Perception checks, to handle whether characters who are not actively searching for something will notice a hidden object or creature. But in theory, this can be applied to any check.

So in this context, a passive Insight check allows a creature who is very bad at being deceptive (in general or in this specific circumstance) to immediately be called out by the DM. This can simplify play in more mundane settings, and while still allowing players to take actions if they have prior reasons to suspect the behavior of the character they are interacting with.

There are general rules for "passive checks" in 5e's skills chapter (Player's Handbook p.175), which "passive insight" is really just a specific implementation of. That didn't start with 5e, though. In 4th Edition, they had a general 'passive skills' concept ("Checks without rolls") that was along the lines of "you're always taking 10 with some skills". Insight was one of those. But the idea may predate even that, since there's long been an issue of knowing that you rolled a 2 on your perception or insight (or equivalent skills by other names) rolls and being aware that the information you got isn't trustworthy.

I wouldn't really call this Mercer's homebrew, both because it's kind of in the book, and because it's hardly Mercer's in specific, as (based on googling) there were discussions of Passive Insight in 5e going back as far as when the game was still being called "DnD NEXT".

Passive ability checks are explained in chapter 7 of the 5e Player's Handbook. While passive perception is the most commonly used passive check, any ability check can be made passive that the DM desires.