Ykhjuy

I am going to be a first-time DM for my group. While I was planning the first session, I just thought about my players interrogating (torturing) a creature.

If a character tries an Intimidation (or Persuasion) check, do they know if they succeeded in intimidating (or persuading) their target? Could the captive enemy act convinced and lie to them?

In my experience, the players should know the result of their roll. You don't need to tell them bluntly yes you succeeded or no, you can add flavour to it - say they rolled a 10 on intimidation and if it's not enough to get the prisoner to spill everything to the PCs, you can comment 'he twitches at you, eyes dancing between you, but his lips are sealed'

Alternatively if you want the prisoner to try and defend against an intimidation/persuasion, you can have him make a roll against the PCs roll.

Usually yes, a player should know if their check succeeded or failed.

You could make it more ambiguous if you wanted to though as you mentioned in your last sentence. If your NPC is trying to hide some information and the intimidation check failed, you could roll a deception check for them. In this case the NPC might pretend to be frightened and give false information under the pre-tense that the player characters believe they intimidated them. In this case, only a successful insight check competed against the deception check would reveal the truth.

They know it worked, but they can't know the consequences

Let's say the PC is scaring a poor goblin. You, the DM, ask the player to roll a Charisma (Intimidation) check. The player rolls 18, 22 total — a pretty high result, so the player can guess he succeeded. You, the DM, describe the goblin being trembling with fear.

From the game perspective, the character can say the goblin is definitely scared.
So yes, he knows his intimidation worked. But does this mean the goblin cannot lie?

Being good at intimidation does not turn you into a lie detector. Moreover, a scared person probably will lie, because of the fear. They won't tell you the truth, they will tell you they think you want to hear.

For example, Horde of the Dragon Queen adventure describes this kind of reaction:

Captured kobolds are terrified: they say whatever they think the questioner wants to hear

The PCs may or may not know whether they have failed depending on how the skill check is set up. Obviously the PCs would know if they failed to intimidate a bandit into backing down, but they wouldn't necessarily know whether they're being deceived or not.

The scenario you gave seems like a good place for a Contest, as described in the Player's Handbook, p.174.

The PC and the monster make an ability roll each, and then compare the results to determine who wins. If the monster is just going to hold out against torture, then it would probably be a Constitution roll versus Charisma(Intimidation). If the monster wants to pretend to break while actually lying, that's obviously a Charisma(Deception) roll versus the PCs' Charisma(Intimidation).

Since the PCs shouldn't necessarily know if they've been bluffed, it's a good idea to roll your monster's deception behind a screen in this case, and of course don't tell them you're rolling Deception.

Players know what players know and Characters know what Characters know.

Joe is playing Silk the Thief. Silk encounters a vault door for Farln the Mad Trapper. Joe rolls Find Traps and gets a decent but unimpressive roll. GM: "You find no traps." Joe: "Wait, There arn't any traps? Or there IS a trap and I didn't find it?" GM: "Silk found no traps." GM leans back with a grin Joe can decide that Silk is feeling paranoid and search again, or he can decide Silk is good enough to find even the worst Farln has to offer and open the door.. its up to him.

Some successes are obvious.. you hit the orc. Some, are not. If the party is successful on the intimidation, the creature should give honest information, or reveal its lack of information. If they fail, the creature may lie, or clam up, unintentionally give bad info, or any other non-helpful response you can think of. Now, if it lies, you would give your PCs a Sense Motive / Detect Lies roll as appropriate, but I would usually make that roll for them so they don't Meta-Game know they failed. Depends on your players honestly.

Whether they got the prisoner to talk is obvious.

Is he talking? Then you got him to talk. (Mechanically you can handle this several ways, but the simplest is to just assume that they crank up the pressure until he talks. Unless you're under very tight time limits or the prisoner is likely to escape, there's not really any other outcome.)

Whether he's lying, or holding something back, is not obvious.

This is where your players learn an important fact about torture.

Before we roll skill checks we should know what success and failure look like. The interrogation is going to continue until the party believes they've gotten the answers they want from the prisoner. Success means that those answers are correct. Failure means they're incorrect or incomplete.

If the guy has no reason to lie or hold back, then they can't fail. Just tell them everything he knows.

Suppose he does want to lie. That's a Deception check. Since the PHB doesn't spell this out: the DC for a Deception check to hide information is the Passive Insight of whoever you're talking to. Hopefully the players will be smart enough to have their highest-Insight guy involved in the interrogation.

(If anyone asks if they can "make an Insight check", I recommend giving a steely glare and asking what exactly their character is doing to "gain insight".)

The Deception check should be a hidden roll. If it fails, then you tell the player a plausible lie. If it succeeds, tell them the truth. Either way, tell them they're pretty sure from the guy's body language and tone of voice that it's true. This summarizes the whole process of asking questions, getting lied to or distracted, scaring the guy again, asking more questions, etc. into one roll that determines the outcome we care about.