I'm planning a new campaign where the villain will force the party to do his bidding, starting from level 3.

The opening will be an encounter with the villain, where he will "ask" them to do not-so-nice things. A good-natured (or even neutral) predictably will refuse to do so, because it involves a massacre of innocent village, for instance.

I want to intimidate them into submission, simply by showing how great the difference in strength between them and this villain, but not kill them. This villain stat should stay the same across every encounter, but slowly become beatable later. Think of it like a Darth Vader, I guess?

I will be using 5e, and this is part where I believe the problem is: the bounded accuracy means that you can't really show how great the power difference is. I mean, the players having a +4/5 attack bonus vs the villain having +7/8 will still allow them to think "well, we are 5 people, we can still try". Ramping up the AC has its limits.

At first, I was planning to "reveal" the attack bonus, "okay, so that's a 7 plus 8. I believe that's a hit?" to freak them out, but I have doubt that it will deliver the message. A very damaging spell/ability may freak them out alright, but at level 3, I fear killing them out outright (I almost always roll in open and never fudge an open roll)

Is there a way for me to accomplish this without railroading them into the choice? Or is 5e the wrong system to do this?

You don't

I usually hate this type of answer but unfortunately this sort of thing is rarely done well at tables and should be avoided. Particularly in a first session this will be considered railroading.

No matter how strong the initimidation is the players will still think they have a chance to win and are extremely unlikely to give in. This results in three options:

1. TPK, the players will refuse to do what is asked and would rather fight to the death.


2. Your players micraculously win and destroy your carefully planned campaign.


3. The PCs run away. This might be a welcome outcome for you but it is still unlikely they will do what you want them too.

... the villain will force the party to do his bidding ...

This is almost a definition for railroading. You shouldn't be forcing your party to do anything. You can present the villian as evil and describe his actions but you should never dictate how your PCs respond to it. For experience it is extremely unlikely this scenario will go the way you want without railroading.

Some Alternatives

Straight up forcing your players to do something through intimidation is a bad idea. But there are other ways that you could achieve a similar result.

• 
  Blackmail them. Hold a town to ransom, capture the PCs little sister, something suitably evil and compel the PCs that way. This may still not get the result you want but will portray the villain as suitably evil.


• 
  Trick them. Have the villain pose as the quest giver, give them vague or inaccurate reasons for what he wants done. Maybe the people they are sent to kill "killed my son", which is of course a lie. A villain lying to the party is totally ok and something I have done in the past. This is the most likely approach to get the players to commit terrible acts, make them think they weren't terrible while they were doing them. Warning some players may not takes this well. You will need to judge based on your own group of players.


• 
  Run it as a cutscene. Before the game starts you describe the situation, the players are in the midst of the preparation to commit this terrible act. Describe how the powerful enemy inspired terror in them and they choose to do his bidding. Most importantly they regain control of their character before they commit the act. They can then choose to not do it if they want to. Never force a PC to do something they don't want to, if you do you are denying them their agency.


• 
  Have it part of the campaign setting. Estabilish the dominance of the villain as part of your campaign setting, the PCs are already living under the thumb of the bad guy. Have it part of the characters backstory how they came to be under his control. _{Thanks to Nitsua60 in chat for the idea}
  

I'd also suggest you watch this video from Matt Coville on Running the Bad Guys. He introduces a villain called Karalel the Vile and its a great example of how to introduce a powerful and evil enemy. You could also check out the episode of dice camera action where Chris Perkins first introduces Strahd (I'm not sure which).

Session 0 and Campaign Expectation

It is possible that all the advice I have given is totally inaccurate, your players may enjoy this kind of play. If so they wouldn't be like any of the players I have ever met but that doesn't mean it is wrong.

If this is something you believe your players will be onboard with you need to set it up in the campaign expectation during session 0. Explicitly explain that you would like to be able to force the characters to do thing through intimidation. If they aren't onboard then you shouldn't do it. I expect this will be the case, if I'm wrong however, I wish you luck.

Fish with a worm and a hook rather than with a stick of dynamite

Is there a way for me to accomplish this without railroading them into
the choice?

No. What you described is a railroad. If your players are OK with a railroad game, then press on. If not, stop now and change your approach. Put the hard choices in front of them and see how they respond. Then riff off of their response.

Or is 5e the wrong system to do this?

A lot of systems are the wrong system for this.

I would advise you that you can try to influence them to go in a particular direction, but if they resist, you need to be flexible enough to see where those decisions by the players takes the game. You will most likely find some great new things happening based on how they try to get out of this mess / difficult choice.

(This approach served me well in numerous games and in numerous campaigns as a DM).

Let their efforts to evade or overcome this compulsion become the new "plot" such as it is.

Fun all around.

Is the plot or the story more important to you, as the DM?

I think you might want to answer that question before the next session. The story that comes from your game, your campaign, is what the players did when confronted with hard choices, dangers, challenges, etc, and also the consequences of their decisions.

Have you and your players discussed the chance that this campaign may end in grand and glorious failure? In other words, is the story about what they did, even if in the end their greatest efforts weren't enough? (Feanor's story was like that).

That's a story too. Some of the most memorable D&D campaigns I was in had catastrophe as the end play. The trick to making this successful and fun is in making this about the choices the players make, and maybe a few unlucky rolls if the dice, rather than about your domination of their characters. You own all of the cards in the deck: what's the point in dealing them a losing hand?

Mike Mearls has this to say about how fickle the dice(Fate) can be:

the dice will be cruel to you, but you will soldier on (PHB, p. 4)

“I’d rather die!”

Player’s choose their character’s response to situations and the above is a perfectly sensible choice. Your only sensible response is the kill the PCs, complement the players on their moral integrity and start planning your new campaign. Are you cool with that?

I’m thinking of Darth Vader and I cannot think of any time the heroes voluntarily helped him. I remember Leia being tortured, I remember Han futility firing his pistol at Vader until forcefully (ha) disarmed and Luke choosing to plunge to his death (as he thought) because, you know, heroes.

It’s not the mechanics of D&D that work against you here - it’s the expectations.

“Sign me up!”

There’s the other type of player who when tempted by the dark side will grab it with both hands. You know, like Darth Vader.

If they go this way then you need to be prepared to find out just how depraved your friends can be which can be confronting for everyone.

Not railroading means letting the story go where the player’s want

Which might not be where you want or expect. Or are even comfortable with.

Don’t use mechanics to communicate

As I used to say to my children: “Use your words.” Talk to the player’s about the themes you want to explore and how you want to explore them. Listen to what they want.

Let the player’s make the running and have the villain enjoy it

Let the villain make it clear that every achievement the player’s have made has only served the villain’s agenda.

They saved the town? That allowed the villain to capture the princess. They saved the princess? That allowed the villain to steal the Sword of Ultimate Awesomeness. They recovered the sword? That allowed the villain to unleash the etc. etc.