If a spellcaster is riding a flying creature with flyby (peryton, for example), can the caster deliver a touch spell to a target along the flyby path and then get out without provoking an opportunity attack? Can I use my action in the middle of my mounts flyby action, or does the mount's movement have to be finished before I use my action?

Yes it can, even without the Flyby feature

Can I use my action in the middle of my mounts flyby action

First and foremost, there is no specific "flyby action" the peryton has to take, it just doesn't provoke opportunity attacks. See Monster Manual page 251:

Flyby. The peryton doesn't provoke an opportunity attack when it flies out of an enemy's reach.

A peryton rider doesn't provoke opportunity attacks as well. See Do I draw opportunity attacks when my mount uses flyby? for more details.

Considering casting a spell in-between, you are definitely allowed to do that:

You can break up your movement on your turn, using
some of your speed before and after your action. For
example, if you have a speed of 30 feet, you can move
10 feet, take your action, and then move 20 feel.
(PHB 190, "Breaking Up Your Move")

So yes, you can fly in, cast a spell and fly out without provoking an OA.

The fun thing is — your mount doesn't even have to have the Flyby feature in order to evade opportunity attacks. Any controlled mount can take the Disengage action:

It moves as you direct it, and it has only three action options: Dash, Disengage, and Dodge.

Therefore, when it isn't busy dashing or dodging, it can disengage and be safe from opportunity attacks for the whole turn:

If you take the Disengage action, your movement doesn't provoke opportunity attacks for the rest of the turn.

That is one of the advantages of being mounted in combat.

First things first.

If your mount doesn't provoke Opportunity Attacks, neither do you

The mount is using its own movement, not yours, so irrespective of whether the mount is being directly controlled, or if it is being independently controlled, opportunity attacks against you are contingent on whether your mount can be targeted.

So there's nothing wrong with you flying down, delivering a touch spell, and then flying away...

... Unless you're letting your mount act independently. In that case, it's still possible, but you need to modify your Actions.

In this case, what you instead need to do is Ready the touch spell in question, and set the trigger condition to "When my mount brings me adjacent to the creature". Then your mount flies down to them during its own turn, and when it gets adjacent, you use your Reaction to deliver the spell. Then the mount flies away, and because it doesn't provoke Opportunity Attacks, you're not at risk either.

Can you attack in the middle of your mount's movement? No.

You and your mount have separate turns on the same initiative, so you decide which one goes first, but you can't act in the middle of your mount's turn, because it isn't your turn. This was clarified by Jeremy Crawford in his twitter feed a couple of years ago.

You could ready an action to make your attack when you get close enough, though.

Does Flyby protect you from opportunity attacks? Yes.

1) When you are mounted, your mount moves and you are taken with it; you aren't using your own movement.

[Your mount] moves as you direct it, and it has only three action options; Dash, Disengage, and Dodge. A controlled mount can move and act even on the turn that you mount it. - Player's Handbook, p.198

2) You only draw opportunity attacks when you use movement to leave an enemy's reach.

You can make an opportunity attack when a hostile creature that you can see moves out of your reach... You also don't provoke an opportunity attack ... when someone or something moves you without using your movement... - Player's Handbook, p.195

3) By the combination of 1 and 2, you do not draw opportunity attacks while mounted; only your mount does.

4) Because of a special rule relating to Mounted Combat, an opportunity attack against your mount can be directed at you instead.

If the mount provokes an opportunity attack while you're on it, the attacker can target you or the mount. - Player's Handbook, p.198

5) By the contradiction of 4, if your mount does not provoke an opportunity attack, you cannot be targeted by one.

6) Thus, by the combination of 3 and 5, if your mount has an ability (such as flyby) or uses an action (such as Disengage) that prevents opportunity attacks against it, it also prevents opportunity attacks against you.

(You could also have the mount use Disengage, but Flyby allows you to use Dodge or Dash at the same time as not provoking attacks.)