For anyone with the same problem as me, you have to kick the servant out of your party. Then they can be assigned to things.
Viewing post in How do I assign servants?
I'm just playing the browser version. But I figured it out so its all good.
Btw, you might want to explain somehow that servants can't be in your party if you want to assign them. Either that or make it so that they automatically get kicked out of the party if you assign them.
Love the game btw, strikes a good balance between fun and hard.