If the quest for the construction brigade is not active, going to to yellow blocks would just open the tram map.
First you should click on the quest (the icon at the top center), regulate (or not) the emotion of contempt and then chose where to go. Yes, maybe it's a bit too complicated, I'll think about making it more straightforward.
"pressing what you were prompted rather than being about making the correct decision" <=== yes, for now it is more or less like that. I'm planning to expand it in the next few weeks.