I think this happened when I lifted them really up, after some point the game started to play itself
Maybe restricting mouse movement to a certain area could help in these cases too
