Hmm weird, for me it works great with an xbox 360 and playstation 4 gamepad (tested both on ubuntu and the default gnome terminal).
Would be great if you can try it with one of those, just to check if it's a gamepad specific issue or still occurs regardless