Oh, now I got the root cause of the issue: I am playing on a laptop's touchpad and sometimes tap the right bottom corner of the it which is causing a right mouse button click. I guess the easiest fix is just to add the same behavior on the rmb.
