The game really needed some effects on hit and at least some basic scoring.
Also, your bug might be tied to the fact that you seem to autostart the music and the timing might get wrong depending on how much time it took the browser to do so. Manually running the song on player pressing the first button might fix this.