Not just the tempo, but when you actually try to press in time with the rhythm, the input timing feels a bit off, which adds to the difficulty.
The 1-2-3-4 count makes sense! I think I could get by without it, though if the game demands a strong sense of rhythm and the input window is super strict, it might feel pretty tough for players.
Maybe you could make the stages shorter, or even wrap things up after just a single battle per round. Honestly, stopping at a point that feels "a little too short" to you as the creator might actually be the sweet spot for players.
It does feel like features have piled up a bit, so trimming things down could make the experience much smoother and more comfortable. Keep up the good work!