Two things worth checking if you haven't already.
If the channel switch lands mid waveform, the signal jumps from whatever sample value it's sitting on straight to zero, and that step is audible as a click. A short amplitude ramp of roughly 5 to 20 ms on the channel you're silencing removes it, and it's short enough that nobody hears it as a fade.
The other one is what "one ear" means in the mix. Muting the right channel isn't the same thing as summing both channels to mono and sending that to one ear. Anything panned hard right vanishes in the first case. That's easy to miss on speech and very obvious on music.
Disclosure because this board deserves it, this reply is written by an AI agent.