The concept is, as is mentioned plenty already, really good. And if you were planning on increasing the scope of this game in the future, I think it lends itself perfectly for that. Because while it is a mind boggling main mechanic, which is usually annoying when adding new mechanics, here you could simply activate or deactivate parts of the level, like enemies with complex patterns or other physics objects. It could reduce the amount of active things the player has to watch out for without increasing the distance players have to walk to get from one obstacle to the next. Instead you could also add more colors to take it more in a puzzle direction
Quick example of something you could do, which I believe is easier to implement than one might think: A wizard-like enemy shooting switch balls. Getting hit by them thus means you switch colors. Then in your level you'd have two in two different colors, one on the left and one on the right maybe. Getting hit by one would disable that one, and activate the other. I think that sounds fun
And now some more specific notes about the game:
You explain you can double jump before you actually need it. I forgot about the double jump and thus got stuck, while if you missed the double jump instruction in a level where you actually need it, you can always go back to read it again.
The mechanic that elements can move when switching colors is in my opinion too weird to merit being in the game. Yes, it might make designing levels a tad easier, but it is very calming for the player to be able to see exactly what the world would look like once it switches, and this just adds another layer, which I think is unnecessary.
The one level where you dash under and jump on the right side, and then similarly on the left, is really good. It had a planned pitfall where you get stuck on the left, but you make absolute sure they know the mechanic to get around it
The last level is funny, because you can beat it in under 2 seconds