From the holy scrolls:
You can only have four colors on the screen at once. Black is a color. The four colors include all shades, all tones, all hues. Gameboy does not support semi-opacity. The four colors apply to the whole screen, including sprites. You cannot have one four color palette for your background and another for sprites. This jam does not allow for a fifth experimental color like 'default screen background'. Four is the total amount of colors you are allowed. You may use three or two. Five is right out.
The four colors you choose do not have to be the default palette you've found on Wikipedia. Lospec.com has loads of four color palettes you could try. You can even make your own.
You can change your palette mid-game so long as the new palette is four colors. You cannot change your colors mid-scan-line or any other sneaky trickery. If you have more than four colors on the screen at any one time it is not obeying the rules.
Yes, the gameboy had three colors plus transparency on sprites. No, this is not a restriction you need to obey in your competition entry.
You can change the palette with a light switch or when it's night time so long as there are only four colors on the screen at one time. Are there four colors on the screen? Yes? Congratulations, you abide by the rules. Are there five colors? Do two of the colors blend with opacity to make a new color? Wow buddy, looks like you're in more than four color territory!
'Four colors maximum' means 'four unique colors'. A color is a unit measured by its unique properties such as hue, saturation and value (or whatever other color system you subscribe to). You could have red, blue, green and yellow on your screen, but 'a slightly darker shade of red' would count as a new color. You could not have fifty shades of blue and argue that you're only using 'one color'.
Yes, the illusion of a fifth color counts as a fifth color. Do you quickly flip the palettes to make an extra color? Does it have an extra color on the screen? Welcome to five coloursville friendo, population: you.
No, dithering does not count as an illusion of a fifth color, as long as you are sticking to the 160x144 resolution.
If you want to argue how your five colors are special, do it in your game jam submission. I'm sure the reviewers (your fellow game jammers) will understand... 👀
If your next question starts with "but what if" the answer is no. No, you can't do your special idea. It breaks the rules.
"Can I use shaders?"
Does your game have four colors? Is the resolution 160x144? Does your game art adhere to the spirit of the jam - i.e. does it look and feel like it's on a gameboy? Yes? Go ahead.
"But-"
Then no.