Thank you for playing!
Yeah, only the unchanging one adds to the flower score with all others adding to the pasture score. Filling the lower bar does slightly increase the chance that modifiers are applied to other squares, but that is the only effect of clicking the unchanging one other than the score difference. Clicking the different flowers only increments the "level", but there are more modifiers that show up at higher levels regardless of which you click.