I like the concept! Except I couldn't really pass the final door in the first level for some reason. Might just be a skill issue on my part, but the button wouldn't activate whatever I put on it. Thus, this comment is based off only the first level.
I like the concept of the literal flow of time though. But get why it makes you bigger. Maybe you just grow bigger the longer you live and that accelerates your age or something. Thats my idea of it.
Neat idea, good luck with the jam!!