There's a really cool concept at the heart of this one - a shame it doesn't really work out in practice. The core mechanic is definitely the kind of thing good jam entries are normally made of!
Sadly, sometimes when we try and do all the bits of something at once, the result is none of the bits end up shiny. It might be better in future to focus on either story/character/setting or funky mechanics rather than both, especially under jam conditions.