After finding the seal in the temple, you needed to head down to the middle room below the starting area in the cave. If you get close to either wall, the golem breaks it for you :)
The idea was to open a path while moving slowly, with an unkillable enemy nearby that tracks you only by sound. I should have made that much clearer
