Your save is screwed now. You can continue without Adder and the memoir pages quest until the end of the game.
If you really want to get that two thing, in the first world, you need to also get into the Vortex on top of Branch castle. Not only go to the throne room to take the castle back. By doing that, the bug will not happen in the Merged world.