Try to go back to the head maid then she'll instruct you to do something.
I figured it all out. So I'd been massively confused. I found the other door on the bottom right of the dining room that I somehow missed and got the signet ring to get the key.
Also got a gamebreaking bug since I thought the solution was to change back in the bathroom. Ended up getting the armory and weaponry keys but when I came back from the weaponry I went to the south west entrance and my character got locked into walking back and forth because of the guards there.