If that doesn't work, enter the fortress with the invisibility potion FIRST, THEN use the allure potion.
