You can side with him only if you aren't on Lady Hale's route (the concept is that the lady is corrupting you).
Oh that makes a lot of sense now, thanks