I have the Katryx blade equipped on Ryen, and am using the Armor Crushing skill it gives him on Lace, however, each time I use it Ryen says he didn't hit it correctly to break her armor, I've tried it maybe 5-ish times with no change, what am I doing wrong?

It has a random chance to hit or miss. I guess you were just unlucky. Keep trying and it should work eventually.