I don't remember clearly why it works this way but someone also said that "H SKILL ATK" also depends on how much you slept + Hole Exp uses non-natural numbers but fractional ones like -> 0.4857x2, so that's why h-damage bonus is small therefore the increase of h-damage is felt only after a long play time.
BTW - I didn’t really dive into sex stat topic so much because it wasn’t that interesting for me, since in principle all the h-damage depends mainly on CONST stat than something else and the maximum hole skill value you can reach seems to be only 150 but you can’t realize this big h-damage if you don’t have a lot of CONST stat which is why you are still going to rely on CONST stat more instead of these hole skill values.
I may be wrong about something but it’s kind of like this.