How fascinating. This man (and let's keep in mind that we're looking at an adult man here, not a kitten, a bunny, or some other cute little animal meant to make you go "aww") actually managed to start a family. He apparently knows how to have sex, given that he has a son. (Though, considering his wife doesn't give a damn about him, Toby could literally be anyone's.) Yet, he has zero clue what to do with his own life: all he knows is how to let people walk all over him, and then cry and whine about how much it hurts and how bad he has it. That is not the mindset of someone who once dared to take on the responsibility of a family.
By the way, it's not even a given that Toby meant him when asking about his father. Maybe Harvey was originally just a virtual husband created for Toby's mother, who was abandoned by her real husband and is coping with the crisis this way. Even that scenario strikes me as far more interesting.