Generally speaking? Only one, and only for specific jobs. You want charm on the social jobs, physics on most of the work jobs. Mainly because higher factors means gaining stats faster in those areas.
Plus wits for exp gain.
The game's calculations are exponential, so doubling the number far more than doubles the output. Getting there sooner is valuable enough to be worth it.
Or, y'know, just skip a few days at a time and don't worry about all that nerd shit.
Otherwise? No, it's not really useful for most of your characters. The ones on the combat team, maybe, and that's about all.