A lot of people dismiss these scandals by saying, "That's between him and his wife." But I think that viewpoint is crazy, and we absolutely should care when a politician cheats.
The most important quality that we want in elected officials is for them to be trustworthy. We need to have people in office who will put their constituents' interests above their own. We can't watch over their shoulders every second. Anyone capable of getting himself elected will generally be better at hiding his crimes than we are at discovering them. (The exceptions are the scandals we know about. But obviously there are far more incidents that should have been scandals, except the corrupt politicians successfully kept them hidden.)
For a government official, trustworthiness is much more important than intelligence, competence, ability to communicate, or any other quality. If a politician is incredibly skilled but untrustworthy, he'll only be better at screwing us over for his own gain. That's the exact opposite of what we want.
Ordinarily, it's difficult to assess someone's trustworthiness. A successful politician will be very skilled at making people think they can trust him, whether or not that's the truth. Since we can't distinguish between candidates on the most important quality, we fall back on secondary issues - usually whether they claim they'll do stuff we agree with. Lacking better information, we simply have to hope this sub-optimal method of picking who to vote for works out for the best.
But occasionally we do get better information. When a politician cheats on his wife, he is demonstrating that he finds his own personal pleasure to be more important than the promises he made to and happiness of someone he ostensibly loves and sees every day. Once he's shown himself to be that kind of person, why would you think he would act in a trustworthy manner toward millions of people he's never even met?
Would you hire a pickpocket to manage your bank? Would you hire a tax-cheat to oversee the Treasury Department? (Oh, wait...) So why would you hire someone who lies to his wife to tell you the truth?
And this is true whether the cheater is a Democrat or Republican.
By contrast, it doesn't matter when an athlete or actor has an affair. Tiger Woods' job is to whack a ball into a hole with a stick. Nothing about his job requires us to trust him. (There are too many witnesses and cameras for him to have an opportunity to cheat at golf.) And since an actor's job is to pretend to be someone he's not, you could make the case that an ability to fool his wife demonstrates just how talented of an actor he is.
So when you're talking about people outside of the government, I would agree that any extra-marital affairs are none of our business. But with government officials, basic common sense dictates that once they've demonstrated that they can't be trusted, we shouldn't trust them.
Also, I should clarify that I'm only talking about when a politician cheats on his or her spouse. I don't care and don't think anyone else should care if it turns out a politician posed for racy pictures, or went to a sex club, or (for unmarried politicians) are outed as gay or found to be promiscuous. None of those are issues of trust, and they have nothing to do with how they would govern.