Kinda depends on how you are measuring it.
It sounds like you are just going by elected presidents, but quite a few were multiple-term presidents, and those presidents had multiple elections with different opponents.
And sometimes, a losing opponent would go on to win a later election.
Also, no one ran against George Washington, twice.
Out of 59 elections (if you include Washington), I think there have been 49 white guys and one white lady who have lost a US Presidential election at least once (and may or may not have gone on to be President in a later election).
https://en.m.wikipedia.org/wiki/List_of_United_States_presidential_elections_by_popular_vote_margin
I am exactly as fun at parties as you would imagine.