He's speaking more in the context that each candidate has a group that are going to vote for them no matter what, what is left is those in the middle. If some of the people here are in the middle then their vote- technically due to our 2-party system (not saying it's right, but that's the way it is... for now)- should go one way or another, not a 3rd way, and thus their vote that could have gone to Romney instead gets tossed aside and lowers his total by one in that state. Meanwhile, another person in the middle, for whatever foolish reason, votes for Obama, and there is one more vote for O and one less vote for R because the first person, more or less, "threw their vote away."