The vote you cast is not "largely irrelevant". When you cast your vote you may not be voting for candidate X directly but you're still casting a vote that supports that candidate.

Also, in many states, electors are required to vote for the candidate that receives the most votes. As a matter of fact, so called "faithless electors" (those that buck the system) are very rare (total of 8 since 1900) and have never influenced the outcome of a presidential election:

From History.com:

Presidential electors in contemporary elections are expected, and, in many cases pledged, to vote for the candidates of the party that nominated them. While there is evidence that the founders assumed the electors would be independent actors, weighing the merits of competing presidential candidates, they have been regarded as agents of the public will since the first decade under the Constitution. They are expected to vote for the presidential and vice presidential candidates of the party that nominated them. Notwithstanding this expectation, individual electors have sometimes not honored their commitment, voting for a different candidate or candidates than the ones to whom they were pledged; they are known as “faithless” or “unfaithful” electors. In fact, the balance of opinion by constitutional scholars is that, once electors have been chosen, they remain constitutionally free agents, able to vote for any candidate who meets the requirements for President and Vice President. Faithless electors have, however, been few in number (in the 20 century, one each in 1948, 1956, 1960, 1968, 1972, 1976, 1988, and 2000), and have never influenced the outcome of a presidential election.
Honestly, it sounds like you're not really clear on how the system works and that's leading to some confusion on your part.

If it helps make up your mind, just about every die-hard liberal in the land is all for replacing the electoral college with a popular vote. If that doesn't sway your opinion, nothing will.

Here's another little tidbit that's very interesting and should completely put to rest any thoughts of eliminating the electoral college:

Barack Obama received 3.3 million more votes than Mitt Romney in the Nov. 6 election, but won 3.6 million more votes than Romney in just four cities — Chicago, Philadelphia, New York and Los Angeles. He won those margins without much of a campaign. Now, imagine an Obama candidacy free of the need to appeal to Ohio factory workers, Colorado cattlemen, Iowa hog farmers and Virginia police officers, and you start to get the picture.