Sorry, but I would have let the first guy have a piece of my mind. Something along the lines of "Hey asshole - that's me and you're fucked up for thinking that way."

I guess this comes under the heading of "No good deed goes unpunished".