You asked a fair question and that guy is a stand up guy as well. It certainly doesn't sound like you were being confrontational. What a great interaction. Hope he comes back so the two of you can chat.

I have a hat VERY similar to that. It was thrown in as a free gift with an online order from some retailer. It fits well so I wanted to wear it. Rather than walk around with an empty velcro space (which looks stupid), I put an OD American Flag patch on it as the Flag seemed like the best option to avoid trying to look like some fake operator and the OD matched the hat. I didn't serve but I do love America and don't mind advertising that. The Flag is a symbol we all share.

If a vet asked me if I served based on my appearance while wearing the hat, I would make it immediately clear that I did not...however I'm not such a stand up guy because I wouldn't give up my hat...it just fits too good and a good hat is hard to come by .