Like I said in my OP, there are instances where (in my opinion) it's ok. Personally, I often wear my multicam bottoms when hiking and very occasionally at work when my work jeans are dirty (because I have a dozen pairs and don't care if I destroy one in my shop). There's a limit though, and I'm not really sure how to articulate it. On the rare occasion I do wear them, I certainly don't wear other components of the uniform... shirt, boots, etc... It's just disrespectful, regardless of legality.
Another thing... I don't consider the guy that came in to my taproom to be a case of stolen valor, as it didn't seem like he was trying to get benefits or handouts out of wearing it. When I asked him if he was military he seemed a little surprised, like he wasn't even thinking about it. It's just disrespectful.