As the very proud dad of an Army E6 I think you're just fine. I wear ball caps all the time. A couple have flags, one has a "punisher" patch which was gifted to me by a friend's son (dad was a SEAL, his son is). I also have a sweatshirt, hat and t-shirts with my son's last unit on them from his most recent deployment. The few times I've been asked if it was because of my service I clearly state "no, it's because of my son's and a friend's son". Always asked to thank them as well as for the families support of them.