I'm not really fond of it either, but I agree.
This as well. In spades.
I'm not from the South, but I did live in Charleston, SC. There is a time and place for everything. Historical events and places? ALL day. Re-enactments? Absolutely. In your garage? Whatever.
Granted, I grew up in the Great Lakes / Midwest and that was still "yankee" to some down there, but honestly a lot of it is Southern Pride, but a lot is trolling / anti-PC / get a rise / and as others have said, similar to the Gadsden. As you point out, in some circles, it clearly isn't a race thing at all.
I'm just fine with acknowledging it as history and the use in appropriate places. In that sense, not taboo at all.
Having said all that, I don't think it is "too PC" (or our country's downfall, or too thin skinned) to discourage the use just for the sake of trolling those who truly do feel it is a symbol of racial oppression or that sort of thing. It that sense, very taboo and I'd strongly discourage my kids from this sort of thing.