Sure there's a difference, but unless you are a take the gun out on special occasions and otherwise a closet/bedside safe queen, I find carrying unchambered practically negligent. (I hate safeties as well but at least they are pretty negligible slow down with practice) Racking before being in the game not only costs you crucial time in an appreciable quantity, it is a liability in a sense because now if it does make the difference, the threat has another weapon (especially if you were drawing against someone with a knife or something). Watch Policeactivity video's and the like (raw bodycam type footage w/ out commentary) Not all of those situations are applicable, but there's enough that demonstrate having both hands free could be questionable especially if someone's in your face already.
It's no skin off my back at the end of the day, but my frustration is out of concern for those doing that potentially costing themselves. I think if you carry you should be able to have the confidence and attentiveness to not have a ND, and if still concerned, opt for a gun with a easily swept safety.