Nothing quite like watching the car in front of you swerve as you see the baseball cap turn to face the passenger. There are situations where it is OK to not face the person you are speaking to.
You are exactly right though, some people just don't care. The problem will go away eventually when self driving cars are the norm, we'll only have the deliberate (and mechanical failure) accidents.