Barbara Ehrenreich at Barbara’s Blog knows why you just don’t talk about people that way:
This doesn’t excuse Imus, because he misses a crucial point: That an insult, used often enough, becomes the exclusive property of the insulted. Take the word “bitch,” as applied to any woman with the guts to offend. At first it stung, but then we appropriated it for ourselves. Elizabeth Wurtzel wrote a feminist book called Bitch, and there’s a feminist magazine of the same name. I can call my sister “bitch” in a jokey, tough-gal, way. But you can’t call her that, not if you’re a guy, unless you want to step outside with me.
In other words, in a normal social context, it matters who the remark came from. By the same token, Chris Rock or Dave Chappelle can make jokes that Imus or Leno or Letterman absolutely cannot make. That’s not unfair, it’s just a question of keeping your footing on the marshy ground of social interaction.
But people who have too much money or status and too little contact with what’s going on at street level can fail to understand that.
And speaking as someone who’s getting crankier, older, and (weirdly) whiter by the minute, I’m entirely within the limits of propriety to call Imus a cranky old white guy.