Technically, even "Hey, you!" can be offensive in the wrong tone of voice. It's just words, true enough, but it's the intent behind them that counts.
A knife is just a knife, they're necessary for most people, but one used in a hostile manner isn't "just a knife" anymore, it's a weapon. And words can be used to wound as well.
I was raised to believe words have power. Mystical/Magickal power on top of very real power.
Take my Great Grandmother (Called Biscuit Grandma because she was always baking). In her 90's she was still going strong, a very nice old lady, sweet, whatever you think of of a woman in an old timey dress with her conservative shoes and her hair always neatly done up in a bun.
She never, ever, ever used profanity.
Well never is not actually accurate: She used it only when most needed.
So when Biscuit Grandma said 'Shit Jamie' (directed at my mother) this is a phrase that had the immediate impact of shutting up my mother (if you ever met my mother you would understand how much magic that takes) and it is one phrase I still clearly recall - and the whole circumstance leading up to it.
or when my Uncle Jerry did something abusive and she threw out the word 'Fuck' as she threw the cast iron skillet at him... Again, powerful word, coupled with a powerful and most accurate throwing arm.
Yeah so what does this have to do with derogatory words - Well all words have a certain power to them. How we use them, how often we use them does a lot.
A person who throws out the word 'fag' all the time tells far much more about self than they do of the people they are throwing the word at.
While its true each time a gay person is called a Fag, God ripped the wings of an angel and throws it screaming and burning down to earth.... or perhaps it only scars the person called a Fag a little emotionally... Either way, its a horrific, damaging thing that takes an angel and ruins it.
We live in a world which is evolving socially at an incredible rate of speed. As it is, society is no longer actually judging the victims of derogatory words, they are judging the person using it, which should be doing more damage to THAT individual, however most who rely on such words are so blind to reality that they don't know enough to be hurt.