A central subpoint of your post that I didn't address in my earlier response is the issue of Deference. To paraphrase what I think is the underlying question, "Should objections by the people who believe they have been harmed by a particular statement be given greater weight?" Kind of parallel to the notion of "standing" in law.

Should the views of Black people on the use of the N-word, for example, be given more weight than the views of White people who use the term?

Do women have a special right to object to a rape joke?

I don't think a "yes" or "no" answer gets to the root of the problem. The real question for me is "Why isn't everyone angry over a racial slur or the trivializing of a crime against women?”

Words are not just words. They are actions. An offensive statement is not first and foremost a statement of opinion. It is the pushing of a course of action, and just as actions have consequences for the person who is being addressed in the statement, these consequences should have consequences for the speaker.