for marginalized people, seeing that oppressive behavior is not tolerated makes them feel more safe. If it makes you feel unsafe, you might be relating more to people who perpetuate oppression than you are to people who are being oppressed. There's also the aspect that the more privilege you have, the more you feel entitled to a space, so when you are told that you are only conditionally allowed, that is threatening to that feeling of entitlement. But most marginalized people feel only conditionally allowed all the time in every space, so no one is giving you an experience that they have not already had. It might seem unfair or uncomfortable but that is because re-balancing when you are used to being the 'winner' means having some perks taken away.
The privileged poor are some of the worst exploitation and oppression sympathizers. Economics is the oppression that most rarely leads to a rejection of the system, probably because it's the one where the carrot is largest and closest. People really think they can get it
I have dealt with many people who use the same derails when discussing slurs. I am sick of the same old shit. Derails like "well how do we discuss these important things if I can't even use the word 'l*me'" which is a red herring because a slur-use of the word l*me is not ever going to help fight ableism, and the same applies to other slurs. Or "I would have listened and cared about this important issue if you had just been more polite" which has never happened. People who are going to change when someone points out their error are not going to refuse to change because someone didn't point it out in the right way. Calling out oppressive behavior results in defensiveness or concern, and the only thing that decides which it is is the values of the person who is being called out.