When I read your question, I was confused for a moment when you said “prejudiced toward” instead of prejudiced against. Clearly you mean prejudiced against.
But it raised an interesting question to me. Is it equally bad to be prejudiced towards someone, as it is to be prejudiced against them? If you are prejudiced towards, then you will favor them, perhaps over others. Perhaps on spurious grounds.
It occurs to me that for every prejudice towards, there could be an equal and opposite prejudice against, and vice versa.
This question is difficult for me to talk about because it paints with such a broad brush. It suggests that all prejudices are inherently unfair. It seems to me that there are many grounds on which prejudice is perfectly acceptable. So, without specifying what kind of prejudice(s) you are talking about, it’s nearly impossible to answer the question in a meaningful way.
I also am struck by the last part of your question, where you ask what gives people the right to let their prejudices affect others. This use of “rights” is interesting. It suggests there are some kind of inherent rights to live free of prejudice that we are born with. This, of course, is not the case. Rights have nothing to do with it. People beat up other people until other people organize to prevent it.
It’s all about power. I don’t mean mere physical power, though. I mean the power of mutual self-interest and logic and persuasion and education. No rights exist unless you have the power to protect them.
You can’t keep your prejudices from affecting others. You are who you are. The only way to change someone is to educate them, or use other means to modify their behavior. Asking people “what gives you the right” is the powerless way to try to get something. It’s begging. It rarely works. To stop behavior that bothers you, you have to project your own power in any way you can. Sometimes a simple request is enough. Other times, you have to work much harder, organizing, and agitating, and staying in other people’s faces until they give you what you want.