Ignorance of others comes from never meeting others. You live in your own community and hear shit about others, and you believe it because they look different. They look like they could do that stuff they are rumored to do.
Fear is intertwined with ignorance. We tend to be afraid of that which we don’t know. We get all angry about them, and how they want what we have and are going to kill us if they get a chance. Not all of these fears, I should point out, are unreasonable.
There’s something else I would throw in the mix, and that is the economy. When people are not doing well, it can make them feel better if they find a scapegoat. Right now, the major new scapegoat is immigrants and Muslims. They are the fearful others who come from outside, and we must defend ourselves, so the thinking goes. And worse, some people think “we must kill them.” Thus lynchings and hate crimes and intimidation. Get out you dreadful other (fill in with the hated epithet of your choice). You don’t belong here. Get out before I kill you.
So yes, racists are afraid of others. Often times, they remain afraid of others even when sitting down to dinner with them. That’s pretty scary.