Would you have a problem with a mosque being built in your area? Not more than any other type of building.
From what I’ve heard, a lot of people are up in arms about mosques being built around the US. Why? People tend to be afraid of what they don’t understand.
What’s the big deal? People are afraid that a mosque would change their lives, some akin to @worriedguy‘s points. The reaction is caused the same fears that generate assumptions and judgments about a person based on their apparent religion/beliefs is the source of this issue.
This is human nature, and the more we fight it, the more we will spin our wheels. The problem is the unwillingness to gain the understanding about something so that there is no fear around it.
Take the one example of the morning call to prayers, if I understood the purpose behind it, I would not mind. I used to live in a neighborhood with a church that chimed 3x/day: 6am, 12noon, and 6pm. Sometimes it woke me up in the morning, but I never minded because I chose not too; I took it as a sign that all was well. We all have the freedom to see things from a perspective of love and compassion rather than fear.