I think every community benefits from having “challenging” members. Not only ideologically challenging, but also just plain irritating. They keep us socially and intellectually flexible, which is a healthy thing. It’s like the body’s immune system: if it’s overly sheltered from environmental irritants as it develops, it becomes overreactive and creates problems where none exist.
There are limits, though. It’s reasonable to ask of members that they not go out of their way to cause trouble. Baiting others for sport, perpetuating feuds, hounding and serial-flaming, etc. are blatantly anti-social behaviors and nothing good will come of that.
As a mod, it’s not always clear when to take action (beyond merely modding a question or comment) against a member for anti-social behavior. We get quite a few calls to rein in this or that member because they’re offensive in some way, and we often agree that they’re offensive. But “offensive” can’t be the standard for taking action against someone.
In practice, the mods sometimes act as a firewall against the kind of social vigilantism that would drive away irritating members. There too, it’s often hard to draw the line between legitimate critique and irritation-fueled attack; people are sometimes dismayed when posts “calling out” irritating members get taken down, while offensive posts are left standing. While we understand that this doesn’t win us any popularity points, we have concerns other than popularity at heart. And we’ll certainly admit that we’re far from perfect at all this.
So no, I think it would be a terrible idea to allow popular consensus to determine who stays and who goes.