Having already been a member of a site where other users can actually take away points for answers with which they disagree, I vastly prefer the Fluther system.
It comes down to this for me: some people (like, almost all of them) would rate answers based on ideology instead of fact, although I can see how some would confuse the two. Sometimes, well-thought-out answers of various schools of thought would be downgraded because the rest of the people who came along didn’t believe in that system.
As a result, the level of points (or gold or lurve or whatever) would come to mean less about the person’s overall quality, and more about how well they fit into the dominant paradigm. I’m not sure this is something I, personally, would enjoy here any more than I did at the other site.
Remember, sometimes there are only a few champions for an idea that turns out to be correct. If we always put the smackdown on dissent, we miss a lot.