I think it goes against the nature of the site. I have repeatedly seen people comment on what a warm and welcoming community this is, especially juxtaposed against the rest of the internet. Most anonymous-ish community websites turn into places people go to vent their frustration with the rest of the world in a place that has no real-life consequences. It’s not pretty.
Not having a downvote option is one of many design decisions that I think has helped to foster the gracious and thoughtful spirit of the community. If you really disagree with a comment your only recourse is to take the time to write out exactly why you don’t like it. More than once, I have done this and had a very good dialogue with the person I disagreed with, and we’ve come to a mutual understanding of each other. Clicking a “Crappy Answer” button would never be able to accomplish that.
And believe me, I have wanted to “Crappy Answer” people very badly, and have lost my temper and said nasty things I later regretted more than once.