My neighbor is an elementary school teacher, and she works in the inner-city school district and she said dealing with students from different racial and ethnic backgrounds, whose parents may have different views than most in western society, have limited English ability, etc.
From someone I worked with for a very limited time who was a special ed teacher for students with mild disabilities (who spent most of their time in a normal classroom), it was the office politics, and not being taken advantage of by those who knew just what they were doing. And also balancing work with having a life.
My mom was a special ed teacher for those with severe to profound disabilities, and she said figuring out how to not give your life away to deal with the paperwork. For a regular teacher I can imagine this is comparable to classroom plans and grading.