Hmmm… I know a couple computer programmers who feel the same way you do. If you’re pretty right-brained that could be a good one for you. As long as your work is sound, you’ll mostly be left alone. If you’re more of a creative type, you could perhaps get into web design.
If you’re a bit more “hands on,” you could work a back of the house job in a restaurant. You’ll still have to communicate with the rest of your team and your kitchen manager/ head chef, but if you find the right restaurant the back of house team can become a pretty tight-knit group. It’s definitely a lot more casual types of interaction than working with customers, but the flip side of that is that everyone says whatever is on their mind. If you just need a quick job right now, starting as a dishwasher will let you work your way up to greater responsibility, and dishwashers are for the most part left alone. It’s a way to get your foot in the door to becoming a cook/chef.
You could work with the Humane Society or be a vet. Animals can be a lot friendlier than people. Or you could do landscaping.
As far as construction, there are a lot of different types of construction work. My uncle is a contractor; he mainly does custom-built houses and decks or remodeling projects with a small team. If you learn a skilled/specialized trade, you’re looking at a much different work environment than a construction site.