I have the same problem for the last few years. I’m a woman so my specific complaints are different than yours, like why the f$!k can’t they get the waist back down at the waist on dresses and shirts! Flippin’ fashion has put the waist at my ribs for too long, time for that to change.
But seriously, if you shop in some better stores you might find some more clothing that you like. It will cost you more, but if you are looking in places like Target, Walmart, and Penneys, you have not really given fashion a chance. There is so much garbage out there now. Try Express Men or Bloomingdale’s. Let a salesperson help you, they will make suggestions, they usually know the stock very well, what fits tight or loose (although during Xmas you can hit a less experienced xmas employee) and they will get you a new size if you need it while in the dressing room. Maybe enlist a friend who is a good dresser to go shopping with you.