Wear and let wear is an excellent ideology, but judging people on what they wear is deeply rooted in human culture. Even before the development of civilization people were using various forms of body adornment (jewelery, body painting, etc) to show social status. In every single culture there has been an “elite” who is allowed to wear the nicest things, whether it be the rarest shell in the area or the latest Ralph Lauren fashion line.
Even if there aren’t strict sumptuary laws in most modern, industrialized cultures, they are still kind of there in ways. Middle class people, because of their income, generally cannot wear the same things the movie stars wear, those $10,000 dresses with $5,000 sunglasses… It’s just out of the question unless you have some secret source of income that nobody should know about.
Similarly, people dress in certain ways to distinguish themselves. We acknowledge that people judge based on appearance and clothing choice, so we use that element of our culture to our advantage (sometimes). When people want to stand out from the norm, they wear crazy clothing; when they want to show how wealthy they are, they go for the most expensive and “in” clothing they can get. Or, they’ll dress in imitation-high-class clothing to give the impression that they are better off than they may actually be.
Self-adornment and its implications can’t be erased from human nature. It’s like telling someone not to eat McDonalds because others might think they’re a slob. I’m sure you show quite a bit of yourself in the way you dress, whether you want to (or are aware of it) or not. As long as people wear anything but their birthday suit, this will be true.
Edit: I’d give you a source for my information on the info in my first paragraph, but I’m remembering it from a History of Craft class I took last semester, and we didn’t use textbooks. Sorry!