I purposely wear them mismatched. Why not? Please show me the rule that says that one needs to wear matched socks at all times. Oh yes, that rule doesn’t actually exist! As with a lot of other “rules”, when you look at their root, you realize that it exists only in others’ minds. Why should I ever be obliged to conform to someone’s mental image of what I should be like or what I should dress like? I can’t think of one good reason, hence I don’t follow these arbitrary rules, especially fashion rules, which aren’t really rules at all.
As Mr_M elaborates, most people are too concerned with how they look and appear to the world to pay attention to you and spot your foible.