Welcome to Fluther.
I can’t even begin to explain the “why” of your anxiety, and I would suggest that it’s not even a good question for you to attempt to answer. (Though I would compliment you for your apparent capability to recognize the different ways that it manifests; a lot of people would not have that self-awareness, I think.)
Contrary to your feeling, I don’t have any greater than so-called “normal” anxiety: I don’t particularly like to speak in front of groups, though I have done that, and I don’t like to be unclothed in groups of people, though I have also done that. Even so, I don’t particularly care to go to restaurants by myself; I just don’t like to waste the social aspect of “going out to eat” ... and not having a companion to share the experience with. (Or maybe that’s just my own rationalization; I’m also self-aware enough to realize that not everything we do is driven by pure reason, but our reasoning faculty helps us to explain perfectly logical-sounding reasons for every weird thing that we do.)
But getting back to your question, and the reason why I suggest it’s “the wrong question” to answer. Let’s assume that you find the perfectly valid reason “why your anxiety is so weird” – what good would that do you? (I’m not saying that there’s zero value in answering the question, because it could be a good exercise, but… you probably won’t be able to answer it anyway, and even if you could, then it probably wouldn’t help anything.)
But here’s a question that, even if you also can’t fully realize a full answer to it, working towards an answer would help you with every step: “How can I deal with this in ways that make my life better?”
Maybe (for example) you could start to walk past restaurants at lunch time or at other times during the daylight hours. Perhaps you could step inside just to ask the hostess if there are any specials that day, even if you don’t intend to eat there. Maybe you could sit at the bar and talk to the bartender while you smell the aromas around you. I don’t know; you think of something. (Or think of another question, which “the process of answering” all by itself would lead you to either resolving the issue entirely – it’s possible – or at least more effective ways to deal with the issue.)
Asking the question of “What’s wrong with apples?” just leads to a never-ending list of all of the problems with apples… and no solutions. If you ask instead, “How can I make apples better?” then you can start an orchard, a bakery, a juice factory… you name it.