You do not have to be around them if you do not want to. You may not have to be up front with them and tell them that you do not want to be their friend unless they are trying to be around you a lot. Unless they are trying to hang out with you all the time, I do not feel it necessary to tell them to go away and that you are not their friend. It may come across as you being a jerk if you just tell someone they are not your friend, especially if they are not trying to be your friend. It’s okay to be nice to someone by saying “hi” without hanging out with them at all.
Also, if these “annoying” people are friends of your friends, you may end up isolating yourself by saying you don’t want to be friends with “everyone” in the group. May come across as selfish. I agree with you though, some people are okay in small doses, but after so long it is best to just not hang around them.