First determine what your boundaries are.
For example, having a cocktail or two before dinner, or having a couple of glasses at wine during dinner, even every day, doesn’t seem to me like “alcohol abuse”. Does that seem like “abuse” behavior to you? If so, then your standards are tighter than most of the American population. It’s not impossible to find friends who share your standards, but it will be more difficult in some parts of the country.
Likewise, firing up a doobie, which is technically illegal in most of the US, hardly rises to the level of “drug abuse” to me, but maybe I’m rationalizing because I was at a party yesterday where some of the party-goers were doing that. (I definitely want to steer clear of the guy who was trying to sell the stuff – even to people he didn’t even know! – but knowing that someone smokes a joint on the weekend doesn’t bother me. I just don’t do it myself.)
I had two bottles of “hard lemonade” in the afternoon and nothing else to drink, and stayed away from the smoke. Does that make me “part of the abuse crowd”?
You want to know very clearly what your boundaries are so that you don’t have to make case-by-case judgments. Set the line and then monitor it.
The other thing is that you can’t monitor what people are going to do in their lives away from you. It’s no trick at all to host parties where there is “no smoking; no drinking; no drugs”, and (if you can throw a good party, which isn’t so difficult, really) have good parties. But that doesn’t mean that those people won’t also attend other parties where the behavior is a lot different. You can’t control that.