Passive aggressiveness comes from a place of fear. It’s a way to get out your feelings without really committing or admitting that you are angry, which can result in confusion for the other person & frustration for you. Just say exactly what’s on your mind, without worrying what the other person will think or how they will react.
I, personally, am the queen of passive aggressive and couldn’t bring myself to ever say something without caring how the other person feels. So that advice is much easier said than done, I know.