I never had this problem with my mom. My clothes never fit her because she is taller and bustier than I am, and even when I was in high school and college there was enough of a weight difference between us that she couldn’t even attempt to fit into my things.
On the other hand, I could easily wear a lot of her clothes. Not everything, but tops and skirts. I had a summer job in college with a no jeans/shorts rule. Really stupid, because it was a daycare center and I spent all my time doing messy things with kids. The director wanted us to look like actual teachers. I didn’t own anything other than jeans and casual shorts and didn’t bother to go buy them since I had ready access to my mom’s closet. Yes, it drove her crazy because a lot of the things were dry clean only and I’d manage to get fingerpaint or grass stains on them. If she’d put her foot down, I would have stopped, but she never did anything other than express mild annoyance.
Anyway, like others have suggested, I think you should calmly ask your mom to check with you before borrowing something. You can always say no if she wants to wear something you plan on wearing. Does she at least take good care of your things? Wash them and put them back?
The mom in me wants to know who pays for your clothes. If you do, you have all the more right to expect her to respect your things. If you parents still pay for your clothes, you may need to be a bit more understanding about loaning them out.