That’s what my husband always called ordinary mushrooms we bought at the grocery store. I never heard that usage before he said it, so I thought it was his nickname (he nicknamed a lot of things), but I guess other people do use it.
I think it’s context sensitive. I generally only use it to refer to the psychoactive mushrooms just to avoid ambiguity, in the same manner I only really use the word “gay” to refer to homosexuality.
When I first heard the term back in the 1970s it was used for the magic ones. About the 80’s I heard it used for any type of mushroom, the meaning was taken by context.