If you’re inventing a different fantasy world, then it’s not copyright infringement. There are thousands of books and probably millions of stories about wizards.
You may want to avoid doing things that are very similar to popular stories you know of, though, not because you will be sued (you probably won’t) but because it will feel too much like the other story.
For examples:
* A story about a young wizard-ess who lives in a world with magic and who studies at a magical academy and has various adventures and coming-of-age issues could be very unique and would be the same genre as Harry Potter, but nowhere near a copyright infringement.
* If an episode in that story has a room full of flying keys, and the wizard-ess notices one has a broken wing from being handled, snatches it and runs as the other keys start chasing her, and she closes the door just in time, and the pursuing keys all hit the door right after… that also would not be copyright infringement. It would just be weirdly and annoyingly copying a scene from Harry Potter, and would make everyone groan. So you wouldn’t be sued, but you might be disliked, so I wouldn’t recommend that. You could have flying keys though if you wanted.
* If you use the exact same setting, and/or call the school Hogwarts, then you’ll be writing derivative or fan fiction, and would want to get permission, or you might get threatened, or sued if you tried to sell it. So ask or look for permission (or don’t sell it) if that was what you wanted to do.