Ah, quick answers are never good. So urusakunai might be the word for not annoying but to actually say “It’s not annoying!” you might say anything from “uruasakunai yo!” to a friend or ”(sore wa) urusai ja arimasen”.
…a little flight of fancy, if the mods permit, for earthduzt, merging this with another question How would one say “Sean Hannity really annoys me.” in Japanese? :)