Gaaahh, I hate earbuds! I don’t wear them because they make one ear sore and they just fall out of the other. I can’t remember which ear does what, but they’re so much a pain in the butt that I just don’t use them.
To answer your question, I’m not sure that they would make a true “callus” unless they were rubbing a lot. If they’re just sore from something being in there that isn’t shaped like your ear (stretching and poking, sort of like nose rests on glasses) then, you may still get used to it. I wear headbands sometimes and they pinch right behind my ear and it gets sore. Sometimes I get used to it though. Same thing when I wore glasses. I never got a callus, but I was mostly used to it, still getting somewhat sore from time to time.