Interesting question!
I think the main reason is simply energy versus effort. Chicken eggs are easily produced and provide a great amount of energy per unit. It’s pretty much set it and forget it – feed the chicken, give it a good home, and it’ll produce eggs for you regularly. Can you imagine any attempt to harvest the same quantity of chicken semen? Pardon my course language, but you’d have to spend all day beating off hundreds of roosters, if you even can beat off a rooster, to get one egg worth of semen. The energy to effort simply favours eggs over semen.
As for fish eggs, again they are more easily harvested than fish semen. But furthermore, caviar was itself more a sign of prestige. There is no similar prestige with fish semen, because it’s hard to collect (again, if you even CAN collect it).
I don’t think there’s a taboo about eating semen though. After all, there are those prairie oysters linked by Tits. I also don’t have a natural gag reflex against the thought of eating semen, although I will admit that I don’t like the feeling of human semen in my mouth (it’s gritty and pasty and weird).
One last thought, I would consider it animal abuse to forcefully harvest mass amounts of semen for human consumption. So perhaps this contributes to the negative desire to consume it.