Eggs are a fertility symbol. I hope I don’t have to explain why.
Rabbits are also a fertility symbol, because they are prolific breeders.
We use a cluster of fertility symbols around a spring holiday that old Pagan cultures used to celebrate fertility because Spring is the time of increasing fertility. Back when people farmed for their lives (instead of thinking food came from a grocery store), this meant it would be possible to grow food again and you wouldn’t die of eternal winter. This seems to be a point lost on city slickers. ;)
Then at some point, Christianity came along with its Reborn God (see: fertility symbol) that it may well have copped from other religions too, and attached their fertility festival to the Pagan fertility season, in hopes that the Pagans of old Europe would somehow become Christians instead. (They did the same thing for Christmas, by the way.) And it seems to have worked.
Here, History.Com explains your symbols for you. It was the first hit on Google. ;)