I like gorillapaws’s answer which is quite good, I’m just going to add to it. The sahara isn’t quite impassable, but it’s such an endeavor, it may as well be. North Africa traded throughout the med in ancient times when the Phoencians set up trading outposts all over (Carthage was Phoenician), and then again later when the Muslims conquered it. It’s only in the post-Islamic Golden Age era that North Africa has fallen on hard times.
But sub-saharan Africa has a harder time connecting to the rest of the world. We know some trade occurred over the Indian Ocean, but even that is harder than the easy connections in the Med and over the near eastern land routes. It’s easy to forget how FREAKIN’ HUGE Africa is because we’re all used to the mercator projection maps that make it look smaller than it really is. And much of sub-saharan Africa’s coast is rockier, making for less quality ports.
Then you get another factor – rainfall. The best places to build large stable civilizations is in an area where moist air hits a mountain range. The result is a ‘rain shadow’ where rain falls predictably and the rivers that feed it also become predictable. Africa has no such feature until you get to the far eastern edge. As a result, while Africa gets a decent amount of rain below the sahara, it’s erratic and undependable from year to year. Rivers surge some years and go dry others. Imagine how hard it is to build a stable empire under those conditions.