Our ears do so much more than allow us to hear the speech of others or listen to music. The inner ear organ is very primitive and is formed in the first trimester in humans… it detects not only sound waves, but also vibrations relative to motion and spatial positioning relative to gravity, and it is doing this many months before we can see. We close our eyes when we go to sleep, trusting our auditory-vestibular system, which continuously collects and analyzes sound and motion, to alert us if something is awry.
Our eyes are limited to what is in front of us, but our ears provide us with awareness of the world all around us and beyond visual obstacles. Typically, we hear a falling rock (to use @Seek_Kolinahr‘s example) and are able to reflexively locate it with our eyes. There are creatures that are blind who rely solely on their ears to navigate through the world. I recently learned that there are parks that are cutting back on noise pollution because of the impact it has on wildlife.
Our brains may need to use more space and energy on visual processing, but that may well be because the ears became incredibly efficient many millennia ago. However, since our species has designed our world based on vision, the loss of that sense would likely be more handicapping to humans.