I was a child prodigy and taught myself to read and write at the age of two from comic books. I also have strong memories of crawling around on the floor, pooping my rubber pants, sucking on my pacifier, and jumping up and down in my crib dating all the way back to the age of six months. I think these two facts are connected.
It’s believed that we all have memories reaching back to birth and possibly beyond, but that they are inaccessible because the way the brain stores memories changes when we learn language. As a prodigy with precocious language skills, I think I can remember much farther back than most people because my brain had installed its linguistic operating system so early.
Given this, I think it’s highly likely that the age at which one acquires language matters very much in the development of other intellectual capacities. In Chomsky’s seminal Syntactic Structures, he describes the way the mind creates overlapping clouds of meaning around words, with more concrete meanings towards the centre and more abstract meanings around the edges, where they often overlap with other words and concepts. Given that so much of our cognition is linguistic, it’s unsurprising that overall intelligence might be affected by our development of language skills.
Julian Jaynes, incidentally, notes that the corpus colossum is not, as commonly believed, the only connection between the two lobes of the brain. The two language centres are also connected by a thin bundle of neurons, and Jaynes argues that language in general and words specifically are actually part of the mechanics of brain function, as they form a neat way of compressing complex thoughts into small bundles capable of being transmitted across the gap between language centres, and the reason (he argues) that we experience the gods as audible voices speaking to us.