It varies a lot, so “average” is difficult to really determine. I know plenty of adults who can’t really grasp that concept though, so the average rises dramatically higher than you might think.
Personally, I understood that quite well by age 3, but I don’t think that it’s normal/average to grow up in a small town in a small state and travel extensively to visit friends/family in places ranging from tiny villages to the metropolis of Boston, and I knew distances and directions better than many adults. Many people spend practically their entire lives in one place. And many I’ve talked to who were South and/or West of Pennsylvania don’t even know where Vermont is on the map. Seriously, if I had a nickel for every time I’ve gave up and said, “About 100 miles from Boston.”, I’d be filthy rich!
As it’s a matter of broadmindedness rather than age, I’d also say that the average is even higher in the US than in many other countries. Other places treat education as something to improve one’s quality of life; we Americans see it as either something that drags us away from Facebook for a few hours a day or as a luxury item that allows the children of the rich to get a piece of paper that will allow them to get a high-paying job of the type that is off-limits to those that can’t afford to plop down $50–200k. It doesn’t help that many here see education as a threat since educated people capable of critical thinking are less gullible and thus will ask uncomfortable questions.
That attitude towards formal education also affects culture. Those who don’t bother to expand their minds through academic means are also less likely to open themselves up to new experiences. Why bother grasping the concept of other countries when no other country on Earth matters? Why learn about different cultures when we are the best culture in the universe and everyone should be like us? Regardless of how book-smart one is, one’s cultural upbringing will have a profound effect on what concepts one can/does grasp.