The phrase is taken from the title of a novel by Thomas Wolfe. Here’s how Wikipedia explains it:
“The title comes from the finale of the novel when protagonist George Webber realizes, “You can’t go back home to your family, back home to your childhood… back home to a young man’s dreams of glory and of fame… back home to places in the country, back home to the old forms and systems of things which once seemed everlasting but which are changing all the time — back home to the escapes of Time and Memory.
“You can’t go home again” has entered American speech to mean that after you have left your country town or provincial backwater city for a sophisticated metropolis, you can’t return to the narrow confines of your previous way of life, and, more generally, attempts to relive youthful memories will always fail.”
I don’t know if it’s always true, but the idea contains great wisdom about the human condition in a social context.