Right, it doesn’t “remember” you from last time.
Regular polyurethane foam is like a spring: the more you compress it, the harder it pushes back. That means that the parts of your body that sink in more deeply are going to be under more pressure as the foam pushes back harder in those places. Your body ends up having to realign itself in response to those pressures.
Memory foam is less like a spring. Under pressure, it just kind of relaxes and stops pushing back. So when the more protruding parts of your body dig into the memory foam, they’re able sink in as the foam gives way to the pressure. Your body can maintain it’s natural geometry since it’s not being shoved here and there by the pressure of the foam.
Once you get off the foam, it does go back to it’s original state (though this happens over a few seconds). Body temperature does affect the stiffness of memory foam, though, so the spot where you’ve been lying will remain considerably softer until it cools back down.