I suppose it comes from years of living with family or in a dorm room or having a housemate or having a job that requires staying in hotels. In these cases, it is a matter of modesty. On the other hand, they are a source of additional warmth. I prefer to sleep in a chilled room and burrow under layers. Pajamas do not bind like a nightgown does.
Note that this is only when I sleep alone. If the SO is in the same bed, he generates so much heat that I can’t sleep unless some of the covers are pulled off, even in the dead of winter with the windows open.