Often, minorities were not allowed in the same places so they didn’t need 4 bathrooms, because only white people were present. Some places still had outhouses for Black people in more remote areas. Large buildings had separate bathrooms for whites and Blacks. Asians and Latin Americans would usually be expected to use the “colored” designated bathrooms too.
Very few places outside of the Southern states had this set up. My parents and grandparents never saw any such thing growing up in NYC. My friends who grew up in the South grew up with separate water fountains and bathrooms and remember their schools getting desegregated.
Slightly of topic: You might be interested in the movie The Green Book. Also, @chyna mentioned Hidden Figures, which is a fantastic movie if you have never seen it. It shows the bathroom situation and I think it also was a commentary on how not only Black people were treated and women also.
Hidden Figures shows some brilliant minds from other parts of the country and even an engineer who was an immigrant working on the space program for NASA in Virginia. Most of the senior people, including John Glenn the astronaut, were from northern and midwest states, but they were working in Virginia so the custom to have separate bathrooms was portrayed. The truth is the Black women who worked directly with the engineers and mathematicians used the white bathrooms even though Black bathrooms existed. You can read about the historical accuracy here https://en.m.wikipedia.org/wiki/Hidden_Figures I found it interesting.
The Green Book I think has even more emphasis on the racism and separation of races, because the racism is in the South by Southerners who live in the system and worry about being judged by their peers. The dynamic is slightly different.
Both are based on true stories.