This is a recognized phenomenon, yes. It’s not because bacteria grow faster in skim milk, though; in experiments under controlled conditions, the growth rate is the same for both. The difference is in the flavors that result from the bacterial growth.
Bacteria release enzymes that break down various components of the milk. One of these is protease, which targets the milk proteins. The by-products of this enzymatic reaction are particularly bitter tasting. After a given period of bacterial growth in the two kinds of milk, the number of germs will be about the same, but the skim milk will have considerably higher levels of protease and proteolysis byproducts, so it will tend to taste nastier. The speculation is that the fat globules in whole milk inhibit the access of the protease to the casein proteins. Whole milk develops spoilage flavors that tend more toward sourness than bitterness, and these are less likely to be interpreted as alarming.