I don’t think the people using it are necessarily using it in a derogatory way, it’s that a lot of us look down on the situation in the culture where the term is used.
Like a jelly above said, it sounds ghetto, and part of the culture that uses it is that it is common for people to have children while not married, the child is usually unplanned, and the children are growing up in not very good circumstances all too often.
Not to be confused with single people who choose to have a child and it’s planned, and can afford to take care of the baby and simply didn’t have a husband, but want to have children. An upper middle class single mom isn’t being called a baby mama by her peers, or the man who fathered her child. Not likely anyway.