The problem with the term is that people used it, and others interpreted it, as a racial slur because it sounds like one, not that it actually is.
Envision a racist calling a black child a “tar baby” thinking himself clever and demeaning. He intended to mean it derogatorily, even though he is ignorant as to its true meaning.
Now consider an ignorant over reacting politically correct liberal hearing John Kerry properly describe talk about altering Social Security as being a “political tar baby”. The ignorant PC Liberal mistakenly accuses Kerry of being racist.
Both are wrong, but the term sounds inappropriate. And because it does sound that way, it now carries that connotation.
This is how language evolves.