Theoretically we would not know the condition of a persons “soul” until evaluation after death so this question must be purely for philosophic discussion.
This being said I think “soul” damage from music would entirely depend on the person and the effect the music had on that person.
For instance Led Zeppelin would not in any way effect me but listening to Barry Manilow long enough could possibly drive me to commit homicide. Then that act of homicide would harm my “soul”.
Even in this scenerio the music was not what caused the damage but the ACTION inspired by the music.