You would get a positive feedback loop, probably resulting in an audible tone. The amplitude would be limited by the transmitter’s modulation limiter, so all you’d have is a very ordinary oscillator.
Rex nailed it. Think of rock stars on stage who hold their guitars up to the mics or amps and it starts squealing until they touch the strings again. Same principle.
I think he means you use the headphones as a mic for the FM transmitter so that what happens is noise is picked up by the radio which generates audio picked up by the headphones(or mic) that the FM transmitter transmits to the radio, which makes audio of it….ect, ect.