It is probably derived from flutter, a word whose original meaning was ‘to be borne or lie tossing on the waves; to float to and fro’ according to the Oxford English Dictionary.
Some more modern meanings for that word would be, for example: ‘to move with a light quivering motion through the air’; ‘to move about aimlessly, restlessly, sportively, or ostentatiously; to flit, hover’; and finally ‘to move about or to and fro with quick vibrations or undulations; to quiver’ (OED). I think that pretty well describes the movement of a jellyfish.
Supporting that, this website notes how ‘jellyfish float’ and ‘move with the ocean currents’. This would go along with the original but now obsolete definition of flutter as well as its newer definitions.
The word flutter is also etymologically linked to the word fleet, the original meaning of which was essentially ‘to float’. Another obsolete definition is ‘to drift or be carried by the current or tide on the surface of the water’, while another (also obsolete) indicates ‘to swim: said of fish, occasionally of other animals and men’ (OED).
And, finally, you’re probably familiar with our use of fleet as the collective noun for a group of ships. I think if you compare the way ships move with the way jellyfish move (jellyfish ‘sails’ catch water rather than wind), the connection is apparent. I can just imagine a bunch of jellyfish moving together like an underwater naval fleet or something.
A fleet becomes a flutter becomes a fluther of jellyfish. Maybe.
This is my personal conjecture after doing a little research. It seems to make sense, though. Hope it helps some.