For one, if you live in a place that has even a small amount of humidity, they never really dry fully in between washes. This can lead to the growth of mold. Blech. And second, when you dry yourself off, you’re also removing dry, dead skin. I for one don’t want to use a towel with 26 years of built up dead skin.
Try not washing your towels for a while—just hang them in your bathroom instead—and see how long before they smell odd. That’s why we wash bath towels.
I hang up my towel and use it again, maybe two more times after a shower. But they start feeling rough and not clean after that. Why would you not wash them. That’s just icky.
We wash bath towels to get rid of dead skin.
That does not, however, explain why we wash dish rags. (Especially after we rinse the heck out of them in the rinse water.)