Sounds to me like you've got a soap and/or softener build-up.
First things first, don't use fabric softeners on your towels.
My favorite trick is to wash all my towels and washcloths together (separated by colors). I add about 1/3 cup of distilled white vinegar when the rinse cycle starts.
It might take a couple of tries, but it rescued some pretty nasty washcloths for me!
PS, turtlegirl beat me to it!