Candida likes to grow in dark and damp places--a wet towel is a virtual petri dish for yeast. Sweaty or wet clothes also create a good environment for yeast to grow.
That said, there are a few things you can do to reduce your changes of getting an infection: dry off quickly and thoroughly with a clean, dry towel after showering or swimming. Try wearing 100% cotton underwear--it lets your body breathe. You can also change your underwear every day and after exercising, and avoid wearing tight underwear, thongs, or pantyhose. It's also a good idea to wear loose-fitting clothes to bed.