Brown sheep cotton fleece is AMAZING for sweaters. It has a bit of wool in it, so if you're alergic, it won't work, but it adds SPROING to the yarn. Cotton can stretch out if there's nothing to help it NOT stretch. I made this sweater with it for my mom and it's cool, comfortable and cute (imho!). It also comes in TONS Of colors, depending on where you get it.
No matter what you get, I'd probably still recommend gentle washing. After putting that much work into it, a little TLC for the finished garment is worth the little extra time.