Amanda, have you tried Lakeside Fibers and The Knitting Tree?
I think there's a place on Willy St. too. I left Madison 11 years ago, but my in laws still live there, and the MIL is a totally Ironman knitter. We still talk from time to time about the LYSes there.
In the more outlying areas, there's Prairie Junction in Sun Prairie...um, I think it's called Susan's in Columbus...Blackberry Ridge Mill in Mt. Horeb...Yarn Nook/Trollhaus in Stoughton.
Also if you're just looking for materials, rather than say knitting circles, you've got a zillion options for buying yarn and fiber directly from livestock farmers. You can probably find leads to that via the Farmers Market or the Wisconsin Women's Sustainable Farming Network if it still exists. Or the Upper Midwest Organic Farming Conference.
Personally, I'm fond of yarn.com/WEBS online. I live a ways from town and am not about to give oil companies money if I don't have to. The UPS guy, by contrast, drives past here four times a day.