I like charts just fine for color work: they do help you visualize what the knitting is supposed to look like.
Cable charts are not bad either; they're pictorial and once you get the concept of how many stitches to cross for each cable turn, I find them easy.
It's the lace charts that drive me nuts, and not least because different pattern writers use different symbols for ssks, k2togs, skpsso and so forth. (Symbols for knit and yarnover are pretty consistent.) Glancing back and forth from the work to the chart to the symbol key is slow and frustrating and leads to mistakes. I have to keep things under strict control if it's going to work at all. Which means enlarging the chart, keeping it on a magnetic board in front of me, keeping track of repeats and rows with a million markers, throwing in lifelines everywhere.
After a couple of repeats, however, I usually begin to see the logic of the pattern and can continue by reading the knitting instead of checking every stitch on the chart. That's when it becomes fun. Some complex patterns, however, are impossible to internalize, at least for me, and I usually give up in disgust after the third frog.
I wonder whether visually oriented people do better with charts and verbally directed people are more comfortable with written-out instructions.