I think gauge is the bane of many knitter's existence.
I was just reading Knitting Rules by Stephanie Pearl-McPhee last night and need to go a re-read her section on gauge.
I have always had a problem with getting my garment gauge to match my swatch gauge and while I was pretty tired when I was reading last night, the gist of it is this: gauge swatches lie.
I always thought it was just me. But you do your little swatch and then you make your garment and the gauge is not the same. You don't knit the same when you have a hundred or more stitches on your needle as when you only have 20. The weight of the garment as it grows will stretch your knitting out and change your gauge as well.
I think the only answer is to check and recheck your gauge while knitting your actual garment. That and make sure you don't fudge your swatch at all.
The Knitting Harlot lists 8 things to know about swatches and the first one (and the last one) is that swatches lie.
The second one is that a bigger swatch is better. Don't cheat on your swatch because you want the yarn and pattern to work and so you will knit a little looser or tighter to make up for that extra half stitch. Measure your swatch and as she says "suck it up if it's wrong."
Third is always wash the swatch. This will keep you from finding out what happens to your garment the first time you wash it.
Fourth is a swatch is not absolute. As you have found out, you can do a perfect gauge swatch and still have a problem with your sweater. Paying attention to your garment as you work is the only way to catch it before you have most or all of it done and it just doesn't fit.
This is just a small part of what she says on gauge. It's worth getting the book and reading it.
I think even the most experienced knitters have to deal with the issue of gauge. It gets easier to deal with the more you knit but I'm sure most of us have had to deal with ill fitting garments due to gauge that just didn't work out the way we planned. Stephanie calls it "The Element of Surprise." She says: "The way swatches lie is simply another one of knitting's little jokes designed to keep things interesting."
I hope this helps a little.