One thing that I think might be happening is that I knit the swatch in a different tension than when I actually start knitting the garment?
This sometimes happens. Usually we're really tense when knitting the swatch, trying for gauge, then relax and loosen up when knitting the project. If you're working on a sweater, cast on for a sleeve and check your gauge after several inches. If it's okay, go ahead, if it's off, change the needle size, but keep knitting on the sleeve and note to change the needle for the rest of the sweater. Don't worry about the difference just on that sleeve, it should come out looking okay when you're done.
You can also knit on something else for a while before you start your gauge swatch, that might help you be more relaxed. And you can block your swatch to see if the yarn changes after and how it will look in the finished item.