If my gauge swatch ends up being 4 stitches shy of what the key states it should be, should I go up a needle size or down?
Down, I think. You’re getting (just an example here) 16 stitches to 4 inches instead of 20 like the pattern calls for, right? If you want more stitches in the same space, they have to be smaller, so you use a smaller needle. Or vice versa, as the case may be.
Yes, more stitches are needed so a smaller needle should be used.