Think about it this way: measure how wide the 25 st are on your swatch. Are they 3.75"? About 1/4" too narrow for every 25 st?
Then multiply that error across the entire width of stitches for all the body pieces. It is staggering how just 2 st off in gauge can make a garment either way too small, or way too big, in the end results.
Keep trying to get the correct gauge with a different needle size, except that I always just start a new swatch.
FYI: sometimes a real tight (or a real loose) cast-on row can affect the resulting gauge immediately above the cast on row. You could try your new swatch with a looser cast-on and use the same needles. Are you knitting a full 4" in length swatch? This also helps reveal the true gauge.
Swatching is a chore, to be sure…but incorrect gauge can destroy the ‘fit’ of a garment. Been there, done that!