First, a swatch should have more stitches than what it gives for 4"; the edge stitches shouldn't be measured. The goal is to figure out how many sts in 4", not to make a 4" square.
You could go down in needle size, but if your yarn is heavier than in the original pattern, you might not get the gauge called for, but it would still be too stiff. Finding the needle that gives you a fabric you like, then making a smaller size can be done. What is it you're making?