Yes, you need to change needles (if you want the finished item to fit)
If you should have 24 stitches= 4 inches and you have less than 24 then change to a smaller size needle and redo the swatch. If you have more stitches per inch than you should, then change to a larger size needle and reswatch.
Just keep trying until you get it right
The "suggested" needle size on the ball band is just that: a suggestion of where to start.
Happy Knitting (and Swatching) :XX: