Changing needle size for guage

I had to change to size 6 needles, from size 7 for the body of the sweater to make guage. Do I also change the ribbing needle from size 5 to size 4?

Yes.