you would usually start a project (a hat) with the smaller needle size because that would be a ribbing row, and then switching to larger needles, would mean that your work would be smaller (tighter) on the band and larger (looser) on the main portion.
Depending on the pattern, I'm not sure if you should continue on smaller needles. Is it actually possible to go up 1 or 2 needle sizes instead?
Really, it would depend on your gauge swatch. Did you do one of those?