Colourwork hat riding up

Hello! I’ve just finished knitting my first adult-sized colourwork hat. I like it, but unfortunately it keeps riding up on my head when I wear it! It’s plenty long enough, and the 1x1ribbed brim has lots of stretch to it. I’m wonder if it is a. too big, b. too small or c. the fair isle portion of the hat is fitting tighter than the brim, causing it to ride up.

Any tips or suggestions?

Welcome!
Can you post a photo of the hat? Use the landscape icon in the top middle of the Reply box. It may help us to see it.
If the stranded knitting is too tight, it tends to pucker so that’s fairly easy to see. To me, that seems the most likely.