The number you cast on has more to do with the weight of yarn and needles you are using. For a standard hat made with worsted weight yarn and size 7-8 needles you’d cast on about 80 stitches. That’s what I use for the hats I’ve made. If there is a pattern involved you need to decrease accordingly so the pattern will come out right.