Actually both are helpful as are your written directions for the repeat. It is indeed: k,k,k, k2tog, yo,k. That takes 6stitches to do one repeat and will leave you with 6 stitches. That'll work evenly into 96sts.
The yarn over may be causing the problem. It occurs between stitches and so it won't use a stitch. Here's a video that may help.
Absolutely gorgeous hat. Thanks for the pattern link.