It's common for a knitter's purl stitches to be a little looser than her/his knit stitches, particularly when using the continental method. This can cause a knit next to a purl look larger than normal. The easiest way to remedy the problem is to consciously make your purl stitches a little tighter.
You can also try combination purling (or another method which involves purling in the wrong direction and then correcting the mount when working the stitch above), but I wouldn't bother unless purling tighter doesn't work. Combined knitting works well for simple or flat knitting, but complex stitch patterns are difficult to convert. (This is according to Annie Modesitt, who literally wrote the book on the subject.) Some people use combined just for working straight stockinette or ribbing, but IMO it makes more sense to just work on your continental purling. (This was my conclusion after experimenting with combined.)
Just to be thorough, a final possibility is to purl using a smaller needle, but that won't always be convenient, unless you're doing straight stockinette.
BTW, I think your hat looks great! People will be too impressed with the cables to notice any minor (and I emphasize minor) issues with laddering.