If you were to simply cast on the total neck stitches in a circle, you would not be able to get the completed sweater over your head! This problem is solved by having the front of the neck dropped. To achieve this, the neck is knit as an incomplete circle, increasing one stitch on each end every other row until half of the front stitches have been added. The remaining front stitches are then cast on to complete the circle.
This is part of your pattern and I think explains your problem. You do knit back and forth at the beginning, and then the rest of the stitches are added and then you make the join.