As suzeeq points out the ONLY stitch you should be following is the one on the edge. Find the edge and move over one. You can check if you are staying matched up vertical wise by seeing if you have the increases happening at the same time on both sides.
The single crochet seam is really pretty and gives more structure for the sweater. I like the directions in Vogue Knitting.
I know you are antsy to finish the darn thing, but you may want to set it aside for a bit. (Says the woman who was up until 11 pm last night finishing a shell. )