If you work to the armhole, then begin your cast-offs, you will be bringing the yarn with you, so the yarn will be where you want it to be for the back. You work across the back, and then you'll come to the place for the second armhole, bind off there (the yarn will follow) and then work across the front.
So to bind off for the armhole, knit 2, pass the first stitch over the second, knit another, etc, the way you would bind off. Count the final stitch left as one of the back stitches.
Rereading your original post, I realize that you are knitting a cardigan.
You should have the 120 stitches for the back, and after the decreases will end up with the 102.
Work all the way around the sweater, across the front, the first armhole, the back, the second armhole, and then the other front.
Cut your yarn and put the fronts on holders or a spare piece of yarn. Then start working the back only, doing the decreases as listed. After that you'll put one of the fronts on the needle and work whatever armhole decreases there are on the armhole edge, and then the other front.