Leave the center 16sts on a holder, reattach the yarn and knit across the next 20sts. At the armhole edge, turn and purl back. On the next row, cast off 3sts at the neck edge, knit to the end of the row. Turn and purl back. Again at the neck edge, cast off 2sts, knit to the end, turn and purl back. Cast off one stitch, knit to the end, turn and purl back. Repeat the last 2 rows once more and you'll have 13sts left. Work 4 rows straight and cast off. All of the decreases are at the beginning of knit rows.
Sometimes it helps to make a diagram of the decreases or cast offs for the neck. Then it becomes easier to see how to reverse the shaping. Don't worry that the shaping is offset one row, it'll never show in the finished sweater.