Welcome to KnittingHelp!
You're knitting back and forth on the circular needles at this point?
You want to decrease at the beginning of the row on the neck edge (not the armhole edge). Cast off 3sts, work to the end of the row, turn and work the next row, turn and then cast off one stitch at the beginning of the row. Repeat the decrease of one stitch at the neck edge every other row 5 times more (6 times total). That'll decrease your stitch count by 9sts and take 13rows to complete.
Can you give us the name of the pattern or a pattern link, please?