Another, simpler way to down-size a pattern is to use smaller needles. This way you can follow the pattern as written, yet end up with a smaller garment.
If you don't want to do that, I would do what you're already doing. Check out the differences between the other sizes and work from there. If they all say the same, I'd go with that. Better yet, wait until you get to the armhole and see if three would be appropriate.