My bet is once you try it on with the sleeves done it will be fine, especially after blocking.
I just finished a top-down and I thought the neckline looked too small/close fitting. I cast on 96 stitches and did 6 rows of twisted rib...it seemed small when I tried it on but when it fit over my head I kept going.
After getting the whole thing knitted, the weight of the garment pulled the neckline down and after blocking it was bigger still.
I guess it depend too on th eyarn. I was using King Tut cotton which grows a little, but not too much.