I think you're best bet is to drop a needle size or two--however many it takes to get gauge. It's hard to say how many, since each person's knitting is different.
This way you'd know for sure what size you're going to end up with.
I personally like my sweaters looser, so I tend to lean that way. I always figure I can wear a sweater that's loose but never wear one that's too tight.
If you want to stick with the 20 per 4, I wouldn't worry too much about where you end the armhole and such. You can do that by inches. Patterns often specify where to end on a chart to make it easier to keep track. You can keep track of any changes that you make to suit your size.