Welcome to KnittingHelp!
That's a tough question. It depends on how much too big and how complicated the pattern stitch is. If you think the sweater would work without the buttonbands, you might sew ribbons on the fronts and use them for closures.
For myself, I'd have to re-knit part of the sweater, maybe the back.
I'm sorry for the mistake. Unfortunately, we've all had things like this happen.