Once it's buttoned, nobody's going to notice anyway. You're the only one who cares about it, and you shouldn't be so hard on yourself. There's no knitting police here. Sit back, relax, and enjoy the compliments you're going to get about the beautiful sweater you've just completed.
It's done when you can leave it alone.