After you bind off the last 5 stitches for the top of the buttonhole panel, you're yarn is in the wrong place to work the top of the front. That's why you'd need to rejoin.
However, I don't really understand why you can't just work all the way to the end, turn, bind off the first 5 and continue from there. :??