I would cut the colors after each use. It is quite a ways from one use of a color to where it is used again.
One thing you can do is as you begin a new row with a new color to hold the color you are dropping up along the left hand needle and knit with the new color by knitting over the tail on one stitch and then knitting under the tail for the next stitch. Keep alternating that way until you have covered the tail for a couple of inches. You can cut the tail off before or after you work over it. You can adjust the tension on the worked over yarn by pulling on the end of the tail and then stretching things a bit, you don’t want any loops to appear at the end or anything. That will leave a tail of the new color at the edge. I think I would just leave that tail there and work it in to its own color before you sew the vest sides together. But that will cut the number in half that you have to work in.
The colors used with this vest (if you use similar colors) will probably work fine for working in some of the tails the way I said. If the colors are high contrast like red and white sometimes the carried color will peek through a little.