It might look odd if the sleeves were totally different color. I found, too, that with colored yarn, like the Karaoke I used, that the stripes were naturally different thicknesses around the body and sleeves. It ended up looking cool, but I didn't anticipate it.
Maybe you could start the body with the ROY, then start the sleeves with ROY and then switch back and forth for the G. BIV part. KWIM?