I did the knit cast on as you suggested and its looking a lot firmer.
However, I think I found a more serious problem that would cause me to re-do the whole sweater.
After I tried it on the last time, I noticed some loose yarn strands on the wrong side about 4 inches below the neck. I've tried the sweater on several times, and this just appeared now. I think its showing above where I changed yarns so it doesn't appear to be an unraveling of the Russian join. However, the stitches on the right side look very even which would indicate the join unraveled.
The only other thing I can think of is that I must have inadvertently split the yarn and where the yarn split, it broke off. I temporarily wove in the yarns but I'm concerned the structure of the sweater is compromised.
I was thinking about unraveling just that one column of knit stitches to see if I can untangle that one stitch but this stitch is about 50 rows back. I've unraveled a stitch up 10 rows before but not this many.