Yes, I think you are right, you have to work more than one row of a color for that to work.
I haven’t done a lot of one row stripes lately, except one row each of two different colors throughout the whole hat or mitts. I found that when I did that the jog thing wasn’t that noticable. I’m not sure how it works if you do only one stripe alone. Maybe it won’t be that bad.
There are some different ideas for handling the jog out there. Do a Google search and you should find some of them. They may be helpful.
Studies in Jogless Stripes
There are the 3 I have bookmarked on my computer.