Help with switching colors

HI
I’m knitting some phentex slippers ( the one’s my grammy usta knit a looonng time ago ). I understand how to pick up and drop the color on the wright side but when i’m on the wrong side and trying to knit i dont understand the concept of switching the colors. I am currently switching the colors between the needles? Is that what I am suppose to do?
I finished one slipper by following the pattern but it came out way tooo small and i’m wondering if it’s because i’m switching the colors the wrong way?
Thanks
Christine

Size wouldn’t be affected by color switching. I’m not sure of the answer though, but I’m sure someone who knows will be along soon!