When you switch colors, you have to hold the one you're stopping over to the left and bring the new one up from under it. This will twist the stitches and avoid the hole. Do it on both sides.
Check out Amy's video on intarsia if you're confused.
As for the holes that are still there, just stitch them closed.