The entire front is an inch too narrow?
What is the name of your pattern?
I don’t thing keeping the twists tight at the color change would make this much difference unless there are multiple color changes across the rows. Then it might be affecting adjoining sts and causing gauge problems. There’s not a need to pull the yarn tightly when you change color.
If the entire front is off by an inch, that’s not so bad and could probably be dealt with in blocking. Loosening tension is best done by changing needle size but I’m afraid that would leave a line across your knitting. If you decide to rip back and change needle size above any ribbing that may be your best bet.