Staying in pattern

Hi, I’m new to the forum and join because I need assistance with my knitting project.
I have just started knitting a cardigan with raglan sleeves. The pattern is embossed rib or fishermen’s rib which is fun to do, however now I have cast of 8sts at the start of the arm hole I can’t seem to keep in pattern. I’ve patterned to the end then knit the next row to last 5sts then knit two together then knit the other three. The next row is pattern with purling two together. The 6 rows I follower over 69 rows form the decreasing with two stitches decreased over these 6 rows. My problem is keeping in pattern. I have undone this 17 times and then put the knitting back on the needle starting back with the 8sts cast off. As you knit one stitch below which is the pattern I have ended up with loops on the rib side which is the front.
1)Should I undo all? 2)How do I keep in pattern?

Will be most grateful for some advice thank you

Welcome to the forum!
One of the best things to learn is looking at your knitting and reading the pattern. Sounds like you’re already doing this and recognizing where you’re out of pattern.
For decreases in fisherman’s rib, the best thing may be to give up the rib pattern is the area of the decrease. So you may have to cast off, knit a stitch or two and then get back into the rib pattern. The same would work for casting off on the wrong side. The regularity of these knit or purl sts works with the overall fisherman’s pattern.

Thank you salmonmac, I have followed your tip on knit a stitch or two after the decrease and finally! I have stayed in pattern. So a big thank you, to you.