The accidental YO is most likely being caused by not moving your working yarn between the needles when changing from K to P or visa versa.
Those Xs are probably being caused by forgetting to move the working yarn from front to back or back to front when switching from K to P or visa versa.
Boo, I’m doing a cabled sweater and I’ve been doing the same thing! It’s annoying, but I’m learning how to tell the difference between the YO and the actual stitch, so that when I get back to it I know which strand to drop. I have been known to drop the wrong one, then I have to get out the crochet hook and pick up the stitch again.
The x can also be caused by not fully taking the loop off the needle. It sounds like you need to slow down and concentrate on what you’re doing for a little while.