This is a common problem in raglans. Some designers and knitters have experimented with the basic structure to improve upon it.
I think the reason that it doesn’t show on the non-sleeve side is that the extra fabric has somewhere to go - see how the edge of the “armscye” is sticking out several inches from your arm, like a sort of flared cap sleeve? Once the sleeve is knitted, that sticking out bit is held flat against the top of your arm by the weight/structure of the sleeve and the extra fabric has to go somewhere.
If you hold your arms out to the side, as though you were doing the “A” in the YMCA dance, the extra fabric will probably redistribute itself.
This is an eternal problem with sleeves. Set-in sleeves look good when your arms are straight down by your sides, not so good when you raise your arms. Raglans look good when your arms are slightly extended but a bit rumpled if your arms are straight down or raised. Drop shoulders look good when you hold your arms out like an “R” in semaphore, but rumpled when they are up or down.
Regardless, I wonder if you would be be better off making a smaller size, and then doing a few short rows at the bust, to maintain the ease there, and keep the looser-fit feeling, if that is what you are aiming for.
I find this video tutorial helpful for top-down short row bust shaping:
Lovely colour!