Would I have to adjust the pattern at all?
To work the patt to the desired gauge and sz you'd have to sub yarn that's the same weight. Subbing yarn that's very diff in weight would necessitate completely reworking all aspects of the pattern as the worsted will give you a smaller garment overall (length, width, position and number of incs/decs, neck and sleeve shaping, etc.) Super bulky to worsted could mean the difference between an adult sweater and one that fits a child. You might be able to get away with making a much larger sz if you can get the worsted gauge to match the measurements of the garment (check the schematic). (IOW, if your super bulky was going to be made up in a Med, an XL in the worsted might get close to the Med sz but you'd still probably have to adjust for length by adding rows, etc.)
ETA: similar issues raised in this thread