I think you are being too hard on yourself. It may be a little loose when you attach the yarn, but you'll have to pickup stitches along that edge anyway when you work the neckband and that will obscure any problems. You can also tighten things up when you weave in that end.
If this just won't work for you, try making a loop with a slip knot, like you are about to start casting on. Place this look next to your first stitch on the left needle and knit it with your first stitch (like a k2tog). This will give you a nice clean start.