The only time I don't keep left-over yarn is: if I decide it's ugly and I'll never use it (then I donate it), or if it's a complete skein of really expensive leftover yarn and I can return it to the LYS.
It doesn't matter how little the left-over ball is. If I like it, I'll keep it! I have a couple of zip lock bags of all my teeny-tiny balls; one day I'll make some wild project with them! I like to sort them by similar type, yarns that work well together.
I've just been sorting through my yarn today, and getting more organized. I've recently decided to use zip-lock bags galore. All of my works-in-progress are now in their own bag, with their associated balls of yarn. All of my left-over skeins are in their own bag, with same-color or otherwise matching skeins.
The nice things about keeping my yarn in zip-lock bags, besides keeping it well sorted, is 1) it never gets tangled, 2) it keeps the skeins in like-new condition, even when they're carried in my knitting bag for weeks. This is especially good if you think you'll want to return extra skeins to the store--they'll still look great. And 3) if the yarn labels fall off, you can keep them in the bag, so they don't get lost or mixed up with other labels. (Until I get organized enough to write down all the yarn types I've used with something I'm working on, I save the labels.)
I've got a big basket with all of my acrylic yarn in it. I don't use acrylic very often, but I have a huge stash of it from my younger crochet-afghan days. But since I don't like mixing it with my wools, in projects, I've got it seperated into it's own basket.
I've also got my "Stash Trunk," an old-fashioned trunk which I lined with a nice, mellow wrapping paper, and have organized with a few cardboard dividers (covered with matching paper). I've got three sections: my favorite wool stash, waiting to inspire me; my works-in-progress; and my finished objects that I haven't given away yet.
The best organizing decision I ever made as far as my yarn and spinning-wool goes, was to put up a couple of high shelves in my office/studio. They're about 16" from the ceiling, on opposite sides of the room. They are perfect for keeping baskets of yarn and wool-ready-to-spin. As well as a couple of boxes of things. I stained the shelves, and hung them up with nice brackets. They weren't that expensive, and they look great. And I don't know what I would do with all that stuff, if it weren't up and out of my way! It would probably be in the basement (tragic!). It's so nice to have it all within reach, and it actually adds to the atmosphere. Nothing like baskets with yarn peeking out, and nice wood, to look at!