Pound of love is a worsted weight #4. If you knit it on a smaller needle it may be somewhat stiff and dense.
The needle given on the yarn label isn't really the one 'recommended' for the yarn, it's the one that gets the gauge which classifies the yarn into a weight class. The gauge and needle size for this yarn would put it in the worsted range. Most baby yarns are fingering weight and would use a size 3 or 4. Seethis chart, and this one.
Many patterns use a different needle than the one on the label and it also depends on what you're making. Sock yarn used for socks is knit on size 0 to 3 because you need a dense knit for longer wearing. The same yarn used in a lace shawl can be knit on size 4 up to a 10 to make it loose and open.