I sympathize. I've been knitting for a new granddaughter who's on the way. Haven't had little babies around for quite a while. I made newborn baby booties, thinking they looked gargantuan, only to find that on an actual baby they were too small. I say trust the pattern. A lot depends, too, on how much the baby/toddler weighed when first born. A newborn pattern for a typical 7 pound baby isn't going to fit an 8 or 9 pounder. For a toddler who was a big baby, you might even want to go up a size.
I knit things for the little babies and toddlers in the neighborhood to test the pattern. The kids love the surprise gifts. I also ask the mom's permission to try projects on their kids. They all know that I'm a crazy knitter now. Although (while utilizing one of my test subjects recently) I did get a comment about why I'm knitting mittens in the middle of a hot summer. They just don't get it. Ya gotta get 'er done early. It's too late when the first snows come screaming outside your door.