I've stuffed all my toys with Polyfil without complaint on any of the recipients' parts. It's fairly inexpensive at the Michaels that is my only source for crafty stuff in town.
When you say "the holes in the fabric", do you mean the little holes that you can see between the stitches when you stretch out the knitted fabric? If so, that's why most patterns recommend using a needle smaller than what the yarn would usually require - it makes the fabric tighter, so you can stuff it without worrying about the filling showing through. If it's a light-coloured object, you don't need to worry too terribly much about the filling showing because it won't be very noticeable anyway, and it's pretty difficult to pull it out unless the holes are really big.