Fingerless mitts question

My fingerless mitten pattern says to cast off one stitch at end of row and continue knitting or purling. Says to do this four times. If I cast off four stitches and I originally cast on 24 stitches, I will only have 20 cast ons. Does not call for any increases. Would my mittens be too small?

What pattern are you using? Can you post a link please?

If the given dimensions for the mittens fit your size and you’re on gauge, you should be fine just following the pattern. If the mittens look small to you it may be that you need to adjust the pattern.
I can’t tell where you are in the pattern. What pattern are you using? Can you give us a link to it or a pattern name, please?