If you do this it won't slow them down, plus it just means that it takes longer to get to 6, which is when they disapear.
Actually there is always a fairy near the warp points, sometimes in a jar or in a bush, warping to the same place makes the fairy reapear.