Virtual villagers 3 population help?

My population is only 6 and no matter how many times I get the adult villagers to try and make babies it never works. I've used fertility potions and that hasn't worked either. All the huts have been built so I can support a larger population but the villagers just aren't making any babies. Any help?