Why do we still vary?

Razib Khan:

I notice that last summer Karl Smith asked “Why Are There Short People?” His logic is pretty good, except for the fact that the fitness variation seems to be much starker in males than females (there is some evidence I’ve seen that shorter women can be more fertile, though that’s balanced by the fact that larger women seem to be able to manage gestation better). In any case, height seems to be a fitness enhancing trait which is highly heritable, and yet the variation in height remains!
Karl’s readers offered some reasons. What do you think? Mind you, something which immediately comes to mind is that the logic presented for why everyone should be tall and vary only a touch is logic. Not all the assumptions need to hold. For example, has the advantage to height been invariant at all times and places? I have posited for example that the fact that humans became smaller after the Ice Age may have something to do with increased morbidity and declining mortality, where agricultural settlements “hugged” the Malthusian boundary more consistently than hunter-gatherers. In this sort of environment smaller individuals may have gained a fitness advantage because they required fewer resources to make it through the inevitable “starving times.”*