The relative importance of ancient demography and climate in determining worldwide patterns of human within-population phenotypic diversity is still open to debate. Several morphometric traits have been argued to be under selection by climatic factors, but it is unclear whether climate affects the global decline in morphological diversity with increasing geographical distance from sub-Saharan Africa. Using a large database of male and female skull measurements, we apply an explicit framework to quantify the relative role of climate and distance from Africa. We show that distance from sub-Saharan Africa is the sole determinant of human within-population phenotypic diversity, while climate plays no role. By selecting the most informative set of traits, it was possible to explain over half of the worldwide variation in phenotypic diversity. These results mirror those previously obtained for genetic markers and show that 'bones and molecules' are in perfect agreement for humans.