This paper measures the impact of inefficient spatial distribution of crops on aggregate agricultural productivity in Uganda. By combining village level data on land use and on crop specific land suitability, I show that agricultural TFP could be increased by one third just by reallocating crops according to the underlying structure of comparative advantage. Interestingly, a regional decomposition indicates that half of these gains can be achieved just by redistributing crop production within narrowly defined areas serving the same urban markets. The empirical analysis suggests that differences in market access are a good candidate to explain these inefficiencies: in line with the qualitative theoretical model, more isolated farmers devote systematically more land to non-perishable food crops and their production is less aligned with the agro-climatic conditions they face.