Research shows that the association between adverse climate conditions and human migration is heterogeneous. One reason for this heterogeneity is the differential vulnerability of populations to climate change. This includes highly vulnerable, “trapped” populations that are too poor to migrate given deep and persistent poverty, the financial costs of migrating, and the erosion of already fragile economic livelihoods under climate change. Another reason for this heterogeneity is the differential vulnerability of places. However, despite the growing list of studies showing that the climate-migration relationship clearly varies across places, there is surprisingly little research on the characteristics of places themselves that trap, or immobilize, populations. Accordingly, we provide the first account of the “holding power” of places in the association between adverse climate conditions and migration flows among 55 districts in Zambia in 2000 and 2010. Methodologically, we combine high-resolution climate information with aggregated census micro data to estimate gravity models of inter-district migration flows. Results reveal that the association between adverse climate conditions and migration is positive only for wealthy migrant-sending districts. In contrast, poor districts are characterized by climate-related immobility.