Studies of poverty dynamics relying solely on household income–expenditure surveys can overestimate transient poverty and underestimate persistence of poverty, especially for the poorest. In this study, we make use of an approach that relies on community based change ranking to explore various directions and levels of change experienced by almost 6000 households living in over 100 communities. We find that changes are initial condition dependent and that improvement, even small ones, are far less likely to happen over time for the poorest. An intervention that combines promotional and protective supports can bring such improvements in the livelihood of the ultra poor. Understanding the extent of and the forces that drive such smaller movements is important since it is the cumulative dynamics that ultimately lead to the larger movements of ascent, descent and trap, especially for the ultra poor.