Protected forests are sometimes encroached by surrounding communities. But patterns of cover change can vary even within one given setting – understanding these complexities can offer insights into the effective maintenance of forest cover. Using satellite image analyses together with historical information, population census data and interviews with local informants, we analysed the drivers of forest cover change in three periods between 1973 and 2009 on Mt Elgon, Uganda. More than 25% of the forest cover of the Mt Elgon Forest Reserve/National Park was lost in 35 years. In periods when law enforcement was weaker, forest clearing was greatest in areas combining a dense population and people who had become relatively wealthy from coffee production. Once stronger law enforcement was re-established forest recovered in most places. Collaborative management agreements between communities and the park authorities were associated with better forest recovery, but deforestation continued in other areas with persistent conflicts about park boundaries. These conflicts were associated with profitability of annual crops and political interference. The interplay of factors originating at larger scales (government policy, market demand, political agendas and community engagement) resulted in a ‘‘back-and-forth’’ of clearing and regrowth. Our study reveals that the context (e.g. law enforcement, collaborative management, political interference) under which drivers such as population, wealth, market access and commodity prices operate, rather than the drivers per se, determines impacts on forest cover. Conservation and development interventions need to recognize and address local factors within the context and conditionalities generated by larger scale external influences.