Forest cover in Sri Lanka has decreased rapidly during the last century and only fragments of the once widespread natural forest cover remain. This paper analyzes forest cover change and small scale farmers’ relation to natural forests around two protected forest areas in Sri Lanka; Kanneliya Forest Reserve and Knuckles Conservation Forest. Methods used are spatial analysis to observe changes in forest cover from the 1980s until 2010, interviews with small scale farmers and key informants as well as field observations. In Kanneliya Forest Reserve, a decrease in forest cover is observed, particularly due to population increase and expanding tea plantations. In Knuckles Conservation Forest on the other hand, we find an overall increase in forest cover due to expansion of tree plantations, a ban on shifting cultivation and emigration from the area followed by natural forest regeneration. Agriculture is the most common source of income in both study areas and there is a clear link between conversions of forests to agricultural expansion. The profits from agricultural activities are in general insufficient to sustain small scale farmers’ needs and the most common alternative source of income is achieved through resources extracted from the forest. Since 2001, demarcation of forest boundaries around the two forest reserves has reduced encroachment and illegal felling of timber. However, this policy has simultaneously threatened the livelihoods of peripheral communities in the forest buffer zones, especially in the investigated villages around Knuckles Conservation Forest. Despite successful attempts to reduce deforestation rates through governmental interventions, further incorporation of local people into the management of forests as stipulated in the current forest policy should be continued.