Recent trends in Ugandan agriculture indicate output is increasing for most crops but at a declining rate. The pervasive decline in output per unit of input (factor productivity) threatens the economic well-being and food security of producers and consumers of food in Uganda. Many households are both agricultural producers and consumers and have little access to other sources of livelihood (UBOS 2007). For these households, the declining productivity is particularly serious. Evidence is presented in this paper that improved agricultural inputs and extension information contribute to higher yields, but rates of adoption of improved agricultural inputs are low and many farmers have no access to extension services. Despite the low productivity, marketed surplus as a share of total agricultural output has increased for many food crops (PMA 2007). On the basis of the regression results, we address policy-related “levers of productivity” that influence agricultural output in Uganda. All agricultural inputs besides land (labor, fertilizer, chemicals, improved seeds, and agricultural assets) have a positive and statistically significant impact on output per acre on farms in at least one of the three size categories. Schooling, agricultural know-how, and credit also have positive effects on per-acre output. Land is the exception among inputs: the finding that expanding cultivated acreage is associated with a decline in output per acre is robust across a large number of model specifications with which we experimented during the analysis. Clearly, Uganda's food security problems cannot be solved at the "extensive margin," but must be addressed by greater use of modern inputs and technology.