Smallholder farmers face a number of challenges in spite of the crucial role they play in ensuring food security. An appreciation of the far-reaching consequences these challenges have on food security will inform the formulation of policies aimed at addressing the problem of food insecurity. In Ghana, research on food (in) security issues has mostly focused on the Northern part of the country partly because of the number of development challenges that face the north. This has shadowed the food insecurity situation in the South. I have therefore attempted to understand the underlying causes of food insecurity by examining the challenges confronted by smallholders in the Wioso and Woraponso areas of the Asante Akim North Municipality in the Southern part of Ghana. The research relied on qualitative methods such as focus group discussions and semi-structured interviews in constructing data with smallholders, officials of the Ministry of Food Agriculture and other relevant stakeholders. The rights-based approach which encompasses attempts to reduce poverty through good governance and safeguarding the interests of marginalized groups constituted the main theoretical basis for this research. The research identified a number of difficulties faced by smallholders including inadequate access to credit facilities, poor access and use of modern inputs such as planting materials and agrochemicals. Concerted efforts by government, NGOs and farmer-based organizations are needed to address these difficulties. Empowering smallholder farmers so that they are able to explore other livelihood strategies and adapt to new innovations in their farming activities will help ensure food security.