This paper examines the use of remote sensing satellite data to predict food shortages among different categories of households in famine-prone areas. Normalized Difference Vegetation Index (NDVI) and rainfall estimate data, which can be derived from multi-spectral satellite radiometer images, has long been used to predict crop yields and hence famine. This gives an overall prediction of food insecurity in an area, though in a heterogeneous population it does not directly predict which sectors of society or households are most at risk. In this work we use information on 3094 households across Uganda collected between 2004-2005. We describe a method for clustering households in such a way that the cluster decision boundaries are both relevant for improved-specificity famine prediction and are easily communicated. We then give classification results for predicting food security status at a household level given different combinations of satellite data, demographic data, and household category indices found by our clustering method. The food security classification performance of this model demonstrates the potential of this approach for making predictions of famine for specific areas and demographic groups.