A person’s physical and social environment is considered as an influencing factor in terms of rates of engagement in physical activity. This study analyses the influence of socio-demographic, physical and social environmental factors on physical activity reported in the adult population in Andalusia. This is a cross-sectional study using data collected in the Andalusia Health Survey in 1999 and 2003. In addition to the influence of the individual’s characteristics, if there are no green spaces in the neighbourhood it is less likely that men and women will take exercise (OR = 1.26; 95% CI = 1.13, 1.41). Likewise, a higher local illiteracy rate also has a negative influence on exercise habits in men (OR = 1.39; 95% CI = 1.21, 1.59) and in women (OR = 1.22; 95% CI = 1.07, 1.40). Physical activity is influenced by individuals’ characteristics as well as by their social and physical environment, the most disadvantaged groups are less likely to engage in physical activity.