Objective: To determine the difference in delivery success of health messages delivered through pushed SMS, pushed voice messages sent to personal phones, and voice messages retrieved from a community phone (“retrieved voice messaging”), as well as the difference in quality of the user experience.

Methods: We analyzed the project's electronic monitoring data between September 2011 and June 2013, including demographics, enrollment data, and messages sent and successfully delivered. We also collected and analyzed information from quarterly phone-based surveys with users to assess quality of the user experience, including acceptability, comprehension, new information learned, and reported behavior change.

Results: More than half of subscribers enrolled in the retrieved voice messaging service while nearly one-third enrolled in the pushed SMS service and less than 10% in pushed voice messaging. Message delivery success was highest among pushed SMS subscribers and lowest among retrieved voice subscribers. Overall, 99% of survey respondents reported trusting messages they received, and about 75% of respondents recalled the last message they received and learned something new. Almost 75% of respondents reported that they had already changed or intended to change their behavior based on received messages. Intended or actual behavior change was significantly higher among pushed SMS enrollees than among pushed or retrieved voice messaging enrollees (P = .01).

Conclusion: All message modalities led to high levels of satisfaction, comprehension, and new information learned. Due to lower cost, higher delivery success, and higher levels of intended or actual behavior change, SMS is the preferred delivery modality. However, the majority of users included in this study did not have access to a personal phone, and retrieved voice messages provided an opportunity to access a population that otherwise could not be served. Providing multiple methods by which users could access the service was crucial in extending reach beyond literate personal phone owners.