Data for Find the Fake: Boosting Resistance to Health Misinformation in Jordan with a WhatsApp Chatbot Game, 2022

Judgements of Misinformation
Adapted from Roozenbeek et al. (2022), participants were presented with six headlines: three using misinformation tactics and three that did not use common misinformation tactics. When presented with each headline, participants were asked to respond to the question ‘Does this headline use any misinformation techniques?’ on a 4-point scale: Definitely is misinformation, Probably is misinformation, Probably is not misinformation, Definitely is not misinformation. For ease of interpretation, ratings were scored such that higher scores represent stronger belief that a headline was misinformation (Definitely is misinformation = 4 and Definitely is not misinformation = 1). We chose phrasing that highlighted the term misinformation to increase relevance to policymakers and practitioners, but it is worth noting that much prior research on inoculation measures perceptions of misleadingness rather than misinformation classifications per se (e.g., Roozenbeek et al., 2022).
In line with prior literature, we compute three scores to assess accuracy in misinformation detection (Basol et al., 2021; Maertens et al., 2021). First, we calculate a measure of discernment, defined as a participant’s average misinformation scores for misleading headlines minus their average score for headlines without misleading content. With this operationalization, discernment scores could range from -3 to +3 where a score of +3 indicates a participant rated all misinformation headlines as ‘Definitely misinformation’ and all non-misinformation headlines as ‘Definitely not misinformation’ for perfect discernment.
We also examine the disaggregated discernment score including the average ratings for the three misleading headlines and ratings for the three non-misleading headlines. As higher ratings correspond to judgements that a headline is using misinformation tactics, more accurate scores would be represented by higher scores on the misleading headlines (representing true positives) and lower scores on the non-misleading headlines (representing true negatives).

Sharing Misinformation
Adapted from Basol et al. (2021) and Roozenbeek and van der Linden (2020), sharing of misinformation was assessed with two headlines: one that did not use misinformation tactics and a headline that used misinformation tactics. Each participant was randomly assigned to one of two misleading headlines, a headline that used extreme emotion or a headline that used a false expert. Participants were asked to rate their likelihood of sharing each of the headlines on a four-point scale: (1) Very unlikely to share, (2) Unlikely to share, (3) Likely to share, (4) Very likely to share.
As with judgements of misinformation, we report three scores for sharing—discernment of sharing, likelihood of sharing misleading headlines, and likelihood of sharing non-misleading headlines. Discernment of sharing was calculated as the sharing score for non-misleading headline minus the sharing score for the misleading headline. Accordingly, participants with high positive discernment of sharing have lower intentions of sharing misinformation, relative to sharing more non-misinformation.

Self-Report Outcomes
Complementing the evaluation of the intervention’s impact on detection and sharing of misinformation, we examined differences in attitudes toward the chatbot game.

Confidence
Participants were asked to indicate the extent to which they felt more, the same, or less confident in detecting misinformation after completing the game.

Perceived Difficulty
Participants were asked to report whether they thought the game was too difficult, the right level of difficulty, or too easy.

Recommending the Game
Participants were asked to indicate the extent to which they would recommend the game to others with three possible response categories: yes, maybe, and no.

For more information, please find our Working Paper.