Learning or performance gains are often measured in improvements of test scores over time. However,
learners may also answer questions incorrectly which they appeared to master previously, making
claims about overall learning gains problematic. Using data from 1,211 grade 6 learners in the
Umgungundlovu district in KwaZulu-Natal, we interrogated the consistency of the improvement of their
test scores after a year in grade 6. The learners had completed the same 40 question multiple choice test
at the onset and the end of grade 6, with no interventions only normal schooling in-between. The content
of the test was grade 5 and 6 mathematical content. We know from previous research that the learners
were not randomly guessing, yet we found that on average, the learners only got 6 out of 40 questions
correct on both test 1 and test 2, did worse on 5 questions and better on 6. Further, almost half the
learners both improved on 5 or more questions and declined on 5 or more questions. On 35 questions,
more than half the learners changed their answers between the two tests. In other words, their
performance is very ‘unstable’. This indicates that learners are not consistent in their thinking, possibly
reflecting guessing, albeit not random. Furthermore, the findings point to a methodological issue of
making claims about average performance gains, as these may disguise very unstable learner
performance. Such claims should therefore be interrogated before they are applied in making statements
about links between learning and teaching or other background factors.