A trial where personalised teaching helped some students, not all
A large trial found that personalised teaching made no difference to maths results on average. For the students with the lowest scores at the start, it made a large one.
In 2023, de Barros, Ganimian and Muralidharan published a trial of personalised teaching with 3,331 students. Students were put into groups at random, and the researchers published their plan before the trial began. The main finding was no significant average effect on maths results.
Most people would stop reading there. If you sell personalised teaching, that is an awkward number. We think it is the most useful number we have.
The average was hiding two different results
The trial also reported results by where students started. For the students with the lowest scores at the start, the effect was +0.22 standard deviations, the unit education research uses to compare results across studies. That is a large effect in education research. The average was close to zero because, for the other students, personalisation added little.
Both halves are true at once. Personalised teaching does not improve teaching for everyone. It does most of its good for one group of students.
What that means for who we build for
Chinese education has a phrase for this: 補底, not 拔尖. The first means supporting the students with the most to gain. The second means pushing the students already doing well further ahead. They are different jobs, and they need different tools.
We build for the first. The second is worth doing too. But this trial found the effect in the first group, and those are the students the current system serves least well.
This is also why we design for students with SEN first, rather than adapting the product for them later. That decision is ours, and the evidence for it is weaker. The trial says where personalisation helps most. It says nothing about SEN specifically. We hold the two claims at different strengths on purpose.
The uncomfortable part
Companies often leave a result like this out. We cite it for two reasons. Without it, we would be claiming that personalisation works for everyone, and this trial says it does not. And a tutor who tries our software with a student it does not help will rightly decide we oversold it.
The claim we make is narrower, and it can be tested. In this trial, for the students with the most to gain, structured personalisation had a real effect. Everything else we say about our own product is still unproven, and we label it that way.
Source: de Barros, A., Ganimian, A. J., & Muralidharan, K. (2023), randomised trial with its plan published in advance, n = 3,331. The finding about who gained most is measured, with a comparison group. The decision to design for students with SEN first is our own reasoning, not a measurement.
Bloomate