In 1994, Benjamin S. Bloom wrote,
Using the standard deviation (sigma) of the control (conventional) class, it was typically found that the average student under tutoring was about two standard deviations above the average of the control class (the average tutored student was above 98% of the students in the control class). The average student under mastery learning was about one standard deviation above the average of the control class (the average mastery learning student was above 84% of the students in the control class)
The reader who sent me the link to this paper asked whether it invalidates the Null Hypothesis. The researchers experiments rather than observational studies, so I will give them that. But
1. Usually, educational interventions have small effects. The effect of “mastery learning” of one standard deviation seems rather implausibly high, considering its definition.
Formative tests (the same tests used with the conventional group) are given for feedback followed by corrective procedures and parallel formative tests to determine the extent to which the students have mastered the subject matter.
That does not sound like something that would cause a one-standard deviation difference. My guess is that these findings would not replicate if they were undertaken by different researchers.
2. Even when interventions show large effects for a single subject in a single year, the effects tend to fade out. That is, if you examine the experimental group and the control group three years later, any difference has vanished. Even if these results replicate in the short term, they do not invalidate the Null Hypothesis if they suffer from fade-out.
3. We do not know how what would be necessary to enable tutoring to scale. Bloom seems to believe that tutoring works by adapting to the needs of the student. If so, then my guess is that the process of matching tutoring style to student characteristics would be quite a challenge.
This is Bloom’s two-sigma problem, which we’ve mentioned before.
One problem with Bloom’s two-sigma problem is that Bloom compared classroom mastery learning against one-on-one tutor-based mastery learning, without holding the goal-level of mastery constant.
His Developing Talent in Young People finds that prodigies didn’t start off with prodigious skills, but they were allowed to pursue their interests at their own accelerated pace.
I haven’t read the linked article but the quote under 1. suggests that the experimenters drilled the students about a number of facts, tested for memory of those facts, drilled again, tested again, and continued until the students tested perfectly (or up to some standard they defined as “mastery”).
That is basically what teachers do today, just on steroids. Teachers teach, test (“formative assessment”), reteach, but since time is limited then give a final test (“summative assessment”). Young people have a wonderful capacity for short term memory and much of what this does is just test willingness and ability to hold knowledge in short term memory.
Teachers then pretend (to themselves, to their students, and to the educational establishment) that this knowledge stays with the students. Even though all of us in the business know that knowledge “decays” when it is not used. Our ability live with, and hardly notice, the contradiction is a fact of life.
Almost all educational research is short-term. Which is fine if the aim of education is to develop the ability to memorize and use information for a limited period of time and then forget it–which is actually a very useful skill. However, if the aim of education is to learn things you don’t forget, such research is largely useless.
Slate Star Codex: “If there is any possible way it can be selection bias, it’s selection bias.”
= – = – = – =
I think tutoring can help people learn things over the long term if they really want to learn it. I tend to be agnostic about this stuff.
You know the kids in high school who want to own their own calculus textbook so they will have it with them in college, where they expect to use it again? They can benefit from tutoring.
You know the people who are planning to take a CPA exam or an actuary exam, and they think every topic they learn is important, because passing the exam is a personal goal and brings a big boost in income? They can benefit from tutoring.
= – = – = – =
Some of it might be time on task, and the benefits of coaching. A lot of people benefit from coaching. Can tutoring sometimes be the same as coaching?
Yes, no one is addressing learning loss. One on one tutoring, properly matched, would help unengaged kids learn more for the immediate test. As a teacher, I’ve found the learning loss to be astounding.
I believe we should try teaching far less each year (what we teachers usually call Big 5), teaching it over and over again in different contexts. Add in some new material but don’t expect mastery until it’s the Big Five for the year.
So in algebra one: lines in three forms (counting as two), systems, factoring, basics of coordinate geometry.
Geometry: special rights, right triangle trig, pythagorean theorem, similarity, geometry test facts.
A2: quadratics, power laws, factoring, function concepts & notation, something else I’m forgetting about.
That sort of approach would allow us to better assess long-term understanding and also interventions like tutoring, extra time, etc.
@Arnold: Thanks for formulating The Null Hypothesis in Education. When I first ran across it a few years ago (or maybe more now), it clicked together a lot of things I’d read about educational optimizations.
Do you think the null hypothesis works in both directions?
That is to say, could educators do anything that would derail students’ long-term educational progress, at the scale of a large number of students?
Presumably the threshold of physical harm to the brain (which is pretty morbid) is a line beyond which the null hypothesis would be invalid. But if we’re just talking about really, really bad teachers, or eliminating school altogether, do you think students would ultimately wind up in about the same place?
Beyond a certain point, the null hypothesis definitely doesn’t work in both directions. Very, very few schools in America are at that point but many schools around the world are. Lant Pritchett’s The Rebirth of Education: Schooling Ain’t Learning (Center for Global Development, 2013) is about how the successful campaign to build schools around the world has not been accompanied by successful use of the schools. It is full of sad statistics on educational achievement.
Chicago Tribune is running a sad series on the state of public high schools. This one is more evidence for the null hypothesis.
http://www.chicagotribune.com/news/ct-chicago-schools-funding-grants-20171204-story.html
My personal experience with mastery learning is that it is very effective. I was able to progress twice as fast as the standard pace through math for several years, before it was discontinued in the school district. It simply allowed people that can learn faster than average to move at the pace they are capable of, instead of waiting for the pace of the class. In addition, it simply saves time by preventing people who can already pass a test at a 90% level from taking the class again. I would imagine tutoring is even more flexible in that direction.
For people that are not making progress at the standard pace, I think the benefits of mastery learning show up on a longer timeline, as they are not “socially promoted” to the next topic or level without demonstrating mastery, despite a gap in foundational knowledge or skills.
So despite significant evidence of success, why is it not practiced? This suggests a corollary to the null hypothesis…”Despite the empirical evidence, many mastery programs in schools have been replaced by more traditional forms of instruction due to the level of commitment required by the teacher and the difficulty in managing the classroom when each student is following an individual course of learning.” So, if an educational intervention requires significantly more work from the teacher, it is unlikely to be implemented well, regardless of its effectiveness.
Individual mastery learning, fully differentiated, could see each student working on a different topic. It would reduce the opportunities for the teacher to lecture to the class, and increase the level of individual support needed to answer questions for each student…almost like tutoring.
“”Despite the empirical evidence, many mastery programs in schools have been replaced by more traditional forms of instruction due to the level of commitment required by the teacher and the difficulty in managing the classroom when each student is following an individual course of learning.”
I think the same thing goes on in the Judicial system with things like the HOPE program that Mark Kleinman wrote about.
In terms of education the book “Teaching Needy Kids in Our Backward System” is at once a story of one of educations greatest success stories, and a depressing story of political failure.
the difficulty in managing the classroom when each student is following an individual course of learning.
Difficulty? Difficulty?!?! Impossibility! Perhaps if you had 24 extremely self-motivated kids, but then you wouldn’t need a teacher in the first place.
The obvious way to get closer to an “individual course of learning” is to “track” students into faster and slower groups. But tracking is a very big no-no in modern education. Partly because blacks and hispanics wind up “over-represented” in the slower groups, while whites and Asians are “over-represented” in the faster groups.