“Research-based” does more harm than good

Natural science produces knowledge about cause and effect. Biologists, chemists, and physicists use the scientific method to discover laws that can be used to shape the environment to human desires.

A good example is the invention of air conditioning. Compressed air is cooler than non-compressed air, so air conditioners compress air and blow this cooler air into the room. So now we have air conditioning and more comfort on hot days.

Social science produces knowledge about correlations. Social scientists statistically control multiple factors to discover a single factor’s causation probability. They create theoretical situations in which, “when all other things are equal,” a correlation can be described as one factor having a certain probability of causing another.

In TSVOTEP I explained that taking social science findings out of the situation in which “all other things are equal,” strips them of what makes them true, which is being in a situation in which “all other things are equal” (chapter 8). In the real world, all other things are never equal, so educators can’t know if a specific social science correlation holds true in their situation. So far, no harm done. Educators can ignore social science findings, realizing that all other things are never equal in the real world.

For years teachers and principals have ignored educational research, but now that they are supposed to use “research-based” methods, they need to challenge this idea with a stronger argument for why they should pay no heed. As this blog explains, one reason is that social science findings sometimes do more harm than good.

Correlations are all around us, and they are important to understand. The links between experience and correlation are usually direct and concrete, so we don’t need social science to discover them.

For example, we understand the correlation between a final score (one team won) and the higher skill level of the winning team’s players. In childhood games we understood this correlation, so we “chose sides.” We wanted to equalize skill levels so games were more fair, more challenging and more fun. We understood those correlations, too.

Social science correlations can be instructive; but they do more harm than good, if they are interpreted as wrong-way cause-effects.  The “effective” principal literature of the 1970s and 80s is an example. Correlations were misinterpreted to mean, “an effective principal is the key to an effective school.”

Addressing the following three questions explains why this misinterpretation caused more harm than good.

1. Why did scholars and school reformers reach cause-effect conclusions from studies that showed correlations?

2. Why was the correlation turned into a wrong-way cause-effect?

3. Why do superintendents, teachers, school board members, state legislators, and education reformers still believe “an effective principal is the key to an effective school,” even though it is not?

The first question asks why we expect social science findings to do what natural science does — provide cause-effect findings.  In the “effective principal” studies we looked at a common school experience — the correlation between effective schools and effective principals.  It is common because we often find both in the same place.  They are, in fact, correlated.  But social science research findings are more valued if they find something new, instead of what we already know.

In this case, we already knew the key to being an effective principal is being in an effective school.  So researchers turned the correlation into a wrong-way cause-effect — “the key to an effective school is an effective principal.”  Later we will see that this is not true, and believing it does more harm than good.

First, here is a different example.  A recent headline said my shirt color says something about my personality. I already know my choice of shirt color is influenced by my mood and my mood is influenced by my shirt color. This correlation is a common experience.  But a headline suggesting that my choice of shirt color is caused by some deep personality trait makes the same experience uncommon.  Its purpose, of course, is to entice me to read the article, which is the same purpose for claiming the key to an effective school is an effective principal.  There is no surprise if the headline reads, “Your mood influences and is influenced by your shirt color.”  Similarly, there is no surprise if the headline reads, “Effective schools correlate with effective principals.”

Think about the social science findings described in popular magazines (Psychology Today, Women’s Day, etc.). The headline never reads, “Researchers claim a cause-effect relationship for a correlation you experience all the time.” But that is what social science articles often report.

This leads to the second question — why are correlations turned into wrong-way cause-effects. Researchers want to surprise with their findings. For example, if they say my shirt color reveals some deep personality trait, I am surprised that a “study” could substantiate that. And the easiest way to surprise the reader is to take a correlation we experience all the time and turn it into a cause-effect in the unlikely direction.

Again, the findings of effective principal studies are an example. We know that “being in an effective school is the key to being an effective principal,” so researchers ask if the less likely cause-effect is also true. In a few cases it is, but this still does not shock us, until this finding is misinterpreted to mean “an effective principal is the key to an effective school.” There you have it — surprise and wrong-way cause-effect.

The third question asks why well-educated people believe this, even though they know it is not true.  If we look at who benefits from believing it, we can see why education research findings do more harm than good.

Many people benefit from believing “the key to an effective school is an effective principal.” If that is true, superintendents, legislators, school board members, teachers, parents, and students bear little responsibility for making a school more “effective.” According to this finding, until an effective principal is in place, there is little reason to try to make a school more effective. This misinterpretation serves the interests of those who (1) are powerless to make a school more “effective;” (2) want to appear as though they know how to do it; and (3) want no responsibility for making it happen. That covers a lot of people.

Some may think I am stretching things. But how else can it be explained that well-intentioned educational administrators and policymakers consistently repeat: “the key to an effective school is an effective principal,” even though they know an effective principal is not the key to an effective school? They know this because, if given a choice between sending their own children to a school with a mediocre principal and an effective faculty, or a school with a mediocre faculty and an effective principal, they will choose the first. An effective principal became the key to an effective school in the 1980s, even though we know an effective faculty is the key to an effective school.  Clearly, heeding this wrong-way, cause-effect conclusion has done more harm than good. Think of the time spent trying to hire an effective principal, which could not be spent building an effective faculty, which we know is the key to an effective school.

The natural world is complex but predictable. The social world is complex and unpredictable. We need to understand this, or we do more harm than good by pursuing surprising, wrong-way, cause-effect conclusions.

The “effective” principal literature is a case in point. We know school effectiveness is complex and unpredictable, but we want social science to discover something new and predictable. Researchers accommodate with a finding that says “an effective principal is the key to an effective school” — a statement that (1) serves the interests of powerful people, (2) is rarely true, and (3) does more harm than good.

In conclusion, I did not need a single study to tell me there is a correlation between good schools and good principals. It took multiple studies, however, to proclaim what is NOT true — that the key to an effective school is an effective principal. I love irony.