It's only a model...

Shaw once wrote that “England and America are two countries separated by a common language.”. To a lesser extent, scientists and lay people are two similarly separated groups. The problem is that often the word one uses doesn’t mean what the other person thinks it means [1].

Take, for example, the word “theory”. To a lay person, the word “theory” is roughly equivalent to guess, idea, hunch, or glimmering. Thus, calling something a theory means that you aren’t sure what really happened, but this explanation is as good as any. To a scientist, the word “theory” means that an idea has been sharpened using multiple tests and developed until it offers the best possible explanation of the observations. Thus, when a scientist says that evolution is a theory, she means that evolution is the best explanation for the diversity and variation of the natural world, just as saying that relativity is a theory means that it best explains the way that matter behaves at very high speeds [2].

Another good example is the word “model”. To a lay person, a scientist’s model is just a crude approximation and shouldn’t be used until it can provide “perfect predictability”. For example, take these comments on the use of models in climate change research:

But when I requested facts, again, I didn't mean to put you on the spot. I only wanted to demonstrate that to my knowledge, no one can point to a single fact regarding man-made warming. It's all speculation no matter how "close" they get with their statistics and modeling.

It's only a model and models cannot account for everything, which actually causes whatever warming or cooling takes place in our atmosphere. Then there is the bias of the creator of the models and the motives behind their results

But just getting one to mimic the past would be a good start, but here's the problem even with that - it would have to track actual past events -perfectly- to even have a remote chance for predicting the future, and even if one did, it shouldn't be taken as... um... gospel, as members of the religion no doubt will.

To a scientist, models allow us to predict how things will behave. Thus, everything is a model, from the laws to the theories to the hypotheses to the very measurements that we make. A necessary result of this is that we must use the best models we have, but be aware of their limitations.

The first limitation of models is that they need measurements. This is a limitation because every measurement is a model, too! For example, take out your ruler and measure a sheet of paper. What were the dimensions? Did you get 8.5” x 11” x .004”? Or did you get 8.51” x 10.58” x .003”? Or some other number? Try measuring another sheet of paper. Do you get the same dimensions? Or different ones? Now ask someone else to measure those same two sheets of paper –I’ll bet that they get different results.

The reason is  that all measurements have two characteristics: accuracy and precision. An accurate measurement is one that comes close to the true value. For ordinary note paper, that’s 8.5” x 11” x .004” on average. But not everything is as easy to measure as paper! Scientists typically use different types of measurement to determine how accurate their measurements are. If all of the measurements give the same result (within the margins of error), then they know that the results are good enough to test their ideas with. If you have gotten a value of 6” x 13” x .1” when measuring the paper, then odds are your measurement wouldn’t have been very accurate.

Similarly, measurements of something are precise when they all cluster around the same value. If you have gotten a value of 6” x 13” x .1” when measuring the paper then 8.51” x 10.58” x .003” when you repeated the measurement, your measurements wouldn’t have been very precise. Precision is important because it allows us to lower the error bars on our measurements and know that we are truly measuring what we think we are measuring.

A classic example of this is the neutrino deficit problem in solar physics. According to the theory for our Sun’s fusion reaction, it should be producing more neutrinos than we detect. One of three things could be the answer:

I)       Our measurements could be inaccurate

II)     Our measurements could be imprecise

III)    Our theory could be wrong

Scientists tested the measurements several ways and determined that they were precise. And they tested the theory using other data and decided that it was probably right. The only thing left was that the neutrino measurements weren’t accurate for some reason. So they shot a beam of neutrinos at a detector from various particle accelerators and have discovered that neutrinos change “flavor”. Because the detectors can only “see” one type of neutrino, they were missing the others, leading to the apparent deficit[3].

Why are accuracy and precision important to models? Because models are like viaducts – what you get out of them depends on what you put into them [4]. Let’s take a simple model with a linear function:

Y=X

This is what happens when measurement error is introduced:

Model1

The true answer lies somewhere inside the yellow zone. But all we can say from the model is that, for X=15, the result( Y) is somewhere between 13.5 and 15.5. This is part of why scientists spend so much time concentrating on reducing measurement errors; buy doing so, we can reduce the uncertainty zones.

Now let’s try it for something a bit more challenging – the non-linear function Y=X*X:

Model2

The error range has increased significantly. Now when we “know” X=15, we can only say that Y is somewhere between 182.25 and 272.25.

Let’s examine one more example of how measurement errors can make models more challenging. This time, we’ll use a simple trigonometric function:

Y=tan(X) 

Model3

Notice that this time the errors make it very difficult to tell what the true value of Y should be. This function is what scientists call “chaotic” – a very small change in the inputs can cause a large change in the outputs. If we knew what the true value of X was, then it would simplify to this:

Model3a

Or would it? You see, there’s another bias that’s been built into the measurements – how often do we make them? The graph above shows what we get if we measure X only for the integers. This is what we get when we measure it twice as often (every 0.5):

Model3b

And this is what we get when we measure it every 0.1:

Model3c

Notice that the patterns are starting to resemble each other. That’s when we know that we are close to the best measurement interval for a particular phenomenon. If adding more measurements doesn’t make the pattern change significantly (i.e., doesn’t mean that you would predict something else based on the results), then you have enough measurements..

But even with enough measurements taken with enough precision and enough accuracy, can you achieve perfect predictability using a model? No.

The reason is that those limits pass through the model and create a level of blurriness that limits what can and cannot be said. That’s why scientists use multiple models run multiple ways and feed them with multiple measurements from multiple sources. Only when the majority of the models agree can we say that what is predicted is likely to happen [5]. The most that we can say about a model’s output is called its resolution. That tells us the dimensions that we can predict, including volume, time period, and outputs.

It isn’t just climate modeling that is subject to these problems with modeling resolution. It is seen in cosmology, seismology, biology, chemistry, and every other field of quantitative science.

Does this therefore mean that we can’t use the predictions of climate change models to decide what is happening now and what might happen later? Not unless you are also willing to refuse to use the medical models that tell us how to use vaccines, and the chemical models that formulate your vitamins, and the physics models that power your lights.

Does it mean that we should blindly accept what the models say? Heck no – no more than you would blindly accept the advice of a doctor. Instead, you should check, and be skeptical. Just don’t expose your ignorance and ask for impossible things.

John

[1] This is not the Humpty-Dumptyism of groups such as politicians. Nor is it the simple erroneous usage of the well-meaning but ill-read that give use such egregious boo-boos as “octopi” [a] and “enervatingly strong” [b]. Rather, it is in the Vizzini-inspired sense.

[2] Amusingly, the word “hypothesis” has almost exactly opposite meanings in the two groups as well. To a scientist, a hypothesis is just a working idea that has some support but really, really needs to be tested to get the bugs out. To a lay person, calling something a hypothesis frequently elevates it to near “law” status.

[3] The good news about this is, as one physics wag put it, “Now we know the Sun isn’t going to go out any time soon.”

[4] Geek points for the reference!

[5] This is another one of those words that means different things to lay people and scientists. To lay people “likely” typically means better than even odds that something will occur. To a scientist, “likely” means that there is a 95% probability that something will happen (unless they state lower likelihood, such as the 90% probability that is common in climate modeling or the 80% that is seen in many education works).

[a] As any good linguist will tell you, the proper plural of “octopus” is either “octopodes” or “octopuses”. The word “octopus” comes from the Latin “octo” for eight and the Greek “pus” for foot. Greeks do not conjugate their nouns the way the Latins did; thus, using “octopi” for the plural of “octopus” is as erroneous as using “womans” for the plural of “woman”.

[b] “Enervatingly” means lacking in strength; however, a surprising number of erstwhile scholars use it to mean the exact opposite.