Is there value in using a model that fails at validation against measurements?

Question

Say we have a model that describes a physical or dynamical system, for example, a hydrological model. If the model results do not match the measurements of the quantity being simulated, i.e. the model validation fails, is there any value in using the model for some other case or application?

Answer 1

The late George Box famously said:

All models are wrong. Some models are useful

All models are wrong.

That's important.

All models are wrong.

In any real-world system that we're interested in, the only accurate model of the system, is the system itself. Anything else, any simplification, gives wrong answers.

Some models are useful.

That's important, too.

Some models help us structure our discussions of the uncertainties (H/T Prof Neil Strachan)

Some models allow us to test "what if" scenarios, and look at what the relative changes might be. Now, in the real world, exogenous influences that the model can't account for, would mean that the actual outcomes would be different: but if the model gives a near-enough account of the scale of relative changes, we can still make informed judgements on the basis of the models. Even when it involves pesky things such as turbulence.

Some models give us insights into emergent properties of systems: a lot of the really interesting problems are emergent properties that aren't necessarily obvious from just looking at the basic rules of the system, and any insights we can gain are valuable.

Some models allow us to do experiments on systems that we couldn't possibly experiment on. And although we know the models are wrong, they can at least give us an indication of the range of possible outcomes.

That's not to defend the use of bad models to do bad work. There's far too much of that in my field, and I do not like it. As Simon W says, it is possible to test models to see if they are suitable for a particular purpose. To continue using a model for a purpose where it is known to be inapplicable, is charlatanry, not academia.

Answer 2

The answer depends very much on where the model fails. If it is very accurate in some regimes but very inaccurate in others, that model can still be used in those regimes where it is known to be very accurate.

A good example is Newtonian mechanics. Newtonian mechanics now known to be wrong (sometimes very wrong) as a universal model. Using Newtonian principles to build a collider that studies quantum events that result from relativistic collisions doesn't make much sense. On the other hand, Newtonian mechanics works very well in the ordinary, everyday world of earthly events. We still build bridges using Newtonian mechanics. It would be downright silly to invoke general relativity in bridge design.

What if the model doesn't work anywhere, or if the interval over which it does yield somewhat reasonable results is vanishingly small? In that case, the model isn't very good. A model needs to have some space over which the results are acceptable to have even limited applicability.

Answer 3

Yes there is value.

1)Often times we use models to check our own understanding of a process. If the model is built with all known factors included, it might still fail to produce results that can be verified with measurements. When the model is unable to reproduce a phenomenon, the test case can be used to assess what parts of the model need further development.

2)Sometimes systems are too complicated to model accurately (or perhaps there are too many unknowns in the initial or boundary conditions). Although the model might fail to reproduce absolute values that are measured, it might still be able to produce realistic dynamic variance. So, the model is still useful for studying the relationships between the dynamic processes that are involved. This is sometimes how we find secondary effects that are not typically direct outcomes of the model processes.

Answer 4

A slightly different slant:

As energynumbers has pointed out, all models are wrong in one way or another. A fully accurate model of reality would be as complex as reality, so some simplification is always needed. This means that there cannot be any useful model whose predictions in every way are perfectly accurate (except possibly by accident).

Validating a model, therefore, is not a process for saying "Is this model correct" with a yes/no answer. Rather, it should be about,

1. Deciding criteria the model must fulfill to be useful for a given purpose.
2. Assessing whether it fulfills those criteria.

If the model fails validation for one application, it may still be useful for another.

Some examples of different validation criteria, taking a regional ocean model as an example:

One obvious choice is whether the predictions of current speeds or of water levels is most important. Even after selecting one of these, the way in which the predictions are assessed may vary. A reasonable default might be to assess the fit at every time point between measured and observed water levels in a number of places, and accept the model if measurements such as bias and RMSE are below defined thresholds. However, here are two examples of specific scenarios that justify different approaches:

• NOAA runs models whose primary application is the production of depth information for navigational aids. The critical statistic for them is not the general accuracy of their models, but the frequency with which the model overpredicts the water level - as this form of error could result in vessels running aground. [1]

• Vested et al (1995) [2] give an example of a storm surge model that would provide flood warnings. It was tested not for the accuracy of all of its water level predictions, but for the accuracy of its predictions of peak water levels, as those are what would matter in operation.

[1] NOAA NOS STANDARDS FOR EVALUATING OPERATIONAL NOWCAST AND FORECAST HYDRODYNAMIC MODEL SYSTEMS Silver Spring, Maryland October 2003 noaa National Oceanic and Atmospheric Administration U.S., NOAA Technical Report NOS CS 17, Oct. 2003

[2] H. J. Vested, J. W. Nielsen, H. R. Jensen, and K. B. Kristensen, “Skill Assessment of an Operational Hydrodynamic Forecast System for the North Sea and Danish Belts,” in Quantitative Skill Assessment for Coastal Ocean Models, vol. 47, D. R. Lynch and A. M. Davies, Eds. Washington DC: American Geophysical Union, 1995, pp. 373–396.

Answer 5

Great question, with some interesting answers. It's obviously struck a chord around here.

I'd just like to add the abstract from one of my favourite papers, by Oreskes, Shrader-Frechette, & Beiitz (1994):

Verification and validation of numerical models of natural systems is impossible. This is because natural systems are never closed and because model results are always non-unique. Models can be confirmed by the demonstration of agreement between observation and prediction, but confirmation is inherently partial. Complete confirmation is logically precluded by the fallacy of affirming the consequent and by incomplete access to natural phenomena. Models can only be evaluated in relative terms, and their predictive value is always open to question. The primary value of models is heuristic.

Basically, in a complex system, you'd never expect a model's results to match observed data - there is noise everywhere, but also these systems contain deterministic chaos. You can simulate the behaviour of such a system, but you can't simulate the results. Even if your model is exactly correct, with the correct inputs, you won't get the same behaviour, because even the slightest value truncation will lead to a significant divergence, eventually.

Also, even if you do get the exact result you can't know that you got the right result for the right reasons - equifinality is a real problem. Especially in a model with more than two interacting calibrated parameters. There is no guarantee that the calibration process will give you better parameters, you can only be sure that it will give you better results for the calibration data. Even calibration using a cross-validation methodology (train-and-test) with actually independent data (rare) might fail, because the validation part can fail for the above reasons.

Anyway, everyone interested in that this topic should read that paper, I'm sure I've missed points here. It is a bit heavy-going, but that's because of the concepts - the paper is actually pretty enjoyable to read.

References

• Oreskes, N., Shrader-Frechette, K. & Beiitz, K., 1994. Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), pp.641–646. Available at: ProQuest.

Answer 6

Since there are already two answers discussing possible merits in further analysis let me just add the other side of the coin; it could also be prudent to see the simulation as a failure and not analyse it any further.

That could be the case if it's state of the art that similar simulations reproduce these experimental values you were mentioning and if these simulation can reproduce the values you want to look at as well. In that case, it only makes sense to try to find out why you can't reproduce your experimental test values. If you do not want to spend much time in developing the model, then drop it altogether; no one will trust your results.

Trust is, in my opinion, the crucial point here. As mentioned in the previous answers it is not uncommon for simulations to give only (poor) estimates of experimental results. But it is all important in these cases that you can scrutinize why you still trust the results you want to get out of it.

To be honest, I do read your question that way that you have no good reason to be confident in further results. If that is the case my advice would be to step back and first think about whether you can trust them. To do that no general answer will help you; it depends too strongly on your system and what the state of the art is for these kind of simulations.