2
$\begingroup$

I want to validate a model and compare several parameterizations. I have some observations so I calculated the RMSE for each parameterization with the best meteorological forcing. However I want to nuance with the variability of the forcing. I suppose that more the forcing has an impact on outputs in its range of uncertainties, less the parameterization explains the amelioration of the RMSE.

In my case, I have 5 forcing files and 2 different parameterizations, the output $Y$ is a time series. I name $Y_{ij}(t)$ each output with the forcing $i$ and the param $j$ and I name the mean of $Y_{ij}(t)$ for each time step between all the forcing with the parameterization j as $\bar{Y_j}(t)$ and the time series of the observation is $y(t)$

I want to compare the RMSE with the following formulae (where I replace $y(t)$ by $\bar{Y_j}(t)$):

$$\frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2$$

Finally, I want to conclude that if the ratio $ \frac{RMSE}{\sqrt{\frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2}} $ is near to 1 so it is difficult to conclude because the error on the forcing lead to the most of the sensitivity on the outputs and if it is near to 0, the parameterization really improves the model to fit to the observations.

Is this approach relevant?

$\endgroup$
2
$\begingroup$

I think you might be getting terms mixed up. Let's follow the standard convention of data assimilation and call your model output from the ($k^{\text{th}}$ parameterization/forcing) $X_k$ and the observations $y$. For simplicity's sake, let's just say that the observation operator is unitary ($H=I$). The equation for the Root Mean Squared Error (RMSE) for the $k^{\text{th}}$ model ($k$ could be a unique index that is a combination of forcing and parameterization) is $$RMSE_k=\sqrt{\sum_i\frac{\left(x_{i,k}-y_i\right)^2}{N}}$$ where $x_i$ and $y_i$ are indices paired 'samples' from $X$ and $Y$ respectively. The sampling that you use performs RMSE sampled over time, but you can just as easily sample over space.

What is a metric that is easily comparable to RMSE that measures model variation? Let's follow what you suggested (more for the narrative). If we let the mean state for the $k^{\text{th}}$ model be $\bar{x}_k=\frac{1}{N}\sum_i x_{i,k}$ then the formula you suggested would actually be $$\sqrt{\sum_i\frac{\left(x_{i,k}-\bar{x}_k\right)^2}{N}}$$. Now that looks a lot like the formula for the standard deviation. Rather, the way that we notated this formula shows the model-dependent standard deviation fluctuates on time. However, if we make a model mean variable $\hat{x}_i=\frac{1}{N}\sum_k x_{i,k}$, then we can determine how much the different forcings and parameterizations cause the output to vary with time: $$\sigma_i=\sqrt{\sum_k\frac{\left(x_{i,k}-\hat{x}_i\right)^2}{N}}$$.

So really, your formula should look like (reverting back to your notation): $$\frac{RMSE}{\sqrt{\frac{1}{N}(Y_{i,j}(t_k)-\bar{Y_j}(t_k))^2}}$$

Now, I think your idea is fine, but the interpretation is incorrect. You can legitimately have a number that is greater than one (for example, if you see no difference, the standard deviation is 0, therefore your metric reaches infinity). You also cannot say if the inclusion of the parameterization makes the model better based on this metric. That would require the examining the RMSE of each parameterization + forcing. Such an experiment shows relative roles that the parameterizations/forcings of similar kind have in creating the number of possible model outputs that could be causing model errors.

An example that I know of where such an analysis was conducted was Thomas et al. (2019). In it, the RMSE was computed and the model standard deviation was compared, with the standard deviation being smaller than the RMSE (therefore leading to numbers greater than 1, per the corrections [namely the square root] to your logic).

$\endgroup$
3
  • $\begingroup$ I should also note, it is best to treat a perturbation in the forcing as if it were a new parameterization. There shouldn't be a distinction between the two, until it comes to something like a sequential analysis or time analysis. $\endgroup$ Jul 21 at 19:41
  • $\begingroup$ Thank you for the explanation and correction. Yes I forget the root square to compare with the RMSE to get a ratio homogeneous. In fact, I mix up the interpretation with this ratio with the same ratio by replacing each term by their mean over the forcing. $\frac{1}{N_f} \sum_j \frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2$ where $\bar{Y_j}(t_k)$ will be better to any set of observation. Thank you for the article $\endgroup$
    – AnthonyB
    Jul 22 at 22:50
  • $\begingroup$ The problem with give a perturbation the forcing. It is difficult to keep a coherence between all the meteorogical variables and often the difference between the forcing is difficult to model it as perturbation. $\endgroup$
    – AnthonyB
    Jul 22 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.