Multi-model inference

No ‘best’ model

Your analysis may show there is no best model

You may have:

  • One highly-ranked model, but several others are plausible, or
  • Many models of reasonable dAICc/Likelihood, with AIC weights shared evenly across the model set

In this situation, you can adopt a multi-model inference approach

Multi-model inference

In multi-model inference, you use your entire model set to calculate a weighted average of each parameter estimate

By using all your models, you acknowledge the uncertainty in which of your hypotheses is correct, and emphasise that there is still useful information even in the less-preferred models

  • Including less plausible models is balanced by giving more weight to hypotheses with higher likelihoods
  • Parameter estimates reflect the range of values predicted by different models
  • Models with larger AIC weights have a stronger influence on the final parameter value
  • Poorly-supported models with AIC weights close to zero have little influence on parameter values

Model-averaging equation

You can calculate a model-averaged estimate of a parameter as1:

\[ \hat{\bar{\theta}} = \sum_{i=1}^R w_i \hat{\theta_i} \]

Where:

  • \(\hat{\bar{\theta}}\) (theta-bar-hat) is the model averaged estimate of the parameter of interest \(\theta\) (theta)
  • \(w_i\) is the weight for model \(i\), and
  • \(\hat{\theta_i}\) (theta-hat-i) is the parameter estimate from model \(i\)

Calculate a model-averaged estimate

In simple terms, you:

  1. Extract the parameter estimate from each model.  If the parameter doesn’t occur in that model, use a value of zero
  2. Extract the AIC weight for each model
  3. For each model, multiply the parameter estimate by the AIC weight
  4. Sum the result for all models in the a priori set, to avoid unwanted subjectivity

\[ \begin{align} \hat{\theta_A} &= Estimate_{Model A} * Weight_{Model A} \\ \hat{\theta_A} &= Estimate_{Model B} * Weight_{Model B} + \\ \hline & \text{Model-averaged estimate, } \hat{\bar{\theta}} \\ \hline \end{align} % \text{spacer} % \hrule % \rule{\textwidth}{0.4pt} \]

Full model set

How would we apply a model-averaging approach in our Chinese water deer case-study?1

With a choice of two detection functions, one density covariate and one detectability covariate, our full model set contains eight models:

Parameters

  • Only intercepts for density and detection
  • Distance from the coast affects density
  • Landcover affects detection: Curve parameters
  • Landcover affects detection and distance from the coast affects density

Half-normal

  1. Null model
  3. SD

Hazard-rate

  1. Null model
  3. Shape/Scale
  4. Global model

Model-averaged estimates in R

Calculating model-averaged estimates is simple in R, using AICcmodavg’s modavg() function

Use the modavg() function to calculate the weighted average of the coefficient for the distance to coast:

1modAvgDens <- modavg(models,
2    "DistToCoast",
3    second.ord = TRUE,
4    c.hat = cHat,
5    parm.type = "lambda")
1
Model set
2
Covariate of interest
3
Use small-sample correction factor
4
Use QAICc
5
Calculate density

View output


Multimodel inference on "lam(DistToCoast)" based on QAICc

QAICc table used to obtain model-averaged estimate:
    (c-hat estimate = 1.240156)

          K  QAICc Delta_QAICc QAICcWt Estimate SE
hn._DTC   4 283.08       49.65       0        0  0
hnLC_DTC  5 233.43        0.00       1        0  0
haz._DTC  5 280.27       46.84       0        0  0
hazLC_DTC 6 247.36       13.93       0        0  0

Model-averaged estimate: 0 
Unconditional SE: 0 
95% Unconditional confidence interval: 0, 0

The first column in the table shows which models contain DistToCoast as a predictor, and are therefore contributing to this model-averaged parameter estimate

Note the model weights

When we consider this subset, the QAICc weight is all loaded on the half-normal model with Landcover as a predictor of detectability

The output below the table provides the model-averaged estimate, and its SE and 95% CI

Take-home messages

There are two important take-home messages

Weak models have little influence

Weak models fail to contribute to the model-averaged estimate

Confidence intervals are wider

Confidence intervals of the model-averaged estimate include zero, emphasising what we already know from our previous analyses: Distance to coast is not a good predictor of water deer density

Unfortunately modavg() only works for density, not detectability, but you can calculate model-averaged estimates for detectability by hand using the equation provided earlier