www.cmm.bristol.ac.uk/forum

Hi all,
Just a brief question: I have a hard time finding a good statistic to evaluate the overall model fit/quality for multilevel models using runmlwin. In STATA I normally use the command estat ic to obtain the Bayesian Information Criteria (BIC) which has served well in comparing models – for example, whether a random slope inclusion improves the model fit. Is there an equivalent to BIC or AIC that I can obtain after using runmlwin? I am aware of the Deviance Information Criteria (DIC) but this does only work for MCM models and usually I use the regular IGLS/RIGLS models.
I am currently running multilevel Poisson and Logit models and here I don’t even obtain a logLikelihood measure – so I can’t calculate the Deviance manually.
Any help would be highly appreciated!
Thanks so much!

Best,
Raphael

Hi Raphael,

First note that for continuous response models the log-likelihood is provide and so for these models you can calculate LR tests, AIC and BIC statistics by making use of Stata's lrtest and estat ic commands.

It is for discrete response models (i.e. models for binary, count and categorical response variables) that issues arise.

MLwiN does not print the log-likelihod statistic after fitting discrete response models as MLwiN does not use maximum likelihood to fit these models. MLwiN uses quasilikelihood estimation to fit these models. The log-likelihood statistics associated with these models are therefore not correct log-likelihood statistics which would mean that LR tests, AIC statistics and BIC statistics based on these log-likelihood statistics would also not be correct. This is why the log-likelihood is not displayed after fitting these models.

We recommend that any final discrete response model should be fitted by MCMC. You can then compare these models using the Bayesian DIC statistic in much the same way that you would interpret AIC and BIC statistics when models are fitted by maximum likelihood.

For more information see

. help runmlwin##quasilikelihood_estimates

Best wishes

George

Hi George,
Thanks a lot for the helpful response and the additional insights!
Have a nice day!

Best,
Raphael