Comparing two models

Welcome to the forum for R2MLwiN users. Feel free to post your question about R2MLwiN here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to R2MLwiN: Running MLwiN from within R >> http://www.bris.ac.uk/cmm/software/r2mlwin/
Post Reply
adrianrus
Posts: 4
Joined: Tue Jun 07, 2016 8:55 pm

Comparing two models

Post by adrianrus »

Hello,

I just run three models with R2MLwiN and I would like to compare each other.
I have tied to find the command but I didn't find something that works.

Could you please help me with that.

Below you have the models:

(m.empty <- runMLwiN(logit(EdPerform2c) ~ 1 + (1 | iID), D = "Binomial", estoptions = list(EstM = 1), data = EducationNoMissingData))


(m.level.1 <- runMLwiN(logit(EdPerform2c) ~ 1 + AgeO + Gender + Provenance2c + exPunishment2c +

cEdType4c + KnowStay2c + FriendAdult2c + Homework2c +

(1 | iID), D = "Binomial", estoptions = list(EstM = 1), data = EducationNoMissingData))


(m.level.1and2 <- runMLwiN(logit(EdPerform2c) ~ 1 + AgeO + Gender + Provenance2c + exPunishment2c +

cEdType4c + KnowStay2c + FriendAdult2c + Homework2c +

REducation + RHealth + iChildStaffNight2c +

iOrganization + (1 | iID), D = "Binomial", estoptions = list(EstM = 1),

data = EducationNoMissingData))


Thank you so much!
Adrian
kaiserdominici
Posts: 18
Joined: Thu Feb 06, 2014 9:37 am

Re: Comparing two models

Post by kaiserdominici »

Have you tried with

Code: Select all

deviance()
...?

This manual says that the command "Returns the deviance from "mlwinfitIGLS" objects", which I guess is what you get using R2MLwiN.

k.
ChrisCharlton
Posts: 1351
Joined: Mon Oct 19, 2009 10:34 am

Re: Comparing two models

Post by ChrisCharlton »

MLwiN calculates the DIC statistic (see chapter 2 of the MCMC guide - http://www.bristol.ac.uk/cmm/media/soft ... mc-web.pdf) for models fitted via MCMC. In R2MLwiN this is returned in the BDIC slot of the returned object (e.g. m.empty@BDIC). As described in this chapter a lower value of this suggests a better model fit, however if the difference is small then you should confirm the difference with further checks.
adrianrus
Posts: 4
Joined: Tue Jun 07, 2016 8:55 pm

Re: Comparing two models

Post by adrianrus »

kaiserdominici wrote:Have you tried with

Code: Select all

deviance()
...?

This manual says that the command "Returns the deviance from "mlwinfitIGLS" objects", which I guess is what you get using R2MLwiN.

k.
Hello,

Yes, I tried the compare the models using the command you provided, but it didn't work.
It seems that it would work only with Normal models.

Thank you!
adrianrus
Posts: 4
Joined: Tue Jun 07, 2016 8:55 pm

Re: Comparing two models

Post by adrianrus »

ChrisCharlton wrote:MLwiN calculates the DIC statistic (see chapter 2 of the MCMC guide - http://www.bristol.ac.uk/cmm/media/soft ... mc-web.pdf) for models fitted via MCMC. In R2MLwiN this is returned in the BDIC slot of the returned object (e.g. m.empty@BDIC). As described in this chapter a lower value of this suggests a better model fit, however if the difference is small then you should confirm the difference with further checks.
Hello Chris,

Thank you for the guide. I read the chapter.

Below I listed the values of DIC for each of the models.
Most importantly, I am interested if the difference between m.level.1 and m.level.1and2 is significantly different. Just looking of the results makes me to think that this is not the case.
What further check do you have in mind?

Usually, I do these analyses using lmer4 and it is very convenient to compare the models.

Thank you.
Adrian


m.empty@BDIC
Dbar D(thetabar) pD DIC
1555.87805 1524.98560 30.89253 1586.77063


m.level.1@BDIC
Dbar D(thetabar) pD DIC
1441.29187 1410.08850 31.20344 1472.49536



m.level.1and2@BDIC
Dbar D(thetabar) pD DIC
1432.89331 1405.77820 27.11516 1460.00854
ChrisCharlton
Posts: 1351
Joined: Mon Oct 19, 2009 10:34 am

Re: Comparing two models

Post by ChrisCharlton »

The MCMC guide suggests fitting the models using different random number seeds and starting values (see the MCMC chapter 5 replication examples on http://www.bristol.ac.uk/cmm/software/r2mlwin/examples/ for details of how to change these with R2MLwiN). I would also suggest the normal MCMC diagnostics such as checking that the chains have converged to a stable distribution and that the effective sample sizes aren't too small.
Post Reply