Hello everybody,
I'm running a multilevel logit model with crossedrandom effects using runmlwin.
I would like to let Stata calculate the Area Under the ROC curve afterwards as explained here: http://www.ats.ucla.edu/stat/stata/faq/roc.htm
using the lroc, nograph command. Similiarly, there is a lstat command in stata giving helpful measures for the validation of a logit model.
However, it cannot be used after runmlwin with the error: "last estimates not found".
Is there a way to save my runmlwin estimates so that Stata can use its other implemented commands?
Thanks in advance for your help,
Best regards.
Using Stata commands lroc and lstat after runmlwin

 Site Admin
 Posts: 432
 Joined: Fri Apr 01, 2011 2:14 pm
Re: Using Stata commands lroc and lstat after runmlwin
Hi sujust18,
Am not familiar with ROC analysis, but I note that it doesn't look like you can do a ROC after xtmelogit, Stata's own multilevel modelling command.
http://www.stata.com/help.cgi?xtmelogit_postestimation
After fitting model using runmlwin, you can access all parameter estimates with the following commnads
In a multilevel logistic regression you should be able to retrieve the linear preditor as...
Best wishes
George
Am not familiar with ROC analysis, but I note that it doesn't look like you can do a ROC after xtmelogit, Stata's own multilevel modelling command.
http://www.stata.com/help.cgi?xtmelogit_postestimation
After fitting model using runmlwin, you can access all parameter estimates with the following commnads
Code: Select all
ereturn dir
ereturn list e(b)
ereturn list e(V)
Code: Select all
predict xb1, xb
George

 Posts: 16
 Joined: Tue Mar 09, 2021 5:54 pm
Re: Using Stata commands lroc and lstat after runmlwin
Dear George,
I am also doing a series of multilevel analysis to identify the availability of diagnostic tests in primary care surgeries in different countries. Surgeries characteristics are level 1 and country level 2.
I would like to do it with MCMC, because the prevalence of my binary dependent variable is very high (>90%)or very low (<10%)
After using runmlwin I would like to plot ROC curves so I can compare visually the ROC of a single level model (with primary care surgeries characteristics only ) with the ROC of a 2 levels model in which I add country as the second level variable. This is to see visually how country improves the ability of model 2 to predict the outcome. The approach is based on this paper by Juan Merlo:https://europepmc.org/article/PMC/PMC4847925
When using
that includes the fixed effects only.
How can I have a calculation that includes both the fixed and the random effects?
After  melogit  , the syntax would be:
How can I do this with runmlwin?
Thank you
Manuel
I am also doing a series of multilevel analysis to identify the availability of diagnostic tests in primary care surgeries in different countries. Surgeries characteristics are level 1 and country level 2.
I would like to do it with MCMC, because the prevalence of my binary dependent variable is very high (>90%)or very low (<10%)
After using runmlwin I would like to plot ROC curves so I can compare visually the ROC of a single level model (with primary care surgeries characteristics only ) with the ROC of a 2 levels model in which I add country as the second level variable. This is to see visually how country improves the ability of model 2 to predict the outcome. The approach is based on this paper by Juan Merlo:https://europepmc.org/article/PMC/PMC4847925
When using
, as suggested below, stata calculates the predicted mean based on a linear predictorpredict xb1, xb
that includes the fixed effects only.
How can I have a calculation that includes both the fixed and the random effects?
After  melogit  , the syntax would be:
mu being an option that allows to include both the fixed and random effects.predict xb1, mu
How can I do this with runmlwin?
Thank you
Manuel

 Posts: 1279
 Joined: Mon Oct 19, 2009 10:34 am
Re: Using Stata commands lroc and lstat after runmlwin
As you have found, running predict after runmlwin will only give you the fixedpart prediction. If you want a prediction with both fixed and random effects then you will need to manually request that the higherlevel residuals are returned via the residuals() option and then use these to extend the prediction. George gives an example of this in his presentation: "8th International Amsterdam Multilevel Conference (17th March 2011)", available on the following page:
https://www.bristol.ac.uk/cmm/software/ ... entations/
You can find the syntax for this in the provided Stata dofile or on slides 19 and 26 of the presentation.
https://www.bristol.ac.uk/cmm/software/ ... entations/
You can find the syntax for this in the provided Stata dofile or on slides 19 and 26 of the presentation.

 Posts: 16
 Joined: Tue Mar 09, 2021 5:54 pm
Re: Using Stata commands lroc and lstat after runmlwin
Dear Chris,
Thank you very much for your prompt reply, much appreciated!
I am using the dataset provided in the paper by Merlo and colleagues to learn from their approach : https://doi.org/10.1371/journal.pone.0153778.s004
I generated the following syntax :
and then
The higher level residuals and SE are provided back to stata as u0 and u0SE
I am trying to generate a variable which includes the fixed and random effects
I tried
How can I generate a variable that includes both the fixed and the random effects, that I could then use with roctab?
Thank you very much.
Manuel
Thank you very much for your prompt reply, much appreciated!
I am using the dataset provided in the paper by Merlo and colleagues to learn from their approach : https://doi.org/10.1371/journal.pone.0153778.s004
I generated the following syntax :
runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) nopause
and then
This gives the same outputs that are presented in the paper.runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons, residuals(u)) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) mcmc(burnin(1000) chain(10000)) initsprevious nopause nogroup
The higher level residuals and SE are provided back to stata as u0 and u0SE
I am trying to generate a variable which includes the fixed and random effects
I tried
but I get :predict newvar, xb +u0
So I tried to adapt (probably wrongly) the syntax provided in the slides of the conference you mentionedoption + not allowed
r(198);
but when then I usegenerate prediction = [FP1]cons + [FP1]male + [FP1]agegroup2 + [FP1]agegroup3 + [FP1]agegroup4 + [FP1]agegroup5 + [FP1]agegroup6 + u0
to get the AUC and ROC curve, I get values that are smaller than in the paper, I get an UAC of 0.57, while in the paper is 0.630. I think it is because my new variable prediction in fact only contains the random effect residuals u0.roctab Psycmed prediction, graph summary
How can I generate a variable that includes both the fixed and the random effects, that I could then use with roctab?
Thank you very much.
Manuel

 Posts: 1279
 Joined: Mon Oct 19, 2009 10:34 am
Re: Using Stata commands lroc and lstat after runmlwin
I suspect that your problem is that you have forgotten to multiple the fixedpart parameters by their associated data, so you'll want something more like:
or more simply if you get Stata to do half the work:
You can find a couple of more detailed examples from George in the following topics:
Can you use Stata's predict command after runmlwin?
Predictions via the runmlwin interface: a clarification
Code: Select all
generate prediction = [FP1]cons*cons + [FP1]male*male + [FP1]agegroup2*agegroup2 + [FP1]agegroup3*agegroup3 + [FP1]agegroup4*agegroup4 + [FP1]agegroup5*agegroup5 + [FP1]agegroup6*agegroup6 + u0
Code: Select all
predict xb
generate prediction = xb + u0
Can you use Stata's predict command after runmlwin?
Predictions via the runmlwin interface: a clarification

 Posts: 16
 Joined: Tue Mar 09, 2021 5:54 pm
Re: Using Stata commands lroc and lstat after runmlwin
It works!
Thanks a lot Chris this is very useful!
Thanks a lot Chris this is very useful!