www.cmm.bristol.ac.uk/forum

Hello everybody,

I'm running a multilevel logit model with crossed-random effects using runmlwin.

I would like to let Stata calculate the Area Under the ROC curve afterwards as explained here: http://www.ats.ucla.edu/stat/stata/faq/roc.htm
using the lroc, nograph command. Similiarly, there is a lstat command in stata giving helpful measures for the validation of a logit model.
However, it cannot be used after runmlwin with the error: "last estimates not found".

Is there a way to save my runmlwin estimates so that Stata can use its other implemented commands?

Thanks in advance for your help,

Best regards.

Hi sujust18,

Am not familiar with ROC analysis, but I note that it doesn't look like you can do a ROC after xtmelogit, Stata's own multilevel modelling command.

http://www.stata.com/help.cgi?xtmelogit_postestimation

After fitting model using runmlwin, you can access all parameter estimates with the following commnads
In a multilevel logistic regression you should be able to retrieve the linear preditor as...
Best wishes

George

Dear George,
I am also doing a series of multilevel analysis to identify the availability of diagnostic tests in primary care surgeries in different countries. Surgeries characteristics are level 1 and country level 2.
I would like to do it with MCMC, because the prevalence of my binary dependent variable is very high (>90%)or very low (<10%)
After using runmlwin I would like to plot ROC curves so I can compare visually the ROC of a single level model (with primary care surgeries characteristics only ) with the ROC of a 2 levels model in which I add country as the second level variable. This is to see visually how country improves the ability of model 2 to predict the outcome. The approach is based on this paper by Juan Merlo:https://europepmc.org/article/PMC/PMC4847925

When using

predict xb1, xb

, as suggested below, stata calculates the predicted mean based on a linear predictor
that includes the fixed effects only.
How can I have a calculation that includes both the fixed and the random effects?

After - melogit - , the syntax would be:

predict xb1, mu

mu being an option that allows to include both the fixed and random effects.

How can I do this with runmlwin?

Thank you

Manuel

As you have found, running predict after runmlwin will only give you the fixed-part prediction. If you want a prediction with both fixed and random effects then you will need to manually request that the higher-level residuals are returned via the residuals() option and then use these to extend the prediction. George gives an example of this in his presentation: "8th International Amsterdam Multilevel Conference (17th March 2011)", available on the following page:

https://www.bristol.ac.uk/cmm/software/ ... entations/

You can find the syntax for this in the provided Stata do-file or on slides 19 and 26 of the presentation.

Dear Chris,

Thank you very much for your prompt reply, much appreciated!

I am using the dataset provided in the paper by Merlo and colleagues to learn from their approach : https://doi.org/10.1371/journal.pone.0153778.s004


I generated the following syntax :

runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) nopause

and then

runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons, residuals(u)) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) mcmc(burnin(1000) chain(10000)) initsprevious nopause nogroup

This gives the same outputs that are presented in the paper.

The higher level residuals and SE are provided back to stata as u0 and u0SE

I am trying to generate a variable which includes the fixed and random effects
I tried

predict newvar, xb +u0

but I get :

option + not allowed
r(198);

So I tried to adapt (probably wrongly) the syntax provided in the slides of the conference you mentioned

generate prediction = [FP1]cons + [FP1]male + [FP1]agegroup2 + [FP1]agegroup3 + [FP1]agegroup4 + [FP1]agegroup5 + [FP1]agegroup6 + u0

but when then I use

roctab Psycmed prediction, graph summary

to get the AUC and ROC curve, I get values that are smaller than in the paper, I get an UAC of 0.57, while in the paper is 0.630. I think it is because my new variable prediction in fact only contains the random effect residuals u0.
How can I generate a variable that includes both the fixed and the random effects, that I could then use with -roctab-?

Thank you very much.


Manuel

I suspect that your problem is that you have forgotten to multiple the fixed-part parameters by their associated data, so you'll want something more like:
or more simply if you get Stata to do half the work: You can find a couple of more detailed examples from George in the following topics:

Can you use Stata's predict command after runmlwin?
Predictions via the runmlwin interface: a clarification

It works!
Thanks a lot Chris this is very useful!