Using Stata commands lroc and lstat after runmlwin

[sujust18]

Hello everybody,

I'm running a multilevel logit model with crossed-random effects using runmlwin.

I would like to let Stata calculate the Area Under the ROC curve afterwards as explained here: http://www.ats.ucla.edu/stat/stata/faq/roc.htm
using the lroc, nograph command. Similiarly, there is a lstat command in stata giving helpful measures for the validation of a logit model.
However, it cannot be used after runmlwin with the error: "last estimates not found".

Is there a way to save my runmlwin estimates so that Stata can use its other implemented commands?

Thanks in advance for your help,

Best regards.

[GeorgeLeckie]

Hi sujust18,

Am not familiar with ROC analysis, but I note that it doesn't look like you can do a ROC after xtmelogit, Stata's own multilevel modelling command.

http://www.stata.com/help.cgi?xtmelogit_postestimation

After fitting model using runmlwin, you can access all parameter estimates with the following commnads

Code: Select all

ereturn dir
ereturn list e(b)
ereturn list e(V)

In a multilevel logistic regression you should be able to retrieve the linear preditor as...

Code: Select all

predict xb1, xb

Best wishes

George

[ManuelDewez]

Dear George,
I am also doing a series of multilevel analysis to identify the availability of diagnostic tests in primary care surgeries in different countries. Surgeries characteristics are level 1 and country level 2.
I would like to do it with MCMC, because the prevalence of my binary dependent variable is very high (>90%)or very low (<10%)
After using runmlwin I would like to plot ROC curves so I can compare visually the ROC of a single level model (with primary care surgeries characteristics only ) with the ROC of a 2 levels model in which I add country as the second level variable. This is to see visually how country improves the ability of model 2 to predict the outcome. The approach is based on this paper by Juan Merlo:https://europepmc.org/article/PMC/PMC4847925

When using

predict xb1, xb

, as suggested below, stata calculates the predicted mean based on a linear predictor
that includes the fixed effects only.
How can I have a calculation that includes both the fixed and the random effects?

After - melogit - , the syntax would be:

predict xb1, mu

mu being an option that allows to include both the fixed and random effects.

How can I do this with runmlwin?

Thank you

Manuel

[ChrisCharlton]

As you have found, running predict after runmlwin will only give you the fixed-part prediction. If you want a prediction with both fixed and random effects then you will need to manually request that the higher-level residuals are returned via the residuals() option and then use these to extend the prediction. George gives an example of this in his presentation: "8th International Amsterdam Multilevel Conference (17th March 2011)", available on the following page:

https://www.bristol.ac.uk/cmm/software/ ... entations/

You can find the syntax for this in the provided Stata do-file or on slides 19 and 26 of the presentation.

[ManuelDewez]

Dear Chris,

Thank you very much for your prompt reply, much appreciated!

I am using the dataset provided in the paper by Merlo and colleagues to learn from their approach : https://doi.org/10.1371/journal.pone.0153778.s004


I generated the following syntax :

runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) nopause

and then

runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons, residuals(u)) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) mcmc(burnin(1000) chain(10000)) initsprevious nopause nogroup

This gives the same outputs that are presented in the paper.

The higher level residuals and SE are provided back to stata as u0 and u0SE

I am trying to generate a variable which includes the fixed and random effects
I tried

predict newvar, xb +u0

but I get :

option + not allowed
r(198);

So I tried to adapt (probably wrongly) the syntax provided in the slides of the conference you mentioned

generate prediction = [FP1]cons + [FP1]male + [FP1]agegroup2 + [FP1]agegroup3 + [FP1]agegroup4 + [FP1]agegroup5 + [FP1]agegroup6 + u0

but when then I use

roctab Psycmed prediction, graph summary

to get the AUC and ROC curve, I get values that are smaller than in the paper, I get an UAC of 0.57, while in the paper is 0.630. I think it is because my new variable prediction in fact only contains the random effect residuals u0.
How can I generate a variable that includes both the fixed and the random effects, that I could then use with -roctab-?

Thank you very much.


Manuel

[ChrisCharlton]

I suspect that your problem is that you have forgotten to multiple the fixed-part parameters by their associated data, so you'll want something more like:

Code: Select all

generate prediction = [FP1]cons*cons + [FP1]male*male + [FP1]agegroup2*agegroup2 + [FP1]agegroup3*agegroup3 + [FP1]agegroup4*agegroup4 + [FP1]agegroup5*agegroup5 + [FP1]agegroup6*agegroup6 + u0

or more simply if you get Stata to do half the work:

Code: Select all

predict xb
generate prediction = xb + u0

You can find a couple of more detailed examples from George in the following topics:

Can you use Stata's predict command after runmlwin?
Predictions via the runmlwin interface: a clarification

[ManuelDewez]

It works!
Thanks a lot Chris this is very useful!