getting credible intervals around postestimations

Welcome to the forum for runmlwin users. Feel free to post your question about runmlwin here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to runmlwin: Running MLwiN from within Stata >> http://www.bristol.ac.uk/cmm/software/runmlwin/
Post Reply
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

getting credible intervals around postestimations

Post by becciaa1992 »

Hello! I am fitting a null two-level multilevel logistic model using the following example code:

Code: Select all


runmlwin outcome cons , /// 
  level2(group: cons) ///
  level1(id:) ///
  discrete(distribution(binomial) link(logit) denominator(denominator) pql2) ///
  rigls maxiterations(100) ///
  nopause
  
runmlwin outcome cons , ///
  level2(group: cons, residuals(u, savechains("m1u.dta", replace))) /// 
  level1(id:) ///
  discrete(distribution(binomial) link(logit) denominator(denominator)) ///
  mcmc(burnin(5000) chain(50000) thinning(50) ///
    savechains("m1beta.dta", replace)) initsprevious /// 
  nopause 
  
  * pull residuals
egen pickone = tag(group)
sort u0
generate u0rank = sum(pickone)
sort group
list group u0 u0se u0rank if pickone==1 

I am interest in comparing specific level 2 groups using odds ratios and odds difference, is it possible to get credible intervals around these estimates in runmlwin? Thanks!
ChrisCharlton
Posts: 1285
Joined: Mon Oct 19, 2009 10:34 am

Re: getting credible intervals around postestimations

Post by ChrisCharlton »

I think that you should be able to do this by loading the .dta file where you have saved the parameter chains and then running the mcmcsum command on the variable that corresponds to the parameter of interest. For details on the options for this command run:

Code: Select all

help mcmcsum
from within Stata.
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

Re: getting credible intervals around postestimations

Post by becciaa1992 »

Thank you for your reply! I have tried mcmcsum, and I get the following error:

Code: Select all

Estimate results must come from runmlwin
Furthermore, when I load the .dta file where the parameter chains I stored, the only variables included are: iteration, deviance, FP1_cons, RP2_var_cons_, and OD_bcons_1; I don't see the residuals.
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

Re: getting credible intervals around postestimations

Post by becciaa1992 »

To give you some more context: After I fit the models and pull residuals:

Code: Select all

runmlwin outcome cons , /// 
  level2(group: cons) ///
  level1(id:) ///
  discrete(distribution(binomial) link(logit) denominator(denominator) pql2) ///
  rigls maxiterations(100) ///
  nopause
  
runmlwin outcome cons , ///
  level2(group: cons, residuals(u, savechains("m1u.dta", replace))) /// 
  level1(id:) ///
  discrete(distribution(binomial) link(logit) denominator(denominator)) ///
  mcmc(burnin(5000) chain(50000) thinning(50) ///
    savechains("m1beta.dta", replace)) initsprevious /// 
  nopause 
  
  * pull residuals
egen pickone = tag(group)
sort u0
generate u0rank = sum(pickone)
sort group
list group u0 u0se u0rank if pickone==1 
I would like to calculate an OR between two groups by exponentiating their residuals, and then get a credible interval around that.
ChrisCharlton
Posts: 1285
Joined: Mon Oct 19, 2009 10:34 am

Re: getting credible intervals around postestimations

Post by ChrisCharlton »

In your estimation command you requested that the residual chains at level-2 are stored in the file m1u.dta, so this is what you will need to open in order to calculate what you need. When running the mcmcsum on variables from the currently loaded dataset you need to specify the variables option when running the command, otherwise it will look for the chains saved as a mata matrix.
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

Re: getting credible intervals around postestimations

Post by becciaa1992 »

thank you so much! so that makes sense, although it looks like I can only use mcmcsum (with the variables option) if I use all the chains; when I try to use it on a subset (e.g., on the residual values for one group) I get an output that looks like this:

. mcmcsum u if group==1, variables
------------------------------------------------------------------------------
| Mean Std. Dev. ESS P [95% Cred. Interval]
-------------+----------------------------------------------------------------
policy | 0 0 0 45.000 0 0
------------------------------------------------------------------------------

So I'm assuming that means I cannot get credible intervals around group-specific effects?
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

Re: getting credible intervals around postestimations

Post by becciaa1992 »

Actually, I can get credible intervals around a group-specific effect using this code:

Code: Select all

bysort group (iteration): egen p_mean = mean(p)
bysort group (iteration): egen p_lo = pctile(p), p(2.5)
bysort group (iteration): egen p_hi = pctile(p), p(97.5)
where p is a variable for the group specific-predicted probability of the outcome (calculated from the residuals).

But then I run into issues when trying to compare two groups.
ChrisCharlton
Posts: 1285
Joined: Mon Oct 19, 2009 10:34 am

Re: getting credible intervals around postestimations

Post by ChrisCharlton »

mcmcsum should work with subsets, so it may be a bug if that isn't working for you. A workaround would be to remove rows that do not correspond to the desired group before running the command, although the alternative code that you identified should be equivalent to what mcmcsum is doing for the mean and credible intervals anyway.
becciaa1992
Posts: 21
Joined: Thu Oct 17, 2019 12:21 am

Re: getting credible intervals around postestimations

Post by becciaa1992 »

got it to work, thank you so much for your help!
Post Reply