Confidence Intervals vs. Significant p-values

Welcome to the forum for R2MLwiN users. Feel free to post your question about R2MLwiN here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to R2MLwiN: Running MLwiN from within R >> http://www.bris.ac.uk/cmm/software/r2mlwin/
Post Reply
adrianrus
Posts: 4
Joined: Tue Jun 07, 2016 8:55 pm

Confidence Intervals vs. Significant p-values

Post by adrianrus »

Hello all,

I need to interpret a significant p-value that has a confidence interval (CI) for the estimate that crosses 0 (see below).
Usually, I place more importance on CI.

Knowing that MCMC was conducted in this model, should I place more importance on the CI or the p-value?

MLwiN (version: 2.36) multilevel model (Binomial)
N min mean max N_complete min_complete mean_complete max_complete
iID 60 1 23.05 77 55 1 20.36364 74
Estimation algorithm: MCMC Elapsed time : 12.08s
Number of obs: 1120 (from total 1383) Number of iter.: 5000 Chains: 1 Burn-in: 500
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
1415.712 1401.830 13.881 1429.593


Coef. Std. Err. z Pr(>|z|) [95% Cred. Interval] ESS
Intercept 0.79868 0.35035 2.28 0.02263 * 0.16612 1.50846 28
...
MalnutritionPYes 0.31611 0.16009 1.97 0.04832 * -0.01020 0.62371 663
...


Thank you so much!
Adrian
richardparker
Posts: 61
Joined: Fri Oct 23, 2009 1:49 pm

Re: Confidence Intervals vs. Significant p-values

Post by richardparker »

Note that one-sided Bayesian p-values are another option which can be helpful; these can be toggled on/off in R2MLwiN via z.ratio. There is an example in Section 6 of the R2MLwiN article (http://dx.doi.org/10.18637/jss.v072.i10) recently published in the Journal of Statistical Software; as the article states, these return the proportion of chain iterations which are of the opposite sign to the chain mean (the coefficient estimate). I.e. they use the same information as the credible intervals (insofar as they both use the values of the chain iterations), but just emphasise a different element of that information.

E.g.:

Code: Select all

library(R2MLwiN)
data("tutorial")
F3 <- normexam ~ 1 + standlrt + (1 | school) + (1 | student)
print(standlrtMCMC <- runMLwiN(Formula = F3, data = tutorial, estoptions = list(EstM = 1)), z.ratio = FALSE)
...would yield the following output (truncated):

Code: Select all

The fixed part estimates:  
              Coef.   Std. Err.  pMCMC(1-sided)  [95% Cred.   Interval]    ESS 
Intercept   0.00514     0.04246           0.444    -0.07933     0.08896    211 
standlrt    0.56321     0.01250               0     0.53862     0.58791   4060 
So the proportion of chain iterations which were below zero for the Intercept was 0.444, whilst none of the 5000 iterations were below zero for standlrt.

Compare this with (note that z.ratio = TRUE is actually the default here, but just including it to be explicit):

Code: Select all

print(standlrtMCMC, z.ratio = TRUE)
...which would yield (again, truncated):

Code: Select all

The fixed part estimates:  
              Coef.   Std. Err.      z   Pr(>|z|)       [95% Cred.   Interval]    ESS 
Intercept   0.00514     0.04246   0.12     0.9036         -0.07933     0.08896    211 
standlrt    0.56321     0.01250  45.05          0  ***     0.53862     0.58791   4060 
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1  
Best wishes,

Richard
Post Reply