post hoc power calculation question

[sasso4499]

Hey Everyone,

I am trying to use MLwIN 2.30 to conduct a post hoc power analysis on repeated measure regression models, originally implemented in SAS PROC MIXED. In each of these models (using separate ones for each measure of interest), I am examine two variables, one representing between-patient variability in the measure and the other reflecting within-patient variability in the measure, as fixed effects predictors of a repeated measures DV.

The code for the model is as follows:

proc mixed data = ks_adh;
class patno session;
model bdi = between_patno_F1 within_patno_F1/s ;
repeated session / type=un subject = patno ;
run;

One of my measures has a high amount of within-patient variability (.72) while the others have relatively low within-patient variability (.31-.33). Reviewers have questioned my power to detect an effect under these different circumstances (i.e., different amounts of within-patient variability in the measure of interest). Are you all aware of any ways in which I might estimate differences in power as a function of differing levels of within-patient variability in the predictors? That is, how can I index the extent to which the power of these within-patient predictors is compromised due to low within-patient variability in the raw scores? Is this something MlWin is suited for? If so- any tips on how to calculate would be very helpful. I would greatly appreciate any feedback or suggestions you all may have!! Thanks
-a confused psychologist

[ChrisCharlton]

You might want to look at the MLPowSim software (http://www.bris.ac.uk/cmm/software/mlpowsim/) to see whether this covers any of your requirements.

[sasso4499]

Thanks for the response- we took a look at MLPowSim and realized that this would only allow us to do a post-hoc simulation. The main issue raised by reviewers was that, given the the low within-patient variability in some of or predictors, were our null findings regarding certain within-patient predictors merely manifestations of low power. A post hoc power calculation would allow us to determine what sample size would be needed to find an effect for these within-patient variation in this predictors (that had low within-patient variability). Presumably, given an infinitely large sample size, these effects would differ from zero. However, this wouldn't allow us to know whether we failed to find an effect because our sample size was to small, or for other reasons (i.e., our measure of these predictors weren't sensitive to real within-patient deviations, or possibly these predictors our irrelevant to the repeated measure changes we are trying to predict). Given these concerns that would remain even after a post hoc power simulation was conducted, we decided it would just be best to highlight these concerns in more detail. However, I would welcome a difference of opinion, if you feel that there is something else that could be gained in this case by going through with the post hoc power calculation

[billb]

Hi Sasso,
Post-hoc power calculations are generally frowned upon in the stats community - see for example http://research-repository.st-andrews.a ... tprint.pdf although in practice we often talk about using pilot studies to get estimates for a power calculation for the main study and the difference is somewhat subtle! Multilevel power calculations are difficult as one has so many things to consider and that is why we wrote MLPowSim which performs such calculations via simulation - there are analytical alternatives for some simple balanced cases and for example the package pint (see http://seis.bris.ac.uk/~frwjb/esrc.html for links to several packages including Pint).
Regards,
Bill.