MCMC-estimation and memory

[IngarB]

Hi,

I'm running a 4-level model with a binary response variable. There are 5400 individuals and 20 explanatory variables at level 1, and no explanatory variables at level 2, 3 and 4. I run MCMC-estimation and have asked for 200 000 iterations - when I reach 164 200 iterations I get this message:

"MCMC Error 0063: Not enough memory to continue"

I have tried it at two different PC's - and it stops at the same number of iterations (164 200), so I don't think it has to do with the memory or capacity of the PC. What can I do to continue estimation??

Details:
Under “Estimation Control” and “MCMC” I have put the following numbers:
Burn-in-length: 500
Monitor chain length: 200 000
Thinning: 1
Refresh screen every: 50

Under “Model” – “MCMC” – “MCMC Options” I have chosen:
Use hierarchical centring
Centre at level: 4

Is it any solution to this problem?

Best regards,
Ingar

[Lydia]

When MLwiN uses MCMC estimation, for each MCMC iteration it stores the estimate at that iteration of each parameter. This is how it is able to plot out the trajectories for each parameter. So in your case, it will need to store (20 [coefficients of explanatory variables] + 1 [coefficient of cons] + 4 [a variance at each level]) x 200 000 [number of iterations] = 25 x 200 000 = 5 000 000 estimates. The estimates are stored in a column in the worksheet. 5 000 000 is, obviously, a large number of estimates to have to store, which is why you're experiencing problems. You have two possible solutions:

1) Increase your worksheet size. There's an FAQ on the CMM website that explains how to do this; note that there is a limit to how big worksheets can be though and you may come up against this

2) Use thinning. This means that instead of storing the estimates for every iteration, you only store the estimates for every 10th iteration or every 100th iteration or whatever. To do this, enter n into the Thinning box in the Estimation control window, where n is the number of iterations you want to go in between storing the estimates, that is you want to store the estimates for every nth iteration. MLwiN will allow n to be any positive whole number (I believe). I believe that the final estimates you see in the Equations window and elsewhere will still be based on all the iterations, not just the stored iterations, but you'd better check that in the MCMC manual. If that's the case, then you only need to store as many iterations as you can usefully plot on a graph (when you look at the trajectories), unless you are planning to do anything for yourself with the stored estimates such as calculate distributions for functions of parameters. I'd recommend starting with a thinning of perhaps 5 or 10 and increasing if you still can't run (though it would seem that if it's crashing after 164 200 iterations then 5 should certainly be enough- even 2 should really be enough).

[IngarB]

Thank you so much!! It worked:)

Ingar

[IngarB]

I forgot to say what I actually did: I tried thinning of 5 and then I could run 500 000 iterations without a problem.

As to the question of what the estimation are based on - the MCMC manual (Browne 2009) says on page 26:
"Note that the parameter mean and standard deviation use all the iteration values,
no matter what thinning factor is used. All other summary statistics and
plots are based on the thinned chain only."

Thanks a lot!

Ingar

[Lydia]

Glad to hear it worked, and thanks for passing on the info about what it uses for the actual estimates - useful to know!