Very slow transition from burn-in to inference chain

[NilsGYork]

Hello,

as always, I'm not quite sure whether this is a runmlwin or MLwiN problem.

I'm estimating a complex (i.e. multivariate 2-lvl) model on a large sample (~100k obs) using MCMC via the runmlwin add-on. MLwiN 2.29 is run in batch mode using mlnscript.exe (64-bit). Estimation works, but convergence is very slow. I expected this from MCMC but have now spotted something that makes me wonder if I'm waiting unnecessarily.

I noticed an odd behavior of mlnscript.exe: The command window does not show any iterations after the last burn-in iterations (e.g. gets stuck at 'Burning in for 4900 iterations out of 5000'). Then, after several days, estimation suddenly finishes and I get results - I think I've seen once or twice that the command window was showing the iterations for the main bit of the chain. However, I cannot confirm this with confidence because MLwiN runs on a server cluster and I'm not monitoring it all the time. So I don't know if the command window is just not being updated but the sampler is working in the background, or if there's a problem transiting from the burn-in to the estimation of the chain that is used for inferences. Doesn't seem to be a problem with much smaller samples and univariate models, so this pointed me to the latter hypothesis.

Are any data being copied when moving from burn-in to the 'main chain'? If so, is that a very slow process? Is there a known work-around, e.g. set burn-in to zero and discard the first 5000 iterations manually from the stored chains?

Cheers
Nils

[ChrisCharlton]

You are correct that the script version of MLwiN does not display anything while performing the main MCMC iterations. This is a left-over from the way this information is displayed in the GUI version as the script version is a thin layer over the back-end. Unfortunately there is currently no way to know how far it has got using this interface, other than timing how long the burnin took and calculating an estimated time from this. We might look into providing better feedback in a later version. If the model is taking a lot longer than you expect then it is worth checking the parameter chains when the results are returned to make sure they look sensible.

[NilsGYork]

Thanks Chris! Nothing to worry then - mixing is an issue but that's a problem of the model specification, not one that is due to MLwiN.