Very slow transition from burn-in to inference chain
Posted: Thu Jan 30, 2014 12:18 pm
Hello,
as always, I'm not quite sure whether this is a runmlwin or MLwiN problem.
I'm estimating a complex (i.e. multivariate 2-lvl) model on a large sample (~100k obs) using MCMC via the runmlwin add-on. MLwiN 2.29 is run in batch mode using mlnscript.exe (64-bit). Estimation works, but convergence is very slow. I expected this from MCMC but have now spotted something that makes me wonder if I'm waiting unnecessarily.
I noticed an odd behavior of mlnscript.exe: The command window does not show any iterations after the last burn-in iterations (e.g. gets stuck at 'Burning in for 4900 iterations out of 5000'). Then, after several days, estimation suddenly finishes and I get results - I think I've seen once or twice that the command window was showing the iterations for the main bit of the chain. However, I cannot confirm this with confidence because MLwiN runs on a server cluster and I'm not monitoring it all the time. So I don't know if the command window is just not being updated but the sampler is working in the background, or if there's a problem transiting from the burn-in to the estimation of the chain that is used for inferences. Doesn't seem to be a problem with much smaller samples and univariate models, so this pointed me to the latter hypothesis.
Are any data being copied when moving from burn-in to the 'main chain'? If so, is that a very slow process? Is there a known work-around, e.g. set burn-in to zero and discard the first 5000 iterations manually from the stored chains?
Cheers
Nils
as always, I'm not quite sure whether this is a runmlwin or MLwiN problem.
I'm estimating a complex (i.e. multivariate 2-lvl) model on a large sample (~100k obs) using MCMC via the runmlwin add-on. MLwiN 2.29 is run in batch mode using mlnscript.exe (64-bit). Estimation works, but convergence is very slow. I expected this from MCMC but have now spotted something that makes me wonder if I'm waiting unnecessarily.
I noticed an odd behavior of mlnscript.exe: The command window does not show any iterations after the last burn-in iterations (e.g. gets stuck at 'Burning in for 4900 iterations out of 5000'). Then, after several days, estimation suddenly finishes and I get results - I think I've seen once or twice that the command window was showing the iterations for the main bit of the chain. However, I cannot confirm this with confidence because MLwiN runs on a server cluster and I'm not monitoring it all the time. So I don't know if the command window is just not being updated but the sampler is working in the background, or if there's a problem transiting from the burn-in to the estimation of the chain that is used for inferences. Doesn't seem to be a problem with much smaller samples and univariate models, so this pointed me to the latter hypothesis.
Are any data being copied when moving from burn-in to the 'main chain'? If so, is that a very slow process? Is there a known work-around, e.g. set burn-in to zero and discard the first 5000 iterations manually from the stored chains?
Cheers
Nils