error 'cannot allocate smatrix(fs)'

[xinyuzou]

Hi,

I bumped into some problem in fitting a cross-classified model. Any advice or suggestions are appreciated.
My data has more than 2 million observations with the number of the two classifications being around 300 and 30000. Besides, I have 112 covariates for the model. When I tried to run the IGLS first, MLwiN returned the following error message: cannot allocate smatrix(fs). And the program crashed after I clicked OK.

I am using MLwiN 2.3 on Win 8.1. The computer has two processors, 2.27GHZ and 2.39GHZ, with 20GB RAM.

What can be the problem here? Is it due to the large data or the number of covariates?

Thanks!

[ChrisCharlton]

I can't give a definite answer without looking at the data, but I suspect that the problem is due to the large number of records per level-2 unit. If you have access to Stata or R I would suggest that you try using runmlwin (http://www.bris.ac.uk/cmm/software/runmlwin/) or R2MLwiN (http://www.bris.ac.uk/cmm/software/r2mlwin/) to call the 64-bit scripting version of MLwiN (mlnscript) which will be able use use more of the memory in your system.

[xinyuzou]

Hi Chris,

I tried using runmlwin on Stata, but MLNScript stopped working with the following message:"error while obeying batch file C:\User\XY\AppData\Local\Temp\ST_02000007.tmp at line number 164"

Do you know the cause of it?

Thanks!

I can't give a definite answer without looking at the data, but I suspect that the problem is due to the large number of records per level-2 unit. If you have access to Stata or R I would suggest that you try using runmlwin (http://www.bris.ac.uk/cmm/software/runmlwin/) or R2MLwiN (http://www.bris.ac.uk/cmm/software/r2mlwin/) to call the 64-bit scripting version of MLwiN (mlnscript) which will be able use use more of the memory in your system.

[ChrisCharlton]

Unfortunately that error message doesn't provide enough information to track down the cause. Can you find the file that it refers to (C:\User\XY\AppData\Local\Temp\ST_02000007.tmp) and check what is on line 164?

For your data you will probably also have to add the optimat option in mlwin_settings (see runmlwin help file) to reduce some of the memory allocations. Note that to use the 64-bit scripting version of MLwiN you will need to set your MLwiN_path global to something like "C:\Program Files (x86)\MLwiN v2.32\x64\mlnscript.exe", but while you are checking that your syntax is correct you may prefer to use "C:\Program Files (x86)\MLwiN v2.32\i386\mlwin.exe" without the nopause option as this should provide more informative messages.

[xinyuzou]

Hi Chris,

When I set the MLwiN_path global to i386\mlwin.exe and specified the optimat option, MLwiN returned the error message of 'not enough memory available for the command' after the 1st iteration. The result after the 1st iteration was shown in Stata with the warning message that the IGLS algorithm failed to converge. Increase the number of iterations.

I checked the tmp file, from line 161 to 167 are: NOTE Fit the model // STAR // BATC 1// NEXT // MONI 1// ITNU 0 b21 // CONV b22
On line 164 it was 'NEXT'.

[ChrisCharlton]

Thank you for the additional information. It would appear from this that the cause of the problem is indeed most likely to be due to the large data size. I would suggest trying with the x64 version and the optimat option turned on, and if you still have problems I think that you would need to look into working on a subset of your data instead.

[xinyuzou]

Hi Chris,
So I just tried the x64 version with the optimat option on, but the program halted with the message: The model did not run properly in MLwiN. You most likely clicked the 'Abort Macro' button in MLwiN, rather than the 'Resume Macro' button.

I guess that is again due to the data size?

[ChrisCharlton]

I can't say for certain, but I think that there is a very high probability that this is the case. If I remember correctly if add the batch option to runmlwin instead of nopause it should display any error message generated from MLwiN when it returns to Stata, so you could try doing that to confirm this.

[xinyuzou]

Hi Chris, yes I have tried the batch option and I think the error log confirmed it: error at line: NEXT work matrix too small (rmatrix::zbz)

[ChrisCharlton]

Yes, that would be my interpretation of this error message too.