Optimat

[as3601]

Hi,

Im running some simple models on a big dataset (1,000,000 obs). There are some very big level 1 units largest is just over 5000.

I get an overflow problem, so i specify the Optimat option.

The models now fit, but i am wondering what happens to the units with greater than 800 observations?

Are the units completely ignored. or are the first 800 observations of each unit only used?

thanks
Adrian

[ChrisCharlton]

If you don't get any error messages then as far as I am aware you will get the correct results. If you take a look at the following paper:

http://www.sciencedirect.com/science/ar ... 01548?np=y

you will see that most of the time MLwiN will not need to do direct inversions of the block and will therefore need less memory (for examples where this isn't possible see section seven). The optimat option just specifies whether or not MLwiN should allocate enough memory for the worst case where this is required. If you don't allocate enough and it is needed then you will get an error, or possibly a crash. Alternatively sometimes the required maximum block size is so large that there isn't enough memory to allocate as in your example.

[as3601]

Thanks Chris,

How does mlwin allocate memory, is this all done automatically with non-contiguous blocks, or does it like a contiguous block to work with?

I also see there is a 64bit mlscript.exe I assume this means it can use inexcess of 2gb of RAM at anyone time to fit a model.


Thanks A

[ChrisCharlton]

If you turn on the tempmat option in mlwin_settings then it will all be allocated within one large contiguous block (you can specify the amount of memory with the size option). Otherwise each matrix, etc is allocated its own block, and these could come from anywhere in the address space.

Yes, the 64bit mlnscript.exe can allocate much larger amounts of memory and is not limited by the 32bit address space.

[as3601]

Hi Chris,
Thanks for the useful help.

i assume size() is reffered in kb of space
i.e. 10 GB = 10485760 KB

Cheers A

[ChrisCharlton]

If I remember correctly the units used by size() are KCells (as in the MLwiN options menu), where each cell is eight bytes.