Multilevel Item Response Models in runmlwin

elisa1977 · Post by **elisa1977** » Sat Jan 25, 2014 4:28 pm

Hi,
I’m writing a brief tutorial on Multilevel Item Response Models.
I would like to ask some information about the possibility to estimate Multilevel Item Response Models with discrimination parameters (factor loadings) using runmlwin.
Item Response Models can be formulated as mixed logistic regression models by stacking
the responses (y_ij) to different items i of person j in a single response variable (y).
Thus, the resulting data can be modeled using a two level model, where item-person responses (level1 units) are nested into persons (level2 units).

Considering a set of binary items, a basic IRT model specifies the
probability to provide a positive response to an item as function of a
person parameter “ theta_j “, an item parameter "beta_i" and a discrimination parameter (factor
loading) " lambda_j"

logit(y_ij=1)= beta_i+ lambda_i theta_j

where beta_i and lambda_i can be specified as fixed parameters and theta_j as a random term
at person level which usually follows a normal distribution with constant
variance.

If factor loadings are fixed equal to one, any software for
generalized linear mixed effect models can be used to estimate IRT models.

Indeed, a multilevel IRT model is a three- level model with logit link and factor loadings at both levels.

MLwiN should allow to specify and estimate mixed effect
models with factor structure .

I tried to have a look to runmlwin information material to find examples related to IRT models but
I have experienced some problem in understanding how to specify factor loadings.
I attached the data set (MLIRT), the code (IRT.do) I used to estimate 3
multilevel IRT using gllamm in Stata and the results I got in GLLAMM (IRT.txt)

The models set in the" IRT.do" file have the following characteristics

1) Model 1 fixes factor loadings equal to one at level-2 and level-3 (I could replicate this model using runmlwin and I got results similar with both routines)
2) Model 2 considers factor loadings at level 2 and level 3
3) Model 3 constrains factor loadings at level 3 equal to factor loadings at level 2

I would be grateful if you can provide me with suggestions to specify the models in runmlwin

Thank you in advance for your support.

The data set (MLIRT.dta) contains the following variables:
id: person code

Class: class code
q1 q2 q3 q4 q5 : 5 binary items

In long format:
y: (item-person combination) responses to 5 items (q1,q2,q3,q4,q5) stacked in long format

i1,i2,i3,i4,i5 : indicator variables i_x=1 if responses refer to item q_x (x=1,2,3,4,5) 0 otherwise

GeorgeLeckie · Post by **GeorgeLeckie** » Mon Jan 27, 2014 2:10 pm

Hi,

You can fit one- and two-parameter (multilevel) probit IRT models in MLwiN. However, you cannot fit the usual logistic versions of these models. To fit these models, you need to specifying them as a multivariate response model with one response variable for each item. You specify a diagonal covariance matrix. You then introduce a factor and allow this factor to have different loadings on the different items/responses.

I recommend that you first read Chapter 20 of the MLwiN MCMC Manual. This chapter shows you how to fit very similar models for continuous responses (you will need to specify binary responses).
http://www.bristol.ac.uk/cmm/software/m ... mc-web.pdf

Then work your way through the runmlwin do-file which shows you how to replicate those results
http://www.bristol.ac.uk/cmm/media/runm ... delling.do
http://www.bristol.ac.uk/cmm/media/runm ... elling.log

In terms of your three models, you should be able to fit your Model 1 and 2 using runmlwin. You can't fit your Model 3 as that model constrains one set of factor loadinds to equal another set and that is not currently possible in MLwiN.

Best wishes

George

GeorgeLeckie · Post by **GeorgeLeckie** » Mon Jan 27, 2014 3:18 pm

Hi,

I have thought more about this.

I don't think you can currently fit multilevel IRT models in MLwiN

You can fit single-level 1- and 2-parameter IRT models, but only if you are willing to use the probit link function

Best wishes

George

GeorgeLeckie · Post by **GeorgeLeckie** » Wed Jan 29, 2014 5:43 pm

Hi,

Just seen that you can use Stata's -gsem- command to fit 1- and 2- parameter single-level and multilevel logistic IRT models

http://www.stata.com/manuals13/semexample29g.pdf

As I said in previous post, you can only fit 1- and 2-parameter single-level probit IRT models in MLwiN.

Below is the code to replicate a probit version of the 2-parameter single-level IRT model illustrated in the gsem example

Code: Select all

* Load the data
use "http://www.stata-press.com/data/r13/gsem_cfa", clear

* Fit 2-parameter probit IRT model
gsem (MathAb -> q1-q8), probit var(MathAb@1)

* Generate a student identifier
gen student = _n

* Generate cons
gen cons = 1

* Fit single-level mutivariate binary response model to the 8 items, using a 
* probit link function and quasilikelihood (MQL1) estimation
runmlwin ///
	(q1 cons, eq(1)) ///
	(q2 cons, eq(2)) ///
	(q3 cons, eq(3)) ///
	(q4 cons, eq(4)) ///
	(q5 cons, eq(5)) ///
	(q6 cons, eq(6)) ///
	(q7 cons, eq(7)) ///
	(q8 cons, eq(8)) ///
	, ///
	level1(student:, diagonal) ///
 	discrete( ///
 		distribution(binomial binomial binomial binomial binomial binomial binomial binomial) ///
 		link(probit) ///
 		denom(cons cons cons cons cons cons cons cons) ///
 	) ///
 	mlwinsettings(variables(50)) ///
 	nopause

* Store the results 
estimates store m1mql1

* Specify initial values for discrimination parameters
matrix flinit = (0\0\0\0\0\0\0\0)

* Constrain specific discriminationfactor loadings to their initial values
matrix flconstr = (0\0\0\0\0\0\0\0)

* Specify initial values for ability variance
matrix fvinit = (1)

* Constrain ability variance to its initial value
matrix fvconstr = (1)

runmlwin ///
	(q1 cons, eq(1)) ///
	(q2 cons, eq(2)) ///
	(q3 cons, eq(3)) ///
	(q4 cons, eq(4)) ///
	(q5 cons, eq(5)) ///
	(q6 cons, eq(6)) ///
	(q7 cons, eq(7)) ///
	(q8 cons, eq(8)) ///
	, ///
	level1(student:, diagonal ///
		flinits(flinit) ///
		flconstraints(flconstr) ///
		fvinits(fvinit) ///
		fvconstraints(fvconstr) ///
	) ///
 	discrete( ///
 		distribution(binomial binomial binomial binomial binomial binomial binomial binomial) ///
 		link(probit) ///
 		denom(cons cons cons cons cons cons cons cons) ///
 	) ///
	mcmc(on) initsmodel(m1mql1) ///
 	mlwinsettings(variables(50)) ///
	nopause

Output:

Code: Select all

. * Load the data
. use "http://www.stata-press.com/data/r13/gsem_cfa", clear
(Fictional math abilities data)

. 
. * Fit 2-parameter probit IRT model
. gsem (MathAb -> q1-q8), probit var(MathAb@1)

Fitting fixed-effects model:

Iteration 0:   log likelihood = -2749.9336  
Iteration 1:   log likelihood = -2749.3708  
Iteration 2:   log likelihood = -2749.3708  

Refining starting values:

Grid node 0:   log likelihood = -2646.9568

Fitting full model:

Iteration 0:   log likelihood = -2646.9568  
Iteration 1:   log likelihood = -2638.7186  
Iteration 2:   log likelihood = -2637.5627  
Iteration 3:   log likelihood = -2637.5587  
Iteration 4:   log likelihood = -2637.5587  

Generalized structural equation model             Number of obs   =        500
Log likelihood = -2637.5587

 ( 1)  [var(MathAb)]_cons = 1
------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
q1 <-        |
      MathAb |   .8723121   .1404351     6.21   0.000     .5970643     1.14756
       _cons |   .0223258   .0743509     0.30   0.764    -.1233993     .168051
-------------+----------------------------------------------------------------
q2 <-        |
      MathAb |   .3418926   .0826392     4.14   0.000     .1799228    .5038624
       _cons |   -.283741   .0603029    -4.71   0.000    -.4019325   -.1655494
-------------+----------------------------------------------------------------
q3 <-        |
      MathAb |   .4473721   .0881417     5.08   0.000     .2746175    .6201267
       _cons |   .0936626   .0615378     1.52   0.128    -.0269492    .2142744
-------------+----------------------------------------------------------------
q4 <-        |
      MathAb |    .297041   .0793395     3.74   0.000     .1415385    .4525435
       _cons |  -.1992927    .058952    -3.38   0.001    -.3148364   -.0837489
-------------+----------------------------------------------------------------
q5 <-        |
      MathAb |   .7418947   .1188161     6.24   0.000     .5090195      .97477
       _cons |  -.0299771   .0697734    -0.43   0.667    -.1667305    .1067763
-------------+----------------------------------------------------------------
q6 <-        |
      MathAb |   .5745361   .0996388     5.77   0.000     .3792476    .7698246
       _cons |    -.19134   .0653362    -2.93   0.003    -.3193966   -.0632834
-------------+----------------------------------------------------------------
q7 <-        |
      MathAb |   .7161802   .1157557     6.19   0.000     .4893032    .9430571
       _cons |   .0645506   .0690537     0.93   0.350    -.0707921    .1998934
-------------+----------------------------------------------------------------
q8 <-        |
      MathAb |   .5157084   .0933105     5.53   0.000     .3328231    .6985936
       _cons |  -.0146761   .0630579    -0.23   0.816    -.1382674    .1089152
-------------+----------------------------------------------------------------
  var(MathAb)|          1  (constrained)
------------------------------------------------------------------------------

. 
. * Generate a student identifier
. gen student = _n

. 
. * Generate cons
. gen cons = 1

. 
. * Fit single-level mutivariate binary response model to the 8 items, using a 
. * probit link function and quasilikelihood (MQL1) estimation
. runmlwin ///
>         (q1 cons, eq(1)) ///
>         (q2 cons, eq(2)) ///
>         (q3 cons, eq(3)) ///
>         (q4 cons, eq(4)) ///
>         (q5 cons, eq(5)) ///
>         (q6 cons, eq(6)) ///
>         (q7 cons, eq(7)) ///
>         (q8 cons, eq(8)) ///
>         , ///
>         level1(student:, diagonal) ///
>         discrete( ///
>                 distribution(binomial binomial binomial binomial binomial binomial binomial binomial) ///
>                 link(probit) ///
>                 denom(cons cons cons cons cons cons cons cons) ///
>         ) ///
>         mlwinsettings(variables(50)) ///
>         nopause
 
MLwiN 2.29 multilevel model                     Number of obs      =       500
Multivariate response model
Estimation algorithm: IGLS, MQL1

Run time (seconds)   =       3.10
Number of iterations =          4
------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
q1           |
      cons_1 |   .0150403   .0560522     0.27   0.788    -.0948201    .1249006
-------------+----------------------------------------------------------------
q2           |
      cons_2 |  -.2689086   .0567913    -4.74   0.000    -.3802175   -.1575996
-------------+----------------------------------------------------------------
q3           |
      cons_3 |   .0853289    .056124     1.52   0.128    -.0246722      .19533
-------------+----------------------------------------------------------------
q4           |
      cons_4 |  -.1916709   .0564248    -3.40   0.001    -.3022616   -.0810803
-------------+----------------------------------------------------------------
q5           |
      cons_5 |  -.0250689   .0560563    -0.45   0.655    -.1349372    .0847994
-------------+----------------------------------------------------------------
q6           |
      cons_6 |  -.1661995   .0563315    -2.95   0.003    -.2766072   -.0557918
-------------+----------------------------------------------------------------
q7           |
      cons_7 |   .0501535   .0560755     0.89   0.371    -.0597525    .1600595
-------------+----------------------------------------------------------------
q8           |
      cons_8 |  -.0150403   .0560522    -0.27   0.788    -.1249006      .09482
------------------------------------------------------------------------------

------------------------------------------------------------------------------
   Random-effects Parameters |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
Level 1: student             |
                var(bcons_1) |          1          0             1           1
                var(bcons_2) |          1          0             1           1
                var(bcons_3) |          1          0             1           1
                var(bcons_4) |          1          0             1           1
                var(bcons_5) |          1          0             1           1
                var(bcons_6) |          1          0             1           1
                var(bcons_7) |          1          0             1           1
                var(bcons_8) |          1          0             1           1
------------------------------------------------------------------------------

. 
. * Store the results 
. estimates store m1mql1

. 
. * Specify initial values for discrimination parameters
. matrix flinit = (0\0\0\0\0\0\0\0)

. 
. * Constrain specific discriminationfactor loadings to their initial values
. matrix flconstr = (0\0\0\0\0\0\0\0)

. 
. * Specify initial values for ability variance
. matrix fvinit = (1)

. 
. * Constrain ability variance to its initial value
. matrix fvconstr = (1)

. 
. runmlwin ///
>         (q1 cons, eq(1)) ///
>         (q2 cons, eq(2)) ///
>         (q3 cons, eq(3)) ///
>         (q4 cons, eq(4)) ///
>         (q5 cons, eq(5)) ///
>         (q6 cons, eq(6)) ///
>         (q7 cons, eq(7)) ///
>         (q8 cons, eq(8)) ///
>         , ///
>         level1(student:, diagonal ///
>                 flinits(flinit) ///
>                 flconstraints(flconstr) ///
>                 fvinits(fvinit) ///
>                 fvconstraints(fvconstr) ///
>         ) ///
>         discrete( ///
>                 distribution(binomial binomial binomial binomial binomial binomial binomial binomial) ///
>                 link(probit) ///
>                 denom(cons cons cons cons cons cons cons cons) ///
>         ) ///
>         mcmc(on) initsmodel(m1mql1) ///
>         mlwinsettings(variables(50)) ///
>         nopause
 
MLwiN 2.29 multilevel model                     Number of obs      =       500
Multivariate response model
Estimation algorithm: MCMC

Burnin                     =        500
Chain                      =       5000
Thinning                   =          1
Run time (seconds)         =       70.9
Deviance (dbar)            =          .
Deviance (thetabar)        =          .
Effective no. of pars (pd) =          .
Bayesian DIC               =          .
------------------------------------------------------------------------------
             |      Mean    Std. Dev.     ESS     P       [95% Cred. Interval]
-------------+----------------------------------------------------------------
q1           |
      cons_1 |    .019493   .0770604      344   0.403     -.125589    .1712961
-------------+----------------------------------------------------------------
q2           |
      cons_2 |  -.2854443   .0626853      618   0.000    -.4100058   -.1590155
-------------+----------------------------------------------------------------
q3           |
      cons_3 |   .0917589   .0633281      529   0.071      -.02831    .2151516
-------------+----------------------------------------------------------------
q4           |
      cons_4 |  -.2057524   .0592854      657   0.000    -.3227843   -.0876708
-------------+----------------------------------------------------------------
q5           |
      cons_5 |  -.0310241   .0694335      454   0.318    -.1561329    .1142676
-------------+----------------------------------------------------------------
q6           |
      cons_6 |  -.1964042   .0641209      540   0.001    -.3241906   -.0721632
-------------+----------------------------------------------------------------
q7           |
      cons_7 |   .0610268   .0696582      429   0.193    -.0692531    .2040409
-------------+----------------------------------------------------------------
q8           |
      cons_8 |  -.0165442   .0634667      501   0.385    -.1448423    .1051276
------------------------------------------------------------------------------

------------------------------------------------------------------------------
   Random-effects Parameters |     Mean   Std. Dev.   ESS     [95% Cred. Int]
-----------------------------+------------------------------------------------
Level 1: student             |
                var(bcons_1) |         1         0      0          1         1
                var(bcons_2) |         1         0      0          1         1
                var(bcons_3) |         1         0      0          1         1
                var(bcons_4) |         1         0      0          1         1
                var(bcons_5) |         1         0      0          1         1
                var(bcons_6) |         1         0      0          1         1
                var(bcons_7) |         1         0      0          1         1
                var(bcons_8) |         1         0      0          1         1
-----------------------------+------------------------------------------------
Level 1 factors:             |
                        f1_1 | -.8931589  .1448504    230  -1.207734 -.6334296
                        f1_2 | -.3431035  .0821794    833  -.5100589 -.1896402
                        f1_3 | -.4491213  .0878105    802  -.6320073 -.2849912
                        f1_4 | -.3018873  .0813674    782   -.470306 -.1464515
                        f1_5 | -.7565907  .1273306    340  -1.034768 -.5261147
                        f1_6 | -.5873139  .1042238    487  -.8035995 -.3950064
                        f1_7 | -.7354952  .1183328    366  -.9992513  -.523307
                        f1_8 | -.5167831  .0957167    603  -.7130892 -.3364975
-----------------------------+------------------------------------------------
Level 1 factor covariances:  |
                     var(f1) |         1         0      0          1         1
------------------------------------------------------------------------------

The results from -gsem- and -runmlwin- are pretty similar. Note that the discrimination parameters in the -runmlwin- version are -1 times those in the -gsem- version.

Best wishes

George

elisa1977 · Post by **elisa1977** » Fri Jan 31, 2014 4:49 pm

Hi,
I really appreciate all the support you gave me.
I was waiting to have some time to work through the runmlwin tutorial on multilevel factor analysis following your suggestions, but I have realized you posted on line the whole code to estimate IRT. Many thanks for the time you have devoted to my question. It is very useful also to know that IRT models with discrimination parameters can be estimated using runmlwin (even if with probit link function).

Is it perhaps possible to adapt the code for fitting the same model with order responses by adopting an ordinal probit link? I could not find any reference to the ordinal probit link function in the on line material (on line course, MLwiN manual and do files..)

Thank you in advance for any suggestion
ELisa

GeorgeLeckie · Post by **GeorgeLeckie** » Fri Jan 31, 2014 5:47 pm

Hi Elisa,

You're welcome

No I'm afraid you won't be able to fit IRT models for ordinal items in MLwiN because MLwiN only allows multivariate responses models for continuous or binary outcome variables (or mixtures of the two).

Best wishes

George

www.cmm.bristol.ac.uk/forum

Multilevel Item Response Models in runmlwin

Multilevel Item Response Models in runmlwin

Re: Multilevel Item Response Models in runmlwin

Re: Multilevel Item Response Models in runmlwin

Re: Multilevel Item Response Models in runmlwin

Re: Multilevel Item Response Models in runmlwin

Re: Multilevel Item Response Models in runmlwin