The thurstonianIRT package allows to fit various models from Item Response Theory (IRT) for forced-choice questionnaires, most notably the Thurstonian IRT model originally proposed by (Brown & Maydeu-Olivares, 2011). IRT in general comes with several advantages over classical test theory, for instance, the ability to model varying item difficulties as well as item factor loadings on the participants’ traits they are supposed to measure. Moreover, if multiple traits are modeled at the same time, their correlation can be incorporated into an IRT model to improve the overall estimation accuracy. The key characteristic of forced-choice questionnaires is that participants cannot endorse all items at the same time and instead have to make a comparative judgment between two or more items. Such a format comes with the hope of providing more valid inference in situation where participants have motivation to not answer honestly (e.g., in personnel selection), but instead respond in a way that appears favorable in the given situation. Whether forced-choice questionnaires and the corresponding IRT models live up to this hope remains a topic of debate (e.g., see Bürkner, Schulte, & Holling, 2019) but it is in any case necessary to provide software for fitting these statistical models both for practical and research purposes.
In the original formulation, the Thurstonian IRT model works on dichotomous pairwise comparisons and models the probability of endorsing one versus the other item. This probability depends on parameters related to the items under comparison as well as on parameters related to the participants’ latent traits which are assumed to be measured by the items. For more details see Brown and Maydeu-Olivares (2011), Brown and Maydeu-Olivares (2012), and Bürkner et al. (2019).
As a simple example consider a data set of 4 blocks each containing 3 items (i.e., triplets) answered by 200 participants.
data("triplets")
head(triplets)
#> i1i2 i1i3 i2i3 i4i5 i4i6 i5i6 i7i8 i7i9 i8i9 i10i11 i10i12 i11i12
#> 1 1 0 0 1 0 0 1 1 1 0 1 1
#> 2 0 0 1 0 0 0 0 0 1 0 0 0
#> 3 0 0 1 0 0 1 0 1 1 0 0 0
#> 4 0 0 1 1 1 0 1 1 0 0 0 0
#> 5 1 1 1 0 0 1 1 1 0 1 0 0
#> 6 1 1 1 0 0 1 1 0 0 0 1 1In the data set, a 1 indicates that the first item has been selected over the second item while a 0 indicates that the second items has been selected over the first item. In order to fit a Thurstonian IRT model on this data, we have to tell thurstonianIRT about the block structure of the items, the traits on which the items load, and the sign of these loadings, that is, whether items have been inverted. For the present data, we specify this as follows:
blocks <-
set_block(c("i1", "i2", "i3"), traits = c("t1", "t2", "t3"),
signs = c(1, 1, 1)) +
set_block(c("i4", "i5", "i6"), traits = c("t1", "t2", "t3"),
signs = c(-1, 1, 1)) +
set_block(c("i7", "i8", "i9"), traits = c("t1", "t2", "t3"),
signs = c(1, 1, -1)) +
set_block(c("i10", "i11", "i12"), traits = c("t1", "t2", "t3"),
signs = c(1, -1, 1))Next, we transform the data into a format that thurstonianIRT understands.
triplets_long <- make_TIRT_data(
data = triplets, blocks = blocks, direction = "larger",
format = "pairwise", family = "bernoulli", range = c(0, 1)
)
head(triplets_long)
#> # A tibble: 6 × 11
#> person block comparison itemC trait1 trait2 item1 item2 sign1 sign2 response
#> <int> <int> <int> <dbl> <fct> <fct> <fct> <fct> <dbl> <dbl> <dbl>
#> 1 1 1 1 1 t1 t2 i1 i2 1 1 1
#> 2 2 1 1 1 t1 t2 i1 i2 1 1 0
#> 3 3 1 1 1 t1 t2 i1 i2 1 1 0
#> 4 4 1 1 1 t1 t2 i1 i2 1 1 0
#> 5 5 1 1 1 t1 t2 i1 i2 1 1 1
#> 6 6 1 1 1 t1 t2 i1 i2 1 1 1Finally, we can fit the model using several model fitting engines. Currently supported are Stan, lavaan, and Mplus. Here, we choose Stan to fit the Thurstonian IRT model in a Bayesian framework.
As basic summary and convergence checks can be obtained via
Finally, we obtain predictions of participants’ trait scores in a tidy data format via
pr <- predict(fit)
head(pr)
#> # A tibble: 6 × 6
#> id trait estimate se lower_ci upper_ci
#> <int> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 1 t1 0.302 0.515 -0.607 1.36
#> 2 1 t2 -1.26 0.548 -2.38 -0.197
#> 3 1 t3 0.338 0.502 -0.685 1.33
#> 4 2 t1 -0.966 0.532 -2.04 0.0653
#> 5 2 t2 0.881 0.598 -0.174 2.05
#> 6 2 t3 0.703 0.596 -0.401 1.92The thurstonianIRT package not only comes with model fitting functions but also with the possibility to simulate data from Thurstonian IRT models. Below we simulate data with a very similar structure to the triplets data set we have used above.
sim_data <- sim_TIRT_data(
npersons = 200,
ntraits = 3,
nblocks_per_trait = 4,
gamma = 0,
lambda = runif(12, 0.5, 1),
Phi = diag(3)
)
#> Computing standardized psi^2 as 1 - lambda^2
head(sim_data)
#> # A tibble: 6 × 19
#> person block comparison itemC trait1 trait2 item1 item2 sign1 sign2 gamma lambda1
#> <int> <int> <int> <dbl> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 1 1 1 1 3 1 1 2 1 1 0 0.970
#> 2 2 1 1 1 3 1 1 2 1 1 0 0.970
#> 3 3 1 1 1 3 1 1 2 1 1 0 0.970
#> 4 4 1 1 1 3 1 1 2 1 1 0 0.970
#> 5 5 1 1 1 3 1 1 2 1 1 0 0.970
#> 6 6 1 1 1 3 1 1 2 1 1 0 0.970
#> # ℹ 7 more variables: lambda2 <dbl>, psi1 <dbl>, psi2 <dbl>, eta1 <dbl>, eta2 <dbl>,
#> # mu <dbl>, response <int>The structure of the data is the same as what we obtain via the make_TIRT_data function and can readily be passed to the model fitting functions.
To install the latest release version from CRAN use
The current developmental version can be downloaded from github via
if (!requireNamespace("remotes")) {
install.packages("remotes")
}
remotes::install_github("paul-buerkner/thurstonianIRT")After reading the README, you probably have a good overview already over the packages purporse and main functionality. You can dive deeper by reading the package’s documentation perhaps starting with help("thurstonianIRT"). If you want to perform a simulation study with the package, I recommend you take a look at vignette("TIRT_sim_tests").
To ask a question, propose a new feature or report a bug, please open an issue on GitHub.
If you want to contribute to thurstonianIRT, you can best do this via the package’s GitHub page. There, you can fork the repository, open new issues (e.g., to report a bug), or make pull requests to improve the software and documentation. I am grateful for all kinds of contributions, even if they are just as small as fixing a typo in the documentation.
Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460-502. https://journals.sagepub.com/doi/10.1177/0013164410375112
Brown, A., & Maydeu-Olivares, A. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44(4), 1135-1147. https://link.springer.com/article/10.3758/s13428-012-0217-x
Bürkner P. C., Schulte N., & Holling H. (2019). On the Statistical and Practical Limitations of Thurstonian IRT Models. Educational and Psychological Measurement. https://journals.sagepub.com/doi/10.1177/0013164419832063