2021 Conference on Test Security
Alpine Session: Wednesday, October 6
A Practical Approximation of Response Similarity
Presented by:
- Russell Smith, Ph.D.
- 11:00 am EST
This presentation will describe a straight-forward proxy for response similarity index analyses based on reasonable estimates of expected matching responses, and discuss the advantages and disadvantages of each approach. Response similarity index analyses can be challenging for multiple reasons. It relies on a nominal response model to accurately estimate the probability of a specific response across the ability continuum. For each pair of test takers, the joint probability of matching responses is calculated and compared to the actual number of matching responses. The model can require substantial sample sizes, can be computationally intensive, and the current software and
model fit can be finicky.
The expected count of matching responses should be less than the expected count of matching scores. Further, a test taker’s score on the exam should be an accurate estimate of their average expected score on any given item. For example, on a 100-item dichotomously scored test, a candidate with a score of 70 would be expected to score 0.70 on each item on average. Total scores can, therefore, be used to estimate the expected number of matching scores. The proposed method uses total scores to estimate expected number of matching scores for pairs of test takers. Then, the number of matching responses are compared to the expected number of matching scores. Though the model does not have a specific underlying distribution, the ratio of actual to expected is a useful surrogate that is easy and relatively efficient to calculate.
No matter your industry, you will find valuable content that is relevant to your area of expertise at COTS 2021. Join experts from all types of organizations as they discuss the test security capabilities and enhancements that protect the validity of test results and brand integrity. The 2021 conference will be hosted in partnership with ETS. We hope you will join us virtually at COTS 2021 to explore emerging test security capabilities and research on Oct. 6-7, Nov. 3-4, and Dec. 1-2.
To learn more, please click here.