Wolkowitz, A., Davis-Becker, S., Zurn, J. N., & Terry, J. E. (2015). If at first you don’t succeed: An analysis and interpretation of candidate retake response patterns. Presented at the 2015 Association of Test Publishers conference, Palm Springs, CA.
The testing industry supports and encourages examinees to be offered at least one exam retake opportunity. Credentialing programs often define and enforce specific exam retake policies that include requirements such as waiting periods, retake fees, or education/retraining. The intent of these policies is to limit candidates’ exposure to the exam content for security purposes and encourage candidates to make a serious effort at the earlier attempts. Although programs typically provide failing candidates with guidance on preparing for their second attempt, there are a number of strategies that candidates may use to achieve a higher score. The research in this area supports the idea that examinees perform better on a second attempt at an exam, but there is little empirical research providing insight as to how examinees’ response patterns change or remain the same between attempts. This information can help program stakeholders better understand how candidates prepare for a second attempt at an exam and if having prior exposure to some content provides an advantage. From a program maintenance perspective, analysis of retake performance and response patterns can help identify compromised items or help detect cheating. In the session presented at ATP, representatives from two credentialing programs provided an overview of their exam retake policies, including the rationale for such policies. Presenters explained how response patterns were analyzed for repeat test takers for both programs, including in-depth investigations of these response patterns by item response time, item difficulty, item type, and exam length. The main focus of this presentation was a review of the key findings that emerged from this investigation, along with the resulting impact for these programs. From a psychometric perspective, specific findings were detailed that raise validity and security concerns. From a policy perspective, key findings were interpreted with regards to program impact and future policy discussions. Guidance was provided to program leaders as to why such investigations should be conducted, how to analyze response patterns, and suggestions for psychometric and policy interpretations.