Originally published in Credentialing Insights – The online journal for ICE
Interview by Tina Riner, Pearson VUE – Interview with Adobe, NCARB, and Alpine Testing Solutions ahead of ICE Virtual Exchange 2020.
This year, I.C.E. is continuing their tradition of providing an opportunity for individuals in the licensing and credentialing industry to expand their knowledge through the I.C.E. Virtual Exchange (Nov. 9-11). One session being presented is “Converting 4-Option Multiple Choice Questions to 3-Option and Integrating Both Into Your Exam,” presented by Amanda Wolkowitz, senior psychometrician at Alpine Testing Solutions; Brett P. Foley, director of professional credentialing and senior psychometrician at Alpine Testing Solutions; Jared N. Zurn, AIA, NCARB, CAE, vice president of examination at NCARB; Corina Owens, psychometrician at Alpine Testing Solutions; and Jim Mendes certification development manager at Adobe.
I.C.E. Publications and Editorial Committee member Tina Riner recently spoke with the presenters about their upcoming session, which will explore the benefits of using three-option multiple-choice (MC) items and explain how to psychometrically convert existing operational four-option MC items to three-option MC items without re-pretesting. The presenters will also discuss when it is appropriate and beneficial to make the switch, provide an overview of the psychometric process on how to convert items and share practices to begin building new three- and four-option MC items.
Why is this topic important to the profession?
Many credentialing exams contain four- or five-option multiple-choice items. However, research clearly indicates that three-option multiple-choice items are just as good, if not better, than four- or more option items. Two presumed hesitations for moving to three-option items is the need to re-pretest items, as well as convince stakeholders that the items are just as valid as measuring ability as four- or more option items.
What do you hope attendees will take away from your session?
We hope attendees will take away that it is possible to convert items in a bank to three-option MC items without re-pretesting them; however, careful design decisions and certain precautions need to be made before implementing such a conversion process.
Did you find that the items, once modified, performed statistically similar to the original items, or was the performance of the item improved?
We have our predictions, but the real data results will be known soon!
Did you find that some items were not able to be converted? Or alternatively, did you find that a certain type of item was more conducive to being converted?
Yes. We were very careful on which items we permitted into the conversion pool. You’ll have to attend the presentation for details, but we will tell you that we did a very careful review of the existing option statistics before selecting items that were potential candidates for conversion.
What was the reason for converting the item? (Ex: poor performance, future cost savings, etc.)
There were multiple reasons for converting items, including (but not limited to) test taking time and future item development time.
Was it confusing for candidates to see a mix of three-option and four-option items on the same exam?
Other changes were occurring with the exam at the same time. All stakeholders were notified of upcoming changes, including the use of three-option MC items. Actual candidate reaction is to be determined. However, we have worked with several programs that use MC items with varying numbers of options, and have not received negative candidate feedback.
Not giving away the session, how was it possible to convert without pretesting? Is there anything you can share?
All we’ll say now is that we did a very careful analysis of the option statistics, we used item response theory (IRT), we did not convert all four-option items to three-option items and we have/had a backup plan in case the statistics did not turn out as planned after administration. For the punch lines, you’ll have to come to our session!
Any considerations for others considering this change?
When exam developers learn that research supports the use of three-option items, many are justifiably excited by the prospect. However, it’s clear that removing options from items has the potential to affect item difficulty. Therefore, it’s prudent to be thoughtful and cautious when changing items in an active assessment program. Rather than making a wholesale shift, this session will focus on adding three-option items to your arsenal of item types in a way that minimizes disruptions for you and your candidates.