Originally published December 02, 2021 in Credentialing Insights – The online journal for I.C.E.
Interview by Robert C. Shaw, Jr., PhD, National Board for Respiratory Care (NBRC)
After graduating from the University of Nebraska with a doctorate in quantitative, qualitative and psychometric methods, Brett Foley, PhD, has worked with programs in the education, certification and licensure areas. He currently serves as the director of professional credentialing and a senior psychometrician at Alpine Testing Solutions, and is the chair of the Nebraska Board of Engineers and Architects as a public member. Formerly, Foley served as president of the Northern Rocky Mountain Educational Research Association. His research interests include standard setting, policy considerations in testing and using visual displays to inform the test development process.
Robert C. Shaw, Jr., PhD, vice president of examinations for the National Board for Respiratory Care (NBRC) and member of the I.C.E. Publications and Editorial Committee, recently spoke with Foley on how he got into psychometrics, what being a public member of a board of directors is like and more.
RS: You have worked with programs that had missions in education, licensure and certification. What have been some memorable programs from these three areas?
BF: One of the best parts of my job is getting to work with experts in a variety of different fields and playing a small role in assisting in their important work. Early in my career, I had the opportunity to work with the Hawaii Department of Education on their state-level summative assessments for kids with severe cognitive disabilities.
More recently, I’ve shifted focus to credentialing. While there are many programs I’ve enjoyed working with, I’m especially proud of the long-term work I’ve been able to do with the National Council of Architectural Registration Boards (NCARB), the International Hearing Society (IHS) and the National Board of Examiners in Optometry (NBEO). These groups respectively produce credentialing exams for architects, hearing professionals and optometrists. In addition to doing essential work in protecting the public, these organizations also have been active research partners for Alpine, advocating for innovation, the dissemination of new practices and presenting at conferences like the I.C.E. Exchange.
RS: In what year did you earn your doctorate from the University of Nebraska and what was your first psychometric gig?
BF: I finished my doctoral work in 2010. My first psychometric gig was at the University of Nebraska at Lincoln while still in graduate school. I was lucky enough to earn an assistantship (and later a full-time job) at the Buros Center for Testing. It was a fantastic experience, giving me my first opportunity do practical, hands-on work in psychometrics. Much of my time was spent working on equating, standard setting studies and content alignment studies for state departments of education.
RS: You are a public member and current chair of the Nebraska Board of Engineers and Architects while having a background as a psychometrician. How does the public member of this board become its chair?
BF: Following a meeting, I was out at dinner with a group of architect subject matter experts who happened to be board members back in their home states. They were discussing the challenges they were facing in recruiting public members to their boards. I asked what they looked for in a public member and if someone with a testing background would be useful in such a role. They were very positive about the idea and encouraged me to pursue it. After getting home, I did a quick search and found that the Nebraska public member position would be opening in the following few months. I filled out an online application and a couple of months later got a fancy certificate in the mail from the governor.
It has been an awesome experience, and I need to take this opportunity to thank Alpine in being supportive of its employees engaging in professional service. For me, it has been more professional development than service alone; in my first six months on the board, I learned more about the non-assessment elements of the licensure (e.g., legislative concerns, disciplinary actions, education, experience) than I had in many years of psychometric work.
An especially meaningful aspect of the role is that I get the chance to celebrate the success of candidates. With all the necessary work we do in the assessment community to set up the hurdles candidates must pass on their way to a credential, it’s fulfilling and humbling to have my signature on the certificate that may hang above the desk of an architect or engineer for their entire career, or to speak to first-time licensees at a ceremony at the Nebraska state capital.
The board’s mission is “To establish requirements for education, experience, examination and enforcement for the practices of engineering and architecture to safeguard life, health and property, and to promote public welfare.” In short, protection of the public. I was able to become chair because both the state statutes and board members recognize the public member as an essential component of and full participant in the board. As such, board leadership shifts between architects, engineers and public members across their five-year appointments.
RS: I attended a session during the Council on Licensure Enforcement and Regulation (CLEAR) virtual meeting in fall 2020 in which a regulator from Canada proposed that half the members of an occupational regulation board should be members of the public to assure that the needs of the public are a priority rather than the needs of occupational incumbents. I propose you are in a unique position to think through such a scenario so what do you see?
BF: I don’t have a good answer as to what the “right” number of public members on a board should be. Clearly, I recognize the value of public members in ensuring that the interests of folks outside of the profession are represented. However, all board members I have met who are working in the profession take their role of protecting the public very seriously. Additionally, they bring essential subject matter expertise. For example, it is difficult to judge the competence of a professional in the field if you lack competence yourself. In my view, the strongest boards feature a mix of public members, professionals in the field in question and representatives from closely related professions. This helps make sure there is a diversity of views represented, there is sufficient subject matter expertise to make informed judgments on technical issues and no one constituency can ignore the concerns of the others.
RS: I saw an announcement about a CLEAR webinar you gave that proposes a different way to arrive at the passing decision within a program that assesses candidates with a test. Anticipating that Brett the psychometrician will describe what he saw during the webinar, how does Brett the public board member look at the proposition?
BF: Great question! Brett the psychometrician has spent much of his career focused on the accuracy of tests for making pass/fail decisions. Brett the public board member is much more concerned about the accuracy of testing systems for making credentialing decisions.
While these ideas are similar, the differences are subtle but important. Testing systems include all the hurdles candidates must pass to earn the credential (for example, architects must pass six different exams) as well as the retake policies for each of these exams. Each of these hurdles have properties that affect associated exam-level accuracy (e.g., reliability), and each additional assessment and retake opportunity influences the likelihood of false-positive or false-negative credentialing decisions. Estimating the accuracy of credentialing systems by taking all these factors into account was the topic of the CLEAR webinar and continues to be a topic of interest for me (both as the psychometrician and the public member).
Increasing the Accuracy of the Licensure Decisions: On the Benefits of More, Shorter Exams
Brett Foley, PhD, recently presented a CLEAR webinar in which he proposed a different way to arrive at the passing decision within a program that assesses candidates with a test. Below are the main takeaways from the presentation.
For most of my professional life, I’ve held to the conventional wisdom that 1) all things being equal, longer tests are better, and 2) it’s a good idea to stay away from conjunctive decisions (i.e., having to pass multiple sections of an exam) because when you set up multiple hurdles, your final decision is only as accurate as your least accurate hurdle. It turns out that this is reasonable advice, but only if you are thinking of a single exam attempt. In the real world of high stakes testing, it is a professional expectation that failing candidates are allowed retake opportunities. Once retakes come into play, conventional wisdom goes out the window.
If we think of the credentialing decision as only being “made” once a candidate either passes their exam(s) or continually fails until they exhaust all retake opportunities, false-negative credentialing decisions (i.e., a qualified candidate fails to earn a credential) almost never happen. This is because a qualified candidate would have to (incorrectly) fail on the first try and the second try and the third try and so on. Because of these “ands,” the rules of probability we learned back in school start kicking in.
Think about rolling a fair die and getting a six. Getting a six on the first try wouldn’t be that unusual, but getting a six on the first, second, third and fourth tries would be pretty unlikely. In the same way, once the number of allowed retakes gets to about three or four, the probability of a qualified candidate failing all of them (even for very unreliable exams) drops to near zero. Awesome! We have almost no false negatives! So, we should just keep cranking up the number of retake opportunities, right? Not so fast.
In most credentialing programs, especially where issues of health and safety come into play, we tend to be just as interested in false-positive credentialing results (i.e., an unqualified candidate earning a credential). In this case, the laws of probability start working against us.
Back to our die-rolling example: Even if the chances of rolling a six aren’t great, rolling a six on the first, second, third or fourth try isn’t as daunting. Similarly, false-positive credentialing decisions become increasingly, and alarmingly, likely, even for fantastically reliable exams, when several retake opportunities are allowed. That is, the unqualified candidate only needs to get lucky once and the wrong decision is made. So, do we get rid of retakes?
It looks like the research is starting to point to a two-step solution for minimizing both false positives and false negatives. First, adopt reasonable retake policies, with things like lag-time between attempts and requirements for remediation/education after multiple failing attempts. Second, adopting multiple exams of modest reliability can be just as (or more) effective than a single, long, highly reliable exam when it comes to credentialing accuracy. The reason for this is that setting up multiple hurdles helps to counteract the false-positive rates induced by retakes: an unqualified candidate may pass one of the exams, but as the number of hurdles increases, it becomes increasingly unlikely to pass all of them.
Robert C. Shaw, Jr., PhD
Robert C. Shaw, Jr., PhD, worked as a respiratory therapist and respiratory therapy educator between 1982 and 1995. After earning a doctorate degree in higher education with emphases in measurement and statistics from the University of Missouri in 1994, Shaw began advising occupational credentialing programs in 1995. He worked for Applied Measurement Professionals, Inc. (AMP), from 1995 through 2015 while providing psychometric direction to several programs including the National Board for Respiratory Care (NBRC). The NBRC owned AMP until the company was sold to PSI at the end of 2015. Shaw is the vice president of the examinations division for the NBRC.
Brett P. Foley, PhD
Brett is the Director, Professional Credentialing and Senior Psychometrician at Alpine Testing Solutions. He has worked with many types of testing programs in licensure, certification, and education. He received his Ph.D. in Quantitative, Qualitative, and Psychometric Methods from the Department of Educational Psychology at the University of Nebraska-Lincoln. He is a past-president of the Northern Rocky Mountain Educational Research Association. He currently serves as a Consulting Editor for the Journal of Educational Psychology and on the Nebraska Board of Engineers and Architects as the Public Member. His research interests include standard setting, policy considerations in testing, and using visual displays to inform the test development process.