As we close out 2022 and head into 2023, we asked six leaders, who are at the helm steering the ship for their programs, to share their thoughts on the future of their respective program and our industry. We had the privilege of hearing from:
Can you share anything about your future (2023-2027) goals, plans, objectives, or challenges?
“If we’ve learned anything over the past 2 years, it is that the future is quite hard to predict. As the leader in analytics, SAS continues to emphasize resilience and adaptability to meet the needs of our customers across market verticals.
Our goal for certification, while maintaining our ‘tried and true credentials’, is to support business transformation by first solidifying our understanding of the skills critical to ‘new’ SAS users and offer them world-class credentials that meet them where they are in their learning journey and help them meet their learning and career goals.”
–Matt Scicchitano, SAS
“We’re faced with the challenges of producing more of the same high-quality content in less time with only a slight increase in resources. We will test methods/approaches to leveraging Artificial Intelligence (AI) in our item development processes. We have a few different theories in mind and will test some of them in the coming year in the hopes of identifying the best approach and addressing the aforementioned challenge.
Producing quality content is good – but producing valuable outcomes that employers and professionals can trust and leverage is the ultimate goal. We’ll explore methods to increase our capacity to meet this goal more broadly across the IT industry.
Intellectual property theft, score validity, and preventing policy breaches remains top of mind. Partnering with legal and other industry organizations that will help us change some of the negative trends we are seeing is becoming a key element of our operation moving forward. We might emphasize in-depth investigations, data forensics, evidence collection, and ultimately legal action to shut down bad actors much more now than in the past.”
–Carl Bowman, CompTIA
“Over the next 4 years, our organization plans to conceptualize an updated approach to our architect licensure programs. The industry has been doing the same thing the same way for so many years —and we’ve (not shockingly) been getting the same result. To make major improvements in the areas of diversity, equity, and inclusion, we know we need to take a fresh look at what we do and how we do it.”
–Jared Zurn, NCARB
“In the next 4 years, I see our program growing commensurate with our projected company growth. That means developing new exams as well as keeping current exams fresh and relevant. It also means many new candidates to manage and satisfy. As our programs grow, security will become even more of a concern. There will be an increased demand for reporting and tracking. I see the social media presence expanding and evolving. We will need to look at new technologies and industry trends to stay on pace. As we formulate plans for an academic program, there will need to be some adjustments or accommodations, and perhaps new paths of learning and assessments for that audience.”
–Joe Cannata, Certification Director
If you had a magic wand and could make one wish to change/improve the industry, what would it be?
“I would love to see more focus on simplifying the integration of testing systems and applications. Most programs likely have various vendors and systems for exam development, exam delivery, candidate data management, etc. In addition, we may need to integrate these with internal systems such as an LMS, CRM, or portal. Making integration across these systems easier would be magical!”
–Michelle Jackson, Lenovo
“Our ability to remove stolen intellectual property (IP) from distribution on a global scale, or completely eliminate the negative impact of IP theft through enhanced exam delivery processes and mechanisms.”
–Carl Bowman, CompTIA
“I think the ‘standard answer’ here is to once and for all end the game of whack-a-mole that we have with brain dump sites/tools. It’s in human nature to want to get a peek at the content that goes into a high-stakes assessment, but the exposure of exam content compromises the integrity and validity of assessments and undercuts the value of the certification for everyone who earns it. One of my primary goals in the design of our program is to avoid the trap of the ‘paper certification’ – and to have our exams recognized by those who take them as being ‘tough but fair’ measures of their skills and knowledge.”
–Jim Henderson, Ping Identity
“Have a digital passport for identification across the various platforms, so candidates would have one login for exams, certification data, digital badges, and CEU maintenance. This way, when they change jobs, or work with new programs, they don’t need to create new accounts on each platform.”
–Joe Cannata, Certification Director
“Consistency all the way. New players in the credentialing space, old players adjusting their business models, and post-pandemic labor market pressures have led to various forms of credentials being available across the IT space. It’s often difficult for consumers and credentialing bodies to navigate the landscape and understand the differences between various credentials and their value.”
–Matt Scicchitano, SAS
“I wish it would be easier and less expensive to capture and measure true performance assessments. We all want an assessment that truly reflects practice with all its messy complexities. However, creating an assessment that allows for messy interactions where successful outcomes are not known until months later seems impossible given time and dollar constraints.”
–Jared Zurn, NCARB
Where do you see things headed in the future for the testing industry?
“I see the future of testing becoming more candidate centric. A recurring theme in discussions with colleagues is making our programs and tests a positive experience for the candidate. Candidates have many choices when it comes to training and certification. For a viable program, the candidate experience must be satisfactory.”
–Michelle Jackson, Lenovo
“The consumers of the certification exams have high expectations for credentials that are valuable in the market, and we maintain that value through the rigorous practices that we have employed for decades. However, the speed of technology has increased, and that has created candidate demand for learning and assessment that fits into their work-life balance and meets the expectations for on-demand, instant gratification that we’ve all come to expect in recent years.
The certification and testing industry will need to balance those practices of maintaining validity and value while becoming more consumer friendly. Technology will help organizations keep up with these changing demands and help certification bodies reach more users – just in time.”
–Matt Scicchitano, SAS
“Ongoing increased focus on performance-based testing, new approaches to PBTs, technology enhancements with the ultimate goal of improving the employee’s knowledge/skills and making the worker more effective. Assessing whether the candidate is truly prepared for the job, is what employers truly need/want. With a potential economic slowdown and the need to run operations with limited resources, employers will need to know that those who remain employed, are fully equipped for the tasks at hand.”
–Carl Bowman, CompTIA
“I think a lot of what has been done will continue to be done. Sure, we love to talk about disruption, but then we go back to tried and true methods because they are safe and defensible. Multiple-choice exams serve a good purpose, checking for knowledge acquisition, and they should continue to be used when that is the goal. I hope the industry is headed towards figuring out ways to measure competency outside of test centers and away from online proctoring cameras.”
–Jared Zurn, NCARB
“The implementation of AI-generated items seems to be gaining momentum. The way SMEs collaborate is moving away from in-person workshops to a distributed model using collaborative tools. I see a need for innovative ways of assessing skills coming, which will make the traditional exam seem ancient.”
–Joe Cannata, Certification Director
“There seems to be a potential trend towards the idea of ‘unlimited retakes’ in the academic world. If the goal is to achieve mastery, then limiting the ability to retake until test takers achieve mastery can hamper progress. For technical certifications, if the goal is to measure someone’s achievement of mastery, then limiting retakes might cut them off from a technology as a career path. How do you make sure that you’re actually measuring mastery if someone has taken an assessment 10 or 15 times, rather than measuring their ability to memorize an item pool?
Practical assessments are the way to go, possibly with some form of AI-based task generation (similar to the idea of using Automated Item Generation in forms-based exams) being used to create equivalent variants of tasks. This is a lot of complexity associated with performance-based testing because you can’t just use the same task and have the candidate fill in different configuration values per variant. But this is something that we need to work towards. Ultimately in the IT industry, what someone knows is less important than what someone is able to do. Knowledge is a prerequisite to the skills but being able to apply the knowledge is the important thing.
For my program, I don’t want to see a disparity between the ‘book answer’ and the ‘real world answer’. Those two answers must be the same, or the exam simply isn’t a valid measure of what we’re trying to measure.”
–Jim Henderson, Ping Identity
How has the pandemic changed the way that you conduct your test development and delivery practices?
“Primarily, the pandemic accelerated trends we were already beginning to see develop.
Our exam development activities were already becoming more asynchronous and geographically dispersed. Now, we operate almost exclusively virtually when creating and maintaining our certification content. While this can present its own unique administrative challenges, it has also resulted in opportunities to engage new technologies and a more diverse global team of subject matter experts.
Regarding delivery, I imagine we’re part of the majority that saw online proctored exam volumes skyrocket seemingly overnight. This was born out of necessity, and like so many others, we had to lean in to the technical and security challenges that this rapid change brought about. The industry has had to take action to flatten the learning curve and make online proctoring a secure, reliable, and convenient method of assessment.”
–Matt Scicchitano, SAS
“Like many, we pivoted to completely remote item writing and review meeting and we began offering an online proctored option along with traditional test center delivery. Online proctoring is here to stay, but all virtual item development and review has already been pushed aside because the value of face-to-face reviews is worth the cost.”
–Jared Zurn, NCARB
What do you consider to be the pros and the cons with any new interventions you have implemented due to the pandemic?
“Really the biggest challenges (pro or con) ultimately had to do with remote IDWs (and C&As and Standard Settings). In the ‘pro’ column, we had the ability to include item writers from outside the US; we had item writers from both EMEA and APAC, and it was great to have the perspectives from people outside the US without having to deal with travel. On the flip side, it was harder to get participants to commit just because of the time requirements and knowing that they’d be expected to maintain their regular job as well. In-person makes it feel ’more legitimate’ and harder for the participants to say, ‘I need to step away for a meeting.’ I think they tend to be more engaged in person.”
–Jim Henderson, Ping Identity
“Even before the pandemic, we did all our exam development workshops online and remotely. Thus, we did not have to adjust or adapt to travel restrictions. In general, remote development always has the advantage of including SMEs that might not be able to travel to an on-site, in-person workshop.”
During the pandemic, we re-introduced online proctoring for some of our exams. The benefit is making the exam more easily accessible to a broader audience. The challenge is setting candidate expectations about this testing environment. Working closely with our test delivery provider was key to successful implementation.”
–Michelle Jackson, Lenovo
“Online delivery is a great option for those that have the resources to support a quiet and stable remote environment. For our program, that accounts for about 25% of exam takers. However, we may have unintentionally created more of gap between the ’haves’ and ‘have nots’ because some candidates with less financial means and who receive less support from their office feel stuck going to a test center. We certainly haven’t figured out how to address the various equity issues at play, but it doesn’t mean we won’t stop asking the questions.”
–Jared Zurn, NCARB
Alpine Testing Solutions is very appreciative of these experienced professionals who graciously shared their thoughts and vision for the future of the testing industry.
As an industry, it is clear that intelligent, cutting-edge solutions for test development, delivery, and maintenance are essential to improve the overall candidate experience and validity and the growth of quality programs. As Garrison Wynn, author and motivational speaker, coined it, “Action and adaptability create opportunity.” Those opportunities act as the catalyst that will drive solutions and that allow programs to thrive. The passion, commitment, and efforts of individuals such as those featured in this article continue to make a difference.