In March 2020, approximately 20 subject-matter expert (SME) volunteers had booked airline travel to Washington, D.C. They planned to meet face-to-face on March 13 and 14 for a workshop to help develop items for the Architect Registration Examination® (ARE®) for the National Council of Architectural Registration Boards (NCARB). However, on March 11, Covid-19 was declared a national pandemic. With immediate travel bans in effect, NCARB staff and consultants had just 3 days to logistically shift the workshop from in-person to virtual.
When asked about the potential consequences of delaying exam development activities until the travel bans were no longer in effect, Jared Zurn (Vice President, Examination) said, “Holding off until travel bans were lifted was never a consideration. We knew it was vital and appropriate for us to keep pace with development of the next iteration of the exam for the sake of our program, stakeholders, and members. The world continues moving, and progressing and hitting ‘pause’ is never a good idea in the field of architecture—or in other disciplines.”
Sheronne Wilson (Manager, Examination) added, “Our SME volunteers were excited to gather in D.C. at the NCARB office to collaborate on exam development. Although there was disappointment on both sides, we all knew the shift was required in order to progress with exam development and maintain the high standards of our program. The upside is that NCARB had some experience holding occasional virtual workshops and so we were confident we could make it work.”
Fortunately, NCARB was able to pivot the March workshop quickly and successfully from an in-person to virtual setting in spite of only having three days’ notice. It took significant effort in communicating the change to the SME volunteers and the timeline needed to be adjusted. Ultimately, with technology (particularly web meeting tools) and cooperative SMEs, NCARB was able to accomplish the meeting objectives.
NCARB’s Success
NCARB, a highly successful nonprofit, has stakeholders and members that include the legally constituted architectural registration boards of the 50 states, the District of Columbia, Guam, the Northern Mariana Islands, Puerto Rico, and the U.S. Virgin Islands. Its mission is to collaborate with licensing boards to facilitate the licensure and credentialing of architects to protect the health, safety, and welfare of the public. NCARB develops, administers, and maintains the Architectural Experience Program® (AXP®) and the universal ARE, which includes 6 exam divisions. Those in pursuit of becoming a licensed architect must complete the ARE, along with education and experience requirements. Upon achieving licensure, architects have the option of being awarded an NCARB Certificate, which is instrumental in helping them get licensed in multiple states. Access to ongoing continuing education is also provided as a benefit of the NCARB Certificate.
In 2015, NCARB adopted the tagline of “LET’S GO FURTHER” to exemplify their commitment to continuous improvement, which contributes to their success. Zurn explains: “It reflects our philosophy and the ongoing question we continually pose to one another, ‘How can we make things better for our program and our members?’ We don’t just talk about continuous improvement; we live it. In keeping with our domain of architecture, we are always ‘architecting the exam’ to make it better. We also expect those we work with to stay at pace with our desire to improve. We continually collaborate with our members, SME volunteers, Alpine Testing Solutions (Alpine), and our other trusted partners as we strategically look for ways to improve and to try new things.”
“NCARB is fortunate to have a large volunteer SME pool. I think SMEs consider it a privilege to provide their expertise and give back to the profession and we trust and appreciate them. They like knowing items they write have the potential to be on the exam. Their dedication and commitment contribute significantly to the success of our program and that is a huge win for all involved,” said Wilson.
“The SMEs also thrive in NCARB’s dynamic and collaborative culture,” explained Zurn. “If you want to be successful, you have to understand the culture of your staff and your volunteers and how they interplay. We have a volunteer culture of wanting to be together, and not just for writing or reviewing items. When we conduct in-person workshops, it is common to take SMEs to dinner and we talk about the kids or dogs and so on, and then during the course of the workshop, we have more networking activities that allow opportunities to chat and collaborate. These activities help build a strong sense of community and comradery.”
“When recruiting volunteers, NCARB identifies expectations of what it means to be a volunteer for different aspects of the program, not just the exam,” explained Zurn. “We break work down into tasks, with the expected level of effort so SMEs can indicate which activities align with their skills and availability. We also collect demographic data, including race, age, gender, ethnicity, background, education, experience, location, geographic region, size of architecture firm, and types of projects worked on. We streamline our selection and data collection processes using Lineup®, a SME management tool NCARB developed that is commercially available via subscription to all interested parties. Lineup® allows us to take a holistic view of the SME data and curate work groups that are balanced and representative of all firms and projects.”
Lessons Learned
“In 2015, we made an attempt to switch from in-person to virtual workshops. It failed. In retrospect, it was due to our mindset and less advanced technology,” said Zurn. “We viewed it as a short-term experiment, and were not fully committed to its success. Over time, our mindset has changed and we’ve discovered it makes sense to conduct some, but not all, work virtually. About 3 years ago, we realized that with appropriate training, mentoring, and guidance we could move item writing to a virtual setting, but wanted to maintain the face-to-face item review.”
The pandemic has been the catalyst that has forced all NCARB work to be virtual in the past year. The team has successfully conducted approximately 40 virtual exam activities, including analysis of practice, item writing, item review, quality checks, forms assemblies, case studies, and cut-score equating. But that success has come with hard work and trial-and-error. Zurn pointed out that you have to think through what is going to work for online vs. in-person work scenarios.
Wilson adds, “In-person workshop settings are conducive and ideal for pairing up a couple of SMEs to collaborate on a complex item writing assignment, but in the virtual environment, that doesn’t work as well. Additionally, facilitators who conduct virtual workshops need an added skill set to recognize when certain SMEs are not engaging in discussions and how best to pull them in.”
Other challenges imposed by going strictly virtual have been a decrease in productivity and an increase in SME attrition. Some SMEs have stepped down because virtual work has spanned a longer timetable than in-person work. Some SMEs are in a temporary home office doing their day job with possible commotion of kids, pets, and other working family members in the background (with some possibly battling the virus) while trying to fit in time to complete virtual assignments. Rather than side step these tough issues, NCARB and their partners have been motivated to look for creative solutions and to be more flexible, open-minded, and sensitive to SME needs.
“Approximately 40% of our current SMEs are new,” said Wilson. “Returning veteran item writers know the process well. But for new SMEs who are learning about item development, with the amount of material and training at hand, it’s been understandably difficult for them to grasp everything in a virtual environment. Given that everyone is trying to balance work/life during the pandemic, we’ve opted to slow walk it a bit and have reframed this as a rebuilding year. Early on in the process, we provided new SMEs additional guidance and training and are pleased to see them responding and making more progress.”
One of the benefits of virtual vs. onsite environment is the flexibility NCARB and their partners can provide to meet various schedules. They are not an organization that functions only 8 a.m. to 5 p.m. on weekdays. They are willing and able to cater to their volunteers and meet whenever SMEs are available, whether it be on the weekend, or in the early morning or evening hours.
“A creative and very effective intervention NCARB developed and incorporated in conjunction with Alpine’s facilitators are community building activities and ice breakers that provide SMEs with breaks to avoid getting ‘Zoom fatigue,’ said Wilson. “Some are designed to increase the comfort level of SMEs with one another and others are designed to help with item writing. Post-workshop SME evaluations have been positive with approximately 64% indicating that these activities have helped increase productivity.”
Strategy for the Future and Maintaining Culture
While NCARB is used to bringing in approximately 50 SMEs four or more times a year for in-person workshops, the pandemic experience has helped them realize they can do things differently.
“When travel bans are lifted, we envision our ‘new normal’ will consist of a balanced hybrid approach where some workshops and activities will be virtual but others, such as reviews and various other exam activities best suited for face-to-face collaboration, will be in-person,” said Zurn.
“Our veteran SMEs are accustomed to traveling to workshops and have shared with us that they miss opportunities to travel and collaborate in person with fellow SMEs and NCARB staff,” mentioned Wilson.
“And that’s back to the culture,” said Zurn. “SMEs want the opportunity to travel to workshops again. We will not take that away from them because that would undermine the culture. We realize the number of in-person workshops will decrease over time. We plan to repurpose some of the funds we would have spent on travel and find ways to reward our SME volunteers and help them understand how much we appreciate them, which will help us continue building a strong sense of community.”
Based on NCARB’s track record and “LET’S GO FURTHER” philosophy, there is little doubt they will continue to be successful in their quest for continuous improvement as they face the challenges of the day.