The management of any assessment programs continues to be a complex, challenging task for any organization. Regardless of whether the program operates in the education, professional credentialing or IT credentialing sector, competing demands place a significant amount of stress on your program. For the program sponsor, there is a consistent need to respond to new demands and find solutions that preserve the value of your testing program while maintaining its credibility.
Throughout the test development and maintenance process, it is easy to lose track of the big picture, and not recognize small shifts in your program that seem innocent at the time, but gradually build into significant changes. Many assessment programs realize the value in bringing in independent reviewers or auditors who can be used as a quality control check for the program and can help ensure that the policies and procedures that have been established continue to be followed, and that new procedures are consistently implemented.
The term audit has a tendency to send chills down all of our spines, and the image of a painstakingly meticulous IRS agent exists somewhere in the subconscious of all of us. However, the goal of an external evaluation or review does not have to stoke fear within your organization. Rather, it should be presented as an opportunity to take a step back and try to look at the big picture. An independent review, provides everyone in your organization the opportunity to demonstrate the quality of the work that is being done, while also shining a light on areas that need improvement or areas where resources are being stretched too thin.
Alpine has conducted independent evaluations and audits that focus on the entire assessment lifecycle, from inception to score reporting, but have also focused audits that focus solely on only one or two critical areas, such as the scoring of an assessment, or the item and test development processes. Regardless of the phases in the assessment lifecycle that we are reviewing, our experience in conducting audits have led us to focus on five key areas.
- Alignment of program evidence with intended interpretation and use.
- Documentation of participants, processes, procedures, results, and decision.
- Internal and external communication
- Monitoring of any external contractors
- Prevention, detection, and enforcement aspects of test security
Unfortunately, we all suffer from too many tasks to complete, with too many competing demands on our time. In this type of work environment, one of the easiest things to let slip is the documentation for the program. No one enjoys sitting down and writing out the details on the procedures followed to complete a given task. The natural tendency is that once a task is finished, it’s time to move on to the next one. But this lack of documentation can have serious implication in the long run, and can lead to inconsistencies over time, and add to the burden of completing certain tasks the second or third time.
One of the consistent themes that exist with almost all of the organizations we have worked with is that the communication among different teams and units can be significantly improved. It is one of those issues that vex us all; and despite our best efforts and intentions, is an area that requires constant vigilance. Using an independent review to help identify any gaps that may have developed, or systems that could be used to enhance internal communications continues to be one of the most valuable outputs of the process.
In a similar vein, for many organizations that rely on external contractors, there is a tendency to become overly dependent upon these contractors and to place too much faith in their procedures. Contractors should provide complete documentation of the procedures they have completed, but not every contractor is as committed to this principle as they should be. All too often, the relationship develops in such a way that the contractor completes some work, while the details on how they have completed the work remain something of a mystery. This can have serious consequences for your program, and substantively impact your ability to move forward with a new vendor. It’s okay to trust, but verify.
There is obviously no need to try to convince anyone reading this of the importance of security for the credibility of your assessment program. In fact, it remains so critical that we will be dedicating some upcoming posts to this topic. For now, it goes without saying that a periodic review of the security protocols for prevention, detection, and enforcement for your testing program remains something that all programs should consider.