Meeting the regulatory principles for e-Assessment

Michael Kayser, VP and director of programme management at Pearson VUE, presents a brief crib-sheet on the regulatory principles for e-assessment, what they mean to awarding bodies, and how your testing partner can help ensure complianceE-assessment is continuously growing and developing all around us, and the 13 regulatory principles introduced in 2007 by OFQUAL marked a milestone in the education industry’s acceptance of the technology. The establishment of regulation surrounding the use of electronic systems in assessment demonstrates that the regulator aims to modernise the qualifications system, encourage and develop innovation and continuous improvement, and ensure that the integrity, reliability and validity of all forms of e-assessment are upheld.
The full April 2007 regulatory principles document is available to download from OFQUAL at, but here we give a brief crib-sheet on what they are, and a summary of what they mean. OFQUAL’s own summaries are outlined in each section, immediately followed with our own comments.

1. Validity and reliability of e-assessment
OFQUAL: Awarding bodies must ensure that assessment delivered and maintained by electronic means is fit for purpose and produces a valid and reliable measure of a candidate’s skills, knowledge, understanding and/or competence. The choice of assessment method must be independent of the technology on which it may be based.
Pearson VUE: In supporting clients with this principle, the testing partner should always propose the method, system or technology that is best suited to giving the most accurate measure possible. Technology should never be used simply for its own sake but rather to measure something more, or better, than before.

2. Security
OFQUAL: Awarding bodies must maintain and review the security of e-assessment systems to ensure authentic test outcomes and protection against corruptive influences. Procedures must be in place to assure the security of hardware and software and the integrity of test data.
PV: This must be a primary area of focus for any respectable test delivery company. Our own systems include E-encryption and item banks with features like pooling and randomisation, to ensure that, from the creation of every item, through test delivery to the final reporting of results and candidate data, there is minimal risk either of content leaking, candidates cheating, or test-takers’ data going astray.

3. Data integrity – input/output
OFQUAL: Awarding bodies must be confident that systems have been thoroughly tested to ensure that they have sufficient capacity to store, retrieve, generate and share all necessary data, including the ability to exchange data securely with other internal and external systems, as required, without endangering the integrity of the data.
PV: Many awarding bodies do not have the capacity themselves to hold and traffic data securely. This can be part of the service of the e-assessment provider, who may be able to take responsibility for this aspect on your behalf.

4. Operation of e-assessment systems
OFQUAL: E-assessment systems must be stable and work reliably to generate valid and reliable assessments and/or results. They must be demonstrably consistent with relevant recognised standards of good practice and be easy to navigate.
PV: This refers to the need to meet with any other regulations and to be able to demonstrate compliance, for example being able to generate key information such as results or system error reports on demand. Your testing partner should be able to show regulators at any time that the systems you are using are sufficiently robust to support high-stakes assessment.

5. Integrity of e-assessment systems
OFQUAL: Systems must allow for flexibility in the light of technological development. System testing must be thorough, and be reviewed at regular intervals once the system is operational. Awarding bodies must ensure that suitable support facilities are in place for centres and that there is a comprehensive contingency plan should any part of the system fail.
PV: This regulation concerns technical support – a formal arrangement must be reached between the awarding body and the e-assessment partner to ensure that the software, hardware and infrastructure have the capacity for the functionality and volume of use with which they are expected to cope.

6. Access to e-assessment
OFQUAL: Awarding bodies must have policies and procedures in place to ensure that disabled learners are not treated less favourably than non-disabled learners when implementing e-assessment. This must include disabilities as defined by the Disability Discrimination Act (DDA) 1995 and subsequent regulations and guidelines.
PV: Here we see an example of how these regulations must be met in conjunction with other existing legislation and best practice wherever relevant. At every stage of the development of an e-assessment programme, it is the duty of all those involved in developing tests to ensure compliance with the Disability Discrimination Act, not only in terms of physical access, but also in such considerations as font size, voice-activated software or any other necessary adjustments so as not to exclude anybody on the basis of disability.

7. Avoidance of barriers to new technology for learners
OFQUAL: Awarding bodies must ensure that the use of technology does not create barriers for learners by providing user-friendly interfaces for centres and learners and by enabling familiarisation and/or training sessions appropriate to the mode of delivery. Provision must be made available for learners with particular assessment requirements.
PV: It is important to make the point that no test-taker should be at a disadvantage if they are less computer-literate or confident with technology. A test should test solely what it is intended to test, and not in any way be a test of the candidate’s skill or familiarity with the testing medium. That medium should therefore be geared up to be as easy to use as possible.

8. Business continuity/disaster recovery
OFQUAL: Awarding bodies must have suitable measures in place to ensure the effective management of business continuity to address business interruption and the need for disaster recovery for their e-assessment services and systems, in the event of a system’s failure. This management should be underpinned by measures to identify potential risks to those services and systems so that they can be managed to minimise disruption.
PV: This is addressed in detail at the Service Level Agreement stage. Between the awarding body and the testing partner, adaptability must be built-in to cope with all worst-case scenarios and risks; and responsibility for this is shared according to that SLA.

9. Automatically generated on-demand tests
OFQUAL: Awarding bodies must ensure that there is a sufficient volume of assessment items or questions to provide consistently secure, robust, balanced and unique on-demand tests, appropriate to the form of assessment.
PV: Electronic item banking means that on any given day, every candidate could be sitting a different test, containing a unique mix of items that can be presented in a random order, and which add up to a test of the same difficulty level as the one being sat by the next candidate. With this model, you could make your test available every single day of the year, without needing to create all-new items for every test. By thorough testing of item banks and of the randomised item selection technology (where applicable), the testing provider must be able to demonstrate that every test is as robust, reliable and valid as the next.

10. Test conditions and environment
OFQUAL: Awarding bodies must have policies and procedures in place to ensure that centres manage the controls on test conditions in relation to on-demand testing, invigilation, secure test environments and health and safety.
PV: This regulation protects both the test-taker in terms of the security of the personal data that they share with the test centre, and the security of the assessment itself. It also relates to current health and safety legislation, ensuring that test centres meet their obligations in this area.

11. System familiarisation for assessors and system administrators

OFQUAL: Awarding bodies must provide suitable support for system users, such as familiarisation sessions and guidance for assessors and moderators.
PV: E-assessment often involves cutting-edge technology, which everybody involved in the administration of the tests needs to understand as much as necessary. While the test delivery company must ensure the required level of knowledge and understanding of the systems and software, the awarding body must take equal responsibility to give clear guidance on judgments and decision making for assessors dealing with different media of work.

12. Adaptive testing

OFQUAL: In addition to regulatory principles 1-11, awarding bodies must ensure that any adaptive testing that they provide produces robust assessment that reliably identifies the appropriate level of each learner and is comparable across different modes of delivery where this is required.
PV: Adaptive testing is an advanced psychometric method for accurately targeting an individual’s knowledge level. The test algorithm selects an item to start and if the candidate gets that one right then a harder one is selected, or if wrong an easier one. This continues until the candidate is consistently getting one right and one wrong, like an oscillating wave of limited height and depth. This is currently used to great success in nursing exams, but is still considered very much cutting-edge rather than industry standard. Where this method is used, the regulation requires that the system be adequately tested to ensure consistency and accuracy.

13. Use of e-portfolios for assessment

OFQUAL: In addition to regulatory principles 1–11, e-portfolio systems should store and maintain performance evidence for access by all required parties securely, meet the evidence needs for a range of qualification types and enable learners to move their portfolios from one centre to another.
PV: This regulation cements the value of the achievement of each test-taker. No test-taker should be bound to one testing system or organisation by any built-in incompatibility. It is up to all of us in the industry to make sure that the test-taker’s e-portfolio can store and maintain a variety of forms of performance evidence or coursework, which is secure but accessible to any authorised party, and can be freely migrated at the test-taker’s choice.
Pearson VUE is a global leader in computer-based testing for information technology, academic, government and professional testing programmes around the world. Pearson VUE provides a full suite of services from test development to data management, and delivers exams through the world’s most comprehensive and secure network of test centres in 165 countries. Pearson VUE is a business of Pearson (NYSE: PSO; LSE: PSON), the international media company, whose businesses include the Financial Times Group, Pearson Education and the Penguin Group.

For more information