The Reliability Programme

Last updated on: 09/10/2009

There’s a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on. We need to explore whether that sort of expectation is well founded, or whether within the system there are some trade-offs between absolute reliability of that nature and the validity of the way in which we go about assessment.

Ofqual Chair, Kathleen Tattersall

How reliable are the grades that learners get?

What is Ofqual's Reliability Programme?

The Reliability Programme is a two-year research study looking at the consistency of assessments, and factors which may affect the reliability of results. This could include anything which may affect the result a learner receives, including the place of assessment, time of assessment, marker of the assessment and more. For more information look at the archive section.

Ofqual's second reliability seminar takes place (27th January 2010)

Assessment experts gathered for the second time to discuss Ofqual's Reliability Programme on January 27.

Experts discussed the findings from the programme so far and the continuing direction of work. Jenny Bradshaw, from research body NFER, gave a presentation on the international perspective of reporting results, while Mike Cresswell, from awarding organisation AQA, talked about a policy on reliability from an awarding organisation's point of view.

Ofqual's two-year Reliability Programme, which started in 2008, aims to discover more about the reliability of results and to create Ofqual policy on reliability.

Delegates also discussed public confidence and reliability, operational issues for awarding organisations in relation to reliability and the reliability of vocational qualifications.

A report on the discussions held at the seminar will be written and published on the Ofqual website shortly.

For more information contact Annette.Kinsella@ofqual.gov.uk

Assessment experts discuss reliability of results at seminar (7th October 2009)

reliability seminar Senior examiners and some of the country’s leading experts on assessment gathered to discuss Ofqual's reliability programme at a technical seminar held at the University of Warwick on Wednesday 7th October.

Representatives from awarding bodies and research teams, including Cambridge Assessment, NFER and Pearson, gathered to discuss issues surrounding reliability and possible future aims of the programme. Dr Paul Newton, from Cambridge Assessment, Malcolm Hayes, from Pearson, Ben Styles and Sarah Maughan, from NFER, and Sandra Johnson from Assessment Europe, were among the speakers who gave presentations on issues on reliability. The projects which were discussed included:

Comparing reliability across different assessments
Investigating how different sources of variability, such as the time people take tests or the examiner who marks their work, may affect reliability
How results are reported internationally.

The discussions of the day will be written up in a report which will be published shortly. The individual project reports are expected to be completed by December. A second round of projects are currently being commissioned. Subjects include:

Teacher assessment reliability
Reliability of vocational and occupational qualifications
Estimates of reliability of GCE A level and GCSE subjects
Literature review of awarding body research into reliability.

For more information contact Annette.Kinsella@ofqual.gov.uk

Reliability report presentation at the IAEA conference by Ofqual's Head of Standards Dennis Opposs (15 September 2009)

Ofqual Head of Standards Dennis Opposs will be presenting a report on communicating reliability at the International Association for Educational Assessment (IAEA) conference on Thursday (September 17).

The work, entitled 'No news is good news?', looks at issues around communicating the concept of reliability in the examinations, tests and assessments system.

The report explores the issues in communicating the reliability of assessments in public, including:

Potential and realised difficulties assessment professionals have in using existing channels of communication, such as the media
The benefits of communicating in public about reliability, such as increased public understanding of results
Future plans for the reliability programme.

Read the 'No news is good news' report.

Ofqual has also published a supplementary report, written with messaging specialists Blue Rubicon, on how best to communicate about reliability.

Read the 'Talking about reliability' report.

New reliability animation released by Ofqual (15 September 2009)

Ofqual have created a new video to explain the concepts of the Reliability Programme.

The animation covers the ideas behind the programme, which aims to discover more about the reliability of results of exams, tests and assessments.

The animation uses the analogy of a boiling kettle to illustrate the concept of reliability, explaining that the boiling temperature of kettle is widely known as 100 degrees centigrade, but in fact this is only true if the kettle is boiled at sea level. If boiled on top of Mount Everest, external pressure would mean it boiled at just 71 degrees centigrade. This, states the video, is an example of outside factors affecting a final result – just as external factors, such as the time and place of assessment, can affect the results of an exam.

Ofqual head of standards Dennis Opposs said: "We're delighted to have created this animation, which we believe will explain the ideas behind the Reliability Programme clearly for learners, parents, employers and everyone who is interested in our work."

Watch the animation on YouTube.

For more information email reliabilityprogramme@ofqual.gov.uk

Learners advise: "Five exams in seven days" can affect reliability of results

Ofqual met with members of the UK Youth Parliament at their annual sitting in Canterbury in July to ask their opinions on the reliability of exam results. This work was part of Ofqual's Reliability Programme, which aims to discover more about the reliability of the results of exams, tests and assessments.

UKYP members told Ofqual about their perception of the reliability of results, including:

Factors which could affect reliability: learners' concerns largely centred on exams that were timetabled closely together, which they felt may affect their performance.
How reliability could be improved: learners spoke about the need for double or triple marking to eliminate occurances of marker error.
How exam results make them feel: learners spoke of the pressure they were under, including feeling of failure when they did not get high marks.

A report by the UKYP and video will be published shortly.

Reliability: what learners think

Kishan, 16, UK Youth Parliament member for Brent, gave Ofqual a perspective of reliability from a learner's viewpoint. He said:

Kishan with fellow members of UKYP

"So far in my life I have taken SATS, music exams and more recently GCSEs. I personally didn't feel the reliability of results to be an issue – I felt well prepared for my exams and I thought my grades reflected this. Having said that, I do think that other factors can affect the results you get. For example, people suffering from nerves is an issue in all exams, as is pressure, as sixth form colleges always require good grades for places. It's hard to say how far the actions or mood of the marker may affect a result, as I haven't seen my scripts, but the timing of exams affected me in the first week of June, when I had seven exams in five days.

"I think that maths and sciences have the most reliable results as the answers are yes or no. In other subjects like history, where research is required, more weight should be given to coursework to improve the reliability of results."

In January, Ofqual carried out more workshops with marker researchers Ipsos MORI to discover the perception of the reliability of results by different groups, including learners, parents, employers, teachers and examiners.