QUESTION:

I have questions about calibration that I hope you can help me with.  Do you know of a  practice in calibrations that use a reference point? One person is deemed the “correct” evaluator either prior to the calibration session or post calibration session.  Then all other evaluators are measured against that reference person or point?

ANSWER:

We believe that calibration is the process for standardizing and improving evaluation consistency — it’s not about getting it “right.”  But we wanted to see if others felt the same way, so asked some of our other experts for their opinions.  Here are some sample answers:

• We don’t assume one “correct” evaluator, but rather compare against how the majority scores. Those scoring differently from the majority are given the chance to “defend” their scores. Sometimes the majority is convinced that they scored incorrectly and the minority “wins,” but most of the time, the majority “wins.”
• In our center, calibrations are meant to be collaborative sessions where group consensus is reached, not “match the expert” sessions.
• Most that I have seen compare “scoring” among the group and any outlying scores are discussed for potential validity.  You may be able to use a “common” point or metric for scoring as a foundation, but I probably would not consider one individual as that point — rather an excepted best practice or standard that is measurable. 
• I would think if you used one individual as the common reference point, all would have to agree that that individual scored consistently on all points over time and also folks would have the opportunity to challenge scoring based on accepted practices.  There needs to be an avenue for continuous improvement even in the QA process.