Three health inspectors walk into a bar.
You might expect that they order three different drinks and hilarious antics ensue, but what really happens is that three slightly different inspection reports are filed.
It’s no joke – in fields that rely on decentralized decisions and discretionary authority, consistency from one professional to another is a challenge. Becky Elias from Public Health’s Food Protection Program and Stanford Law School’s Daniel Ho recently penned an article in the Boston Review about what it might take to make public servants more consistent. (Spoiler alert: it’s peer review.)
In case you don’t have time to read the whole, very interesting piece, here are some highlights – and a little commentary.
To understand inspection reporting consistency better, we asked staff to conduct inspections side-by-side and observed how they documented food safety practices, including violations.
In our experiment, health inspectors observing identical conditions disagreed 60 percent of the time.
How can this be? After all, our health inspectors are highly trained in food safety and have expert certification to boot.
To understand why, first you must understand how health inspection works. The goal of restaurant health inspections is two-fold: One is to make sure that a restaurant meets the minimum standards to remain open. The other is to enable restaurant to improve their overall food safety. So when any one of our health inspectors shows up at a restaurant and sees something that warrants closure – a major health risk – they close the restaurant until the issue is fixed (this happens during only 1% or so of inspections). The majority of our work is documenting what we observe and providing ongoing food safety education to help restaurant operators improve.
Health inspectors use many tools to address food safety with restaurants, and they always use a standard inspection form to document their observations. That’s where the 60 percent discrepancy occurs. What we learned in the peer review is that staff members were each addressing food safety, and while they agreed generally on risk conditions in establishments, the documentation of potential violations could vary in many subtle ways (60 percent of the time).
Bottom line – all health inspectors are addressing food safety. They make corrections as they see them and work with operators to improve practices.
The inconsistency also isn’t unique to food safety. The Boston Review article goes into detail about how this inconsistency is evident in a range of fields where professionals exercise judgment in decision making, such as courtroom judging, patent examinations, policing, and teaching, to name a few.
One possible solution is peer review. If frontline government officials could review and deliberate over each other’s work, the quality and consistency of decision making might improve. While isolated examples of such peer review exist, we have regrettably little systematic evidence of peer review’s effectiveness in the public sector.
To improve consistency in our food safety program, our inspectors participated in a randomized, controlled trial to test the peer review process. More specifically, inspectors would conduct inspections side-by-side in a restaurant, each filling out their own inspection form. The two inspectors would then discuss the outcomes of their inspection reports and understand how they arrived at their decisions. Through this we were able to track how staff cited violations when observing the same restaurant environment.
The program then used the results from the peer inspections to identify opportunities for training where it mattered most. This rigorous study design provided a fantastic opportunity to gain confidence (statistical and otherwise) in our plan to mitigate variability.
And, surprise, peer review worked!
The peer review process caused an average increase in violation detections of 17–19 percent. Because the increase was driven by inspectors who had previously been loath to cite violations, the net effect was to reduce the variability across inspectors in the peer review group.
So the point is not that the violation detections went up, but rather, that inspectors were more consistent in marking them on the form.
The most unexpected change was cultural. At the outset, a major concern within the inspection department was that peer reviewing would lead to tension within the staff… But we found that the opposite happened. Rather than challenge one another, inspectors began to observe and learn from their partnerships.
The peer review study provided our inspectors with added recognition that their colleagues are great resources of knowledge and experience. Sharing this made it more possible for inspectors to ask each other questions and share ideas, improving everyone’s ability to perform at the highest level.
What’s next?
As we work to make information about restaurant inspections more available to the public, we want to be sure we are providing an accurate portrayal of food safety performance.
Now that we’ve made such remarkable headway into improving consistency, we are one step closer to developing a restaurant “grading” system. With the understanding that inconsistency can never be completely eliminated, our system will build in fixes so that the grading is fair and equitable, no matter the slight variations of how the inspector marks a form.
This peer review work is instrumental for making our food safety program and Public Health – Seattle & King County the best in the country. And the data-driven intervention is another example of how King County is working to become the nation’s best run government. We hope other programs and departments will see peer review as an opportunity to make our work more equitable and transparent, and we are excited to see the results.