The Department of Health gave your favorite restaurant an 85. What does that mean?
Not necessarily. Depending on a region’s rating system, an 85 can be quite low compared to other restaurants, signaling multiple high-risk security breaches, which rarely happens. And if potential customers don’t understand that, then reviews aren’t getting the public health message they want.
Daniel Martin, associate professor of managerial economics and decision science at Kellogg, and a colleague recently investigated the gap between what regulators want to convey with these scores and what consumers actually perceive.
The team focused on San Francisco restaurant hygiene ratings in 2018 and found that people interpreted low ratings (the equivalent of a C) as much less risky – and much more common – than ‘they actually were. And on the other end of the spectrum, they thought the restaurants with the highest scores (the equivalent of an A +) were riskier than they actually were. They also found that consumers believed low ratings happened more often than they actually did.
“These misperceptions made having a C less bad and an A less good,” said Martin. The trends persisted even when the team provided participants with more information about the scoring system, such as descriptive labels and brief explanations of what the different score ranges mean.
The confusing messages provided by ratings could have repercussions for business and public health. If consumers don’t appreciate a high rating enough to compensate a restaurant for the extra health and cleanliness measures, the restaurant has little incentive to continue. And low-rated restaurants may not have as much of an incentive to improve.
Regulators “should care about what consumers internalize,” Martin says.
Regulators may need to change their rating systems or make rating boards more informative about the health risk in order to align consumers’ interpretations with reality. Indeed, the San Francisco Department of Public Health tried just that last year by adjusting the way notes are displayed; descriptive labels are now much larger and signs do not display any numeric score.
Ratings are everywhere: we grade everything from hair salons to ride-sharing drivers to vacation rentals. And regulators issue ratings for organizations or products such as hospitals, nursing homes, and healthcare plans.
Martin, along with Tami Kim of the University of Virginia’s Marketing Department, focused on regulators because the purpose of their ratings is very clear: Scores empower consumers, who do not have the experience or access to judge for themselves, the ability to make more informed decisions.
The researchers questioned whether regulators were achieving their goal of communicating information to consumers. “The regulator says, ‘This nursing home is a 3. This health plan is a 4. This restaurant is a B.’ They’re trying to say something,” Martin says. “But what do consumers get from this message? “
To find out, Martin and Kim obtained data on 1,759 restaurants in San Francisco that had been assessed from January to April 2018. Inspectors had visited restaurants at unscheduled times and assigned a score based on the types of risk to the food. public health that they had observed. For example, a low risk offense might include washing fruits and vegetables incorrectly or having a dirty toilet. Moderate violations could include difficult access to hand washing facilities or reserving food that had already been given to other customers. And problems such as contaminated food or certain pest infestations would be considered high risk.
In the system at the time, scores ranged from 0 to 100 and were grouped into categories. A score above 90 (“Good”) meant that inspectors generally found only low risk violations; 86-90 (“adequate”) meant several violations; 71-85 (“need for improvement”) meant several high-risk violations; and under 71 (“Bad”) was the worst category.
Dashboards displayed in restaurants showed the numeric rating in a large, color-coded box: green for Good or Adequate, orange for Needs Improvement, and red for Poor. Smaller text below the number lists the four possible categories and the corresponding score ranges.
However, the signs did not explain what each category meant in terms of the number or types of offenses. They also did not specify the score of this restaurant compared to others.
So the researchers conducted an online study and asked participants how they interpreted the different ratings, initially focusing only on number scores.
About 380 participants were asked to estimate how many restaurants out of a random sample of 100 would be ranked in different score ranges: 96-100, 91-95, 86-90, and so on. For some of the score ranges, they also estimated the likelihood that a restaurant in that range would have a high risk violation.
The team found that, without additional context, people tended to believe that low scores were more common than they actually were. For example, participants in this benchmark estimated that 11 percent of restaurants had a score of 0 to 70, when in fact less than 1 percent did. And they thought that only 17% of restaurants got a score of 96-100, while the actual figure was 37%.
People may have assumed that the scores are distributed fairly evenly, speculates Martin. Based perhaps on our experiences at school, we would expect “the regulator or the corrector to want to give a range of grades,” he says. After all, it’s the differences in the ratings that make restaurant information better than the rest.
Likewise, these participants misinterpreted the health risks. They estimated, for example, that a restaurant with a rating of 71 to 75 had a 33% chance of having a high-risk violation, when it was actually 91%. And they believed that a score of 96-100 meant a 13% chance of a high-risk violation, when the actual figure was zero, suggesting that the misinterpretation went both ways, which restaurants the better. rated not getting the credit they deserved.
Is more information useful?
Next, Martin and Kim wanted to know how consumers’ perceptions changed when given more context to help them interpret the ratings. So they conducted an experiment with two other groups of online participants.
The two groups were informed of the approximate distribution of ratings given to the restaurants. But one of the groups received additional information on the meaning of each track. For example, they read that a score of 71 to 85 was labeled “in need of improvement” and meant that inspectors had found multiple violations, usually several of which were high risk.
Receiving detailed descriptions of each category pushed participants in the right direction for most score ranges, but people still largely underestimated the risk of high-risk violations at low-rated restaurants. They also overestimated the odds at the top rated restaurants.
“The interventions only slightly improve the effects,” says Martin.
So what else could regulators do?
One possibility would be to provide more details on the number and types of violations found in each restaurant on its rating panel. Or regulators could distribute the ratings more evenly so that they better match people’s expectations of what the numbers mean. For example, restaurants with multiple high-risk infractions might be given scores in the 60s, while only restaurants with absolutely impeccable hygiene would score above 90.
But regulators can face political problems. Giving a rating of 65 (essentially a D) to a family restaurant could seriously hurt the business. Regulators may be under pressure to avoid chasing too many customers, so that the restaurant has a chance to improve and be successful.
San Francisco’s new odds display, launched last year, could be a step in the right direction. Instead of displaying a numerical score, restaurants are now labeled in large print like Pass, Conditional Pass, or Closed and with a lighter color code like green, yellow, or red, respectively. However, the impact of the new system still depends on the accuracy of consumers’ inferences when they see Pass or Conditional Pass. And by losing all nuance between restaurants in the same category, the new system risks punishing the most hygienic restaurants even more than the old rating system.
Restaurant owners themselves could also provide more information to customers if they are concerned that the scores will be misinterpreted. For example, a well-rated restaurant might post a note explaining that this means it hasn’t committed any high-risk infractions.
But customers may not trust this information, says Martin. “Part of the reason we have certifiers is that they have more credibility than companies,” he says. “This is a difficult thing for companies to resolve because they may be perceived as less benevolent. “
The study also raises the question of whether consumers misunderstand the significance of ratings given to hospitals, nursing homes and health care plans. In these cases, making the right decision might be more critical than choosing where to dine.
Yet with restaurant reviews, “there’s a strong enough public health reason to get this information out,” Martin says. “While it’s not as important as retirement homes, it is important.”