Fun With Statistics - Wine Ratings Edition

By Kevin Meyer

I consider myself lucky in many respects, not the least of which is the fact that I live in one of California's premier wine-producing regions with over 220 wineries within 30 miles.  Over the last several years I've been able to partake of many of their products and have refined my taste down to a very small subset.  Those of you that follow us on Twitter occasionally get to hear of my new favorites.

One tool that many vinophiles use is the 100 point wine scale, especially the one developed by Robert Parker.  The scale has an illustrious history.

It was in this climate that in the 1970s a lawyer-turned-wine-critic named Robert M. Parker Jr. decided to aid consumers by assigning wines a grade on a 100-point scale. Today, critics like Mr. Parker exert enormous influence. The medals won at the 29 major U.S. wine competitions medals are considered so influential that wineries spend well over $1 million each year in entry fees. According to a 2001 study of Bordeaux wines, a one-point bump in Robert Parker's wine ratings averages equates to a 7% increase in price, and the price difference can be much greater at the high end.

Pretty impressive how much power that dude has.  But I'm sure many of our more statistically inclined readers are already starting to wonder how something as subjective as taste and bouquet can be accurately determined, repeatably, to single percentage points. 

But what if the successive judgments of the same wine, by the same wine expert, vary so widely that the ratings and medals on which wines base their reputations are merely a powerful illusion? That is the conclusion reached in two recent papers in the Journal of Wine Economics.

Houston (or Napa), we have a problem.  Who is this heathen that dared question Robert Parker and his ilk?

Both articles were authored by the same man, a unique blend of winemaker, scientist and statistician. The unlikely revolutionary is a soft-spoken fellow named Robert Hodgson, a retired professor who taught statistics at Humboldt State University. Since 1976, Mr. Hodgson has also been the proprietor of Fieldbrook Winery, a small operation that puts out about 10 wines each year, selling 1,500 cases.

He started off with a statistical analysis of judges and the judging process.

In his first study, each year, for four years, Mr. Hodgson served actual panels of California State Fair Wine Competition judges—some 70 judges each year—about 100 wines over a two-day period. He employed the same blind tasting process as the actual competition. In Mr. Hodgson's study, however, every wine was presented to each judge three different times, each time drawn from the same bottle.

The results astonished Mr. Hodgson. The judges' wine ratings typically varied by ±4 points on a standard ratings scale running from 80 to 100.  Mr. Hodgson also found that the judges whose ratings were most consistent in any given year landed in the middle of the pack in other years, suggesting that their consistent performance that year had simply been due to chance.

Uh oh.  So then what?  He took on the medal process.

This time, from a private newsletter called The California Grapevine, he obtained the complete records of wine competitions, listing not only which wines won medals, but which did not. Mr. Hodgson told me that when he started playing with the data he "noticed that the probability that a wine which won a gold medal in one competition would win nothing in others was high." The medals seemed to be spread around at random, with each wine having about a 9% chance of winning a gold medal in any given competition.

To test that idea, Mr. Hodgson restricted his attention to wines entering a certain number of competitions, say five. Then he made a bar graph of the number of wines winning 0, 1, 2, etc. gold medals in those competitions. The graph was nearly identical to the one you'd get if you simply made five flips of a coin weighted to land on heads with a probability of 9%. The distribution of medals, he wrote, "mirrors what might be expected should a gold medal be awarded by chance alone."

Now that a substantial part of the framework of our polite society has come crashing down, what should we do?

One answer would be to do more experimenting, and to be more price-sensitive, refusing to pay for medals and ratings points. Another tack is to continue to rely on the medals and ratings, adopting an approach often attributed to physicist Neils Bohr, who was said to have had a horseshoe hanging over his office door for good luck. When asked how a physicist could believe in such things, he said, "I am told it works even if you don't believe in it." Or you could just shrug and embrace the attitude of Julia Child, who, when asked what was her favorite wine, replied "gin."

As for me, I have always believed in the advice given by famed food critic Waverly Root, who recommended that one simply "Drink wine every day, at lunch and dinner, and the rest will take care of itself."

Bottom line: just try some wines.  Preferably a lot of different wines.  And figure out what you like and dislike.  Then buy more of it.  Who cares about points and medals and especially the supposed value correlation with high price?  Just enjoy.

Comments

[Mark Welch]

Kevin,

I was in CA wine country for the first time last month and was introduced to the 100 point scale. I was skeptical and this post confirms my suspicions. Thanks for debunking the math behind it.

One other thing to consider on the pricing... Maybe just simple supply and demand should rule as more of a factor than the bogus scale. I can imagine the politics going on behind the scenes between the wineries and the raters to jack up prices.

[Pete]

This is not news to anyone who reads Consumer Reports magazine. Typically they report the "spread" necessary for a meaningful difference. For those of us who have been involved in taste testing and panels of judges, well this wasn't news 40 years ago. This illustrates very well why statistical thinking needs to be imparted to students at least by the time high school is reached.