Comments on: WSJ: wine-rating system is badly flawed

By: vinoth

vinoth — Sat, 28 May 2011 12:55:00 +0000

@epicuria we can also use this standard deviation calculator http://ncalculators.com/statistics/mean-standard-deviation-calculator.htm

By: Dad

Dad — Sun, 28 Feb 2010 00:14:07 +0000

Dad take a look at this.

By: The New Way To Rate Wines: A Scoring System For Everyone « secret undersea city

Fri, 11 Dec 2009 00:22:46 +0000

[...] recently, DrVino.com acknowledges that the wine system is badly flawed with a link to the Wall Street Journal article, “A Hint of Hype, A Taste of Illusion” by [...]

By: “Hocus-pocus” – The Taste Makers in the NYer | Dr Vino's wine blog

Tue, 01 Dec 2009 16:35:39 +0000

[...] “WSJ: wine-rating system is badly flawed“ Permalink | Comments (1) | SHARETHIS.addEntry({ title: "“Hocus-pocus” – [...]

By: Kimberly

Kimberly — Sun, 29 Nov 2009 00:53:55 +0000

All of this talk is so “sciency,” and much of it above my head, but I’ve been enriched by the discussion. I’m not a fan of rating systems, mostly because so much good wine gets overlooked by consumers if it doesn’t have a point score attached, which is really unfortunate. I work in a gourmet food and wine retail shop, where, luckily, those of us who love and try many wines will try to steer customers to great, but unrated, values. But then there are those customers who only shop for wine by points, missing out on a whole world of great wine. Oh well, what’dya gonna do?

By: Chris Robinson

Chris Robinson — Sat, 21 Nov 2009 15:39:39 +0000

What is amazing is that under the Australian wine judging system based on 20 point scores I have been amazed at the consistency of scores across judges, me being one of them for many years. These are wine judges who sit around after each round of wines, often 100 plus wines, and give their scores in a system which does not allow for much fudging. It is often common to see no more than a point difference between the main judges and their 20 point ratings. So what is going on here? To my mind holding constant the day, the tasting conditions and to some extent the expertise of the tasters will give you consistent scores. The variability in these articles you have been mentioning is surely the situation, the environment, the glassware, etc and not the wine. Science is all about contraining some, indeed most variables and relaxing others. It would seem to me only a single event tasting meets that criteria. What makes Robert Parker so successful is his “single event” scores (e.g. rating a single vintage of Bordeaux releases) are frankly remarkably “accurate” for those who have seen these recommendations mature over many years. I hope his effusive ratings for 2008 Bordeaux turn out to be correct (Robert tell me its true!!). Why would anyone expect a second round of tasting some many months later, under different conditions, under a different tasting environment, almost certainly differing glassware, would produce different scores? And all of those throwing around statistics should go back to Stats 1. What Standard Deviation means is that the chances of any tasting being consistent, meaning within the “real” rating if we knew what that was, it is 19 in 20 times likely within plus or minus 2 standard deviations o be correct. So a wine with a 90 score and a 2 point SD means anywhere between 86 and 94. In this range almost every wine ever rated falls into the point range. Applying statistics is highly misleading for the reason I mention – this ain’t science, cannot be science unless we apply many more controls! And hey let’s not forget the main point – is the wine enjoyable without points and SD’s? I just had a Cahors wine tonight called Ch Lazerrote. Just lovely 14.5 % alocohol, but seamless and cool fresh acidity making this dance on the tongue. Did anyone notice – no point scores? And I am a statistician.

By: epicuria

epicuria — Thu, 19 Nov 2009 06:39:55 +0000

@michael. I did miss your point first time around. Yes, there should be more tastings that demonstrate differences among varieties as such or as complements to various dishes, regardless of where the wine being used in these ways appeals compared to others in its class or to any other wine(s).

P.S. On wines that go better with various kinds of foods, I learned today that the reason red wines don’t seem suitable for white fish has not to do so much with tannins, but with the amount of iron in various wines, and for the most part, red wines have more of this chemical element than white wines.

By: Michael

Michael — Wed, 18 Nov 2009 23:20:11 +0000

@epicuria: Perhaps I am not explaining myself well. I understand your point, agree with it … although it is wholly besides mine. My only contention is with the journalists who conflate tastings which, as point out, may lead to ratings with tasting in general, which is obviously a much broader category. To me, that confuses the issue and devalues the nature of tasting by associating it — in toto — with any ratings system. (Perhaps an aside to the central issue of ratings per se, but this is a blog, right?!)

And I’m with you on complex collaborative systems, having built them myself. While crowdsourcing can be a powerful tool, I believe it’s also prone to gaming and source-trust issues (e.g. the variable you mention). In other words, the success of such systems depends in no small part on the verficiation methods and formal trust model those systems implement to ward off the nutters. And even that doesn’t always work.

But I do agree with you in spirit as I’m working on such a system right now in re: digital mapping. Another really, really good read is Clay Shirky’s “Here Comes Everybody”. Enjoy!

By: epicuria

epicuria — Wed, 18 Nov 2009 20:56:55 +0000

@Eric. Yes this influence is real, as I noted above, but not very significant,IMO. Wine enthusiasts, and consumers of other products for that matter, also have offsetting variable operating: a desire to disagree with the experts. Still, you do need to encourage blind tastings or have a CT section on such.

-tom merle

By: Eric LeVine

Eric LeVine — Wed, 18 Nov 2009 18:41:56 +0000

FWIW, I have read Surowiecki and think he is brilliant. That said, if you look closely at the Wisdom of Crowds, you realize that any current community rating system for wine is tainted by the professional reviewers–many of the amateur raters know that Parker rated a wine highly etc. So that bias creates issues.

Nonetheless, when trying to figure out whether to drink something from my cellar, do I rely on a 15 year old from Parker or the 50 notes from fellow wine lovers who have had the same wine in the past year? Pretty easy decision…

I have always maintained that both professional and amateurs can add a lot of value in helping people to recommend wines to buy and when to drink them. As so many so, your own palate is always your best guide.

Thanks,
-Eric
CellarTracker.com

By: epicuria

epicuria — Wed, 18 Nov 2009 18:14:05 +0000

@michael. Your elaboration of the distinction you’ve drawn still seems murky. We are talking about ratings that emerge from tastings. Sure there is a marketing dimension to such activity, but it is not part of the evaluation, it uses the evaluation.

What makes the “wisdom of crowds”(crowdsourcing) so compelling is the aggregation of various subjective views on a product or service. Yelp.com and Tripadvisor.com, like digg, American Idol and CellarTracker.com, produce a average rating based on the opinions of all sorts of different people each subject to different variables. With a large enough population of opinions, however, like polling, you get a more objective reading of the merits of something.

I recommend that readers surf over to http://www.amazon.com/Wisdom-Crowds-Collective-Economies-Societies/dp/0385503865/ and read the comments about James Surowiedki’s “The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations” (Amazon.com user reviews are another solid example of this phenomenon). To hear the author on the subject go to: http://wamu.org/programs/dr/04/07/07.php and http://itc.conversationsnetwork.org/shows/detail468.html

By: Michael

Michael — Tue, 17 Nov 2009 23:14:39 +0000

@Dr. Vino, all: My mistake! It was not in WIRED, but written by a WIRED contributor, Jonah Lehrer. The piece is here: http://scienceblogs.com/cortex/2009/10/robert_parker.php

@Charlie: Mine was not a personal confusion of how to read a rating, but a comment on a too-often conflation of the method/nature of tasting with the rating of a wine for the express purpose of selling it (and the rating “expert” as a by-product).

It’s squares and rectangles, after a fashion. Some tasting practices can be for the purpose of rating a wine, but that is a portion of all structured tasting which bear no such intent. For example, tasting for instructional/educational purposes. Or for assessing viability with a menu or within a larger wine program. Or out of sheer anal-retentiveness.

Pace Lehrer, tasting is doggedly subjective, as science shows. That certainly chips away at the foundation of ratings and rating-givers. But neither is tasting wholly without objectivity. So the argument is not dispositive of wine assessment more broadly. But if and when, as in these two pieces, the distinction between “tasting” and ratings becomes so blurred as to appear part of the same thing, I believe many readers are losing out on the bigger, more nuanced picture (even if it means a good “gotcha!” on Tannin-Pants Parker).

By: Paul

Paul — Tue, 17 Nov 2009 18:54:23 +0000

I THINK I WILL DEVELOP A WEBSITE FOR USERS TO RATE WINES. I OWN AN INTERNET COMPANY IT CANT BE TO HARD TO DO THAT, THANKS FOR THE IDEA.

By: Ludwig

Ludwig — Tue, 17 Nov 2009 18:49:01 +0000

Consider this: I entered the Ludwig 2008 Gewurztraminer in three competitions and it won three gold medals, including a double gold at the San Francisco Wine Competition and a four star gold at the Orange County Wine Competition. Possibly wine tasters have a clear and precise characterization for good Gewurztraminer; they have reached a consensus on proper attributes and style and that leads to more consistent tasting results. With the global proliferation of the popular grape varietals, winemakers have had to create every style variation imaginable, including creating wines some might considered flawed, to differentiate themselves in wine society. Obviously not every critic, professional or nonprofessional, will champion all styles and there’s no reason to expect consistent ratings when certain wines are broadly grouped by varietal regardless of style.

By: Eric LeVine

Eric LeVine — Tue, 17 Nov 2009 18:20:37 +0000

Tyler,

I am not proud of my work schedule right now, but it is what I need to do.

The actual site design has been a yearlong process with a wonderful design firm (Fellswoop) and a great branding guy (YiuStudios). The ‘plan’ on paper is amazing, now I am just in the midst of breathing life into it with lots and lots of code.

Cheers as well,
-Eric