Do Organic and Biodynamic Wines Get Higher Scores? That Isn’t What the Evidence Says.

by Andrew Chalk

Photo by Adonyi Gábor from Pexels

In issue 183 (2021) of Ecological Economics Magali A. Delmas and Oliver Gergaud purport to present results that show that externally certified organic and biodynamic wines score 6.2 and 5.6 percentage points respectively higher in blind tastings by expert wine reviewers than wines without these certifications.

In fact, their paper does not show this. Rather, it shows, if anything, that more expensive wines get higher scores, an intuitive result that is already well established.

THE TERMINOLOGY AND HYPOTHESES

Delmas and Gergaud (henceforth, D&G) divide wine into four categories by whether and what type of ‘eco-label’ they carry: externally certified organic, externally certified biodynamic, self-certified sustainable (including organic and biodynamic), and conventional agriculture.

They hypothesize, first, that sustainable wine practices are associated with higher quality than conventional wine practices. Second, that third-party certified eco-labels lead to higher quality wine than self-declared sustainable labels.

THE DATA

D&G used a sample of French wines in reviews by three leading French critics: Gault & Millau (GM), Gilbert Gaillard (GG), and Bettane Desseauve (BD) from 2008 to 2015. A total of 128,182 ratings (some wines may have been rated by more than one critic. These cases are not distinguished in the sample).

The data reported by each critic are slightly different. GM report wines by both organic and biodynamic eco-labels, GG lumps organic and biodynamic wines into one category, BD did not report either so D&G referred to winery websites and industry organizations for the data.

THE METHODOLOGY

In order to test their hypotheses D&G had to essentially construct a model to explain the score for each wine review. They list the factors determining the score as:

1. Whether the wine was externally certified as organic;

2. Whether the wine was externally certified as biodynamic;

3. Whether the wine was self-certified as sustainable;

4. The wine vintage;

5. The wine color;

6. The wine varietal;

7. The appellation;

They then specify what in statistics is known as a regression to estimate the individual effects of each of these factors.

THE RESULTS

For the sample of all wine publications combined, D&G found a 3.3% increase in price for being externally certified organic, and a 10.6% increase for being externally certified biodynamic. There is a 1.5% decrease in price for being self-certified sustainable.

D&G also broke out results for each publication, and for wines that changed status (e.g. conventional agriculture to externally certified organic) during the sample period, and by wine color (red or white).

THE PROBLEM WITH THE METHODOLOGY

The full details of D&G’s results are in their article. However, we don’t need to go into these to see the problem. The main determinant of wine scores is price - and D&G omit this from their model! One hundred dollar wines are simply better than $10 wines, nearly all of the time. The score premium that D&G attribute to external certification as organic or biodynamic is simply the effect of a higher price. We already know that organic wine costs 15% more to make than conventionally farmed wines. Therefore its equilibrium price will be higher. Furthermore, organic and biodynamic certification is more likely sought for higher priced wines (you don’t see a lot of certification on the bottom wine shelf in the supermarket). Omitting price from their regression means that D&G attribute the higher score of these more expensive wines to their eco label rather than their price. There are more technical criticisms as well.

THE SOLUTION

To correct this study, price should be a variable in the regression. In addition, there are anomalies in the results and it would make the results more robust if they could be explained. For example, GG found external certification had no effect on the score of red wines, and BT found no effect on white. Those results contradict each other, and the overall results.

Given the importance of the question I am going to contact D&G to ask for their data in order to calculate the effects of price. Eco labels may be important, but that is not shown in the current paper.

Update: Since this was written I have contacted the authors with a view to obtaining their data and analysing the effect of a wine's price on a wine's score. In the Internet age it has become the norm for researchers to publish their datasets for other researchers to extend knowledge, so I do not envisage a problem. I am awaiting a reply.

==============

Andrew Chalk has a Ph.D. in economics and a penchant for wines with a high ‘throw weight’ (to use some missile terminology from the cold war). I.e. a high score and a low price.