Never mind the general election, real geeks want to know who won the battle of the exit polls. The contestants, as many will recall, were:
And judging from the result and the accolades that followed, Today’s Chanakya was the clear winner.
But not so fast. To make election predictions under India’s first-past-the-post system, polling agencies must first estimate party vote shares and then use this data to make seat share predictions. As we discussed in Why you should ignore opinion poll seat predictions, the primary task of a survey is to collect data on vote shares. This is subsequently translated into seat share predictions based on mathematical models that try to guess how these votes will be distributed among different states and regions.
The ideal scenario is one in which you accurately capture the vote share of different parties and convert these into the correct number of seats. But you can also get the right answer if you make two opposite errors that cancel each other out. This is what occurred with CSDS’ 2012 Uttar Pradesh post-election poll, in which CSDS was the only pollster to correctly predict the number of seats that the Samajwadi Party and the Bahujan Samaj Party would win in that complex four-cornered contest. As Yogendra Yadav later wrote:
Sukumar is absolutely right in saying that we over-estimated the lead in terms of votes and under-estimated its impact on seats. These two compensating errors cancelled each other and our final forecast was closer to the outcome than others.
The first step in evaluating the success of an exit poll (or a post-election poll taken some days after voting) is therefore to check how close it came to estimating the correct vote share. There are various ways to do so (e.g. here), but here we rely on two common techniques:
- Comparing the ratio of the votes of the winner and the runner up (the “Mosteller 2” method)
- Comparing the margin in percentage points between the winner and the runner up (the “Mosteller 5” method–preferred by Nate Silver)
In the recently concluded election, the BJP-led National Democratic Alliance (NDA) won 38.3% of the vote while the United Progressive Alliance (UPA) won 22.8%. This is a ratio of 1.68 in favour of the NDA and a vote share gap of 15.5 percentage points.
Under the Mosteller 2 method, we rank the exit polls based on how far they deviated from the vote share ratio in their own estimates. And we find that the NDTV Hansa poll performed the best, while the India Today Cicero poll did the poorest.
Using the Mosteller 5 method, we find that NDTV Hansa did the best again with a deviation of only 0.8 percentage points from the actual vote share gap between the NDA and the UPA. CNN IBN CSDS came second, and India Today Cicero were last once again.
What is interesting is that Today’s Chanakya’s performance was middling according to both measures, even though its seat share estimate came closest to the actual. Certainly the agency is a winner from a marketing point of view, but in this case a subpar vote estimation appears to have been overcompensated for by an aggressive seat conversion formula (as others also noticed with its Uttar Pradesh prediction).
Today’s Chanakya may be doing best in the box office, but the award for best survey goes to NDTV Hansa.
Research note
Only NDTV, CNN IBN and India Today were kind enough to make their national vote share estimates easily available online. I had to endure considerable time watching YouTube videos to locate the vote shares of the India TV Cvoter and ABP News Nielsen exit polls and — even less forgivably — failed to locate the Times Now ORG data in spite of being forced to watch Times Now for a memorable length of time. Equally painful was having to manually calculate Today’s Chanakya’s national vote share estimate from its state vote share data, since the agency inexplicably chose not to disclose its national numbers.
