And the winner is…

Never mind the general election, real geeks want to know who won the battle of the exit polls. The contestants, as many will recall, were:

And judging from the result and the accolades that followed, Today’s Chanakya was the clear winner.

But not so fast. To make election predictions under India’s first-past-the-post system, polling agencies must first estimate party vote shares and then use this data to make seat share predictions. As we discussed in Why you should ignore opinion poll seat predictions, the primary task of a survey is to collect data on vote shares. This is subsequently translated into seat share predictions based on mathematical models that try to guess how these votes will be distributed among different states and regions.

The ideal scenario is one in which you accurately capture the vote share of different parties and convert these into the correct number of seats. But you can also get the right answer if you make two opposite errors that cancel each other out. This is what occurred with CSDS’ 2012 Uttar Pradesh post-election poll, in which CSDS was the only pollster to correctly predict the number of seats that the Samajwadi Party and the Bahujan Samaj Party would win in that complex four-cornered contest. As Yogendra Yadav later wrote:

Sukumar is absolutely right in saying that we over-estimated the lead in terms of votes and under-estimated its impact on seats. These two compensating errors cancelled each other and our final forecast was closer to the outcome than others.

The first step in evaluating the success of an exit poll (or a post-election poll taken some days after voting) is therefore to check how close it came to estimating the correct vote share. There are various ways to do so (e.g. here), but here we rely on two common techniques:

  1. Comparing the ratio of the votes of the winner and the runner up (the “Mosteller 2” method)
  2. Comparing the margin in percentage points between the winner and the runner up (the “Mosteller 5” method–preferred by Nate Silver)

In the recently concluded election, the BJP-led National Democratic Alliance (NDA) won 38.3% of the vote while the United Progressive Alliance (UPA) won 22.8%. This is a ratio of 1.68 in favour of the NDA and a vote share gap of 15.5 percentage points.

Under the Mosteller 2 method, we rank the exit polls based on how far they deviated from the vote share ratio in their own estimates. And we find that the NDTV Hansa poll performed the best, while the India Today Cicero poll did the poorest.

Screen Shot 2014-05-23 at 1.30.03 pm

Using the Mosteller 5 method, we find that NDTV Hansa did the best again with a deviation of only 0.8 percentage points from the actual vote share gap between the NDA and the UPA. CNN IBN CSDS came second, and India Today Cicero were last once again.

Screen Shot 2014-05-23 at 1.43.26 pm

What is interesting is that Today’s Chanakya’s performance was middling according to both measures, even though its seat share estimate came closest to the actual. Certainly the agency is a winner from a marketing point of view, but in this case a subpar vote estimation appears to have been overcompensated for by an aggressive seat conversion formula (as others also noticed with its Uttar Pradesh prediction).

Today’s Chanakya may be doing best in the box office, but the award for best survey goes to NDTV Hansa.

Research note

Only NDTV, CNN IBN and India Today were kind enough to make their national vote share estimates easily available online. I had to endure considerable time watching YouTube videos to locate the vote shares of the India TV Cvoter and ABP News Nielsen exit polls and — even less forgivably — failed to locate the Times Now ORG data in spite of being forced to watch Times Now for a memorable length of time. Equally painful was having to manually calculate Today’s Chanakya’s national vote share estimate from its state vote share data, since the agency inexplicably chose not to disclose its national numbers.

How opinion polls fared in the last two general elections

As more and more opinion polls hit the headlines, it’s worth recalling how they fared in the two most recent general elections:



Note: The figures in parentheses under the NDA and UPA columns are the respective seat tallies of the BJP and Congress Party.

Should opinion polls be banned?

Despite my critique of their use of communal violence data, the India Today Group graciously invited me to participate in a panel discussion on opinion polling at the India Today Conclave 2014 on Saturday, Mar 8.

I made the following points:

  1. Parliamentary seat projections should be treated with skepticism because there is no simple relationship between vote shares and seats won (as I have argued previously).
  2. About a fifth of the seats in the 2009 Lok Sabha election were won with a margin of 3 per cent or less, which means that fluctuations within the margin of error of most surveys can dramatically change the number of seats a party wins.
  3. Between 20 and 30 per cent of voters make up their minds about whom to vote for a couple of days prior to voting, and last minute vote swings can make a big difference.
  4. Of course I didn’t think that opinion polls should be banned but made more transparent and evaluated by better-informed consumers.

The full video of the event is here: India Today Conclave 2014: Need to regulate opinion polls, say psephologists.

Here is an edited version:

The opinion in opinion polling

As if to underline my warning (in Why you should ignore opinion poll seat projections), the latest ABP News-AC Nielsen opinion poll exemplifies how opaque seat projections are in comparison with more straightforward vote share estimates.

Look at the table below that compares AC Nielsen’s vote share and seat predictions over two successive months in 2014:

Screen Shot 2014-02-23 at 11.35.07 pm

The survey estimates that the vote share of the Indian National Congress (INC) and its allies rose by a percentage point between January 2014 and February 2014, but that its Lok Sabha tally fell nine seats to 92. The vote share of the Bharatiya Janata Party (BJP) and its allies stayed about the same while their count rose ten seats to 236. Other parties’ vote share fell by about a percentage point but their projected seats fell only marginally (by a single seat) to 215.

It’s hard to say why this occurred in the absence of more details from AC Nielsen but what is clear is that the seats projection is being driven not by the vote share expectation but by other, unknown, assumptions about the distribution of those votes. These seat share projections create the illusion of momentum in favour of the BJP alliance when in fact the vote share estimates in effect show no change from the previous month.

Buyer beware.