And the winner is…

Never mind the general election, real geeks want to know who won the battle of the exit polls. The contestants, as many will recall, were:

And judging from the result and the accolades that followed, Today’s Chanakya was the clear winner.

But not so fast. To make election predictions under India’s first-past-the-post system, polling agencies must first estimate party vote shares and then use this data to make seat share predictions. As we discussed in Why you should ignore opinion poll seat predictions, the primary task of a survey is to collect data on vote shares. This is subsequently translated into seat share predictions based on mathematical models that try to guess how these votes will be distributed among different states and regions.

The ideal scenario is one in which you accurately capture the vote share of different parties and convert these into the correct number of seats. But you can also get the right answer if you make two opposite errors that cancel each other out. This is what occurred with CSDS’ 2012 Uttar Pradesh post-election poll, in which CSDS was the only pollster to correctly predict the number of seats that the Samajwadi Party and the Bahujan Samaj Party would win in that complex four-cornered contest. As Yogendra Yadav later wrote:

Sukumar is absolutely right in saying that we over-estimated the lead in terms of votes and under-estimated its impact on seats. These two compensating errors cancelled each other and our final forecast was closer to the outcome than others.

The first step in evaluating the success of an exit poll (or a post-election poll taken some days after voting) is therefore to check how close it came to estimating the correct vote share. There are various ways to do so (e.g. here), but here we rely on two common techniques:

  1. Comparing the ratio of the votes of the winner and the runner up (the “Mosteller 2” method)
  2. Comparing the margin in percentage points between the winner and the runner up (the “Mosteller 5” method–preferred by Nate Silver)

In the recently concluded election, the BJP-led National Democratic Alliance (NDA) won 38.3% of the vote while the United Progressive Alliance (UPA) won 22.8%. This is a ratio of 1.68 in favour of the NDA and a vote share gap of 15.5 percentage points.

Under the Mosteller 2 method, we rank the exit polls based on how far they deviated from the vote share ratio in their own estimates. And we find that the NDTV Hansa poll performed the best, while the India Today Cicero poll did the poorest.

Screen Shot 2014-05-23 at 1.30.03 pm

Using the Mosteller 5 method, we find that NDTV Hansa did the best again with a deviation of only 0.8 percentage points from the actual vote share gap between the NDA and the UPA. CNN IBN CSDS came second, and India Today Cicero were last once again.

Screen Shot 2014-05-23 at 1.43.26 pm

What is interesting is that Today’s Chanakya’s performance was middling according to both measures, even though its seat share estimate came closest to the actual. Certainly the agency is a winner from a marketing point of view, but in this case a subpar vote estimation appears to have been overcompensated for by an aggressive seat conversion formula (as others also noticed with its Uttar Pradesh prediction).

Today’s Chanakya may be doing best in the box office, but the award for best survey goes to NDTV Hansa.

Research note

Only NDTV, CNN IBN and India Today were kind enough to make their national vote share estimates easily available online. I had to endure considerable time watching YouTube videos to locate the vote shares of the India TV Cvoter and ABP News Nielsen exit polls and — even less forgivably — failed to locate the Times Now ORG data in spite of being forced to watch Times Now for a memorable length of time. Equally painful was having to manually calculate Today’s Chanakya’s national vote share estimate from its state vote share data, since the agency inexplicably chose not to disclose its national numbers.

Advertisements

The return of the gender gap

If women alone were to vote in the current elections, the Bharatiya Janata Party (BJP) would win 88 more Lok Sabha seats than the Indian National Congress (INC) (using CSDS data — see Table 3a). If only men voted, the BJP would get 126 more seats than the INC (if we stubbornly assume a constant ratio of voteshare-to-seats conversion). This hypothetical example highlights how gendered voting choices can have a big effect on election outcomes.

As Rajeshwari Deshpande wrote in September 2009:

The Congress has enjoyed an advantage among women voters since the 1996 general election. The party’s popularity among women reached a high point in 1999 when the gender gap in favour of the Congress was 5 percentage points. Since then, the gap has been closing. In the 2009 election, the Congress enjoyed a slight tilt in women’s votes towards it.

That no longer seems to be the case. According to the latest available CSDS data, the gender gap between the INC and BJP has reemerged with a vengeance (we define the gender gap as the difference between women and men voting for the BJP subtracted from the difference between women and men voting for the INC — note that the 2014 gap emerges from less reliable pre-poll data while the rest are from post-poll data):

Screen Shot 2014-05-06 at 1.16.19 pm

So why might such a gap exist? The United States has experienced a gender gap in favour of the Democratic Party since 1980, and in 2000 and 2012 Democratic presidential candidates had a 20 percentage point gap in their favour (that dwarfs the Congress Party’s record eight-point gap in 1998 and 1999). Explanations of the gender gap in the United States and other industrial societies fall into three general (and partly overlapping) categories:

  • Economic. Higher female participation in the labour force, and the fact that female workers face wage discrimination at the workplace, have pushed women to the left of the political spectrum in favour of parties that emphasise redistributive politics.
  • Family. In a similar vein, rising marriage ages, the widespread use of contraception, more out-of-wedlock childbearing and increased divorce rates have driven women voters to the political left, since they are no longer in a position to benefit from the incomes of male partners.
  • Values. The belief that a woman is entitled to reproductive autonomy (particularly salient in the United States) and the spread of feminist ideas have pushed women voters in the direction of political parties that share those values.

To what extent might these factors apply in the India case? At first blush, it would appear unlikely. India’s female labour force participation rate has fallen in recent years (from 44% in 2005 to 36% in 2012), which would undermine the case for an economic explanation. The mean effective age of marriage for Indian women has risen from 19.5 years in 1991 to 21.2 years in 2011 (a 9% increase), and contraception usage has increased between 1992-93 and 2005-06, but the divorce rate remains low and out-of-wedlock childbearing appears rare. So far as values go, abortion is not a partisan political issue in India, and there is no evidence that women’s rights issues shape voting behaviour.

Indeed that India even has a voting gender gap should be surprising: political scientists Ronald Inglehart and Pippa Norris found (using 1990s data) that only “postindustrial” societies have a gender gap that favours left parties; the gap favours ideologically conservative parties in post-Communist and developing societies. (Let’s leave aside the question of how generalisable categories such as “left” and “right” can be.)

But things get interesting when you look at the following chart (drawn from Table 2 in Deshpande’s article):

Screen Shot 2014-05-06 at 1.16.35 pm

The gender gap in favour of the INC in 2009 — when it was small by historical standards — was highest in precisely those groups that you would expect to be most influenced by the same economic and cultural trends that have created the gender gap in Western societies. If this holds up to further scrutiny (beyond the scope of this blog), then it could provide one answer to the puzzle that Deshpande posed about women and the INC: “There is no evidence that women find its policies more attractive than [do] men”.

There are other views. Deshpande argues that it makes more sense to look instead at regional variations, and sure enough, the INC’s gender advantage varies from state to state. She suggests that the INC’s gender gap is really the reflection of caste and class factors, and that there is no independent gender tilt in favour of the INC (although there probably is one against the BJP). Furthermore regional parties matter a great deal, and women in the latest CSDS surveys appear to be leaning towards Mamata Banerjee in West Bengal, Jayalalitha in Tamil Nadu and Mayawati in Uttar Pradesh.

That said, women’s issues are more salient in the 2014 national campaign than ever before (as Supriya Nair pursuasively points out), which has forced political parties to craft more specific appeals to women. An Association for Democratic Reforms-Daksh survey found that women’s safety is the third most important election issue (of ten) for urban voters. And Narendra Modi’s aggressive, patriarchal campaign clearly contrasts with Rahul Gandhi’s persistent outreach to women voters.

How opinion polls fared in the last two general elections

As more and more opinion polls hit the headlines, it’s worth recalling how they fared in the two most recent general elections:

2004

2009

Note: The figures in parentheses under the NDA and UPA columns are the respective seat tallies of the BJP and Congress Party.

Should opinion polls be banned?

Despite my critique of their use of communal violence data, the India Today Group graciously invited me to participate in a panel discussion on opinion polling at the India Today Conclave 2014 on Saturday, Mar 8.

I made the following points:

  1. Parliamentary seat projections should be treated with skepticism because there is no simple relationship between vote shares and seats won (as I have argued previously).
  2. About a fifth of the seats in the 2009 Lok Sabha election were won with a margin of 3 per cent or less, which means that fluctuations within the margin of error of most surveys can dramatically change the number of seats a party wins.
  3. Between 20 and 30 per cent of voters make up their minds about whom to vote for a couple of days prior to voting, and last minute vote swings can make a big difference.
  4. Of course I didn’t think that opinion polls should be banned but made more transparent and evaluated by better-informed consumers.

The full video of the event is here: India Today Conclave 2014: Need to regulate opinion polls, say psephologists.

Here is an edited version:

A question of ordering

A Feb 26 survey by Pew Research’s Global Attitudes Project, titled “Indians want political change” (more detailed pdf version here), got a fair amount of media attention, and no surprise why. The poll shows overwhelming support for the Bharatiya Janata Party (BJP) and its prime ministerial candidate Narendra Modi across the length and breadth of the country.

Note that Pew’s questions relate to party preference, not voting intention, and to candidate favourability, not candidate choice. While these are obviously correlated in the real world this also means that the results of this survey cannot be expected to be identical to, say, the CSDS opinion polls.

There are however other reasons to be cautious about these findings:

  1. Sample errors: The survey has a small sample size of 2,464 which includes an urban oversample of 588. This means that only 1,876 responses were randomly distributed by region and urbanity. You want to oversample a specific population if you are interested in studying that sub-group in greater detail, but in this case I suspect it was done to keep costs down. This makes the margin of error for rural residents higher than that of urban residents. The overall margin of error is 3.8%.
  2. Priming. Question ordering can have a substantial effect on survey results. In the Lokniti election tracker questionnaire, the first substantive question asked of a respondent is whom she intends to vote for, followed by whether she is satisfied with her financial condition, the ruling coalition etc. In the Pew survey, the first question is “Overall, are you satisfied or dissatisfied with the way things are going in our country today?”. With satisfaction levels down to 29% from 51% in spring 2012, a sense of dissatisfaction could arguably influence (or “prime”) the answer to the next question, which asks the respondent to choose between the BJP and INC. In their defence, Pew questionnaires often begin this way and, at least in the US, the evidence suggests that the priming effect of this particular question is modest. But in India, particularly in comparison with the Lokniti method, there could be a question order effect. One way to deal with this is to rotate the question ordering, but we don’t know if that was done.
  3. Framing. More critically, the Pew questions convert the election into an artificial two-horse race, keeping regional parties entirely outside the frame and scoring them only when the respondent volunteers an answer. After a series of questions in which dissatisfaction is channelled towards the BJP we finally get to Q5 in which the respondent is asked whom she thinks should lead the next government. This again creates a national party bias in comparison with asking whom she plans to vote for because it’s perfectly plausible that a voter thinks the BJP should lead a national coalition but nevertheless plans to vote for a regional party. This tilt is misleading because regional parties unaligned with the BJP and INC are currently getting 42% of the vote in opinion polls. In subsequent comments Bruce Stokes, a Pew Research Center director, accepted that the presence of smaller parties “could complicate matters“, but to me this is a fundamental design flaw.

The available evidence certainly suggests that the BJP is ahead of the INC by a decent margin but surveys such as this one are doing us all a disservice by exaggerating the lead.

Note: I have emailed Mr Stokes asking for some clarifications about the Pew survey and will post an update if I hear back from him.

Update on Mar 2

Or you could skip my analysis and proceed straight to this classic clip (transcript here) from the TV series Yes, Prime Minister: