A Feb 26 survey by Pew Research’s Global Attitudes Project, titled “Indians want political change” (more detailed pdf version here), got a fair amount of media attention, and no surprise why. The poll shows overwhelming support for the Bharatiya Janata Party (BJP) and its prime ministerial candidate Narendra Modi across the length and breadth of the country.

Note that Pew’s questions relate to party preference, not voting intention, and to candidate favourability, not candidate choice. While these are obviously correlated in the real world this also means that the results of this survey cannot be expected to be identical to, say, the CSDS opinion polls.

There are however other reasons to be cautious about these findings:

1. Sample errors: The survey has a small sample size of 2,464 which includes an urban oversample of 588. This means that only 1,876 responses were randomly distributed by region and urbanity. You want to oversample a specific population if you are interested in studying that sub-group in greater detail, but in this case I suspect it was done to keep costs down. This makes the margin of error for rural residents higher than that of urban residents. The overall margin of error is 3.8%.
2. Priming. Question ordering can have a substantial effect on survey results. In the Lokniti election tracker questionnaire, the first substantive question asked of a respondent is whom she intends to vote for, followed by whether she is satisfied with her financial condition, the ruling coalition etc. In the Pew survey, the first question is “Overall, are you satisfied or dissatisfied with the way things are going in our country today?”. With satisfaction levels down to 29% from 51% in spring 2012, a sense of dissatisfaction could arguably influence (or “prime”) the answer to the next question, which asks the respondent to choose between the BJP and INC. In their defence, Pew questionnaires often begin this way and, at least in the US, the evidence suggests that the priming effect of this particular question is modest. But in India, particularly in comparison with the Lokniti method, there could be a question order effect. One way to deal with this is to rotate the question ordering, but we don’t know if that was done.
3. Framing. More critically, the Pew questions convert the election into an artificial two-horse race, keeping regional parties entirely outside the frame and scoring them only when the respondent volunteers an answer. After a series of questions in which dissatisfaction is channelled towards the BJP we finally get to Q5 in which the respondent is asked whom she thinks should lead the next government. This again creates a national party bias in comparison with asking whom she plans to vote for because it’s perfectly plausible that a voter thinks the BJP should lead a national coalition but nevertheless plans to vote for a regional party. This tilt is misleading because regional parties unaligned with the BJP and INC are currently getting 42% of the vote in opinion polls. In subsequent comments Bruce Stokes, a Pew Research Center director, accepted that the presence of smaller parties “could complicate matters“, but to me this is a fundamental design flaw.

The available evidence certainly suggests that the BJP is ahead of the INC by a decent margin but surveys such as this one are doing us all a disservice by exaggerating the lead.

Note: I have emailed Mr Stokes asking for some clarifications about the Pew survey and will post an update if I hear back from him.

Update on Mar 2

Or you could skip my analysis and proceed straight to this classic clip (transcript here) from the TV series Yes, Prime Minister: