“Bias it is”: CMIE chief’s defence of CPHS survey elicits fresh critical response from Jean Drèze, Anmol Somanchi

In fact, Vyas’ rejoinder confirms that the survey is likely to be biased, if only because it begins from the “main street” in each sample village. Even if the sample size often requires the survey to go beyond the main street, as Vyas points out, the fact remains that inner-street households will be under-represented. This may or may not be the main source of bias, but it is certainly a smoking gun.

We plead guilty of rounding the CPHS estimate of adult literacy (age 15-49 years) in urban areas from 99.6% to 100%. But even 99.6% stretches the imagination, as do many other figures we presented to illustrate the bias. The accompanying chart, comparing CPHS illiteracy rates among adult women (for rural and urban areas combined) with corresponding estimates from the fifth National Family Health Survey, NFHS-5, speaks for itself. To put things in global perspective, the CPHS adult literacy figures would imply that India has suddenly become a world leader in this field among low- and middle-income countries, leapfrogging China, Vietnam, Sri Lanka and other countries with a sterling record of early investment in mass education.

Guest Contributor

As we pointed out, there is also a major divergence between CPHS and NFHS-5 in terms of ownership of household assets. According to recent tweets and personal communications from Sanjay Kumar, director of the Centre for the Study of Developing Societies (CSDS), CSDS-Lokniti’s recent election surveys are also at variance with CPHS data in this respect, and much easier to reconcile with NFHS-5 data. To illustrate, the proportion of households without television in 2019, in the 11 major states where NFHS-5 is on track, was 29% according to NFHS-5 and 25% according to CSDS-Lokniti’s National Election Survey 2019, but only 6% according to CPHS. Similar contrasts apply to fridges, toilets, and so on. In short, there is a troubling pattern of poverty underestimation in CPHS, vis-à-vis other national surveys.

The representativeness of CPHS data cannot be assumed from the survey methodology. It must be scrutinised through comparisons with other credible sources. We have drawn attention to major discrepancies with the National Family Health Surveys, the Periodic Labour Force Surveys, the census, and now the CSDS-Lokniti surveys as well. Vyas maintains that the CPHS is more reliable than other sources, but it is not clear why this would be the case.

We pointed out that the under-representation of poor households in CPHS may be growing over time. This is an important issue, since one of the main purposes of CPHS is to assess time trends. The survey presents an upbeat picture of economic progress during the last few years, with, for instance, the share of households earning less than Rs 1,00,000 a year in the CPHS sample dropping from 31% in 2014 to 6.6% in late 2019 as Vyas observes. It is possible, however, that this rapid progress is at least partly driven by the growing under-representation of poor households in CPHS data. Even if the bias does not grow over time, it is likely to affect these observed trends. This is just one illustration of why the representativeness of the sample matters. Of course, that does not preclude making good use of CPHS data for some purposes.

In short, we see no need to retract anything we wrote in the light of Vyas’ rejoinder. This is not to dispute the value of CPHS as a large-scale, regular national survey or to ignore the difficulties of executing such a survey. We have learnt much from CPHS ourselves. If the CMIE is committed to further improvement of the survey method, as Vyas assures us, nothing like it. Hopefully, researchers and other users of CPHS data will also help to settle these matters, and meanwhile, exercise appropriate caution.