Delays in the release of India’s census data

1.Introduction

The quality of government data is particularly important for developing countries, where alternative sources of data are either unavailable or unreliable. In most developing countries, non-governmental actors have limited expertise to question government data, let alone build alternative databases. Restrictions on freedom of expression and media, weak judiciaries and lower rates of (statistical) literacy further impede the scrutiny of official statistics. Even countries such as India, which have relatively robust official statistical systems11 as well as tremendous non-governmental expertise in the field of statistics, are struggling to maintain the quality of their official statistics [9, 56, 71, 75, 76, 95, 96].

Figure 1.

Time gap between enumeration and release of India’s census data (1951–2011). Notes: (i) Each vertical bar represents the time elapsed between enumeration and the publication of the data. (ii) The gap is measured in years as we do not have information on the month of release of all publications. (iii) GPT: General Population Tables, PES: Post-Enumeration Surveys. Source: Table 1.

Survey data quality has several dimensions, including, accuracy, credibility, comparability, usability/interpretability, timeliness, relevance, accessibility, completeness and coherence [17, p. 819]. Only a few of these dimensions have been examined in the literature on India’s official statistics. For instance, the quality of Indian census data has been evaluated mostly in terms of accuracy with focus on content errors (errors in the sub-classification of headcount) and, to a lesser extent, coverage errors (errors in the overall headcount).22

In this paper, we examine the quality of Indian census data from the perspective of timeliness, i.e., whether ‘data deliveries adhere to schedules’ [17, p. 819], that has received insufficient scholarly and administrative attention [5],[20, p. 3596],[21],[37, p. 244],[68]. The growing delays in the publication of census data over the past few decades is our point of departure (Fig. 1, Table 1).33 The language tables of the 2011 Census were, for instance, released in 2018, i.e., seven years after enumeration. The migration tables of the 2011 Census were released after the commencement of preparations for the 2021 Census.44

We evaluate the quality of the post-independence Indian censuses along the timeliness dimension. We show that the growing delays are a fact, rather than a mere allegation and situate the problem in its proper context. Section 2 discusses what counts as delay. In Section 3, we discuss how to measure a delay vis-à-vis usual, desirable and feasible schedules of publication. We conclude in Section 4 with a discussion of delays from the perspectives of the unwillingness of the statistical system to face public scrutiny, general decline in the quality of public goods, growing political interference and, in case of data on identity, growing communalisation of politics. While Indian censuses are the primary focus of this paper, we also discuss examples from other countries. The latter examples are neither exhaustive nor systematic, but they help to highlight the problem of delay across developing and, even, developed countries.

2.Delays

The problem of delayed publication of government statistics in India goes back all the way to the 19th Century, when the colonial government began publishing its official data. In “Twenty Years’ Review of Census Statistics” judge and social reformer M.G. Ranade commented that “[t]he Decennial Census Report for 1891, for the Bombay Presidency, loses much of its interest by reason of the great delay that has taken place in its publication” ([74, pp. 209–230] originally published in 1893). The problem persisted in the post-colonial period. Commenting upon the delays in the release of 1961 Census publications, Visaria [97, p. 206] pointed out that contrary to expectations there were delays in the release of the publications of the next census as well: “It was hoped that the 1971 census would eliminate the delays, partly because most of the tables were to be prepared from a 10 percent sample of the rural population and a 20 percent sample of the urban population, with the help of electronic computers, mainly IBM-1401s.” The government itself admitted that the release of the 1981 Census data was delayed due to the “absence of in-house computer” [45, p. 4] and hoped that the problem would be rectified in the 1991 Census. After the 2001 Census, Bose [20, p. 3596] lamented that “the entire data (on a 100 per cent basis) was transferred to the computers for tabulation and this has, instead of speeding up tabulation has resulted in considerable delay.” The National Statistical Commission too noted the problem of delay [52, 56, 82].

Indeed, delays have grown rather than reduced over time despite technological innovation and growing field experience (Table 1).55 Several crucial tables from the 2011 Census, the latest decennial census conducted in India, have been released belatedly in instalments (migration and religion data) or released with considerable delays (language data) even though data are increasingly being released in the form of barebones Microsoft Excel tables.66

Figure 2.

Understanding delays. Note: Delays in conducting census are shown along the horizontal axis, while the delays in releasing data are shown on the vertical axis. Sources: Bosnia and Herzegovina [73], Canada [25], Ethiopia [24], India (see Table 1), PEPSU and Himachal Pradesh [37], New Zealand [77, 78], Pakistan [90], Sri Lanka [61, 8], Turkmenistan [38].

Two basic questions need to be answered for there to be an informed discussion on delays: what counts as a delay and how to measure a delay. The first question is addressed below, while the second is examined in the next section.

A nuanced discussion on delay will have to go beyond a binary treatment of the problem. Delay has to be seen as a kind of continuum. Figure 2 maps the various possibilities of delay. Belated release of census data is commonplace in developing countries. The 2007 Census of Nigeria [24], 2012 Census of Sri Lanka [8], 2013 Census of Bosnia and Herzegovina [73], 2014 Census of Myanmar [89] and 2017 Census of Pakistan [90] proved to be controversial due to delays in the publication of results. Occasionally, developed countries too witness delays in the publication of results. The 2018 Census in New Zealand took place in March and the results were expected in the same year. The Statistics New Zealand admitted that the release is unlikely any time before April 2019 because of “the complex nature of the task” [77]. In Canada, the release of 2006 Census results was delayed because of budgetary constraints, among other things [25].

In several cases, governments do not release results to avoid embarrassing/inconvenient findings. Saudi Arabia probably feared that “publishing an exact count (showing their own population to be smaller than many supposed) might encourage enemies to invade the country or promote subversion [in the Shia-dominated eastern region]” [6, p. 96]. Other countries such as Turkmenistan [38], Bhutan [28] and even early post-colonial United States [72, p. 6] have faced similar dilemmas in the past.

Elsewhere census operations were suspended to avoid ethnic conflict. Nigeria delayed censuses for a long time fearing unrest and eventually conducted censuses that nearly reproduced earlier federal population shares [34]. Pakistan postponed the 1991 Census five times between 1991 and 1998 fearing ethnic conflict in case the existing demographic shares of ethnically demarcated provinces were altered and eventually managed to produce figures that matched the expectations based on earlier censuses [98, p. 687, p. 691].

So, on the one hand, we have timely census and release of data in the ideal case. On the other, there is delayed enumeration and release of data. In extreme cases, data collection exercises are either indefinitely suspended or results are indefinitely withheld. Lebanon is the best exemplar of the former. Its government has not conducted a census after 1932 “for fear that taking one would reveal such changes in the religious composition of the population as to make the marvelously intricate political arrangements designed to balance sectarian interests unviable” [36, 6, 65]. A recent census held in Turkmenistan exemplifies the latter. Turkmenistan’s government did not release the results of the 2012 Census that seems to have revealed that the actual population was far lesser than official claims and the ethnic composition too differed from the government propaganda [38].

In 1951, the government of newly independent India dropped presumably divisive questions related to identity that were part of the colonial censuses. It decided to collect information on scheduled castes and tribes that was just sufficient to fulfil constitutional obligations: “the only relevant question on caste or tribe incorporated in the Census Schedule was to enquire if the person enumerated was a member of any ‘Scheduled Caste’, or any ‘Scheduled Tribe’ or any other ‘Backward class’… While the data on scheduled castes and tribes were published, that on other backward castes was not published… From 1961 onwards, the information was collected only for each Scheduled Caste and Scheduled Tribe” [48, p. 28]. In other words, data on ‘Backward class’ were collected, but not released. In the 1941 Census, in the run-up to India’s religious partition, Punjab witnessed massive manipulation of headcount [39, p. 5]. In 1951, the authorities managed to control coverage errors, but content errors arising out of linguistic/religious conflict affected the headcount. As a result, the language data of the 1951 Census of India for Punjab, the Patiala and East Punjab States Union (PEPSU) and Himachal Pradesh were not published to avoid exacerbating linguistic/religious conflict [37, p. 244]. These were perhaps the only instances of deliberate non-publication/indefinite postponement of the release of census data in India.

Censuses have always been conducted as per schedule in independent India. There are a few exceptions though. Firstly, the reference date of the census was shifted once. Since 1941 the reference date for census has been 1st March except for 1971, when it was 1st April. The reference period for enumeration was changed in that year “to avoid clash with mid-term Parliamentary Election [in March]” [53, p. 5]. It can be argued that enumeration was delayed by a month in 1971. This is the only instance of a nationwide “delay.” However, there was no delay if we consider the fact that the government declared the revised reference date in advance and stated a justifiable administrative reason for the rescheduling insofar as the same staff has to be mobilised for conducting both enumeration as part of decennial census and enrolment of voters in electoral rolls. There have been a few instances of extension of the timeline of enumeration too. The revisional round of the 1961 Census was extended by two days, from March 1 to March 5 instead of March 3 on account of a festival that is celebrated across large parts of the country [100, p. 142]. In the 1961 Census, household enumeration had to be extended by a few months in parts of Nagaland due to political unrest [42].

Secondly, in some cases census had to be delayed in a few districts due to natural disasters [54]. In 1981, census was delayed in the non-snow bound areas of Jammu and Kashmir due to unfavorable weather. In 2001, enumeration could not be conducted in the whole of Kachchh district and a few taluks of Jamnagar and Rajkot districts of Gujarat due to a massive earthquake and had to be postponed until February 2002. Similarly, in 2001, enumeration had to be postponed in the entire Kinnaur district of Himachal Pradesh due to natural calamities. The natural disaster induced delays are few and far between, though. Otherwise the standard practice in India is to conduct census ahead of schedule in snow-bound hilly areas of states such as Jammu and Kashmir, but the schedule and the areas covered are declared in advance.77

Thirdly, census was not conducted in 1981 in Assam and in 1951 and 1991 in Jammu and Kashmir due to political unrest. In, at least, one case (West Bengal in 1971) census was conducted ahead of schedule due to political disturbance [43], while in another the timelines were relaxed under similar circumstances (Nagaland in 1961) [42].88

Fourthly, the North Eastern Frontier Agency (NEFA), which included the present Arunachal Pradesh and Tuensang, Mon, Kiphire, Longleng and Noklak districts (and small parts of Mokokchung and Zunheboto districts) of Nagaland, was barely covered until 1951 due to the remoteness of the region and the sparse presence of the government. Also, until 1971, the Census excluded the Jarawas and Sentinelese tribes of Andaman and Nicobar Islands that were unapproachable [57].99

In most of the above cases there was either no unplanned delay, or the delay was due to objectively verifiable factors such as natural calamities, inclement weather and clash with elections. In other words, India has not seen unplanned delays in enumeration driven by non-verifiable factors. The remainder of the discussion in this paper will, therefore, focus on the delays in the publication of census data.1010 Publication can be delayed because of one or more of the following: political interference, delay in the processing of raw data and delay in the printing of reports. The last seems to have been a bottleneck earlier when “the slow progress of printing” was attributed to “the large amount of work involved and the need to assign it to already overburdened government presses which handle other priority jobs as well” [97, p. 209]. This bottleneck has been largely eliminated by computerised typesetting and internet publishing.1111 Moreover, the Census no longer needs to print a large number of copies as it publishes reports online as well and at times it releases only Microsoft Excel tables that are not supplied in the printed form.

3.Identifying delays in publication

If the government misses a declared, desirable, usual and/or feasible timetable of publication, we can say that there is a delay. The Census of India pre-commits to a timetable for conducting enumeration, which is declared as many as two years in advance. The timetable for publication is not explicitly declared in advance though.1212 In this section, we will, therefore, try to identify delay vis-a-vis desirable, usual and feasible timetables of publication of data.1313 Any assessment of delays has to contend with the diversity of data published by the census. The Indian census data is organised across stages of enumeration (houselisting, household and post-enumeration surveys) and spatial (enumeration block, village/urban settlement, district, state, region1414 and country) as well as socio-cultural (Scheduled Castes, Scheduled Tribes and Others; Religion; Language; Sex; Age groups) levels of aggregation. The data are further organised into general, economic, socio-cultural (caste, tribe, religion and language), migration, fertility and housing tables. The census also publishes several cross-tabulations, atlases and thematic reports. In the following discussion, we will focus on tables generated from household enumeration and post-enumeration surveys, but we will not cover the publication of data for all the levels of aggregation. Moreover, we will mostly restrict our discussion to the first publications in each case.

Figure 3.

Delay vis-à-vis desirable schedule. Note: Release after three years is treated as a delay vis-à-vis the desirable schedule (see text for further details).

Before we identify and compare the extent of delays over the decades, three points are in order. Firstly, the content and scope of the same report published after different censuses could be different. This might explain the variations in the date of release due to differences in the background work needed to prepare the reports. However, we focus on the first round of publications that are mostly restricted to broad categorical and geographical aggregates.

Figure 4.

Delay vis-à-vis usual schedule. Note: Usual schedule is defined as two years for caste and religion data and three for language. There was no delay in 1951 and 1961 vis-à-vis the usual schedule.

Figure 5.

Delays vis-à-vis feasible schedule. Note: The feasible schedule for religion is linked to the release of caste and tribe data, while three years is treated as the feasible schedule for language.

Secondly, there is a need to balance the speed of publication and accuracy of reports [32, p. 7]. In the 1971 Census, “very great importance was attached to the early publication of the provisional totals…Unfortunately some of the Directors were over anxious in transmitting the totals. Revised totals had to be published later” [100, p. 376].

Thirdly, the Provisional Population Totals (PPT) – the first publication of every census that provides provisional figures on headcount, gender and educational attainment–have been released generally within a month of the completion of enumeration and revisional rounds (Table 1).1515 After the 1971 Census, Visaria [97, p. 206] contrasted the “commendable speed” of the release of the PPT with overall publications program that was “behind schedule.” Three decades later Bose [20, p. 3595] too contrasted the early release of PPT and the subsequent delays:

The provisional figures for the total population in India and the states, were released with lightning speed on March 26, 2001. Considering that the revisional round of the [2001] census enumeration was over only on March 5 it was a remarkable achievement indeed. But alas, not even the key tables have been brought out so far and it is unlikely that the main tables will be published before the end of this year [2004]. From jet speed we have descended to auto-rickshaw speed.

PPT are not covered in Figs 3–5 that compare the delays in the release of different reports as the former are released within weeks after enumeration, whereas the rest of the reports appear at least a year later. In the remainder of this section, we will examine the delays in the release of General Population Tables (GPT),1616 Post-Enumeration Survey (PES)1717 report and data on identity – religion, caste and language.

4.Concluding remarks

Delays between enumeration and publication of results limit the utility of census data for administration, policy-making and research [20]. If the delay is long, the data might become obsolete for certain purposes and even suspect in the eyes of users [80, p. 12]. Such obsolescence may affect government and business decision-making. For instance, census data are used by governments for urban planning and deciding federal redistribution, while businesses use the data for information on consumer characteristics and for demand forecasting. India has seen growing delays in the release of major databases amidst technocratisation of policy-making, public professions of faith in evidence based policy-making and fascination with big data. While governments promise more data, they fail to release data collected at a considerable expense. So, there is a mismatch between the practice and policy commitments of the government, including the political class and bureaucracy.

The Census of India takes pride in its pioneering use of latest technologies in data processing, analysis and dissemination, but delays have, in fact, grown after the introduction of advanced technologies. Political apathy and/or interference and, possibly, the time required to learn to handle new technologies2424 seem to have more than neutralised the potential reduction in delays due to technological improvements. Delays in the publication of the 2001 PES report, 2001 and 2011 religion data and post-1971 language data are cases in point.

The delays can be understood in different ways. Firstly, delays might reflect an unwillingness of the statistical system to face public scrutiny. The quality of statistics is affected in the absence of cross-examination [69, p. 43]. The absence of scrutiny pushes the system into a vicious cycle leading to further deterioration of the quality of statistics.

Secondly, insofar as government statistics can be treated as ‘public goods,’2525 the growing delays could be treated as an indicator of the general decline in the provision of public goods in the country. The poor quality of government statistics affects both policy-making and public debate in addition to affecting the quality of data that use government statistics as inputs, e.g., the census serves as the sampling frame for sample surveys.

Thirdly, the increase in delays in the release of government statistics can also be treated as an indicator of the growing political interference with government’s statistical machinery.2626 Recall that the criteria for classifying languages were abruptly revised after the creation of Bangladesh on linguistic grounds and a fresh report was issued that superseded the reports for the 1971 Census based on the earlier criteria. This led to the delayed publication of the 1971 language tables. Engineer [33, p. 4304] argued that the religion wise population figures from the 2001 Census were delayed ‘by the BJP [Bharatiya Janata Party]-led government on account of the Lok Sabha [parliament] elections.’2727 A similar case can be made with regard to the non-release of the 2011 Census data on religion first by the Congress government before the 2014 elections and later by the BJP [68]. Srinivasan [76, p. 304] argues that “publication delays, unless politically or otherwise motivated, do not raise questions of credibility” of data. However, the political aspect cannot be overlooked as the 2001 and 2011 Census data on religion were released only after crucial elections even though in both instances the statistical bureaucracy suggested that the data were ready for release [79, 11, 21].2828

Census is not alone in attracting political interference though. The release of UNICEF-aided Rapid Survey on Children was allegedly suppressed for some states as it showed child immunisation in certain politically and economically important states lagged behind some of the poorer states [85]. In the run-up to the 2014 parliamentary elections, it was argued that the quinquennial round of the 2009–10 survey conducted by the National Sample Survey Office (NSSO) was repeated in 2011–12 due to political interference [27].2929 When National Family Health Survey (NFHS) was allegedly “called off”, the Congress-led UPA government was accused of cancelling data collection to “suppress… unflattering findings,” with regard to health and nutrition [91].3030 The next round finally took place in 2015–16.

Fourthly, the release of census data can be seen as a part of a continuum with prompt release and no release being the two ends of the continuum. Several conflict ridden countries have seen significant and sustained delays, reductions in the scope of census, etc. Unless India’s official statistical system is insulated from government and political interference it might slip toward the wrong end of the continuum. Focusing specifically on religion and language data from census, the growing delays can be read as symptoms of the deepening communal crisis. The delays in the release of religion data have been growing since 1981, which not coincidentally was also the period of the growing communalisation of Indian politics.