Estimating state level indicators from ICT household surveys in Brazil

1.Introduction

We live in an era of unprecedented data availability and accessibility, driven not only by incessant technology development, but also by increased demand from governments and civil society for more and more data and indicators to help formulate, measure, monitor and evaluate policies and decisions. An example of such demand is the set of Sustainable Development Goals Indicators – a framework of measures designed to monitor and evaluate progress in 17 different areas of human living conditions – defined by the United Nations Statistics Division.

A wide range of data and indicators is regularly produced by official and public statistics organizations to meet managers and decision-makers demands. Nevertheless, other data and indicators are demanded but currently not available, leading to substantial data gaps for many topics or areas of interest.

One increasing demand is the provision of statistics for smaller territories and/or with higher frequency. On one hand, this demand coincides with diminishing resources available to collect data, thus limiting sample sizes and increasing difficulties in producing the desired information. On the other hand, some existing data may lack the quality needed for its safe use in decision-making.

Since 2005, the Regional Center for Studies on the Development of the Information Society (Cetic.br), a department of the Brazilian Network Information Center (NIC.br), has collected data about access, use and appropriation of information and communication technologies (ICT) in several segments of society. The Survey on the Use of ICT in Brazilian Households (hereafter called the ICT Households survey) [1] measures the access and use of ICT by household residents in Brazil. With annual samples of close to 33,000 permanent private households, this survey is designed to produce reliable estimates for five Brazilian regions (North, Northeast, Southeast, South and Center-West). These regions represent macro divisions of the country that are made up of states (27 in total). Each state is formed by municipalities (cities and towns), which are the smallest formal government administrative units in Brazil (there are 5,570 municipalities).

Users of the ICT Households survey have been pressing for publication of the survey’s main indicators by the 27 Brazilian states. However, the precision of direct design-based estimates is low for many states, and such direct estimates are considered inadequate for general dissemination.

The aim of the present study was to find a simple method that allows the dissemination of ICT Households indicators by Brazilian states every year. Because the ICT Households survey produces a wide range of indicators and considering the tight timeline available for publication of survey estimates after data collection is completed, no explicit model-based approaches were considered. This decision is justified because model-based estimators, while potentially useful, require careful model fitting and checking to ensure that the resulting estimates meet the quality standards specified for the survey.

To assess feasibility, we applied four simple small area estimation (SAE) methods to obtain state-level estimates for two selected indicators from the ICT Households survey: proportion of households with computers, and proportion of households with Internet access. Details about the definitions of these target indicators are provided in [2] and in Section 2.1. We used data from two survey editions, 2014 and 2015, to produce the estimates in this article. The direct estimates are presented in Tables 3 and 4. The four estimation methods considered were: average of consecutive years – see [3]; pooling samples of consecutive years – see [3]; a single-year composite estimator considering the regions as yielding synthetic estimates; and a composite estimator based on pooling samples from two consecutive years, and using the regions as yielding synthetic estimates – see page 57 on [4]. The estimates derived from the various methods were evaluated using estimated mean squared error (MSE), and by comparing them with similar estimates produced by the 2015 edition of the National Household Sample Survey (PNAD) of the Brazilian Institute for Geography and Statistics (IBGE).

The PNAD was conducted to investigate socioeconomic characteristics of the Brazilian household resident population [5] and its main focus is the estimation of traditional labor force indicators. However, some questions on living conditions are also investigated, including a list of durable goods and services available in the household – including computer and Internet access – thus enabling estimation of a few ICT indicators. PNAD’s yearly sample of more than 150,000 permanent private households enables dissemination of reliable estimates for the 27 Brazilian states. The PNAD survey was discontinued by IBGE after the 2015 edition, and it was replaced by Continuous PNAD [6]. It is important to highlight that the ICT Households survey has many more ICT indicators than PNAD.

This article is divided into five sections. Section 2 describes the data sources. Section 3 presents the methods applied. Section 4 presents the results of the estimates and their analysis. In Section 5, we present the conclusions.

2.Data

The ICT Households survey utilizes a stratified multistage sample carried out independently every year. That is, a completely new sample is selected annually, which leads to the selection of different households. The target population is permanent private Brazilian households and their residents who are 10 years old or older. Detailed sampling design and weighting procedures for the ICT Households survey can be found in [1].

The frame used to design and select the ICT Households survey sample, which is the same as that used by PNAD, contains about 316,000 enumeration areas covering the whole country and is a product of the 2010 Brazilian Census. An enumeration area is defined by IBGE as a very small territory established for control purposes, such that the size of the territory and the number of households in each enumeration area should enable census data collection to be carried out by one interviewer within one month. Every enumeration area is contiguous, located entirely within the urban or rural portion of the territory of a single municipality. Each municipality is formed by a combination of enumeration areas. Also, each municipality is embedded in a state, and the union of states forms the regions that in turn make up the whole country.

The stratification of the ICT Households survey is planned to ensure precise results for estimates by region (5 domains). The municipalities are grouped into 36 strata that are formed by each state, and in nine states (Pará, Ceará, Pernambuco, Bahia, Minas Gerais, Rio de Janeiro, São Paulo, Paraná and Rio Grande do Sul), the state-level groups are subdivided into ‘capital city and its metropolitan area’ and ‘non-capital cities’. The ICT Households survey considered the budget available, the desired quality of the measurements, and the size of the strata to empirically allocate the sample within strata. Due to cost considerations, sample allocation was larger for more populated areas and more accessible locations (e.g. state capital cities). Nevertheless, the survey estimates are expected to be unbiased given the use of probability sampling for all sampling stages and survey weights that reflect the disproportional sample allocation. In this study our aim was to obtain estimates for the 27 states and not for the 36 survey strata. The allocated sample size for each state is available in Table 1.

If a municipality is included in the sample with certainty – because it is a capital city or because of its large size (number of residents 10 years old or older) – then it becomes a stratum, and its enumeration areas are the primary sampling units (PSUs). If the stratum is formed by a collection of non-certainty municipalities, then the municipalities are the PSUs, and they are sampled using a probability proportional size (PPS) systematic sampling method – see [7] – where the size measure is the number of residents 10 years old or older. In both the certainty and sampled municipalities, the enumeration areas are also sampled by systematic PPS sampling where the size measure is the number of households.

Then, within each sampled enumeration area, the first operation carried out in the field is the compilation of an updated list of households to enable selecting the sample of households for interviewing. Households are sampled by simple random sampling without replacement (SRS). Selected households are then visited by the interviewers, who attempt contact and to obtain agreement to take part in the survey. For eligible households, the first step of the survey interview is the compilation of a list of the household residents. From such a household residents list, a single eligible resident is selected by SRS for interviewing. Therefore, all sampling stages are carried out using strict probability sampling procedures.

Field data collection is carried out by an outsourced contractor. The field work is closely monitored by NIC.br/Cetic.br, using a variety of methods, which include field visits to accompany selected data collection efforts, monitoring weekly and monthly control reports, and performing exploratory analysis of preliminary databases. In each of the years 2014 and 2015, 71% of the selected households answered the survey. Such similar rates result from the fact that the contractor providing the data collection services to the ICT Households survey has minimum household response rate targets to fulfill in each stratum. Although this is a contractual arrangement aimed at securing minimum levels of effort by the contractor, field work is carried out in strict adherence to the probability sampling protocols specified, and the response rate targets are used to monitor the efforts, not to change the sampling and contact protocols specified for the data collection.

The weighting procedure of the sample follows three steps:

The PNAD sampling design and methods are detailed in [5]. The PNAD survey target population is formed by permanent private Brazilian households and their residents. The PNAD survey design is also a stratified multistage sample of permanent private households and their residents in Brazil. The sample size for the PNAD of close to 150,000 households (Table 2), as mentioned above, is much larger than that for the ICT Households survey. The coefficients of variation for the estimated Internet indicators are not available by state, only for regions (Table 7). For the estimates of households with computer indicator CVs are available for both states and regions (Table 7). Analysis of these CVs indicates that the ICT indicators estimates have good precision. Unfortunately, no other measures of quality (such as R-indicators, bias assessment etc.) are available for the PNAD.

The PNAD survey collects fewer ICT indicators and uses questions that are somewhat different from those in the ICT Households survey which will be discussed below. Nonetheless, PNAD’s estimates for the two selected indicators were compared to the estimates obtained by the four methods considered here.

3.Methods

First, we present some concepts, definitions and notation, required to describe the four alternative estimation methods considered. Then we present the expressions for the point estimates, and finally, the approaches used for the mean squared error (MSE) estimation.

4.Results

The data on the ICT indicators were collected in PNAD using different questions when compared to the ICT Households survey. However, direct estimates by Brazilian region (see Tables 3 and 4) or for the whole country are not too different between the two surveys. Larger differences appear when comparing direct state-level estimates (see Tables 3 and 4).

Figure 1.

Comparison of state-level point estimates for computer indicator with 2015 PNAD.

Figure 2.

Comparison of state-level point estimates for Internet indicator with 2015 PNAD.

Figure 3.

State-level pooled sample composite estimates and 95% confidence limits for computer indicator.

Figure 4.

State-level pooled sample composite estimates and 95% confidence limits for Internet indicator.

Figure 5.

Comparison of root-MSE of SAE estimates for computer indicator.

Figure 6.

Comparison of root-MSE of SAE estimates for Internet indicator.

Regarding the computer availability estimates, for most regions (except the Northeast) the differences fall within margins of error. The ICT Households survey gives larger estimates for households with Internet access, because it considers the possibility of having Internet access in a household even if it is not accessed by a microcomputer (as is the case for PNAD). In order to evaluate if the differences exceed the margins of error, we included in Table 4 a ‘comparable’ estimate – it is marked with * – for the ICT Households 2015 Internet indicator in which we consider that there is Internet access just for households in which there is at least one computer. Considering this ‘comparable indicator’, the differences do not exceed the margins of error for all regions. This ‘comparable indicator’ is not used in the rest of the work, because ICT Households Survey adheres to the international recommendation (see [2]).

In contrast, the direct estimates by states are quite different, mostly for Brazilian states in the North and Northeast regions where the ICT Households survey sample is small due to the high cost of collecting data. The sample sizes are in Tables 1 and 2 and the estimates are in Tables 3 and 4.

Table 6 shows the ICT Households survey CVs of the direct state-level estimates and the R-indicators [13]. For some states the ICT Households available samples yielded small R-indicators, indicating that the direct estimates for these states are derived from imbalanced samples, something which is not captured when considering simple standard error or CV estimates. On the other hand, for some states the CVs for the direct estimates are high, leading to wide margins of error and large uncertainty in the estimates. CVs range from 3% to 31%, and R-indicators from 0.53 to 0.94. The smallest R-indicator was observed for the state of Amapá in the North region of Brazil, suggesting that the sample available for this state is not sufficiently representative.

These contrasting quality indicators (CV × R-indicators) led us to consider the approaches presented above to try to improve state-level estimates and evaluate the possibility of publishing such estimates for selected indicators using the ICT Households survey as the source. The decision about the best procedure to use was made by comparing the median of the square root of the MSE estimates for the four methods (see Table 8).

The direct estimates have smaller MSE estimates, but these did not include a bias component, which in many cases would be needed, given that the corresponding state R-indicators suggest that there are important differences between the available sample and the population. Hence, the composite of pooled samples method performed best among the approaches considered, when the direct estimator is left out.

In addition, we plotted the state-level estimates for the chosen method and the direct estimates against PNAD’s estimates to evaluate the patterns (see Figs 1 and 2). We clearly see the large difference for Amapá state. The composite of pooled samples estimate is in line with PNAD, whereas the direct estimate for this state is quite far off.

The pooled-sample composite state-level estimates plus their empirical 95% confidence intervals for the proportion of households with computers and for the proportion of households with Internet are shown in Figs 3 and 4 respectively. Margins of error are small (less than 5%) or modest (5 to 10%) for most states, and generally smaller for the proportion of households with computer. Survey analysts consider such estimates adequate for public dissemination.

5.Conclusion

According to the R-indicator, six state samples from the ICT Households survey appear to represent poorly the corresponding state populations (see Table 6). Apparently, our pooled-sample composite estimation reduced the representativeness problem, but at the cost of modestly increased root-MSE (Figs 5 and 6). However, our root-MSE estimates for the direct estimator include rather naïve estimates for its bias, and hence, we consider the pooled-sample composite estimator root-MSE estimates as more reliable.

Pooling samples and averaging data for consecutive years increases the sample sizes at the cost of adding a time-trend bias. Our choice is to disseminate the estimates as regarding a two-year period, in a similar way as is done by the American Community Survey, for example. Also, for some states this did not compensate the representativeness problem. On the other hand, composite estimation includes a bias originating from the synthetic ‘regional’ estimator.

The pooled-sample composite estimator borrows strength across both time and space. The gains in precision and in the comparison with the point estimates from PNAD with this estimator are sufficient to enable dissemination of state-level estimates. Analyzing Figs 1 and 2, we see that the estimates from pooled-sample composite method are closer to the PNAD estimates for all states in comparison to all other approaches, including for Amapá state, which was the state with lowest R-indicator (Table 6). This can be confirmed in Table 9, which shows the correlation between state-level PNAD and ICT Households survey estimates obtained by all estimation methods considered. The correlation between direct ICT Households and PNAD estimates is equal to 0.572 for computer and 0.447 for Internet. For the pooled-sample composite and PNAD estimates the correlations are 0.887 and 0.864 respectively.

Hence, the pooled-sample composite estimator will be adopted to calculate the state-level estimates for the various indicators from the ICT Households survey, that will be disseminated to the public. Since NIC.br/Cetic.br data users have requested regular publication of state-level estimates for ICT indicators, we suggested the adoption of a sample-design that enables faster accumulation of sampled PSUs to obtain more reliable state-level estimates using the proposed estimation approach. As an extension of this work, we will study the possibility of producing estimates for states, following this methodology, but benchmarking to the estimates for regions averaged over two consecutive years.