Positioning household surveys for the next decade

2.1.1Improving accessibility of other data sources for integration

Accessibility of data outside of NSOs for integration has been challenging for many. One of the biggest challenges faced by countries experimenting with the small area estimation of development outcomes such as consumption-based poverty is the lack of administrative data sources that can be considered for integration with household surveys [11]. NSOs in low- and middle-income countries often rely on census-survey integration in small area estimation efforts, as neither administrative data nor non-traditional data are available to them. Strong legislative backing would be helpful to allow NSOs access to other data sources for official statistical purposes, while also addressing privacy risks that are heightened due to data integration [12]. In countries where administrative or nontraditional data sources can be acquired, a cost-sharing schema among multiple surveys within the national statistical office could help reduce the cost burden for individual surveys. Another option for consideration is shifting from sharing data to sharing computation, given the sensitive nature of personal “nano-data” [13]. This shift could potentially limit the level of risk in data privacy breaches while also reducing burden on IT infrastructure.

2.1.2Fostering data interoperability by design

Integration across different data sources demands data interoperability by design. Interoperability is a key requirement for data to be valuable for development and relates to the ease with which data sources can be linked and integrated through geospatial coordinates, common questions, time stamps, common classification standards, or common identifiers for persons, facilities, or firms, among others [14]. Given governments’ budget constraints and the need for more granular data, many countries are pursuing greater integration across different data sources – between censuses and surveys, exclusively among surveys, and between surveys and other non-census/survey data sources.

Examples of survey data integration include: (1) small area estimation methods, which typically integrate survey data with censuses and administrative data source and have been used for measuring consumption-based poverty, mortality, labor force participation, disability, and other areas of the SDGs [15, 16, 17]; (2) survey-to-survey imputation to improve the availability, quality, timeliness, and cost-effectiveness of official statistics, which have been used routinely in applications of proxy-means testing and evaluation of subnational project impacts, typically focused on poverty measurement [18, 19, 20]; and (3) linking individual records of household surveys and administrative records to reduce response burden [6].

There have also been known examples of integration of household surveys with satellite imagery and processed geospatial data for high spatial and temporal resolution estimates of agricultural outcomes, such as crop-specific measures of area under cultivation, production, and yield, which emphasize the collection of precise GPS-based plot outlines [21] and a range of complementary data collected on the ground through objective measurement approaches, such as crop cutting, implemented at scale (i.e., across the entire survey sample) or on a sub-sample basis [22, 23].

Successful data integration by design depends on a number of important elements. Concepts and definitions should be harmonized between the household survey and the other source for integration [24], which could be a population census, a different survey, administrative data, satellite imagery, processed geospatial data, or citizen-generated data [25]. This harmonization should also extend to the list of administrative units used in the data sources. Moreover, common auxiliary variables that can explain a large share of the variations of the outcome indicator should be included in both data sources to improve the efficiency of data integration.

Similarly, if feasible, administrative identifiers for individuals, firms, and facilities should be elicited as part of household surveys to enable unit-record linkages of household surveys with administrative data sources. This can serve multiple purposes, including (1) assessing the quality of survey data by leveraging identical information contained in administrative data, (2) providing a basis for calibration or weighting, or (3) populating certain survey variables (for example, linkages to taxation databases obviate the need to ask extensive questions on income in countries with well-developed and comprehensive administrative sources).

Relatedly, georeferencing should be adopted in household surveys for facilitating the validation and calibration of machine learning models that combine georeferenced survey data with publicly available satellite imagery and processed geospatial data to derive precise estimates of poverty, asset wealth, and agricultural outcomes at high spatial resolution [26, 27, 28].

Privacy risks need to be considered for data integration for two main reasons. First, linking datasets could increase the risk of disclosure. Proper measures should be taken to protect the confidentiality of individuals when disseminating microdata. Second, requesting administrative identifiers may discourage respondents from participating in the survey – this potential risk must be taken into account.

Sampling design for the household survey should also facilitate data integration. When survey-to-survey imputation is being carried out, the samples for the two surveys should strive to have similar designs [29]. Well-designed and high-quality surveys have been successfully used to correct biases from non-probabilistic survey data in several countries [30, 31]. Relevant paradata should be collected and curated for calibration to correct the selection biases and measurement errors generated from citizen-generated data and other types of non-probabilistic data sources [32, 33].

From an institutional standpoint, moving from domain-oriented to process-oriented household survey operations should be considered if the national statistical office is in the process of modernization. Under the domain-oriented approach, an independent team is in charge of the entire process of survey operations for a specific survey (e.g., labor force surveys, living standards surveys, health surveys, etc.), from planning and collection to processing and dissemination. On the other hand, the process-oriented approach establishes units that are in charge of different steps of survey operations, regardless of the type of surveys. For example, one unit would be in charge of methodology and sampling, another unit would be in charge of data collection, and so on [34]. The advantage of this approach is that different sub-processes of household survey operations are coordinated and standardized. For example, the questionnaire design unit ensures that the same definitions and classifications are used for the same variable across surveys. Such an approach improves coordination and standardization across surveys, while also making the surveys more interoperable and efficient. However, for countries running limited numbers of surveys, the resulting gains in efficiency are unclear. More information about this approach is covered in 3.1.2.

2.1.3Establishing a total quality framework for data integration

Although the “total survey error” (TSE) framework identifies each source of error for household surveys [35], other data sources come with their own quality issues, including potential biases and measurement errors for which a similar total error framework is usually absent, except perhaps in the case of administrative data used for statistical purposes [36]. More work is needed to quantify the errors associated with non-traditional data sources such as earth observation data and citizen-generated data, as well as the errors produced during the integration process [10, 37, 38]. For planning any types of data integration, the quality of all input data (surveys and other data sources) should be assessed, in terms of coverage (e.g., sufficient representation of key population groups), timeliness (e.g., regularity of data availability), and measurement errors (e.g., accuracy of data).

Data integration methods, including record linkage and model-based estimates, also involve errors including data linkage error or, for model-based estimates, errors due to the violation of assumptions or poor input data. For model-based estimates, validating model assumptions is an unavoidable step that requires additional time and resources. Estimating the mean square error of estimators is another important aspect that should be further researched and developed. A further aspect of concern is the extent to which the target variable matches the phenomenon captured by the administrative data source – this should be adequately assessed and transparently reported as a key element of data quality. Therefore, a data quality framework for integrating surveys with other data sources needs to be systematically developed.

2.1.4Maintaining high ethical standards and data confidentiality

Data integration increases the risk of data breaches and misuse. Conversely, limited access of certain data sources can hamper data integration. In the first instance, there can be important barriers to access by NSOs, often related to data protection concerns, and strong legal frameworks and institutional arrangements are required to enable access under appropriate conditions. Furthermore, in generating public use datasets, personal identifiers (that allow linkages with administrative records) or precise GPS coordinates (that enable integration with satellite imagery and other georeferenced and processed geospatial data) are considered confidential and are excluded (in the case of the former) or anonymized (in the case of the latter).

While international standards and analytical tools are available for the deidentification of household survey data [39], the risk of disclosure is increasing with enhancements in data interoperability, requiring continued improvement of deidentification techniques and strengthening of NSO capacity to successfully adopt these standards and analytical tools [40]. Recommendations on providing increased access to microdata are provided in Section 2.8.

Another important aspect to consider is related to ethics and respondents’ right to be informed about and are fully aware of the use of their own data, including data integration. For example, when Statistics Canada replaced income survey questions with data from tax records, respondents were informed about this practice during the interview. Addressing respondents’ concerns over the use and protection of their data and the potential benefit of collecting those data is an important issue for consideration [41]. In Zambia, for instance, an ethical clearance is required for data collection, which involves a signed or thumb printed consent form by respondents to demonstrate that they understand the purpose of the survey, how the data will be used, and their rights during the interview.

2.2.1Transforming respondents into collaborators and co-producers

Respondent participation in household surveys largely depends on three elements: trust and trustworthiness, willingness to participate, and accessibility of data collection [52]. Building trust requires a rethinking of the relationship between NSOs and survey respondents, shifting from viewing participants as respondents, to viewing participants as collaborators and co-producers. This runs the gamut from designing surveys that are clear and simple, to being responsive to problems, concerns, and questions, to making surveys more inclusive by bringing respondents with diverse needs and abilities into the survey research and development process [53, 54, 55].

Constraints to willingness to participate in the data collection could include respondents’ inability to relate to survey topics or questions, exhaustion from over-research (respondent fatigue), and competing pressures from other daily activities, among others. While relevant cross-country empirical evidence is not immediately available, on the whole, the extent and drivers of nonresponse are expected to vary across and within countries (both across topics and geographic areas). As such, the issue needs to be studied in each context and various solutions should be adapted and piloted accordingly. The large consultative exercise of the United Kingdom Inclusive Data Task Force [52] is one example of how to make collected data more inclusive, by soliciting input from a wide range of stakeholders. Examples within national statistical offices also include tailoring survey design to meet the needs of respondents, collaborating with behavioral economists and communication specialists, and asking direct questions about response burden to help survey implementers better understand and reduce nonresponse [57].

2.2.2Minimizing the reliance on proxy respondents to improve quality of data

High-quality, individual-disaggregated data that accurately reflect the economic and social roles and choices of men and women are critical for a variety of purposes, including: (1) the targeting and evaluation of policies to provide social protection for raising living standards and mitigating against shocks; (2) promoting access to and ownership of physical and financial assets; and (3) removing barriers to technology adoption, to name a few. Similarly, a clearer picture of the intra-household distribution of labor – across sectors, wage- or self-employment activities, and unpaid care and domestic work – can better inform the targeting of employment and training programs. Furthermore, monitoring progress towards several targets of the SDGs across poverty reduction, agriculture, gender, employment, and inequality, require individual-disaggregated data on asset ownership, labor, time use, and roles in family enterprises.

Household surveys are one of the most promising sources of individual-disaggregated data to analyze these issues and their interactions. However, their reliability and usability are mediated by questionnaire design choices and respondent selection protocols. Regarding the latter, it is common for individual-level household survey modules to allow for proxy respondents to report on behalf of adult household members – a measure that cuts costs and avoids missing information. On other topics, such as asset ownership, it has been common for household surveys to either collect information at the household-level (even when assets are owned by individuals) or to identify intra-household asset owners but elicit information from a single, “most knowledgeable” household member [48, 49].

Momentum has been increasing to improve the availability, scope and quality of individual-disaggregated survey data collected in household surveys on a range of topics including asset ownership, work and employment, time use, and violence against women. Through the formulation of international guidelines on these topics, with a focus on improved approaches to questionnaire design and respondent selection, efforts to promote their adoption in large-scale surveys, and research that has demonstrated the utility of intra-household, self-reported survey data vis-à-vis data that are collected based on sub-optimal respondent selection protocols, countries now have an expanding base of knowledge and experiences to draw from in minimizing the reliance on proxy respondents to improve the quality of data on men and women.

Moving forward, there is a need for NSOs to (1) be more systematic in tracking the reliance on proxy respondents in their survey operations, (2) be critical about their fieldwork implementation protocols vis-à-vis existing international guidance for maximizing rate of self-reporting among adults, (3) draw on documented experiences in improving their approach to interview scheduling with adult household members to minimize the reliance on proxy respondents, particularly for sensitive topics; and (4) to be supported, particularly in lower-income contexts, in the adoption of best practices. Having said that, proxy response may be unavoidable if adult household members cannot be interviewed due to advanced age, poor health and/or temporary migration away from household residence. As such, it can be allowed as a last resort as NSOs put a greater emphasis on the collection of self-reported information from adult household members. And reliance on proxy response will likely continue being the dominant approach to data collection from non-adult household members.

Finally, getting better individual-disaggregated survey data may require additional financial resources, mainly to allow additional time for interviewers to schedule and conduct private interviews. As such, the approach to costing household surveys and securing the required financial resources may also need to be revisited.

2.2.3Improving the correction of nonresponse bias

While calibration and imputation have been widely used by survey organizers to reduce nonresponse bias, two additional approaches could result in its further reduction. The first is to invest in high-quality benchmarking data sources, that is, high-quality data sources (in terms of measurement and representation errors) with auxiliary variables that have prediction power for the outcome indicators [58]. The second is to enhance the collection and use of paradata during electronic data collection, as discussed in Section 2.6. Survey Solutions, for instance, is an example of a CAPI/CATI/CAWI platform that automatically collects and allows the users to download the paradata associated with each survey. While the type of paradata available for nonresponse bias correction varies by mode of data collection, in general, three types of paradata can be used for this purpose: (1) call history data, containing information on interview attempts and outcome of each attempt, (2) interviewer observations of the sample units, and (3) measures of the interviewer-householder interaction [59]. More research and experimentations are needed to better understand how paradata, which may also come with its quality issues such as missing, incomplete and inaccurate information, can be better used to reduce nonresponse bias.

2.3.1Improving sampling frames for household surveys

The most common sampling frame comes from population censuses, through the area frame that contains hierarchical geographical areas from the largest area (at the national level) to the smallest geographic division, usually called enumeration areas (EAs) and a list frame that contains the list of households located within each EA. Address-based sampling frames have also started to gain ground in high-income countries given their efficiency and quality. The addresses are usually obtained from a commercial vendor and updated regularly, and important auxiliary variables are available to help improve sampling efficiency [60]. For countries that do not have the resources or capacity to maintain a comprehensive list of addresses, a master sample frame is often used. With a master sample frame, the address list is updated only for selected enumeration areas. In Brazil, for example, the master sample frame has been used by all household surveys in the country. Master sample frames allow for the cost-sharing of listings, better knowledge of selected areas, and opportunities for richer data analysis [61].

For the 2020 round of censuses, many countries plan to use telephones to follow up with respondents to address missing values and nonresponse [62]. For countries without a good telephone frame for phone surveys, phone numbers collected during censuses (or surveys, as discussed in Section 2.5) can be used for subsequent surveys. However, this requires the following of strict protocols and consent from respondents. More information about obtaining consent is available in Section 2.5.

When a survey aims to sample hard-to-reach populations, the use of multi-frame sampling can improve the cost efficiency of the overall survey and improve the inclusivity of survey data. For example, an epidemiology survey could use an area frame for a general population health survey alongside another list frame of clinics specializing in a certain disease. This method allows for capturing data from a higher number of populations with this specific disease, while reducing the cost of screening to be carried out in the area frame [63]. Multi-frame sampling can also be helpful in post-disaster settings when many respondents are displaced. For example, sampling school districts was found to be an efficient method for reaching families who relocated in the aftermath of Hurricane Katrina [64]. A new project has recently started in the United States that will integrate the current master address frame with the business register, job lists, and the demographic frame. This integration will help reduce response burden, improve collaboration across different agencies, and improve coverage, especially for vulnerable and difficult-to-reach populations [65]. It is important to note that use of multiple frames, and the associated multiplicity weighting are complicated and more challenging for survey organizations to carry out. Guidance and training would be required for the multi-frame sampling to be fully adopted by countries.

Moving forward, efforts should be made to ensure census records are geospatially enabled, i.e., geocoded to a specific location [66] – this can facilitate selecting samples for household surveys as well as data integration (as discussed in 2.1). In a 2019 survey carried out by the UN Statistics Division of 158 countries, 86 percent of NSOs indicated that as part of the 2020 round of censuses, they either have collected or will collect GPS coordinates for enumeration areas, while 70 percent indicated that they will collect GPS coordinates for buildings and housing units [62].

For countries that lack census records for EA selection or have an outdated census frame, high-resolution satellite data can be used to generate estimated population densities and demarcate EA boundaries [67]. For example, the last population census in Somalia was carried out in 1975 with a population count of 3.9 million. Given significant increases in population size (up to an estimated 15 million in 2019) and high levels of displacement within the country, a gridded population approach was developed to create a frame for the first selection stage of the 2017 Somalia Rapid Emergency Response Survey. Geographical areas of Somalia were divided into 100 by 100-meter grid cells and neighboring cells were combined to form primary sampling units (PSUs) [68, 69, 70]. A sample of PSUs was selected using probability proportional to estimated size, with the population figure obtained from WorldPop. Relatedly, geospatial data can also help build equal size EAs to improve field work management and sampling efficiency [72].

2.3.2Adopting innovative sampling methods for difficult-to-sample population groups

Under the SDG pledge to leave no one behind, there is an expectation that surveys should cover various vulnerable population groups, of which many are difficult to survey due to various challenges. These include difficulties in identifying certain populations due to stigma and sensitivity, in locating and accessing certain population groups that are small in percentage or dispersed, and in persuading populations to be interviewed. The need for self-reported (as opposed to proxy-response) data at the individual level, crucial for study from a gender perspective, is also a challenge in household surveys.

There are many ways to improve the measurement precision of certain population groups. For rare populations such as ethnic minorities, over-sampling areas that have more concentrated minority groups can improve sample coverage and reduce costs for expensive screenings. Another option is network sampling, which uses an expanded sample screening process so that information is also obtained from others outside of the household, such as neighbors, relatives, and other connected individuals. This can maximize coverage when collecting data on rare events or sensitive topics. The method has been used to produce estimates on adult mortality and on population groups that are marginalized and face social stigma, such as those using drugs [74]. Another interesting example is using machine learning models to predict ethnicity and assist with more targeted sampling, as covered in 2.7.

For countries that are interested in covering particular population groups, various methods to improve sample coverage should be attempted, in connection with an updated and comprehensive sampling frame. Guidance on sampling vulnerable population groups is currently being compiled by the ISWGHS through a Wiki site [75].

2.3.3Applying responsive and adaptive sampling design

Responsive and adaptive sampling design is an evidence-based approach for guiding real-time design decisions during survey data collection, which takes advantage of advances in electronic data collection such as the availability of geospatial information and paradata. One experiment conducted as part of the 2009 Swedish Living Conditions Survey explored various adaptive survey design strategies, such as terminating data collection as soon as the response rate reaches a certain threshold. During the process R-indicator was used to assess the balance of the set of respondents defined by key characteristics and the distance between respondents and nonrespondents. The results showed that the design reduced data collection costs through significantly fewer call attempts [76]. Another example comes from Nigeria, where a population-based HIV survey adopted a responsive survey design that used paradata to expedite the survey data collection and release [77]. The overall data collection duration was reduced by 1 week from the original plan and saved about $4.4 million in costs.

2.5.1Building sample frames for phone surveys

One of the biggest challenges for telephone interviewing faced by NSOs during the pandemic was the lack of contact information in the sampling frame [106]. In a compilation of national COVID-19 impact surveys maintained by the ISWGHS, only 43 percent of approximately 180 countries used a recent household survey as a sampling frame for telephone interviews; the remaining countries lacked an updated sample frame with telephone numbers [107]. Countries that did not have contact details to reach survey respondents during COVID-19 adopted various methods to obtain the phone numbers, such as through mobile phone service operators, random digit dialing, or using administrative data sources like population and electoral registries. For example, Mongolia used its newly updated household-based registry, which contains one or more phone numbers, to reach respondents sampled in the 2020 MICS Plus [108].

Countries that have long relied on computer-assisted telephone interviewing (CATI) and computer-assisted web interviewing (CAWI) for their official surveys can serve as models for maintaining an enabling survey infrastructure for remote data collection. A recent EU workshop on multi-mode data collection for labor force surveys [109] showed that the most common data collection mode for labor force surveys within the European Union was a combination of CAPI (mostly for the first round) and CATI (for following rounds). Contact information is obtained during the first round. The labor force survey in Canada uses either in-person interviews or CATI (if the phone number is available from administrative files) for the first round, and CATI and CAWI for subsequent survey rounds. Only a few countries, unsurprisingly those with comprehensive registration systems, rely exclusively only on CATI and/or CAWI for data collection.

Relying on face-to-face household surveys as sampling frames for phone surveys implemented during the COVID-19 pandemic revealed the advantages of this approach in minimizing household-level coverage and nonresponse bias, albeit with limits [110, 111]. Going forward, contact information for household members may be elicited in all future face-to-face surveys, which can in turn be used as sampling frames as well as to correct bias [46]. While longitudinal face-to-face surveys routinely collect this type of information to facilitate tracking efforts, cross-sectional surveys should also more systematically collect phone numbers. These efforts should be coupled with revisions to privacy and consent agreements with face-to-face household survey respondents, given the potential for the individuals to be contacted for other surveys at a later time. Countries should also consider collaborating with private data providers such as telecommunication service providers or research institutes with CATI experience to obtain access to data needed for building relevant sampling frames.

2.5.2Developing phone and web survey tools and protocols

Strong phone and web survey infrastructure must be coupled with required survey tools and protocols. For example, phone questionnaires need to be significantly shorter and simpler, given the challenges in keeping respondents engaged during remote data collection, issues with mobile network connectivity, and concerns about respondent fatigue. The flow of questions and visual cues for phone and web questionnaires also vary significantly from face-to-face questionnaires. Meanwhile, protocols should be established for respondent selection, incentive provision, phone and e-mail contact attempts to recruit respondents, proper consent and ethical requirements, and the formulation of scripted introductions and transitions during the interviews. These will be critical for successful survey implementation, as well as to ensure the general representativeness of the data.

When adopting a mixed-mode survey design that includes CATI and CAWI as possible options, NSOs need to decide whether the choice of CATI versus CAWI would be offered initially or at a later time. If the latter, NSOs will need to determine the initial survey mode and the number of days to be allowed before the second mode can be offered. Decisions must also be made on the number of reminders to the respondents and their frequency [112]. When moving towards a mixed-mode system, every decision should be tested, and investments made to develop a survey management system that can support the complexity of mixed-mode data collection. A strong IT system to support the mixed-mode data system is critical [113].

2.5.3Conducting more systematic analysis of mode effects

Phone surveys carried out during COVID-19 raised many concerns of quality, selectivity, and coverage, inter alia. Given increased reliance on phone and web surveys, users must understand the relative accuracy, reliability, and affordability of these surveys vis-à-vis their face-to-face counterparts, ideally through survey experiments that randomize the mode of interview for the same types of questions included in different surveys and questionnaire instruments. These experiments to discern potential survey mode effects can be conducted under the suggested program on survey methods discussed in Section 3.1.f. However, it is understood that even with identical questions, face-to-face, phone, and web survey questionnaires will exhibit generally differences in terms of length and design choices, again as part of a respondent-centric implementation approach. More information on assessing the quality of surveys carried out during the pandemic will be available in the forthcoming ISWGHS publication [114].

3.1.1Strengthening engagement with policymakers and data users

Official statistics are collected to inform policy, promote policy discussions, and increase knowledge. As an integral part of the national data ecosystem, household surveys offer a unique opportunity to respond to the data needs of policymakers and the general public. To ensure that data collected from household surveys are relevant, policymakers and all relevant stakeholders (including marginalized population groups) should be key partners at all stages of survey planning, data collection, analysis, and dissemination. These engagements build co-ownership of data and the entire household survey process with policymakers, which in turn helps to secure financial support for household survey operations in the country.

Various ways of engaging with policymakers and key stakeholders have been adopted in countries. For example, the Canadian Statistics Advisory Council serves as a body for Statistics Canada to engage with the ministries and key experts on matters related to overall quality, including issues related to data collection, data access, privacy issues, and data dissemination. To better understand the needs from regional and local governments, the Federal-Provincial-Territorial Consultative Council on Statistical Policy collaborates with Statistics Canada to determine data requirements, consult on current statistical activities, and coordinate data dissemination [135].

Another way to connect with the public directly is through open consultation. For example, open consultation was carried out in 2020–21 by the Australian Bureau of Statistics (ABS) about the Aboriginal and Torres Strait Islander Health Survey [136]. The consultation helped ABS design a culturally appropriate approach to collect information on health with three components: survey content, biomedical tests, and data integration. The public was asked to complete an online survey and/or submit inputs through email or mail. A report of the consultation is published online. Topics for open consultations are broad, ranging from data collection as in the above example for Australia, data dissemination, or the use of data. Surveys supported by the World Bank Living Standard Measurement Study (LSMS) team establish a Technical Working Group within each country, including NSO officials, line ministries that use the survey data, think tanks, research centers, and academia. The Data Users Group advises on questionnaire design to align with policy needs and increase the use of survey data.

On the whole, strong user engagement can identify not only the data needs on the part of stakeholders but also the ways in which household surveys are used. The latter is critical for identifying the types of household survey data that are needed for ongoing country policy design and evaluation processes.

3.1.2Modernizing national statistical systems

In recent years, many NSOs have taken on the task of modernizing their statistical systems. Drivers for this transformation are both internal and external. Internal drivers may include organizational silos that prevent the reuse of knowledge and data produced under different streams of work, duplication, lack of consistency of solutions, limited interoperability across data sources, and limited capacity for research and innovation. External drivers for modernization include increasing demand for statistical information, the availability of new data sources and methods, and increasing challenges with the traditional data sources that are typically under the auspices of NSOs [137]. Attempts to introduce innovations into household surveys would benefit greatly from a modernized national statistical system. For example, a process-oriented data collection system (discussed in more details under 2.1), which is a key element of a modernized statistical system, can help ensure harmonization across surveys and facilitate data integration.

Another key aspect of a modernized statistical system is an expansive approach to new sources of data, relying on both primary data collection such as population censuses and household surveys as well as on other data sources such as administrative sources and non-traditional data sources. This transformation serves as a catalyst for innovative survey methodologies and better data integration. The shift to multi-purpose sources and multi-source statistics also ensures that the collected survey data are re-used for multiple purposes, hence increasing the value of existing surveys [13].

3.1.3Quantifying the benefits and communicate the value of surveys

National statistical offices must invest in data visualization and data journalism to better communicate the value of household surveys, both in and of themselves and through integration. This is critical for boosting the understanding of the importance of household survey data, and in turn, securing political commitment to and predictable financing for household survey programs. It is also important for NSOs to document how survey data have been used for policymaking, to further demonstrate the value of household surveys.

In an age of abundance of information and misinformation, it is important to develop and maintain a brand for NSOs that is associated with trust, relevance, independence, and quality. Consumer consultations, staff engagement, and communication and marketing strategies are key elements for building such a brand [138].

3.1.4Sustaining financing for household surveys

During the COVID-19 pandemic, 40 percent of NSOs saw data collection costs rise, while 48 percent of NSOs globally experienced decreased government funding [139]. In sub-Saharan Africa, 61 percent of countries experienced increases in data collection costs, with 71 percent seeing a decrease in government funding and 59 percent a decrease in donor funding. These challenges are being experienced by systems that were already insufficiently funded yet are required to produce accurate and timely data. An analysis carried out by the World Bank puts the average cost of conducting a face-to-face household survey, based on a sample of 18 living conditions surveys, at USD 170 per household [140]. Significant variations exist in the cost among these 18 surveys, ranging from USD 64 in Bangladesh to above USD 400 per household in Nigeria. The cost of a national MICS6 survey ranges from USD 29 to USD 370 per household [141]. As we move forward, the cost of household surveys is likely to increase, and many low-income countries will continue to lack the resources to consistently fund their national surveys.

Domestic resource mobilization remains the most sustainable funding resource for statistical advancement. At the national level, systematic funding mechanisms should be in place to support household survey operations as an integral part of the national statistical system. There must be strong statistical advocacy programs in place that can enhance the work done by NSOs for the benefit of politicians and policy makers to ensure that household surveys are priority activities in annual budgets. Demonstration of use cases may be helpful in resource mobilization. In view of ever-present budget constraints and diverse NSO operations and programs that are seemingly in competition for resources, household survey design should be tightly fit-for-purpose such that NSOs could maximize the cost-effectiveness and relevance of survey data production.

Furthermore, public-private partnerships should be encouraged by national governments as a means to drive funding toward statistical development. Special trust funds for the sole purpose of advancing statistics could be established on already existing tax revenue platforms.

The last two years dealt a major blow to the world economy and lower-income countries were hit particularly hard. Funding statistical development must continue being a priority for national governments and the international community if economies are to rely on evidence-based policy responses to emerge from the downturn.

3.1.5Strengthening the capacity of national statistical systems

Quality official statistics can only be offered by a national statistical system that has the requisite managerial and technical skills to deliver on its mandate. Training and capacity building programs can improve on existing skills and help develop new technical skills that can assist NSOs to leverage new types of data sources and new methods.

As seen in the discussion from Chapter 2, the skills mix of NSOs must diversify to increase the development impact of household surveys, expanding to areas such as the use of new data sources (e.g., earth observation data), new techniques (such as machine learning, artificial intelligence, and anonymizing survey microdata), and data integration. Research skills need to be developed to support methodological innovation, including for instance in the use of latent variable models that cater to measurement errors, and communication skills need to be improved to better engage with respondents and users. Such improvements can be undertaken together with establishing a new (or strengthening the existing) business line on experimental statistics, as discussed in Section 3.1.f and as originally called for by the World Development Report 2021: Data for Better Lives [142].

Furthermore, a training needs assessment of 15 NSOs identified additional areas in high demand but often overlooked by statistical trainings programmes, including coordination of the national statistical system, user engagement, and management as some of the key priorities [143]. These issues are particularly relevant for countries as they begin modernizing their national statistical systems (see Section 3.1. Modernizing national statistical system) and as NSOs take on new roles in an Integrated National Data System [142].

There is substantial variation in NSO capacity and in the extent to which national statistical systems provide training for their staff. National training programs range from having an established statistical training institute within or outside of NSOs (as in Brazil, Indonesia, and the Philippines), relying on a small training unit within the NSO (as in Ireland and Ethiopia), to providing only ad-hoc training or no training at all [144]. There is also significant variation on the type of trainings provided to staff as well as their supervision and oversight. Good practices from successful national statistical training programs should be shared and expanded to others.

The challenges of COVID-19 have prompted many statistical agencies, at both international and national levels, to rethink their training programs. At least 75 percent of all statistical capacity development events in 2020 were conducted online, compared with only about 5 percent in 2019, according to the United Nations Statistics Division Global Calendar of Statistical Events, which includes information from major international agencies. Given its efficacy, remote training is likely to continue, even if combined with in-person initiatives [145].

Many e-learning courses have been developed by international agencies on various topics such as phone surveys [146], small area estimation [147], and collecting data through household surveys for various SDG indicators [148]. Bringing these courses together to maximize access for NSOs and to avoid duplication of efforts is essential. The Global Network of Institutions for Statistical Training (GIST) [149] has established a hub that is intended for this purpose: the UN SDG:Learn Statistics Hub [150], which currently holds more than 70 e-learning courses on various topics from different providers, including FAO, UNICEF, UNSD, the World Bank, regional training institutes, NSOs, and others. More courses are in the process of being added, including from the DHS team. The hub also provides micro-learning materials such as brief learning videos, platforms, and blogs. Similar survey-related training materials are also being made available on the ISWGHS website [151].

3.1.6Fostering a program of experimental statistics

This position paper echoes one of the recommendations of the World Development Report 2021 [142] for NSOs to establish a business line on experimental statistics under which a more systematic approach to supporting and conducting methodological survey research can be pursued. In this context, a business line could be established for experimental statistics that use new data sources and new methods to improve methodologies and/or to provide more timely data to better meet user needs.

While experimental statistics are not disseminated in the same way to users, as they are still in the testing phase and not yet fully developed, they nonetheless serve as an avenue for organizing testing and experiments. Experiments can be carried out through a dedicated program on survey methods, under which new measurement tools can be tested and validated vis-à-vis their gold-standard counterparts, ideally through small-scale randomized survey experiments. Many of the priority areas covered in Chapter 2 above require experimentation within the national context, for example, on methods for data integration (Section 2.1), the use of objective measures (Section 2.4), the collection and use of paradata (Section 2.6), and machine learning and artificial intelligence (Section 2.7).

To build a successful experimental statistics business line, NSOs in lower income contexts will likely require technical assistance in building the internal capacity to carry out these activities. Fostering a culture of carrying out small experiments within national household survey programme is also important. Eurostat maintains a website that links to experimental statistics published by NSOs in the European Union and offers a clear example for fostering survey experimentation in countries [152]. Similar exercises can be carried out through other regional statistical organizations or at the global level.

3.1.7Investing in ICT infrastructure

In the third quarter of 2020, 25 percent of NSOs were reported to lack adequate ICT infrastructure for staff to work away from the office effectively [153]. The lack of cloud computing services for data storage and exchange as well as suitable facilities for remote training were common challenges highlighted by the UN-WB survey. Significant disparities exist in the capacities of low- and high-income countries to manage a forced work-from-home situation. Only 57 percent of low- and lower-middle-income countries were able to provide their staff with the necessary tools (such as personal computers, tablets, and monitors) to continue their work after the onset of the COVID-19 pandemic, as compared to 72 percent of upper-middle-income countries and 88 percent of high-income countries. Stronger and smarter technological infrastructure should be available at the national level for implementing mixed-mode surveys. These measures should be coupled with enhancements in data storage, data protection, the creation or strengthening of data dissemination platforms, the establishment of offline data enclaves for the use of confidential survey data (as discussed in Section 2.8), and steps to address the hardware and software needs of NSOs.

Finally, the successful integration of household survey data with other data sources generated by NSOs as well as those from the private sector requires strong legislative backing. The requisite legislation must allow access to other data sources for official statistical purposes while guarding the confidentiality of personal information in both the integrated data and the original data sources.

3.2.1Pursuing a coordinated and systematic approach to supporting national statistical offices

To unleash their full potential, household surveys must be adequately funded and positioned strategically so as to emphasize the critical role of a functional household survey program within the national data ecosystem. For example, during COVID-19, countries that already had existing household survey systems equipped with necessary contact information established were able to use them as sampling frames to rapidly launch and successfully implement phone surveys on COVID-19, taking advantage of investments in capacity building over a decade prior to the pandemic. Just so, investments are needed now to improve the responsiveness and resilience of future data collection systems.

Collaborative efforts at the regional and international level are key to a coordinated household survey program with a medium- to long-term plan at the country level, especially for countries that rely technically or financially on international agencies and the donor community. The ISWGHS should be the forum to foster such collaboration.

3.2.2Sustaining financing at the international level

Shifting the narrative from decrying the “funding gap” to discussing “investment opportunities” creates a better context for statistical development [154]. To build strong national statistical systems in low-income countries that can produce quality and timely data, donors need to think of data in terms of investments into tracking and achieving the Sustainable Development Goals (SDGs) and national development goals.

The Bern Network on Financing Data for Development is supporting a Clearinghouse – a platform that allows for assessing the state of data financing in the poorest and most fragile countries, highlighting ongoing and forthcoming projects, and providing guidance to donors on where to make investments. The function of the Clearinghouse is being augmented through the parallel establishment of a Global Data Facility (GDF) at the World Bank which will provide innovative financing mechanisms to address some of the most compelling funding data gaps identified through the Clearinghouse. By consolidating donor financing towards key priority areas, the GDF will enable greater coordination and create much needed synergies across donors so as to deliver more efficiently on the commitments of the Cape Town Global Action Plan for Sustainable Development, which called for “… [greater] participation of non-state actors in funding statistical activities through innovative funding mechanisms …”. The GDF could be a game-changer in supporting a renewed household survey agenda for the next decade by complementing government resources and leveraging other data investments, including project financing for large statistical capacity building operations.

3.2.3Fostering a stronger coalition of international agencies and countries

Implementing the priority areas identified in this paper requires a strong coalition of international agencies and countries to support such an ambitious agenda, which was also recognized by the United Nations Statistical Commission. During its 46th session, the Statistical Commission established the Inter-Secretariat Working Group on Household Surveys (ISWGHS) to foster improvement in the scope and quality of social and economic statistics as delivered through national, regional and international household survey programmes, with a focus on three areas of work:survey coordination, methodological development, and advocacy and communication [155].

Before exploring possible prescriptions for the ISWGHS, it is important to discuss its relative advantages and limitations; and how to support ISWGHS in fulfilling its mandates The ISWGHS consists of 11 international agencies and 10 member states. International agency members are responsible either for a survey programme (e.g., MICS, LSMS, 50 × 2030, etc.) or for providing regular training and technical support for household surveys in areas under their mandate. Member states either rely significantly on household survey programs for official statistics or offer technical and/or financial support to the work of the group. The ten country members are geographically representative to ensure that the need of the countries in their respective regions are taken into consideration in setting priorities.

All agency members have a mandate to support countries on household surveys, with a focus on specific thematic areas. For example, the International Labour Organization (ILO) is responsible for labor market data, the United Nations High Commissioner for Refugees (UNHCR) on forcibly displaced and stateless people, the United Nations Children’s Fund (UNICEF) for data on the wellbeing of children, UN Women for data on the empowerment of women and girls, and the World Bank for data on poverty and other dimensions of wellbeing. A collaborative group like ISWGHS is well-positioned to focus on the coordination of international survey efforts and cross-cutting methodologies. A good example on how ISWGHS members work together is the COVID-19 impact survey dashboard [156], created in May 2020 with information on surveys supported by members, which has played an important role in coordinating efforts within countries. The ISWGHS has also produced a number of cross-cutting methodologies that are of interest to all areas of work.

The other relative advantage of the ISWGHS, with its secretariat housed within the UN Statistics Division, is its close tie with the UN Statistical Commission [157], which serves the highest body of the global statistical system that brings together Chief Statisticians from all member states. Working closely with NSOs improves support for national needs as well as country adoption of international standards and methods.

More importantly, the ISWGHS has been playing an important role to amplify the impact of the tremendous amount of work undertaken by its members, including through innovative approaches, various channels including its website, webinars, blogs, and regular newsletters.

As we look towards supporting household surveys during the next decade, we suggest a number of priorities and activities that the ISWGHS should carry out to support countries in the short-term (with additional funding needs in italics). These include:

Coordination

Methodological development

Advocacy and communication