The Tool for Assessing Statistical Capacity (TASC)

1.Introduction

We increasingly inhabit a world where almost every activity is captured as a data point, often unknown to us, presenting both promise and challenges. Producers of official statistics must grapple with new sources of data, new methods, and new technologies, even as they continue to implement the surveys and censuses that provide data for development. The challenges of the data eco-system, as well as data demands like producing the indicators for Sustainable Development Goals (SDGs), are thus varied and consequential. It is a difficult environment for all national statistical offices (NSO) to operate in, especially for those whose statistical capacities are not strong.

The International Programs (IP) of the U.S. Census Bureau has been strengthening the statistical capacity of low- to middle-income countries for over seventy years. While the design, scope, and implementation of the assistance programs has changed over the years, IP’s primary mission has remained the same: to build the statistical capacity of NSOs and other statistical units to collect and analyze the data that enable economic and human development. To do this best, one of the first steps is to conduct a gap assessment to understand the needs of an NSO. IP’s best opportunity to build hands-on capacity is usually around the global census rounds, since funding from various sources for statistical capacity strengthening is most generous then. Population and housing censuses are the largest peace time operation of a country, requiring significant government funding and donor support in developing countries. The Tool for Assessing Statistical Capacity (TASC) – sponsored by the U.S. Agency for International Development (USAID) and Inter-American Bank (IADB) – was developed to assess gaps in a country’s capacity to conduct a demographic census or survey [1]. This paper describes the foundational framework, strengths, and limitations of the TASC.

2. Building statistical capacity

The Census Bureau began its program of international technical assistance in the 1930s and its formal training program began in 1947. IP’s training program was initially established as part of the program of assistance to Western Hemisphere countries. In the early 1950s, the program was expanded to developing countries globally. From the beginning, its main goal was to impart the practical skills needed by statisticians and data processing specialists in statistically less developed countries [2].

IP uses many formats for building statistical capacity. It runs a headquarters-based training program on various statistical subjects like sampling, cartography, demographic analysis and population projections (DAPPS), gender statistics, data visualization and storytelling with data, Census and Survey Processing (CSPro) software, Computer Assisted Personal Interviewing (CAPI) adoption for managers, and many others. It also conducts regional training programs in collaboration with international organizations like the United Nations Population Fund (UNFPA) and other regional training centers on these subjects. But most of the assistance provided by IP is structured as comprehensive, multi-year, project-based capacity strengthening, which often focuses on a population and housing census. These projects cover the full census life cycle – from planning and management to dissemination of results. When this is done over successive census and survey rounds, countries can successfully graduate from the need for assistance.

The assistance provided by IP to countries is based on requests from NSOs and funded by aid and development agencies, including strong support from the U.S. Agency for International Development (USAID), or the requesting country government. Before the 2010 round of censuses, IP would send multiple managers and senior staff with various expertise to assess the capacity of an NSO to conduct a census prior to assisting with a census or survey. The delegation conducted the assessment trip at the invitation of the NSO before finalizing the scope of assistance.

Delegations, sometimes as large as seven senior members – experts on mapping, data processing, management, logistics, field operations, analysis, evaluation – were sent to determine the scale and scope of training needs. Assessments could take up to two weeks before a scope of training was drafted. This mode of assessment was not only resource- and time-intensive, it was often subjective, as it depended on the experts sent. It was also frequently impractical to devote valuable resources to such an assessment, which could be otherwise spent on building capacity.

As funding for statistical development became relatively scarce, IP realized the need for a more efficient way to assess an NSO’s capacity to do a household-based census/survey and identify areas that needed strengthening. The solution was to develop a tool that merged expert knowledge of each census and survey operation with standards documented in major international statistical frameworks.

The expertise and standards by which NSOs self-assess became resident in the tool and thus more constant from project to project. The self-assessment design of the TASC also encourages buy-in among the managers and senior staff as the results presented to them are the result of their own introspection and deliberation. So, while the initial emphases of the TASC were cost-effectiveness and objectivity, it also became comprehensive and participatory during development.

3. Frameworks to Assess Statistical Capacity

Frameworks that measure statistical capacity are common in development planning and practice [3]. Before publishing the first version of the TASC, IP considered previously developed frameworks and tools that analyzed issues external to NSO technical capacity, as well as processes and products that reflect generalized statistical capacity. These frameworks considered elements of statistical capacity related to both individual organizations and national statistical systems (NSS). The TASC tightened the focus on measuring the operational – meaning pertaining to census and survey operations – statistical capacity of NSOs to produce quality household-survey but used structure and scoring strategies seen in predecessor frameworks especially relevant at the time of its development:

The TASC was designed to build on the strengths of these tools. TASC drew heavily from Principles and Recommendations for Population and Housing Censuses (P&R) [8], which focuses on good practices for processes likely to result in high quality data, in addition to standards for final products. Each module, section, and item in the TASC is based on such knowledge and standards.

4. Tool for Assessing Statistical Capacity (TASC)

The TASC was originally developed during a period of about two years in response to the needs of the 2010 global census round for low- to middle-income countries conducting traditional censuses. The first version was released in 2013. Senior NSO staff use the TASC to score their census and survey lifecycle operational practices against recommended international standards. It provides a quantitative score of the overall capacity of an NSO, as well as a module-by-module breakdown of areas of strength and weakness. Although this information has many uses, three stand out:

The TASC was developed based on The World Bank definition of statistical capacity at that time as “a nation’s ability to collect, analyze, and disseminate high-quality data about its population and economy” [6]. But, the TASC was specifically designed to measure the capacity of an NSO to conduct household-based surveys and censuses. This focus directed the development of the TASC in two ways. First, the TASC approaches capacity in terms of measuring processes rather than outcomes. Second, the TASC does not measure the ability of NSOs to produce statistics based on businesses, agriculture, and other domains not based solely on household data collection. The scores are specific to relative self-assessment of capacity and not meant to be comparable between NSOs. An understanding of this scope is important when deciding if the TASC is fit for purpose and when interpreting and presenting results.

To successfully conduct a census or survey, an NSO must demonstrate statistical capacity across a defined set of operations. Table 1 shows the TASC modules that assess census and survey taking operations and provides a description of each.

The basic unit of the TASC is an assessment item. Each item prompts participants to consider their NSO’s census or survey operational practice. There are four possible responses for each item representing fully realized capacity to non-existent capacity. NSO participants respond to each item with a rating from 3 (fully realized) to 0 (non-existent) based on the description that most closely reflects their practice. The items are arranged into sections corresponding to factors of capacity, including:

Figure 1 illustrates the format of the TASC by displaying the first few items of Planning and Management along with the scored responses for the Human and Physical Capital factor section.

In addition to evaluating operations, it is important to assess the environment within which the NSO operates. To capture those, the TASC has an additional module on Institutional Capacity, which covers cross-cutting factors not tied to a specific operation in the census/survey process. The module looks at the enabling – again, the underlying legal, political, and resource – environment in which the NSO operates and the extent to which the NSO maximizes its success through planning, practice, and coordination. The Institutional Capacity module has sections unique to this focus, including:

Due to increasing NSO recognition of their importance in improving data quality and coverage, the TASC also has an optional module for Administrative Records. This module is structured identically to the operational modules with assessment items grouped into four sections corresponding to the factors of capacity.

These eleven modules and sections were chosen for two reasons:

Table 2 shows the distribution of assessment items across the operational modules and within four factor sections assessed in all modules except for Institutional Capacity. There are 237 items in the 10 main modules and an additional 14 questions in the optional Administrative Records module. The questions are mostly evenly distributed between the modules, however the greater number of items in Institutional Capacity and Data Processing reflect the complexity of those topics.

4.3Administering the TASC

The TASC is a self-assessment instrument and is available online for free. The toolkit includes an Administrator’s Manual along with the other products – an informational flyer; the TASC Introduction Presentation; Booklet; and Results Calculator – needed to administer a TASC [10]. It should be administered by someone who (1) has many (15+) years of experience on a range of statistical operations; (2) is preferably external to the NSO; and (3) is a respected member of the statistical community. However, since the expert knowledge resides in the tool, the administrator does not have to be knowledgeable on all operations. Their role includes the following – engage with the NSO to set up the TASC meetings; make a presentation on the value and specifics of the TASC and the role of the participants in assessing their NSO; clarify the meaning of items when questions are raised; coordinate and monitor the exercise; calculate the results; prepare the results presentation; and finally, present the results to NSO management and participants.

Managers and senior staff across the various census operations are invited to participate during a TASC administration. The accuracy of the TASC results depends upon the full participation of senior and highly experienced members across the requisite operations in an organization. The administrator first makes a presentation on the importance, purpose, and limits of the TASC as well as the participants’ role in completing the response modules before the administrator facilitated response sessions. Participants are expected to answer only those modules in which they have experience. For example, subject matter specialists may answer questionnaire content and the testing, data analysis and evaluation, and data dissemination modules. If they are also managers, they could also answer the planning and management and institutional capacity modules. Cartographers answer the mapping module or, depending on their experience, may also answer the field operations module.

The TASC is administered in two sequential rounds:

• 1. Individual response round: Each participant answers the modules relating to their expertise individually. They then submit the individual answer sheets to the administrator. The Results Calculator allows up to fifteen individual data per module.

• 2. Group responses round: Participants that answered a particular module come together and answer that same module together as a group. For instance, all the staff/managers that answered the questionnaire content and testing module must agree on a single score for each item based on their organization’s practices. If there are more than six individuals who have answered a module, we recommend they be split into two separate groups that answer on two separate answer sheets. The Results Calculator allows up to three groups per module.

This approach ensures that every participant has read, understood, and scored the practices in their operational area during the individual round, which should provoke a discussion of practices during joint scoring of the group round. The group round ensures that any gaps in TASC participants’ knowledge of practices at the individual level are compensated through discussions. It is important that the administrator observe group dynamics during these discussions. Some interactions are participatory and egalitarian, while others may be hierarchical. Observations of the group dynamics and discussions help the administrator when they develop the results presentation. The TASC produces scores for both (aggregated) individual- and group-based results.

Figure 2.

Example of an overall result chart.

Figure 3.

Example of factors of capacity section scores for planning and management module.

TASC evaluations reflect it being a predominantly positive experience for participants. They feel valued as they are involved in the assessment of their operations. Participants always enjoy the group discussions for various reasons – some learn about the international standards, others because it provokes ideas on how they could enhance their practices.

5. Future of TASC

The TASC offers an efficient solution for capacity assessment for household-based statistical operations because it allows for one or two administrator(s) to obtain a comprehensive picture of NSO strengths and capacity building needs in a relatively short amount of time. TASC administrations are cost effective and the results widely utilized because of its participatory nature, making it an invaluable instrument for assessing NSO readiness to conduct surveys and censuses.

Since TASC version 2.0 was released in 2017, some of the frameworks that it was based on have been updated. The World Bank’s SCI was updated to the new Statistical Performance Indicators (SPI) [9]. Unlike the previous SCI, which mainly focused on the NSO, the SPI expands to measuring the entire NSS and includes new areas like infrastructure and resource components. The TASC, being a process-focused tool, already included many of the SPI’s components in the assessment. However, because it is currently designed only for household-based surveys, measurement of the NSS is only included in the Institutional Capacity module in as much as it assesses the enabling environment for an NSO to conduct surveys and censuses.

Another framework that has been updated is the PARIS 21 Capacity Development 4.0 [13]. The updated framework is broader and more holistic, focusing strengthening capacity at three levels – system, organization, and individual, each targeting five areas – resources, skills, management, power and politics, and incentives. The current TASC is designed to be a self-assessment tool at the organizational level, thus covering that level and the system level from the Capacity Development 4.0 but leaving out the individual. We have no plans to include measurement of individual skill level in TASC. As for the target areas, TASC currently does not measure incentives and measures power and politics only to some extent in the Institutional Capacity module. Future versions may include items to assess both.

TASC will be updated periodically to capture the latest technological and methodological advances, apart from drawing on the enhanced statistical frameworks. The TASC must also incorporate new statistical frameworks such as the 2019 Global Geospatial Statistical Framework and others as they become available [14].

Depending on the availability of funding and interest, future TASC development could include creating supplementary modules for different types of surveys and statistics on agriculture, businesses, or the environment. The Census Bureau would benefit from collaboration with other international agencies to create such modules and to update existing ones, just as we collaborated with IADB on the original version of the TASC.

6. Conclusion

Gap assessments are crucial to effectively targeting capacity strengthening initiatives. The TASC continues to show value in targeting training effectively during its use in the 2010 and 2020 global census rounds. Ongoing updates to account for new technologies – including the adoption of web assisted interviewing in response to the COVID-19 pandemic – and NSO feedback will ensure that the TASC continues to show value through future census rounds. TASC 3.0 will include these updates, and more.