A statistical quality assurance framework for the United Nations

1.Introduction

In 2017, the United Nations Statistics Division (UNSD) reported to the 48th session of the United Nations Statistical Commission (UNSC) that ‘Chief statisticians of the United Nations system have developed and agreed on a generic quality assurance framework that addresses … the statistical work of regional and international agencies and is aimed at ensuring that appropriate quality assurance procedures are being followed’ [1]. Thus, ended an anomalous gap in the official statistics infrastructure, where the UN encouraged member states to put in place a national quality assurance framework to support their statistical systems, but did not have an equivalent framework in place to support the UN statistical system itself.

The journey to develop what was eventually published as the United Nations Statistical Quality Assurance Framework (SQAF) began at the very first meeting of the newly convened Committee of Chief Statisticians of the United Nations System (CCS-UN) in Rome in September 2014. At that meeting, the first substantive order of business was to establish a task team11 to begin work on scoping and drafting a generic SQAF for the UN statistical system. Two and half years later, the SQAF [2] was formally adopted at the 6th meeting of the CCS-UN in New York in Spring 2017.

The need for a SQAF was identified for a number of reasons. Firstly, there was no universally agreed definition of quality employed across the UN statistical system and secondly, there was no equivalent framework to the national quality assurance framework (NQAF) being promoted UN statistical agencies at country level. It was also recognized that an SQAF could be both a statement of intent as well as a description of good practice. An SQAF would help operationalize the Principles Governing International Statistical Activities [3] by introducing a common understanding of quality assurance and a common language to discuss these issues across all UN entities.

It was envisaged that an SQAF would bring several benefits, including:

The remainder of the paper is organized as follows: Section 2 explains the relationship and distinction between the SQAF and the NQAFs. Section 3 builds on this by detailing some of the unique features of the SQAF. Section 4 provides a summary of the components and dimensions of quality as defined by the SQAF. An update on adoption and implementation of SQAFs by UN agencies in provided in Section 5. Some implications and possible future developments are briefly discussed in Section 6, followed by a conclusion in Section 7.

2.Building on NQAF 2012

The SQAF is modelled on the United Nations National Quality Assurance Framework Template and Guidelines adopted in 2012 (UN, 2012) for countries but tailored to the circumstances of UN agencies. The NQAF 2012 was developed by the United Nations Expert Group on National Quality Assurance Frameworks (EG-NQAF). It was ground-breaking as it reflected the understanding that existing quality assurance frameworks shared a common core and followed the same principles despite being organized differently. In March 2019, the United Nations Statistical Commission at its 50th session adopted the United Nations National Quality Assurance Frameworks Manual for Official Statistics (Manual) [4] and the recommendations contained therein. The Manual updates the NQAF 2012 but was amended to reflect the emergence of an enlarged data ecosystem with potential new data sources, data providers, new technologies and methods. It also aims at implementation throughout the entire national statistical system and beyond, not just the National Statistical Office (NSO).

Both the SQAF and NQAF are generic and open for adaptation. They are both descriptive and only constitute guidance in the sense that they provide the components and a general structure within which individual country (NQAFs) – or agency (SQAFs) – specific can be developed. However, the quality principles contained in SQAF and NQAF have a prescriptive nature through the link to their respective governing principles, namely the Principles Governing International Statistical Activities22 and the Fundamental Principles of Official Statistics [5].

The need for a SQAF (as distinct from using the NQAF) principally arose as the challenges facing a statistical unit in an UN agency are very different to those facing a NSO, both in terms of statistical processes and dissemination but most particularly in terms of institutional set-up. Institutionally, statistical units in an UN agency are very different to independently constituted NSOs, as they are embedded in larger policy oriented agencies whose core business is not the production of statistics and often one with a more politicised environment. Furthermore, statistical units in UN agencies will typically focus less on primary data collection, and more on compiling global datasets using data supplied by member states, which in turn involves testing data quality, adherence to international standards, concepts and definitions, and also estimation/interpolation to fill data gaps so that complete globally comparable data sets can be made available. Dissemination often requires elaborate consultation processes with member states to validate and clear estimates. Of course, statistical units in UN agencies also develop statistical indicators, normative standards and provide statistical capacity development to member states (see Peltola et al. [6] for an example of the types of activities a statistical unit in the UN can be involved in). The SQAF was also important as it formally set out, from a UN perspective, how statistical quality is defined. It was also important to show member states that the UN statistical system followed its own advice and had in place a formal SQAF to support the UN statistical system.

3.Some features of the SQAF

This section briefly outlines a number of issues that arose while drafting the SQAF for the UN system, that led to some noteworthy decisions and perhaps signal some important issues for the future.

4.Components and dimensions of quality

As noted in Section 3.2, the SQAF adopted as underlying or defining principles, the principles articulated in the Principles Governing International Statistical Activities – see Appendix 1. The components and dimensions of quality selected were based on existing SQAFs44 – see Fig. 1. Critically, they incorporate not only process and output components, but they also incorporate ‘code of practice’ type or institutional components of quality. Their inclusion was especially important, as increasingly it is this aspect of quality that has proven to be the most challenging, both for NSOs but also for UN agencies.

Figure 1.

Components and dimensions of quality.

The dimensions of accuracy and reliability, timeliness and punctuality, accessibility and interpretability, relevance and coherence are generally well understood as are the trade-offs within each pairing, and therefore do not require any additional comment. Arguably the same is true for the dimensions of process quality. It is issues involving institutional quality however, where some of the biggest challenges for multilateral organisations lie, and where discussion is warranted. In the context of UN statistics, arguably the biggest threat to credibility comes from the insistence that UN agencies use only data or statistics provided by member states (even when those data are not credible or do not adhere to internationally agreed standards or definitions). Another significant threat arises where member states apply political pressure in an attempt to stop UN agencies from fulfilling their mandate by interfering with the publication of objective, impartial and internationally comparable statistics. Thus, institutional quality is the Achilles heel of international official statistics, and of the application of SQAFs. This weakness is typically most acutely felt in specialized agencies, where national counterparts are not NSOs but ministries where a culture of impartial and independent statistics is often not as strong, and where there is less awareness of international statistical standards and methods. So, while chief statisticians of the UN system identified compliance with institutional quality principles as critical to overall statistical quality and identified objectivity and impartiality as critical dimensions for achieving this goal, just as their national counterparts in NSOs would, this remains a source of tension and contestation.

In 2020, the Data Strategy of the Secretary-General highlighted the importance of data quality for all UN organisations, noting that ‘data is not an isolated concern, but permeates our organization and its use is integral to our success’ [13, p. 8]. It was no accident that principle 5 of the 12 basic principles outlined to guide ‘data actions’, stressed the importance of impartiality.

The SQAF includes a generic statistical quality assurance framework (GQAF) to assist individual UN agencies that do not yet have a SQAF to prepare a suitable QAF. As noted in Section 2, the GQAF is modelled on the NQAF template and guidelines adopted by the UNSC in 2012 but tailored to the circumstances of UN agencies rather than NSOs. This approach acknowledged that a ‘one size fits all’ SQAF was not realistic given the varied and complex mandates and governance mechanisms employed across the UN system and hence the logic of a generic QAF that can be adapted to the specific circumstances of each individual entity.

The GQAF provides a template or menu from which a UN entity can adapt to their specific purposes. The GQAF details the broad benefits of having a SQAF along with actions that can be employed to promote the QAF both internally and externally. It details typical quality components and dimensions that an entity might consider and provides linkages to the CCSA statistical principles. It also provides a generic quality assessment checklist and discusses some of the pros and cons of introducing an external quality assessment or peer review.

5.Adoption and implementation

In spring 2022 a short survey was conducted to assess the implementation of the SQAF. Twenty-three of the 29 CCS-UN members responded (a response rate of 79 percent). An additional 8 institutions that are members of the wider Committee for the Coordination of Statistical Activities (CCSA)55 also responded, although it should be noted that the SQAF was designed for UN agencies rather than international organisations more broadly.

Only 10 of the reporting CCS-UN members have a SQAF or a DQAF. As noted in Section 3, the interpretation of ‘data’ and what it means for scope varies across agencies. The survey did not pursue this question of interpretation, but simply quantified whether a QAF exists or not, and whether it was a SQAF or a DQAF. Of the 10 agencies, 4 had DQAFs, 4 had SQAFs and 2 had hybrid S & DQAFs. Six agencies had instituted their frameworks since the publication of the SQAF in 2017, the other 4 existed beforehand. Of these 4, only one is aligned with SQAF, the other 3 are not; one agency has plans to align in the future.

The SQAF recommends that each organization or entity appoint a chief statistician. Similarly, the Data Strategy of the Secretary-General recommended that organisations consider establishing a Chief Data Officer position (Rec #8 Enablers). Only 10 UN entities (43 percent) have a chief statistician or a chief data officer. This is a critical gap in the institutional governance, as it leaves entities without clear mandated statistical leadership and authority. The Data Strategy of the Secretary-General also identified as an action to be prioritized, that organisations develop a framework for strategy oversight and data governance, including a Data Strategy Group, Data Governance Council or a data stewardship model (Rec #13). 78 percent of reporting entities noted they had some sort of statistical or data coordination mechanism, although the majority of these are informal.

Five years after the adoption of the SQAF, only 43 percent of reporting CCS-UN members have implemented such a framework. But it is noteworthy that since the framework was adopted in 2017, the number of UN agencies with a SQAF, has more than doubled from 4 to 10. Several agencies without a SQAF noted that they operate several quality assurance procedures or adopted their own data principles but have not adopted a general SQAF. However, six agencies noted that while they do not yet have a SQAF they have plans to implement one, or in two cases it is already under development. A further 3 agencies noted a SQAF is under consideration. Of more concern though, several agencies noted that a SQAF is not a priority for senior management or would not have senior management support.

These results should not be surprising. As noted above, statistical units in UN agencies operate in a more political environment than most NSOs, so adaptation of principles and quality standards can be a slow and challenging process. In 2014, the CCSA undertook an assessment of the implementation of the Principles Governing Statistical Activities [14]. At the time, 80% of responding international organisations assessed that their level of implementation of the Principles was either ‘high’ or ‘full’. However, when it came to putting in place practical frameworks to support the implementation of the Principles, 55% of the organizations considered adoption to be ‘low’ or ‘not implemented’. It was these findings, in part, that prompted the development of the SQAF.

6.Discussion

The SQAF is only 5 years old, but reflecting the rapid changes in the dataverse, there are areas where the framework might already need to be updated. As noted above, closer integration of ‘new’ data, such as geospatial information, big data and citizen science data with more traditional datasets are just some examples where some reconceptualization of quality of official statistics might be warranted, as it is not yet clear what this might mean for quality assessment and the existing dimensions of quality. If statistical agencies begin to complement co-production models with co-creation models, this too may have implications for how quality is conceptualised. Some additional thought must be given to whether the quality of statistics from these ‘new’ data sources can be meaningfully assessed in the same way as statistics based on traditional data sources have been assessed. What if discussions around the democratization of statistics grow legs, this might dramatically impact on how data and statistical quality is conceptualised; all of the existing frameworks have been prepared by statisticians for statisticians, but what might a SQAF drafted by data users look like?

These ‘new’ data sources may also have a more profound impact on official international statistics, as they may be the catalyst that forces to the surface a debate on the precise role or mandate and methods of international official statistics vis-à-vis the new dataverse and the new opportunities for the production of international statistics, such as the use of remotely-sensed data held by private corporations. This is a debate many might prefer not to have but it is hard to see how it can be avoided for much longer in the rapidly changing dataverse we now live in.

Clearly, the relative space occupied by official statistics in the information value chain is changing. Data was once seen by statisticians primarily as an input for statistics; now statistics are themselves only a small part of many rapidly expanding data ecosystems and economies. In addition, data have become a geopolitical issue [15], with war being waged for control of our data – the ‘currency of the modern age’ [16]. This fact combined with an awakening of governments to the importance of data and official statistics is contributing to a steady politicization of the field. This trend is likely to challenge institutional quality safeguards of international statistics, arguably the weakest link in the chain, even further.

7.Conclusion

The UN system adopted their SQAF in 2017. As noted in the introduction, this the anomalous situation where statistical agencies in the UN system encouraged member states to put in place a national quality assurance framework but yet did not have an equivalent framework in place. The importance of this decision can be gauged by the fact it was the first substantive order of business for the newly constituted Committee of Chief Statisticians of the United Nations System.

The need for a SQAF was politically self-evident but there were also valid statistical needs. The SQAF would bring a common definition of quality shared across the UN statistical system. Today, it is both a statement of intent as well as a description of good practice. It provides an operational tool that helps statistical units embedded in UN agencies to live or operationalize the Principles Governing International Statistical Activities.

Developing principles and standards is an important, often challenging, first step but it is often the easy part. Adopting those standards and principles and implementing them typically takes longer and requires more concerted effort. The SQAF is no exception – since publication in 2017, only an additional 6 UN entities have fully adopted a SQAF, but many entities have indicated that they are in the process of adopting these principles. Now the focus for the CCS-UN must be to assist all UN agencies to adopt the SQAF. The next challenge is to help statisticians to socialize SQAFs and their implications in their agencies. For while the statisticians themselves understand the importance of upholding the UN Fundamental Principles of Official Statistics and the Principles Governing International Statistical Activities, and of objective, impartial statistics, they do not always have the support of their institutions.

Progress has not been as fast or as even as the CCS-UN would wish. Since 2017, only 6 UN entities have adopted institutional wide S/DQAFs. But the publication of the Data Strategy of the Secretary-General reflects a growing awareness of the importance of data and the risks associated with an absence of governance and stewardship. The determination of senior management of the UN to improve data governance and implicitly data quality, presents an opportunity for UN entities to drive implementation and change across entities. A tangible indication of this support is the ‘Senior Managers Compact’ for 2021–2022, where senior management of the UN are being asked explicitly how they intend to operationalize the UN-wide data strategy [17]. Hopefully, with this new drive will come the fair winds that allow institutions that have not yet adopted a SQAF to do so.

In an era where fake news, lies, mendacity, misinformation are abundant and multiplying, the importance of independent, impartial official statistics as a bulwark against this trend, is critical. To fulfil that role, the quality of those data must be demonstrably robust. SQAFs play an important role in supporting this task. Quality assurance is an ongoing challenge; although the SQAF is only 5 years old, some areas of the framework might already need to be updated, to reflect rapid changes in the dataverse. As noted above, closer integration of ‘new’ data, such as geospatial information, big data and citizen science data with more traditional datasets are just some examples where some reconceptualization of the quality of official statistics might be warranted, as it is not yet clear what this might mean for quality assessment.

The CCS-UN is by UN standards still in its infancy. But over the past 8 years, this new committee has achieved a number of critical milestones (outside the scope of this paper). The design and implementation of the SQAF is one of the most important achievements. This work continues but CCS-UN remains committed to improving the quality of international and national official statistics.