Official statistics as a safeguard against fake news¹

2.1Fake news and statistics

Fake news was proclaimed as the Collin’s Word of the Year 2017, published by the Independent [1]. There is no single and generally agreed definition of fake news, but a typical one can be found in the Cambridge dictionary [2]: “False stories that appear to be news, spread on internet or using other media, usually created to influence political views or as a joke”. “Alternative facts” and “post-truth” are concepts associated with fake news.

Fake news is normally the result of deliberate actions, but the severity can vary from cases where the news and the data they are based on are fabricated to cases with correct data or statistics presented in a biased form.

Not all fake news is based on data or statistics, and fighting fake news is, of course, not a task for official statistics and the NSIs alone. Fake news based on statistics or the way these are presented and interpreted, are those addressed in this paper.

Fake news can also be related to a lack of independence and impartiality of the producer of statistics and wrong use of methodology in the production processes. Surveys may be based on small or non-representative samples (e.g. where the participants recruit themselves), biased questionnaires, and also administrative or other data sets with poor coverage of the population to be studied.

Fake news can be based on all sorts of data, from surveys or administrative data traditionally used for the production of statistics to new sources.

Some new data can be characterized as big data or smart data based on electronic devices. Big data can be defined as “large amounts of different types of data produced with high velocity from a high number of various types of sources” (Eurostat RAMON database) [3].

Struijs and Daas [4] have discussed quality approaches to big data used for statistics, and recognised the following main shortcomings linked to such data:

Some possible advantages of using big data for statistics are improved relevance and timeliness and lower costs, including lower respondent burden. Big data are often available in real-time. They are assumed to be used by competitors to producers of official statistics.

2.2Communication

The emergence of communication as a discipline and communication bureaus has been a development in recent years. This has led to a focus on simple messages which easily can reach the users. In statistics, a message focusing on a few numbers can improve relevance and thus the value of statistics. However, there is a balance between simplifying and explaining. Statistics presented out of context can facilitate misinterpretations. This may represent a challenge for the proper interpretation of statistics.

2.3When the fox minds the henhouse

Other institutions than the NSIs have other primary tasks than production of official statistics, and most of them can potentially be instructed by their superior authority in all matters. The quality requirements to processes and output are the same for all official statistics. However, while an administrative body is typically not independent, the unit within these bodies that is responsible for producing statistics should decide on how to produce and when to disseminate its statistics independently. This should be stated by law. Even so, this may be a challenge not only due to the possible influence from political authorities, but also because administrative bodies may be managed according to objectives with statistical measures, and certain statistical results will be in their own interest. The police can, for example, have interest of focusing statistics showing that crime is reduced to show good results of their work, but the opposite can also be the case to substantiate a demand for more resources.

Even if the statistics producers behave impartially, the public may suspect that they are not if there is a system where their statistics are used to measure their own performance. This may damage the credibility of their statistics.

2.4Examples

The examples of fake news given in the following are all based on statistics which are regarded as or close to official statistics. Most of them represent disinformation to influence political views.

One of the perhaps more famous fake news claims was during the Brexit campaign when the Leave side stated that the money saved from leaving the EU will result in the NHS (National Health System) getting £350 m a week. The UK Statistics Authority has afterwards directly commented on this in letters stating that this was at best a gross figure not taking into account the funds that the UK receives from the EU to for instance agriculture and scientific research [5]. Furthermore, it was never clear that the sum would be used in total to health care. This is an example of selective use of statistics (also called “cherry-picking”).

Other typical sources of misused or misunderstood statistics are wrong use or ignorance about definitions. A recent example from Norway is a discussion between politicians about the number of state sector jobs moved out from the capital Oslo. The present Government argued that they have moved more such jobs than the preceding Government led by the opposition. Statistics show that this is wrong if Oslo is defined by the municipality, which is normal. However, the Government claimed that they were right since they had meant the “Oslo labour market region”. This information was suppressed in the original message. The inaccuracy was disclosed by the Norwegian fact-checker faktisk.no [6], which is an example of fact-checker bodies that have emerged in many countries, see the Poynter website [7]. Official statistics are one of their most widely used sources.

In October 2018 the Council of Economic Advisers within the Executive Office of the President of the USA released a Whitehouse report on the opportunity costs of socialism [8], which includes comparisons on income and costs of living between the USA and the Nordic countries. One of the conclusions is that the inhabitants in the Nordic countries have a lower standard of living than people in the USA. On average the standard of living in the USA is said to be 15 percent higher than in the Nordic countries.

The Norwegian Broadcasting Corporation (NRK) asked two employees of Statistics Norway to read the report [9]. They were not able to find out how its conclusion is possible. The definition of standards of living was not clear and the selection of underlying statistics seems to be biased. The results are not in accordance with internationally available statistics. Among other indicators used were the level of taxes and the costs of owning a Ford Ranger pickup truck,22 which is considerably lower in the USA. On the other hand, the Nordic public services such as free or subsidised education, nursery and care for the elderly are not taken into account. For measuring income, average salaries have been used, which also will give a biased picture since the super-rich in the USA pull this figure up. In fact, very few Americans earn as much as the country average. Median income would have been a more relevant measure in this context.

The Whitehouse report was also investigated by faktisk.no [10], which also concluded that the assertion on living standards in the report is in fact completely wrong. Statistics Norway assisted the fact-checkers in their work on this.

A different but slightly similar Norwegian example of misuse of statistics and definitions is an estimation of the distribution of wealth among households carried out by an economist linked to the conservative think tank Civita [11]. In the debate on the increasing inequality in Norway he alleged that wealth is much more equally distributed than normally believed since the Norwegian oil fund (the Government Pension Fund) is publicly owned, and its value could then be shared equally among all Norwegians. This is not the normal and comparable way of estimating personal wealth. A researcher from Statistics Norway has participated in the public debate arguing against the conclusions of the Civita-report [12].

Sometimes it is not a question about fake news, but rather fake and almost irrefutable myths. An example is the assertion that women earn less than men, inasmuch true if women and men are compared as groups. However, this is often mentioned in a context of lack of gender equality. This was recently repeated in a video from the Norwegian trade union for employees in the financial sector. In Norway, a woman in average earns 87 percent of a man. NRK [13] recently presented the interpretation of this on an individual level as fake and interviewed a representative of Statistics Norway. He explained that the difference is almost solely due to the fact that women and men tend to work in different sectors. In the private sector 36 percent of the employees are women, while this share is 70 percent in public sector. There is no evidence that women and men do not have the same salary for equal or comparable jobs in Norway.

3.1Requirements to official statistics

The UN Fundamental Principles of Official Statistics (UN FPOS), first adopted in 1994, describes requirements to official statistics, such as usability and impartiality, emphasising professionalism on methods and production procedures. They also include a principle on the statistical agencies’ right to comment on erroneous interpretation and misuse of statistics [14].

A modern definition describing requirements to official statistics can be found in the UN National Quality Assurance Frameworks (UN NQAFs) Manual for Official Statistics [15]: “Official statistics describe, on a representative basis, economic, demographic, social and environmental phenomena of public interest. Official statistics are developed, produced and disseminated as a public good by the members of the national statistical system in compliance with UN FPOS and accepted quality frameworks such as UN NQAF, as well as other internationally agreed statistical standards and recommendations. In many countries, official statistics are defined and described in a statistical programme”.

In Europe the European Statistics Code of Practice (ES CoP) [16] is normally the basis for the quality requirements to official statistics. ES CoP principles cover the production chain of official statistics from the institutional environment through the production processes to the outputs of official statistics.

For most of the ES CoP principles linked to statistical outputs there is a need to balance compliance between them e.g. accuracy vs. timeliness. An overall requirement can be expressed as use of and compliance with the ES CoP in general. However, some principles are more fundamental than others, and these are normally explicitly mentioned in statistical laws. They comprise the professional independence and impartiality of the statistical institutions, transparency, mandate for data collection and statistical confidentiality. Requirements to official statistics must therefore include these principles.

3.2Independence and impartiality

ES CoP principle 1 states that “professional independence of statistical authorities from other policy, regulatory or administrative departments and bodies, as well as from private sector operators, ensures the credibility of European Statistics”. But what does professional independence mean? The indicators in the ES CoP mention the responsibility of the NSIs and Eurostat for ensuring that statistics are developed, produced and disseminated in an independent manner, in other words how statistics are produced and disseminated but not what or which statistics should be produced.

The UN fundamental principles do not use the word independence, but principle 2 states: “To retain trust in official statistics, the statistical agencies need to decide according to strictly professional considerations, including scientific principles and professional ethics, on the methods and procedures for the collection, processing, storage and presentation of statistical data”. Again, this refers to how statistics are produced.

Professional independence is about the independence of a statistical institution and its head from political authorities or other external parties, to ensure its credibility. However, professional independence is not sufficient to ensure public trust, producers need to act impartially and be objective as well. This is not so easy to measure, but central indicators included in the ES CoP to show impartiality comprise full transparency about data sources and production methods, and equal access to statistics for everyone at the same time according to a release calendar. Possible errors should be corrected and explained as soon as possible.

When a statistical institute shall comment or even correct fake news based on their statistics is an issue as well. Even if it has the right to do so, this must be considered in relation to impartiality and the efforts required.

Users often demand analyses. Analyses add value to and make statistics more relevant. Analysis is a tool to explain statistics and their impact, by presenting them in a suitable form combining data from different sources, interpreting data, and identifying possible causes and effects. Confusing correlation with cause/effect frequently gives rise to misunderstandings or even fake news. Analyses also include modelling and more substantive analyses, such as making projections. Such analyses are not official statistics, but an analysis by a statistical institute shall not advocate policies or take partisan positions.

The relationship between official statistics, analyses and professional independence has been considered by Sæbø and Holmberg [17]. This paper also provides and discusses examples of breaches of the professional independence of official statistics.

4.1Revision of statistical laws

“The European Statistical Law” [18] was amended in 2015 reinforcing the independence of NSIs and other national authorities responsible for European statistics. Amongst other measures in the amended law, a commitment on confidence in statistics should be published by the Member States to ensure public trust in European statistics and progress in the implementation of the statistical principles contained in the Code of Practice. The second round of European peer reviews in 2014–2015 recommended for several countries a revision of their statistical laws, to strengthen their professional independence and clarify the content of “official statistics”, see for example the peer review-report on Norway [19]. Several new statistical laws in European countries, including Norway, have been developed following these recommendations. The UN’s Generic Law on Official Statistics gives a good overview of the rationale behind and recommended content of such laws [20]. Access to new data sources is an important element. The new laws emphasise the requirements to official statistics, i.e. quality in general, and in particular the professional independence and impartiality of the producers of such statistics. These requirements can be considered as core values of official statistics, internationally agreed and captured in the ES CoP and the UN FPOS.

The European Partnership Group has also discussed the role of the statistical offices in a world of “alternative facts”, based on a paper from CSO Ireland [21]. Upholding core values outlined in the ES CoP, how to address misuse and being open to new user demands, data and technology are central elements in their discussion. Legal challenges are also addressed.

4.2Improvements of output quality

Statistics must be “fit for use”. Different users have different needs that must be balanced against each other to give the quality concept a concrete content. Over the past twenty years, statistical institutions have arrived at the consensus that the concept of quality of statistical information is multi-dimensional and that there is no one single measure of quality. For a statistical product, the definition of quality is operationalized by specifying a set of factors or dimensions that characterize its quality: Relevance, accuracy and reliability, timeliness and punctuality, coherence and comparability, accessibility and clarity. These are the dimensions of the quality of statistical outputs described in the ES CoP. As mentioned, the dimensions of quality are interrelated and there are trade-offs between some of them.

Continuous improvement is relevant for all quality dimensions. However, in order to prevent fake news improving some quality dimensions of official statistics is probably more important than others. Without relevance there is no quality. Official statistics must develop continously to satisfy both existing and new user needs.

Official statistics should also be made accessible to different users on relevant platforms including social media.

As already mentioned, statistics based on new data can often be more timely and produced at a lower cost than traditional statistics, though less accurate since there may be methodological challenges linked to coverage and representability. User surveys and focus groups often indicate that timeliness is the main quality challenge of official statistics today given that it is relevant.

An analysis of user perceptions and communication of official statistics in the EU [22], shows that users express positive views on the quality in general, but asked about the quality criteria listed in the ES CoP users are less positive regarding timeliness and punctuality. They also want more transparency and proof of the independence of the producers of European statistics.

If NSIs do not improve timeliness, the chance that someone else will produce the requested statistics and possibly with poorer but may be sufficient accuracy increases.

Other quality dimensions which could be improved are coherence, accessibility and clarity.

Coherence facilitates presenting statistics in a context. Experiences from quality reviews in Statistics Norway tell us that users almost always ask for better coherence and interpretation of statistics in a broader context, which can be achieved by simple analyses of trends and comparison with other statistics. More advanced analyses highlighting correlations casting light on political issues can increase the value for users. But this also calls for caution to safeguard the principles on impartiality and objectivity.

In line with open data standards (see Open Data Charter website [23]) statistical results should be machine readable and accessible in different formats, though without putting statistical confidentiality in danger. Providing relevant metadata together with statistics is crucial to avoid misunderstandings and misuse.

4.3Cooperation

Cooperation is key to meeting the challenges to official statistics. Statistical institutes should be open to cooperation with fact-checkers. Serious fact-checkers can play a role in pointing out and correcting misused interpretation of statistics. It is recommended amongst other innovative actions related to the European Statistical Programme 2021–2027, that “NSIs nominate statisticians or communications staff as Wikipedia editors to insert correct data commentary and links in one set of pages” [24].

The NSIs have a corrective role themselves, as stated in the UN FPOS. However, as already mentioned this must be balanced against impartiality and costs. Cooperation with fact-checkers extends the possibilities to intervene.

Open data initiatives should be supported to ensure that such data satisfy quality requirements, at least are followed by relevant metadata.

Even if statistical laws give access to privately held and new data sources, cooperation with owners of such data and other external experts is necessary to investigate and to utilise big data properly for official statistics. Such cooperation can comprise data processing at the original source, thus relieving the statistical institutions of infrastructure investments and contribute to avoid sharing of sensitive data as well, in line with Eurostat [25].

4.4Branding of official statistics

Independence, impartiality and quality of official statistics must be proven and communicated. At the same time statistical institutes should embrace the data revolution. Baldacci and Pelagalli [26] have written a report for Eurostat on communication of statistics in post-truth society: the good, the bad and the ugly. Business as usual is not enough for the statistical institutions. This represents the bad scenario in the short run. In a longer run it may lead to other actors taking over the statistical market and a loss of trust in official statistics, i.e. an ugly scenario. To keep trust in official statistics the producers must take a pro-active role to satisfy new user needs both regarding the content and how the statistics are communicated.