Official statistics, big data and civil society. Introducing the approach of “economics of convention” for understanding the rise of new data worlds and their implications

1.Introduction

Following up on the questions raised by Radermacher [20], this article asks what role sociology might play for understanding the challenges that official statistics faces in times of big data and profound social and political transformation. It proposes a specific theoretical perspective for thinking about the nature and role of statistics. The key characteristic of this perspective is that it discusses methodological issues and questions of data quality in their interplay with political, economic, social, and moral contexts of data and knowledge production. In contrast to still widespread positivist understandings, the approach we introduce starts from the assumption that statistical facts are always intimately interrelated with norms and values. The numeric representation of social realities cannot be separated from normative orders and political values. Norms and values themselves need to be treated as “facts”: they effectively inform the definition of indicators as well as measurement strategies, the assessment of data quality, or the analysis and presentation of findings. In other words: Both the production and the evaluation of statistical data necessarily depend on epistemic and political conventions. These conventions provide rules and criteria for producing and evaluating (statistical) knowledge about the social world that qualifies as trustworthy and relevant information in a given societal context. At the same time, these conventions are the outcome of socio-historical developments, scientific debates, and political negotiations; accordingly, they can always be contested, criticized, transformed, or (in the long run) replaced.

It is exactly such a kind of “dispute” that official statistics currently faces in relation to big data and commercial data production on the one hand and civil society actors on the other. This dispute has many faces and takes different forms. It is evident in alleged revolutions of social knowledge production that proclaim the coming of a new age of data-driven social physics (Pentland, 2015) as well as in open science initiatives11 or in the activities of “data NGOs” that lobby for the usage of new data sources to address pressing social issues.22 These disputes are not necessarily hostile. To the contrary, the initiative to discuss coming challenges for statistical knowledge production often comes from within the field of public statistics itself (as is evident from the very SJIAOS discussion that this article contributes to).

In the following, we introduce the concept of “data worlds” as a means for thinking and talking about these developments. This notion shall allow to investigate how data infrastructures (historically formed cognitive and organizational frameworks and resources that structure the production, distribution, and usage of statistical data) are anchored and embedded in social rationalities (“conventions”) and for understanding how data are produced, analyzed, evaluated and, eventually, applied as an informational resource for collective action. The concept of “worlds” is informed by the approach of “economics of convention” (EC) which emerged in France in the mid-1980s with the involvement of sociologists and economists (many of whom actually worked at the “Institut national de la statistique et des études économiques”/INSEE) [10]. EC scholars have prominently applied the concept of “worlds”, be it in earlier works to problems of economic coordination [25], be it later on to questions of methodological coordination and data production in survey worlds [26, 11]. In an EC understanding, different worlds of data production vary in their methodological cultures, their epistemic values, their quality criteria and their collective understanding of and relation to the “common good”. As will be discussed in more detail below, this implies that official statistics marks only one of a plurality of co-existing ways of producing and using numeric representations of social phenomena. We argue that official statistics is increasingly questioned by two more recent data worlds which we refer to as the big data world and the civil data world. We present a framework for making sense of the relations and tensions between these three data worlds. The intention of this framework is not to criticize these different data worlds or even to advocate one of them at the expense of the others. Rather, the aim is to offer a basis for thinking about how these different data worlds are currently becoming related to each other, what tensions might follow from these interplays and what social and political implications these new constellations might have.

2.Economics of convention

The French approach of economics of convention offers a contemporary conceptual framework that is closely tied to the field of quantification and official statistics [9, 4]. One of EC’s birth moments actually lied in the empirical analysis of official statistics and early representatives of EC such as Alain Desrosières or Laurent Thévenot were affiliated to the French national statistical institute INSEE. EC is informed by a neopragmatist understanding of science: statistical facts are seen as the result of concrete practices of knowledge production that take place in given institutional contexts and are structured by existing orders of (scientific, political, cultural …) knowledge which they build on, contribute to, and transform. Correspondingly, any over-simplifying understanding of data as pure representations of “social facts” is rejected. Instead, data are regarded as resulting from a complex interplay of conventions, actors, and technologies. One of the notions used to decipher these social processes of data production is the concept of “statistical chains” [5, 8]. Statistical chains are best conceived of as institutionalized social processes which allow to generate data. In this sense, data need to be seen as social artefacts and social constructions [20]. Desrosières pointed to the many tensions which arise in and along such statistical chains because of the different views that the involved actors hold on data and their quality. “Methodological statisticians” are usually highly aware of these processes and of the resulting “conventional nature” of data. However, many users of statistical data – including media experts, “subject-matter-specialists”, politicians and the public – expect data that “hold together”, data that can be treated as true and unambiguous “facts” [7]. Thus, contradictory understandings of the ontology of data are confronted with each other, with widely prevalent expectations to “transform” data from its actually construed and conventional foundations into true and indisputable representations of social reality [7, 9].

The concept of statistical chains points to a decisive point. Statistical data are based on a division of labor between actors involved in the definition of categories, actors and technologies mobilized for and in measurement processes, situations in which data are distributed, communicated, analyzed and interpreted and decisions that are taken on the basis of statistical data. These different actors and stages involved in data production and usage are all involved in their own social and institutional contexts, engaged in different sorts of situations, and faced with different kinds of problems and challenges. As a correlate to the idea of chains, we may hence think of these various actors as embedded in their respective “worlds”. These worlds are “data worlds” [11] in and through which the production and interpretation of data evolves along the statistical chain.

Following Desrosières [6], the production of quantitative data in and across these different worlds is only possible on the basis of conventions of measurement. The definition of such conventions of measurement always happens in concrete social circumstances and is always deeply interrelated with political and normative orders. What counts as relevant, acceptable, and fair process of quantification depends on social and political contexts. Categories and concepts that inform statistical measurement are anchored in public and political discourses and are always linked to specific forms of defining and understanding uncertain and tension-ridden situations. Salais et al. [22] give the example of the measurement of unemployment to demonstrate how institutions of official statistics (and the categories they employ) co-evolve in parallel with other social institutions, in this case the industrial organization of labor relations. The emergence of long-term industrial labor contracts allowed employees and new social classes to develop the expectation that employers provide permanent work and salary. As a complement to this expectation, unemployment as a new social category emerged, a category which actors referred to in order to make sense of their own situation in case they lost their industrial employment. In the course of the 19th century, official statisticians step by step included this category into their procedures of data production by implementing conventions for categorizing human beings as unemployed [22]. Unemployment, in other words, was never a “natural state of being” but a new way of interpreting a novel kind of biographical and social situation. Only by being routinely encoded by official statistics, it turned into a social fact. It is in this sense that EC scholars argue that statistical categories and indicators are expressive not only of social reality, but also of moral and political orders. Facts and values, in other words, are always already intertwined in statistical data.

The identification and exploration of this pragmatic, yet inevitable link between data production on the one hand and political and social normativities on the other may be seen as the crucial contribution of EC scholarship. Conventions are involved in the statistical chain at all stages. They serve as logics of coordination (e.g. between different actors along the statistical chain) and, at the same time, as criteria for evaluation (e.g. of the adequacy and validity of indicators and categories). EC hence applies a normative position to the question of data generation, its application and interpretation. The meaning of data as well as their concrete value for collective action are structured by conventions. Without conventions, data would be just meaningless figures without any anchoring in criteria and logics for assessing their validity, quality and relevance.

The need for a “conventional underpinning” of statistical measures entails that the data produced in order to represent social realities be related to commonly accepted understandings of the social world and established conceptions of justice and the common good. The crucial point again is that there are several different such conceptions that are widely accepted as legitimate and relevant in modern societies; in EC terminology, there is a plurality of “orders of justification”. EC scholars have identified numerous such overarching conventional orders that are commonly established in current societies as basis for governing, justifying, and criticizing the social world and how it is represented in statistical information [3, 2, 14]. Table 1 presents the eight most important of these conventions. The key point is that each of these conventions has a history (it has been developed and established over time) and is mirrored in everyday practices, processes and institutions. For example, the domestic convention is deeply anchored in our understanding not only of family life, but is also mirrored in understandings of how companies or other organizations should care for their employees, take responsibility for the community, and best be built on a kind of natural trusted authority. The industrial convention, in contrast, focuses on the role of standardization and quantification to ensure fair, transparent, and efficient procedures. These different conventions can be differentiated analytically; in real life, actors and institutions face the challenge of combining elements from various conventions and of finding compromises between tensions that arise between these worlds.

These conventions can be regarded as “always already present” cultural resources which structure institutions, including data infrastructures or the different worlds in which data are generated, analyzed and used. Since they have been developed, established, and implemented over the course of decades and centuries, different conventions may be more or less prominent depending on the concrete political and social circumstances. Their relative relevance may vary – both across historical periods and between different political and social contexts (e.g. across the West or between the global North and the global South). But overall, these conventions serve as globally available repertoire of evaluative logics that actors or institutions can draw on when they face the need to legitimize or criticize the relevance and validity of statistical data. Every statistician has an implicit recognition of the presence of this plurality of conventions as normative “substructure” for the data generating process, when acknowledging that there is a legitimate and adequate “definition” of categories, that there is self-evident “space for interpretation”, that data are always (only) more or less appropriate for the representation of a social issue at stake, not because of a lack of accuracy but rather, for example, because of a lack of relevance for collective and public action.

It is evident that the industrial convention is of outstanding importance for any modern institution, situation or “world” in which numerical information is generated and used on a wider basis. But the industrial convention will usually be forged into (sometimes complex) compromises with other influential conventions. This results in changing roles and meanings of the industrial rationality depending on the different compromises that are formed in a given situation. In many scientific contexts, the industrial convention and the convention of inspiration are the most important quality conventions used to legitimize data quality. Other organizations such as official statistics institutes often have to mobilize further conventions to demonstrate their societal legitimacy and relevance. In this regard, the civic convention and the convention of opinion have become more and more important for official statistics, to justify the need for independent official statistics in contemporary societies. This development is mirrored in debates on the future of public statistics. For example, the nascent notion (and job profile) of “information stewards” can be read both as a sort of “interface” between different data worlds and as a “conventional compromise” that combines elements from different worlds, including the market world and the network world.

3.The plurality of data worlds and the rise of tensions

In empirical data worlds, all of these eight quality conventions are (even if “virtually”) present as logics that actors can draw on to design and implement research infrastructures and to successfully perform tasks of coordination. Even if in most empirical cases the number of conventions that become effective is small, quality conventions are nonetheless usually combined; it is their interplay that marks given hegemonic institutional logics. Different such “coalitions” of quality conventions enable and structure different data worlds. By implication, these data worlds need to be seen as the outcome of historical developments. To a certain degree, they are always arbitrary; other configurations would have been possible and could have evolved.

4.Tensions and critiques

The Corona crisis can be mentioned as a typical example of the tensions which can arise due to the plurality of data worlds. Interestingly, the most influential provider of latest data on COVID-19 infections and mortality is not an agency anchored in the field of official statistics, but the Johns Hopkins University, a private US American university which gathers and publishes relevant data for countries worldwide, but also on a regional level. The data of this university were used to criticize information published by national institutions such as the Robert Koch Institute (RKI) in Germany or the Federal Health Department (BAG) in Switzerland, based on the argument that state-driven institutes were too slow or that the data produced by their infrastructure was not efficiently organized. Vice versa, institutes such as the German RKI insisted on the profound and ex-post proven quality of their data. In Switzerland, it was the national media agency (SRF) which implemented a data unit that gathered data on infections and deaths from Swiss cantons to be published as quickly as possible.

These developments mirror important tensions between data worlds that become effective in the form of mutual criticism. Table 3 represents the fundamental critiques which each of the outlined three data worlds voices (in the columns) in relation to the other two worlds. It would be misleading to portray the tensions and critiques as based on controversies about single – and for many evaluations too simple – criteria such as adequacy, timeliness, and so on, or in fact any fixed and finished “criteria catalog”. Instead, we argue that different data worlds are based on quality conventions that express different institutional rationalities, which again are linked to varying deeper logics of evaluation, valuation and interpretation, including forms of methodological reasoning. Investigating these different underlying quality conventions allows the identification of the interlinked normativities and epistemic values, which again articulate different conceptions of public action and the common good.

The plurality of data conventions as well as their complex entanglement with institutional forms, political rationalities, and concrete data practices have important implications. In a nutshell, official statistics needs to actively discuss methodological questions in relation to how statistical data and knowledge production is embedded in political, economic, and social formations and situations. It is not sufficient to criticize other data actors for their evidence lacking rigor or quality or for being “unsound”. Instead, we need to think about how the production of data as well as of “facts” always depends on contextual conceptions of the common good.

Related to the problem of plurality, official statistics also needs to move beyond its state-centered identity which conflicts more and more with the increasingly neoliberal character of societies on the one hand and with an emergence of influential social movements and NGOs on the other hand. Against this background, the political definition of the common good as well as the concrete instruments of “governing by numbers” are themselves becoming increasingly contested. This development has two important consequences. First, official statistics are less and less employed to just represent social realities, but increasingly to consciously transform and govern the social world. Political approaches such as evidence-based policies, benchmarking, and indicator-based regulation of social practices and organizations indicate a profound shift in the role that official statistics play. In this context, facts can never be considered as neutral, pure, or innocent. Second, the function of providing the numbers by which societies are governed is in many cases actually already fulfilled by private actors. For example, the German “Bertelsmann-Stiftung” has become, as a private agent, deeply involved in the public representation and governing of social affairs. In other contexts, official statistics are actually already exploring options for coalescing with the “big data world”; for example, the obligation to deliver numbers on completely new social indicators defined in the UN’s Sustainable Development Goals has led statistical agencies to promote and actively launch “data alliances” in fields such as migration (see e.g. the “Data for Migration Alliance”, data4rmigration.org). The world of official statistics is thus already deeply entangled with other data worlds. As a result, it needs to reflect upon conceptions of the common good as well as of “good data” that prevail in these other worlds. In this context, scientific standards have to be conceived as normative, and not in a positivist manner as self-evident and given, and “data” as unquestioned mirrors of social reality. In this sense, the discussion of “fake news” versus “scientific truth” is misleading. Any claim that official statistics is soundly based alone on science threatens to become a false front as long as it is not linked to opening the “normative black box” underlying any process of quantification or datafication.

Further profound questions arise from these observations. To give an example, the production of data on the social world is necessarily linked to practices of classification. Over the past centuries, the state was in control of implementing and of legitimizing social categories (on the basis of the law, but also reflecting public debates and social movements). With the rise of big data, other actors become involved in the “business” of classification. The states’ power to define categories is vanishing and social categories emerge faster and in a more decentralized fashion in other contexts. This leads to the serious issues of who controls these processes of classification, who monitors their social consequences, and who organizes spaces and possibilities for discussing and criticizing them. The question is, how to include social deliberation into the foundation of national statistical institutes and official statistical categories? The task of the social sciences in this context is to increase the awareness of the convention-based nature of measurement and statistics.

In our view, the key strategical question which will decide about the future role and legitimacy of official statistics is how official statistics engages with struggles for common goods and debates about their definition. Will and can official statistics act as an ally for new political actors such as NGOs, social movements, media agencies, but also for those active in the context of citizen science or open data? Only if and in so far as this perspective is taken, one can truly speak of “public statistics”.

As this contribution has made evident, the important opposition for official statistics is not “fake news” against “scientific data” [21]. The important challenge for the data world of official statistics stems from the opposition that is currently evolving between the big data world “versus” the civic data world. One key future reality test (in the sense of Boltanski [1]) will lie in how official statistics deals with the challenge of big data, which is mainly driven and controlled by big tech companies. What is an adequate institutional response to develop data infrastructures that (1) serve some clearly defined public interest (or common good) but that also (2) can prevail in the face of asymmetrically powerful privatized big data infrastructures owned by big Internet and other data companies? Can official statistics become more “fluid” to form coalitions with new public movements and structures? Which (experimental) institutional principles could provide a basis for organizing these coalitions?