What is the point of benchmarking e-government? An integrative and critical literature review on the phenomenon of benchmarking e-government

3.1The literature review process

The journal articles and conference papers where delimited in a process consisting of five steps. The first step determined the relevance of which central concepts to use. This study concerns e-government and based on previous research the definition of this term varies. At least nine variances of the central concept are found,11 based both on previous discussions on the term e-government (see e.g. Grönlund, 2005a, 2005b; Grönlund & Horan, 2005; Heeks & Bailur, 2007) and the authors’ professional experiences from the field of e-government. A simple search for each term was carried out in the two major databases for IS: Scopus and Web of Science to see the frequency of each term. Using the results and the understanding from previous literature, the three most frequent terms were chosen to be included in the search.22 The latter delimitation was a simple matter of keeping the number of searches to a manageable amount. However, in order to make sure that any possible further studies were not missed, step five of the search processes included studying the references of each article for further studies that might be relevant. This gave no further results and this study therefore draws the conclusion that including more search-terms would not yield further relevant findings. Furthermore, these three core terms are also found to corroborate with the most common terms used in major benchmarks of e-government such as those conducted by the UN, EU, Brown University and Waseda University.

The next step was to find key terms for describing the type of phenomenon of interest. Six terms were given from the start: benchmark, ranking, index, indicator, performance and assessment. Looking at the research on the term benchmark, these concepts can be seen as apt alternatives. All terms, except perhaps assessment, were also derived when looking at some of the most common large international benchmarks of e-government. As in the case of the previous three concepts, it is argued here that by studying the reference-list of the articles finally included in the review should have shown further relevant studies. The third step was to combine the three key-terms with each of the central concepts while executing searches in the two databases respectively. For example e-government AND ranking, electronic government AND index et cetera. This meant 36 searches which yielded 3,590 hits.

In the fourth step the study proceeded by reading the abstracts with the purpose of getting an immediate overview. The main inclusion criterion was to look at journal articles and conference papers written in English that deal with the phenomenon of benchmarking e-government. This focus on the meta-discussion on international rankings and indexes for e-government excludes studies creating their own methods for measuring e-government, or that, for example, assess the economic impact of e-government or look at other effects of e-government and why various e-government projects fail or succeed. It also excludes studies that in passing mention the phenomenon of measuring e-government without this being the main focus. This process left a total of 83 articles and conference papers of possible relevance.

The fifth step meant getting a deeper knowledge of the remaining articles. In practice this entailed another reading of all the abstracts, together with a scan of the main tenets of the content and the conclusion. Articles excluded include those where only the abstract was written in English, and studies that anchored their work in a broad background of measuring e-government but then proceeded to develop a new model and applying it to a certain country or region. This process resulted in the final 27 articles.

These remaining articles were then read in full. The content was sorted in a table that includes three columns (one for each question) and a row for each article (the items). The items were read in chronological order while noting the major findings, and the possible benefits for public sector in the respective columns. The analysis proceeded by comparing how the main findings have changed over time and adding these observations in the final column (for a more detailed view of the findings see appendix “Main findings of research on the phenomenon of benchmarking e-government 2003 to 2016”).

A literature review comes with some important limitations. This has to do with the fact that the answers to the questions in this research are based on what researchers have written on this topic. This means that conclusions drawn e.g. in the “grey literature” or other work done in the public sector are not included.

4.1Waking up to benchmarking e-government – The early period 2003 to 2005

The timing of Janssen’s (2003) research on benchmarking e-government is no coincidence. By the time of his research there had been at least three major measurements carried out since 2001: the EU’s benchmarking of its member states’ public services by Capgemini, the UN’s e-government survey and the Daniel West/Brown University Global e-government survey.

Janssen studies how benchmarks can “influence” or “frustrate” the development of e-government (Janssen, 2003, p. 1). The main issues and findings discussed by Janssen in 2003 are developed further in later studies together with colleagues (Janssen et al., 2004; Snijkers et al., 2007), and will “set the tone” in terms of the core and reoccurring issues and outcomes for subsequent scholars. It is therefore worth summarising his seven points of criticism on the two benchmarks he studied:

A year later Information Polity (IP) publishes two articles on the topic of benchmarking e-government and in the editorial Taylor (2004, p. 119) refers to the articles as critical engagements with “[…] measurement regimes in e-government, regimes which purport systematically to delineate e-government.” One of these articles is by Janssen et al. (2004) who review 18 “international comparative eGovernment studies”.

Their review includes a conceptual framework for how to categorize the various benchmarks, and they raise a number of concerns. One of these is how the overt focus on output indicators risks impacting e-government policy to only focus on “[…] front-office realisations” (Janssen et al., 2004, p. 128). They make a point of how this risks neglecting other, perhaps more important, potentials of e-government such as information-sharing between different public sector entities and the aspect of more efficient or better back-office integration. The authors recommend careful evaluation of the results from the benchmark and hint at the sensibility of avoiding their dictates.

The second article in IP identifies 41 different approaches to “[…] monitoring e-government development […]” (Kunstelj & Vintar, 2004, p. 135). These include international country rankings, intra-country and regional comparisons and even scholarly work, which is a significant difference in definition of the field from Janssen et al. (2004).

One main finding identified by Kunstelj and Vintar is about how a majority of studies apply a too narrow definition of e-government. They also find a lack of theory behind the measurements that can provide a holistic and overall assessment of e-government. More crucially, they find no assessment of back-office integration or transformative potentials such as services tailored around citizens’ needs and challenges (Kunstelj & Vintar, 2004).

Two other articles also published in 2004 are by Peters et al. (2004) and Sakowicz (2004). Peters et al. are the first to provide an overview of the literature of measurements into three broad areas: stage-models studies, service literature and performance indicators (Peters et al., 2004, p. 482). Among other, they conclude that measurements are too simple and lack theoretical underpinning that allows for a richer picture of e-government, as well as noting that the measurements are too focused on assessing e-services (Peters et al., 2004, p. 487). Sakowicz’ (2004) mentions 10 different examples of methods for evaluation/benchmarking e-government and discusses their usefulness for countries in Central and Eastern Europe (CEE). He concludes, among other, that the indicators are too focused on technology and supply-side and missing crucial aspects such as user needs and how technology makes people more engaged. This seems to leave him sceptical of the expediency of these benchmarks for e-government development in general, and CEE-countries in particular.

The last article in this early period is Jansen (2005) who looks mainly at the benchmarks from the UN and the EU with local examples from Norway. What sets Jansen apart from the other early commentators in this field is his insistence on the importance of taking the national context into account which he argues is lacking in current benchmarks. Existing indicators are not able to capture the importance of coping with different conflicts of interests when it comes to “national ICT programs” (Jansen, 2005, p. 10).

While commenting on their findings, none of the articles from this period seems to explicitly mention benefits for the public sector in using benchmarks for e-government. This aspect is of course hard to pin-point since the authors are not always assessing how beneficial the benchmarks actually are via e.g. empirical studies, but are rather doing so indirectly by scrutinizing the benchmarks. The exception here being Kunstelj and Vintar (2004, p. 131) who conclude that the overt focus on measuring electronic services has led to “ […] a significant slowdown of development in most countries” (Kunstelj & Vintar, 2004, p. 131). Others discuss how these rankings and surveys promote a context where “short terms [sic] results” are pursued instead of long term effects, which in turn can create risks of poor infrastructure and security (Jansen, 2005, p. 10).

4.2Deepening the criticism and offering pragmatic solutions – The middle period 2007 to 2012

This period is characterised by two broad developments. The first is how scholars continue to expand and develop the criticism offered in the early period and widening this critique – but also highlighting minor benefits. The second is that there is a focus on the history of the phenomenon of benchmarking, and quantitative research that scrutinizes the methodology of the most common benchmarks and offering solutions on how to improve measurements and indicators.

4.3Theoretically and conceptually informed critique – The late period 2015 to 2016

The final period is characterised by theoretically informed criticism coupled with empirical examples, and a move towards criticising past research for lack of conceptual clarity. An example of the former type of research is Codagnone et al. (2015) who studies the case of the EU and the consequences of using the eEurope benchmark. The authors find that the benchmark creates a large-scale process of government institutions copying each other in what is perceived as success, in pursuit of climbing the ranking-system, and actors imitating each other in this way risk hindering the introduction of new practice (Codagnone et al., 2015). Another critical concern they assert is that “the policy prominence retained by supply-side benchmarking of e-government has probably indirectly limited efforts made to measure and evaluate more tangible impacts” (Codagnone et al., 2015, p. 305).

A different approach to the topic is Schlæger (2016) who looks at e-government benchmarking in China. Observing the criticism of supply-side benchmarks, and how research views benchmarks as problematic, the case of China shows attempts both at measuring back-office processes and a positive view on the use of benchmarks to improve e-government. Benchmarking is even a key area in research on e-government (Zhang et al. referenced in Schlæger, 2016, p. 386).

The last article in this review is Bogdanoska Jovanovska (2016). The author finds a lack of clear demarcation of what she refers to as the study of e-government assessment. This is the only article discussing the problem of defining the area and who recognises the use of many different terms, and the author contends that these terms are used “wrongly as synonyms” by researchers (Bogdanoska Jovanovska, 2016, p. 31). Having made this important remark there is no clarification, and therefore a high degree of uncertainty, as to what the preferred term or terms may be.

Based on their suggestion for improvements, Codagnone et al. (2015) are interpreted as holding an implicit assumption that benchmarks for e-government can be useful, but in relation to the current practice and empirical examples and perspective of the EU-benchmark, they are very sceptical. Schlaeger, on the other hand, listens attentively to scholars arguing for the positive sides of benchmarking in China, but the reader is not given any proof or signs of questioning these assertions. Leaving the question of possible benefits out of the picture altogether is Bogdanoska Jovanovska (2016) and who instead discusses the issue in current practices of benchmarking e-government regarding the lack of indicators and frameworks for measuring back-office processes.

4.4Answering the questions

Having reviewed the literature from the perspective of the three questions posed in chapter two, the following three sections will provide answers to each of the questions.

5.1A general characterisation of research on benchmarks for e-government

The research found in this literature review resembles the conclusion of Heeks and Bailur (2007) concerning research on e-government more generally as that of a “fragmented adhocracy”. This means a condition where there is “[…] limited functional dependence of using the findings of earlier work; limited strategic dependence of researchers needing to convince peers of the importance of their work; and high strategic task uncertainty in terms of prioritized problems to research” (Heeks & Bailur, 2007, pp. 260–261). On a more general level, this lack of common engagement with promoting knowledge and communication between researchers to advance and solve problems – instead of monologues – has also been put forward as a problem for social science as a whole (see Alvesson et al., 2017).

Three brief examples can be given in order to strengthen this point. One is how research on benchmarks takes place in silos with no significant engagement between researchers on the phenomenon being studied in terms of following up on previous findings or trying to solve critical conclusions. There are, of course, instances of referencing to past studies, but not as a way of accumulating knowledge in order to constructively build on previous findings or relate to empirical results. Another clear example is the constant repetition of the same criticism and the lack of a common conceptual theme or research agenda. Thirdly, and with the possible exception of the general remarks by Bannister (2007) and Heeks (2008), there is a lack of calls for devising benchmarks that are tailored for the needs, and positive advancement of knowledge, that is situated in the context and purpose of the public sector.

5.2Benchmarks and beyond? Three thoughts for further discussion

Based on the results of the literature review, there are several questions and concerns still left to be discussed. While it is beyond the scope of this research to address all of them, the article will proceed by providing three additional thoughts for practitioners of benchmarks and researchers on e-government.

The first reflection is related to the understanding and definition of e-government discussed briefly in the introduction, and which is a reoccurring theme in the articles studied in this literature review. If the purpose of benchmarks is to promote “the best” way of conducting e-government, it is argued here that these benchmarks need to be more attuned to the context and purpose of the public sector. For example, rather than focusing on measuring what technological advancement can offer – there can be a firmer connection between what public sector is in general as opposed to the private sector and what it does in terms of public administration, and how new technology fits into this. As can be seen in the summary of the reviewed literature, scholars are generally aware of this issue and the typical answer to this problem is to point towards the importance of also measuring “back-office” processes. This might seem as a relevant response, but the problem could be more complex than this – what if the nature of public administration itself is left out? This is a point made by Yıldız (2012) in his survey of “big questions” within e-government research. One way of doing this more concretely is to see e-government in relation to the functions of a public administration – i.e. the processing of information and case handling, what Lenk (2012, p. 221) calls “the nuts and bolts of administrative action”. This means making an effort towards researching how the information is processed and used as part of the core-purpose of an organisation working under sometimes strict judicial demands, and studying the level of policy, as well as management and operatives (see e.g. Lenk, 2012, pp. 221, 223 and 226). In a nutshell, this requires a more realistic definition of e-government that acknowledges the challenges of ICT in public sector – not just what is deemed as the “best performance” – as well as how it can be applied to the benefit and improvement of a context signified by administrative action and high judicial demands.

The second reflection here has to do with the relation between the nature of the benchmark as a method of promoting “best practice” in the context of the often nebulous, and complex, phenomenon of e-government. Is it possible, or even relevant, to only measure what is “the best way” of conducting e-government? There are plenty of examples from the literature review of criticism of benchmarks and the lack of usefulness for the public sector. It could therefore be argued that there seems to be a need for benchmarks to be more attuned to empirical findings and cases of the use of ICT in the public sector – i.e. research that shows a complex, and far from overtly positive, picture (see e.g. Andersen et al., 2010; Bannister, 2012; Danziger & Andersen, 2002; Gauld et al., 2006).

Taking these arguments together into a third point, there seems to be some indication of a poor connection between the understanding of the context and function of public administration and benchmarks for e-government. A constructive solution to this situation can be to argue for the need for entirely new methods for developing e-government together with the public sector, or continuing the work on improving the benchmarks and complementing them with qualitative research. Both paths, improving the benchmarks or going for alternative methods, can benefit from building collaborations and cross-national studies aimed at directly helping to solve problems. These common efforts should be used to provide an open knowledge base for experts, civil servants and managers in public sector and help them tackle – or at least critically discuss – the challenges and possibilities of how digital technology can benefit public sector.

Besides the work already being carried out on improving the quantitative measures of benchmarks, another brief example of how to continue to improve benchmarks is to emphasise, and to a greater extent apply, the concept of benchlearning. The suggestion here is to create a framework within the practice of benchmarking allowing for collaboration between public administration and research, which can produce environments in which knowledge by researchers helps produce sound policy and promote sensible decision making.

An entirely different alternative to benchmarking – which does not necessarily have to exclude the previous suggestion – can be to pursue research done in the overall theme of pragmatism developed under the heading of action research (AR) (Baskerville, 1999; Baskerville & Myers, 2004; Baskerville & Wood-Harper, 1996). In this sense, for example, the five core principles of canonical action research (CAR) as detailed by Davison et al. (2004) is one way to go. CAR is an interesting alternative because of its synthesises of prior knowledge and experiences from preceding debates on the pros and cons of AR and it offers clear guideline in terms of what principles to follow (see Davison et al., 2004). In addition to this, CAR is a type of research design geared towards an iterative and collaborative process towards helping to produce new knowledge and sharper research questions than the ones currently provided. This would allow for the crucial importance of working in a problem-oriented way that is necessary for dealing with the complex matter of e-government. More in detail, the two core components of CAR are useful because:

AR and CAR are not the only alternative methods to the practice of benchmarking – there are many other potential candidates, and this is not the place to mention them all. The key concern here is to promote the need for more qualitative research that is problem-oriented and attuned to the complexities of new technology in the context of conducting public administration.