Next generation Open Access analytics: A case study

1.1.An era of experimentation

We characterize the current environment as an era of experimentation in relation to OA analytics, perhaps similar to pay-walled analytics before the adoption of COUNTER standards.

We’re seeing innovative new business models developing in addition to the traditional models of funding Gold OA with an Article Processing Charge¹. For example, Subscribe to Open or the Public Library of Science’s (PLOS) Community Action Publishing model².

In turn, this is generating a variety of different goals for usage analytics, ranging from simply counting usage of a particular business model, to identifying organizations that have accessed that content all the way through to granular analysis of topics/subjects to support editorial development.

Organizations are using a variety of metrics to support this activity. Those relying on generic analytical tools like Google and Adobe Analytics will be getting very generic metrics like sessions, page hits, and downloads. Some are using COUNTER metrics. And some are crafting home-brew solutions.

There is a proliferation of stakeholders interested in analytics far beyond the library. And there is a lot of money at stake. Much of that money is currently sunk into traditional pay-walled models, and OA analytics are a crucial tool in informing negotiations on all sides. And another large chunk is sitting with funders who have influence on where the research they fund gets published.

1.2.What about COUNTER Reports?

COUNTER Reports could seem to be an obvious answer to this problem, but they don’t really work here because they’re engineered for a very specific use case - to help a librarian understand and compare usage of paid content in order to make informed decisions on acquiring content to meet their library’s learning and research goals.

This COUNTER use case rightly focuses on the aggregate numbers, and prioritizes scalability over usability because, in most situations, these usage numbers will be further aggregated and analyzed using software that can automate bulk ingestion via SUSHI feeds.

However, this does not mean that COUNTER has no role to play. Far from it, because “metrics are the gold nugget at the heart of COUNTER³”. Think of COUNTER metrics as a defined set of Lego bricks that we can re-combine to create new, valuable analytics reporting.

Fig. 1.

Open Access stakeholders.

2.1.Stakeholders

First, let us recap the stakeholders involved in the OA publishing workflow (Fig. 1):

2.2.Roles

Next, let us walk through some of the roles within this pool of stakeholders that need some form of OA analytics (Fig. 2).

Most obviously, we have the traditional library role, which sits at the core of the COUNTER Reporting use case. Librarians are already tracking usage of OA content as part of licensed collections. Those that also play a role in pure OA publishing will be equally interested in monitoring that usage.

We also have institutional roles that sit outside of the library and are focused on research management, such as the Senior Research Officer. If you haven’t already read Roger Schonfeld’s fascinating Scholarly Kitchen post about this role⁴, we heartily recommend it. This role is going to be more interested in understanding how usage of OA content ties into the institutional research priorities.

On the publisher side, we have development roles that need to understand which organizations are getting value from OA content in order to identify potential future sources of funding. When the publisher is a commercial entity, it is more of a Business Development role. Publishers also have editorial roles that want to understand the subjects and topics that are engaging the community.

Authors want to understand the impact of the research that they publish. In some cases, this information can influence their choice of publisher.

Similarly, funders want to understand the impact of the research that they have funded. Did it reach the communities they were targeting, or perhaps it was also engaged by new communities of which they were previously unaware?

We also have various flavors of intermediaries, such as service providers such as Knowledge Unlatched (KU)⁵, consortia such as Jisc⁶, and distributors such as JSTOR⁷.

Last, but not least, the community itself has an interest in OA analytics. This is particularly important for publicly-funded organizations such as state colleges who need evidence that OA content is reaching the broader community.

Fig. 2.

Open Access use cases and the roles associated with them.

2.3.Metadata

We are seeing a clear need for a broader range of metadata to enhance the traditional COUNTER metrics of searches, investigations and requests.

One is metadata on the audience for OA content. What type of audience is engaging with content; is it identifiable organizations or unaffiliated usage? Geographically, from what region is usage coming. From what categories of organization, such as government bodies vs higher education vs corporate? And on what areas of specialty do those organizations focus, such as biomedical research vs cancer research.

A second example is metadata on the subject/topic of OA content. This would provide very interesting analytics back to Senior Research Officers, as well as to authors, funders, and editors involved in shaping future content priorities.

A third example would be metadata on the source of funding. It would be interesting to see the impact that funders have on OA content, or to do portfolio analysis on funding in terms of the communities impacted by making the content open.

2.4.Other reporting attributes

Finally, here’s a grab-bag of other attributes of reporting that we’re seeing in relation to OA use cases:

3.1.Introduction

The next generation analytics project is a partnership between PLOS and LibLynx to explore this developing area.

The project has three goals:

3.2.OA counter reports

Goal 2 is to provide COUNTER Reports to meet the needs of PLOS’ stakeholders. These COUNTER reports look exactly the same as a normal COUNTER Report - the only difference under the hood is that we are attributing usage to organizations based on matching their IP address to the data in the IP Registry, rather than as a result of controlled access to pay-walled content.

The example Item Master report below (Fig. 3) shows COUNTER investigations attributed in the month of December to IP addresses associated with Stanford University. It lists seven hundred and eighty-three PLOS OA articles that were investigated during that period.

Fig. 3.

Example OA Counter Report for Stanford University.

These reports will be going live to PLOS customers in early March 2021.

3.3.Next Gen Analytics Prototypes

We have also been working on prototypes for more interactive analytics that could meet these broader use cases (Fig. 4). They would provide access to real-time, granular data with interfaces that would facilitate enquiry and support a broader range of metrics and related metadata.

The first example is a report enabling a user to pick an organization associated with a usage and drill down to the articles and authors with whom users from that organization engaged.

Example two flips that around to pick an article and drill down to a list of organizations that accessed that content.

Example three shows the geographical spread of access for a particular article.

In all cases, you can interrogate the analytics with multiple dimensions, such as show me all the usage of an article from governmental organizations in Spain.

Fig. 4.

Examples of prototype interfaces for Next Gen Analytics.