Influence seekers: The production of grey literature for policy and practice

1.Introduction

While digital technologies have radically increased our capacity to produce and disseminate knowledge and information, many of the social and economic benefits are being lost as researchers and policy makers struggle to filter search results, find relevant material and evaluate the huge variety of resources being published online by a wide range of organisations (Lawrence [20]). Given that a great deal of money and resources (much of it public funds) are spent creating knowledge and information to improve outcomes on public interest issues, governments and producers should seek to maximise its benefits for the community. One way in which evidence for policy is produced and disseminated is by academic researchers publishing in books and peer-reviewed journals. Unfortunately access remains a major issue for many policy workers in government, civil society organisations and industry due to costly subscriptions. Another important source of information and forum for debate are newspapers, online sites and blog posts where researchers and others try to translate their work for a general audience. However a large but often overlooked source of knowledge for policy and practice are the documents produced and disseminated by organisations, outside of the commercial or scholarly publishing industry, such as technical and research reports, working papers, policy documents, evaluations and briefings. These are sometimes referred to as grey literature, a term which may obscure more than it illuminates but which is a useful collective noun for a wide variety of formats and publishing approaches which do share some important common properties.

Grey literature may seem to be an issue for a by-gone age of small print runs, fugitive documents and specialist clearing houses set up to try to collect them, of interest only to historians and of little importance in the internet age. In reality the opposite is the case. Like other forms of communication, grey literature has moved online and this has had a huge impact on the way it is able to be produced, disseminated, discovered and used. Despite this, grey literature as a form of electronic publishing is a phenomenon that is often overlooked and has been under researched [25]. As Thompson puts it about the publishing industry as a whole, “A revolution has taken place in publishing and is continuing to take place, but it is a revolution in the process rather than a revolution in the product” (Thompson [32]). Reports and discussion papers have been produced and disseminated for decades if not centuries, with the precursor to the modern version being the political pamphlet used to campaign on all manner of social issues from the 17th century onwards (Briggs [4]). In a similar way to the printing press, the internet has enabled the quiet but steady increase in the number of organisations, government departments and agencies, university research centres and think tanks producing and disseminating policy related publications and other resources. Without widespread awareness they have become central to public debate, informing and contributing to a wide range of topics and disciplines.

Prior to the advent of the internet, one of grey literature’s defining characteristics was that it was costly to print and distribute and difficult to find and access, in contrast to journal articles which may have imposed a charge but were professionally managed and indexed and had stable subscriptions and distribution channels (Lawrence [19]). The internet has turned this on its head, allowing individuals and organisations to cheaply and easily send their reports and documents around the world to peers, partners and other organisations. “Technology has changed the relationship between the provider and the recipient of information but even more radically it has ‘changed the information chain itself’, with new intermediaries taking the place of traditional players” (Feather [13], pp. 84–85). It is unsurprising that many organisations have seized on the potential of desk top publishing software, PDF technologies, email and websites to create, publish and distribute their own publications – particularly where urgent, important or contested social or political issues are at stake. Grey literature has therefore become a key tool for many organisations and should be considered at the forefront of the digital publishing and open access revolution, but because it is not managed or produced in a systematic way, and not counted in academic reward systems, it has often been invisible, undervalued as a resource and under-appreciated as an influence on policy and practice decisions. I would argue that instead grey literature should be perceived as being at the van guard of 21st century electronic publishing. Grey literature is prolific, heavily used and highly valued, particularly for public policy but also for specialist domains with a policy impact such as climate change. The International Panel on Climate Change reports are not only an example of important grey literature documents, they have concluded that they cannot produce their reports without some reliance on grey literature, despite the concerns about its legitimacy (Macdonald [24]). Yet at the same time grey literature creates many problems for producing organisations struggling to understand and meet the standards of professional publishing, bibliographic indexing, online document management, copyright and preservation as well as for users and collecting agencies. The lack of recognition of grey literature’s scale and importance has also led to a serious under investment in long term management of public interest digital content – our collective public knowledge commons.

2.Grey literature research

The concept of grey literature as an object of academic research has mainly been of interest to the discipline of library and information science (Farace [12]) and somewhat overlooked by sociology or media and communications researchers. Perhaps this is due to a general lack of focus on media’s relationship to society by sociologists until the 1990s (Couldry [7]) or a lack of research on contemporary book and journal publishing industries (Thompson [31]). It is certainly the case that there is little use of the term grey literature in either field and it is not generally recognized as a type of media or communication. The approach taken here is to consider media in a broad sense following Couldry who defines media as being much more than newspapers, radio, television and film, it is the “institutionalized structures, forms, formats and interfaces for disseminating symbolic content” (Couldry [7]).

On this basis it is argued that grey literature is not only of interest as a challenge for information professionals but that it operates as a type of media. It is symbolic content produced in a variety of forms and formats in institutionalised structures, however these are not the institutions that we are used to consider as media producers. It is because grey literature is, by definition, produced by organisations whose main purpose is not commercial or professional publishing, which leads to its distinctive interest as a form which requires focused and specialized analysis. As Lobato and Thomas point out, the way that media economies are organized and regulated has important social consequences: “Systems of communication shape our understanding of the world and help us define who we are, as individuals and as communities…Media economies – as systems that organize this communicative capacity – are gateways for power, politics, and pleasure… ” (Lobato [22]).

At the same time the infrastructure that has enabled the explosion in grey literature production and that is required for its long term access and preservation is also part of what needs to be studied. The ‘computational turn’ in the social sciences seeks to examine the structuring aspects of the software, tools and systems that are available and what that enables (Burgess [6]). ‘Infrastructure studies’ or ‘knowledge infrastructure’ (Edwards [10]) involves an examination of how they shape and define our world: “As knowledge infrastructures shape, generate and distribute knowledge, they do so differentially, often in ways that encode and reinforce existing interests and relations of power” (Edwards [9]).

There are some similarities that can be drawn between grey literature production to the discussions of user-generated content or what Castells calls “mass self- communication” – such as its speed, flexibility, targeted audience and open dissemination and circulation, and highly variable standards. Yet they are also significantly different in their institutional basis and the scale and sophistication of their outputs. While some include social media as a form of grey literature, this article is not concerned with social media specifically. Nor will it focus on the widespread production of technical reports in engineering, or field notes in archaeology or other major grey literature production sources. This paper is concerned primarily with public policy and practice oriented organisation-based publications, or grey literature, or what could be described, rather clumsily, as “public publishing”, as in public broadcasting and public libraries – given that much of it is funded by governments and made public in the public interest or with the aim of participating in public debate.

3.Methodology

The impetus for this research has been my 11 year involvement with Analysis and Policy Observatory (apo.org.au), previously known as Australian Policy Online, an open access digital repository and alert service established in 2002 by researchers at Swinburne University of Technology and supported by many other universities and organisations over the last 15 years. APO collects and catalogues policy and practice grey literature resources and as an open access collection is an example of a public knowledge infrastructure project contributing to the global knowledge commons although it may not meet all the principles of open scholarly infrastructures the participants aspire to this level of public service and commitment to public value (Bilder [3]). As the service has developed over the years the need to better understand the proliferation of policy reports and papers and the diverse ecosystem of producing organisations led to an Australian Research Council funded project to investigate the production, use and collection of grey literature for policy and practice, known as Grey Literature Strategies. This research has involved online surveys and interviews with producing organisations, research users and collection services and a survey of digital collections [21]. This paper focuses on the findings for producing organisations. Other results from this research have already been published (see [3,10,20]) and further publications on users and digital collections will be forthcoming.

To establish who is participating in grey literature production, how and why, we conducted an online survey of producing organisations. To complement and probe the survey results we also undertook semi-structured interviews with representatives from organisations engaged in production of research publications on public policy issues in Australia. The survey and interviews asked about the kinds of resources produced, how important they are, how many staff are involved in production, sources of income, and the reasons for direct publishing.

Producing organisations were asked to identify themselves as being in one of four sectors: Government: Federal, State, local government departments or agencies; Education: Universities and TAFES particularly university research centres and institutes; Non-government organisations (NGOs): associations, interest groups, think tanks, charities; Commercial or private businesses: large, medium or small companies; business groups; consultants and research companies, lobbyists. Respondents from all four groups were sought via direct emailing to some of the 3,500 organisations listed as sources on APO at the time of the survey as well as general online promotion on APO and other websites and newsletters including via Linkedin and Twitter. In total 155 organisations responded to the online questionnaire, with 144 from Australia (93%), 7 from New Zealand and 4 from other countries. As the vast majority were from Australia, only these responses have been included in the analysis for this paper (N=144).

Estimates of the population of producing organisations were needed to provide estimates of the scale and value of grey literature production. Calculating the grey literature producer population is difficult as it is likely to be a subset of organisations across all sectors of society. Based on various data sources such as the Australian Bureau of Statistics it is estimated that across government, education, civil society and industry there could be at least 30,000 organizations producing policy-oriented grey literature in Australia. This figure is a conservative estimate based on the following figures: at least 2000 departments and agencies across the Australian federal government, state and territories and local governments; 41,008 not-for-profit organisations (ABS [26]); 56,894 registered nonprofit institutions (NPIs) (ABS [1]); 250,000 businesses in Professional, Scientific and Technical Services (ABS [8]) where at least 10% of these (25,000) may well be producing policy-oriented grey literature; 11,770 management consulting firms (Ibis World [18]); and an unknown number of large companies who may be producing grey literature relating to policy issues affecting their industry.

Survey respondents came from all four sectors with 38% from non-government organisations, 35% from education, 21% from government departments and agencies and 7% from commercial companies. About half of the organisations responding were small, having 10 staff or less, and a further 17% had up to 20 staff. About 10% had over 500 staff, these being mainly government organisations. About two thirds of organisations had 10 staff or less engaged in producing research and information publications, which given their overall size indicates considerable focus on knowledge production. Amongst the surveyed organisations, the production of research and information occurs at a fairly steady pace with a third (38%) publishing on a weekly basis or more often, and two-thirds (62%) producing material quarterly or more often. Staff estimated that they spend around a third of their weekly work time creating grey literature each year.

4.Why organisations engage in the research and publishing

Despite the different drivers and incentives that exist across government, academia, civil society organisations and private companies, the survey showed overwhelming agreement about the motivation for organisations to produce their own publications. As Table 1 shows, the most important reasons for producing research and information are to provide an evidence base for, and inform public policy and practice (92%), translate knowledge for public use (84%), and maximise public access to research and information (79%). Those in education and the NGO sectors also rated sharing findings with peers, raising their organisation’s profile and attracting media coverage highly, more so than those in government or commercial sectors.

NGO’s strongly value using research as an advocacy and lobbying tool (87%) and appreciate the flexibility of grey literature formats (71%). NGOs and governments most valued being able to control the timing of a publication (61%), significantly more than the other two sectors. The education sector also valued meeting organisation or funder requirements (71%). Two thirds of government and commercial organisations rated using publications for internal purposes as important, higher than NGOs or education organisations. This corresponds with the arguments made about the contested nature of policy evidence and the way in which competing interests participate in evidence production (Wesselink [34]). Notably, financial gain was not highly valued with only 11% of respondents overall identifying this as an important or very important consideration and only a quarter of commercial respondents.

If selling content is not a major motivation in the production of grey literature, how is income generated? The production of grey literature is funded mostly from the public funds through grants and commissions and not via sales or subscriptions, unlike commercial or scholarly publishing. Table 2 shows that funding agreements and contracts are the most important source of income for two-thirds (63%) of organisations, followed by grants, which are important for around a half (52%), but important for three quarters (77%) of those in the education sector. Only a tiny four percent actually report earning revenue from sales or subscriptions. While some producing organisations are based in the private sector or represent business interests, it is probable that most of the material produced by government, NGOs and education is paid for through public funds.

When asked who is the target audience for their material, the government sector was rated as the most important for 96% of respondents across all sectors. Even those in government are trying to communicate with their colleagues as a priority. To reinforce this finding, politicians were the second most important audience for 80% of organisations. The third key audience group was practitioners (74%), indicating that producing organisations are trying to influence not only policy, but also its implementation.

5.What gets produced and how important is it?

The types of resources that are produced, circulated and consumed for policy and practice work are many and varied. Some can be characterised as research, defined as creative work undertaken on a systematic basis in order to increase the stock of knowledge and to use this knowledge to devise new applications (OECD [14]). Since the 1990s the demand for research and particularly ‘evidence-based policy’ has steadily increased and is often associated with promoting the use of peer-reviewed journal articles, systematic reviews and more recently a movement promoting the use of randomized controlled trials in public policy in the UK (Pearce [28]). However academic research and peer- review can both exist in publications produced directly by organisations. A range of other kinds of information and contextual knowledge also plays a part such as procedural information, policy or political statements, practitioner experience and so on (Nutley [27]). Less rigorous types of investigation may provide new and useful insights on public interest issues, such as project reports, discussion papers, case studies, submissions or evaluations. And some grey literature is more informational knowledge – the translation of research as information sheets, reviews or guidelines, or the production of non-research content, such as procedures, policies, plans and strategies, stakeholder views and advocacy documents.

Producers were asked to indicate, from an extensive list of 25 resource types (see Table 3) including journals, books, data, reports, briefings, evaluations, news reports and many more, what their organisation produces and how important these are for the organisation’s work. Overall conference papers are the most common resource, produced by 82% of organisations, followed by discussion papers (77%), reports (76%), briefings/reviews (67%), and submissions (63%). Over half of all respondents also produce webpages/websites, news reports or media releases and information sheets. NGOs tend to produce more discussion papers and submissions, indicating the importance of advocacy work. They are also the highest producer of news reports and media releases. Government (80%) and NGOs (61%) are much higher producers of information sheets and summaries indicating the very important role of government as a translator of research and policy information for the wider community as well as an audience for research translation.

In a slightly different order, the most important resources overall for producing organisations are reports (58%), discussion papers (56%), briefings/guides (48%), conference papers (47%), and submissions (47%). If we look at results only for those that produce them, the most important materials, produced by over 50% of respondents are reports (93%), submissions (91%), evaluations and data sets (90%), discussion papers and briefings/reviews (89%), and news articles/media releases (76%). Journal articles are produced by 43% of producers and are considered important by 75% of these organisations. One reason for this is that although a small number of organisations in sectors other than education do produce journal articles, there is often little incentive or reward for this material. As an interviewee from a research company stated: “We want to be held in high regard and for our work to be academically rigorous. Our clients expect that of us. But keeping people publishing in academic journals is hard. The priority is the contract, the client, the deliverable. If, at the end of the day, there’s a bit of time, great, write up some articles. It doesn’t feature as high as it perhaps it should (Producer interview, Commercial sector).”

Interestingly conference papers, which are produced by 82% of organisations are only important for 69% of them. It may be that conference papers are produced more as a requirement of participating or organizing conferences, which supports targeted research and organisation promotion and policy networking, rather than as an output in their own right. This is supported by the figures showing conferences are seen as an important dissemination method for 75% of producers.

6.Production and dissemination

While many organisations have excellent reputations as producers of high quality research and policy materials, grey literature overall can be highly variable and is often considered to be not as credible as journal articles and books (MacDonald [24]). One reason for this is a lack of standards and transparency around the way in which research has been conducted, a lack of detail about whether a publication has been peer-reviewed and in what way, and poor bibliographic details and amateur publishing practices. This makes evaluation of grey literature time consuming and fraught, and citation based metrics of grey literature difficult. It also provides opportunities for marketing and advocacy materials to be disguised as research.

Fig. 1.

Production standards and review processes carried out on materials produced by organisations – % frequency.

Some definitions of grey literature state that it is not peer-reviewed, causing further confusion (often to be found in online guides produced by university libraries). In fact a significant amount, but certainly not all, grey literature is formally peer-reviewed or reviewed in some other way such as by an expert advisory group. While concerns about the quality of grey literature are often raised, and are well justified for some material, just under two thirds of organisations surveyed indicate that they ‘often or always’ conduct an internal review or use an advisory group (60%) or have their work peer-reviewed (55%), with about a quarter (26%) using an external board to review prior to publication (26%) (Fig. 1). Almost all organisations surveyed (90%) undertake basic editing and formatting of their content in-house and professional editing is used by 39% of producing organisations. This is cause for considerable optimism as an indicator of a fairly high level of scrutiny and validation occurring within many organisations producing policy grey literature and makes it clear that we cannot assume it has not been peer reviewed or otherwise quality controlled. However it is often the case that the way in which materials have been reviewed is not made explicit and there is no standard way of indicating what kind of review has occurred so some of the benefits are being lost to consumers when they are evaluating a document with no indicators.

Once content is written, revised and ready, documents have been desk-top published and the ubiquitous PDF prepared, organisations need to make them publicly available and distributed to their audience. Publishing and particularly distribution were amongst the most difficult and expensive tasks for organisations in the print era however the transformative difference of the internet means that once the creation and production costs have been covered, dissemination costs are minimal. Producing organisations overwhelmingly choose to publish content directly on their organisation website free to access to anyone with the link, and this is possible thanks to almost zero copy costs. Seventy percent of the organisations surveyed made between 50–100% of their material available online or in print for free in the last 12 months.

Posting a publication on a website does not necessarily ensure an audience and to achieve policy impact may require a more active campaign of promotion and attempts to alert users to the new material, either directly or via third parties such as the media and information services. In the digital world, publishing, promotion, dissemination and access blur as roles converge and traditional activities and business models are disrupted. As Table 4 shows, the most widely used methods of dissemination, rated important or very important by over three quarters of organisations are: providing access via an organisation’s own website (89%), sharing information directly with contacts (85%) sending email newsletters (83%), presenting at conferences or events (75%) and sending out details via other organisations’ newsletters (72%). Over 90% of NGOs rated all these methods as important or very important.

7.Storage, access and long term management of grey literature

In the print era publishing was often described in terms of a chain, where a publication worked its way from an author’s manuscript, to an agent, then a publisher, printer, distributor, bookstores and libraries and finally to a reader (Thompson [31]). Other participants might assist in this progress such as newspapers and magazines publishing reviews but access to the actual item was channeled through key points. The print grey literature chain had some similarities, beginning with either an author or an organization producing content then having this printed. From there things got harder, distribution of print materials was costly, often involving direct mail to key people and special events. Collection and management was also problematic with many items not having ISBNs and therefore not coming to the attention of the legal deposit agencies. Special libraries, ‘clearing houses’ and private collections were required to bring together key resources and publications for niche audiences.

In the internet era, the publishing business may still involve a range of commercial and public services including authors, agents, publishers and organisations, distributors, booksellers, ebook vendors, libraries, subscription or open access databases, and promotion through media outlets. The publishing supply chain involves original content moving from author to agent, then to publisher, printer, distributor, retailer or wholesaler and library and finally to the customer (Thompson [31]). A variation on this chain occurs for scholarly publishing where the authors are usually academics who conduct research (usually publicly funded) which is published as journal articles or books accessible via commercial subscription databases with access usually paid for by a university or organisation library with public funds. Both chains show the many elements that make up the machinery of commercial and scholarly book and journal production, publication, distribution and sale. Grey literature is also produced by researchers and authors based in organisations and funded by the public, with documents then usually published free to access on websites and disseminated via email newsletters and discovered through search engines or possibly repositories or libraries. Or in some cases not published at all, simply distributed through internal channels of government or organisations. Roles have also converged as publishers and producing organisations can commission, produce, publish, distribute, promote, provide access and discovery systems and store for long term preservation. Organisations that are not professional publishers may, or may not, have the experience, personnel, resources or motivation to meet best practice publishing standards or adequately manage all of the requirements and responsibilities to ensure effective and long-term discovery, access and preservation of their publications. In a number of ways electronic online publishing by organisations has made this situation worse, as more and more organisations now produce content and publish it online without adequate publishing standards and little or no plan for long term management. Online publishing has also conflated activities that were previously separate such as promotion, discovery, access and storage with many organisations now undertaking or being responsible for all four of these to varying standards. This is not to argue that commercial publishing is the answer, but that the skills, standards and practices developed over hundreds of years of publishing need to be better understood and applied by organisations if their works are to be used and valued for public policy and practice.

Discovery and access which was previously the role of the bookseller or library is often now being provided by an organization directly as a ‘retail service’. From the survey of producing organisations we found that storage and access to publications and resources is usually managed by organisations themselves with most surveyed producing organisations (85%) ‘often or always’ using their organisation website to store and provide access to their content. Unfortunately websites are notoriously unstable with redesigns often causing hosted files to be removed from online. As an interviewee summarized things: “The problem with the website is that it’s hard to maintain so once you put something up there it’s this static document that then if the website is changed the document could go”. (Producer interview, government sector) Less than half (46%) report having their own repository software to provide a stable hosting platform or long term management. Just over half of the surveyed producers based in education deposit their material with their institutional repository, suggesting that these systems could be better utilised for grey literature produced by universities’ centres and departments. Beyond this there is little take-up of external databases, libraries or other curatorial services. Only 20% of organisations surveyed comply with their obligation under legal deposit to provide a print copy to the National Library or a state library.

8.Link rot and the digital black hole

In preferring the ease and immediacy of their own websites rather than more stable options such as repositories, producing organisations are major contributors to the proliferation of linkrot across the internet and within the reference lists of many publications. Linkrot or reference rot is the loss of access to online content when it is moved or removed, often as a result of website upgrades or changes. Studies estimate that the rate of loss of digital content is around 30% within a few years of publication online (Bugeja [5]). Despite posting most of their content on their own website, only 26% of the producing organisations we surveyed have a strategy in place to prevent linkrot. Of the rest, 42% know they have no strategy in place and a third don’t know either way.

While it may seem like many producers are mainly focused on the short term this isn’t necessarily the case but neither individual authors or the producing organization are in a position to be able to provide long-term management. This is particularly the case for academic research projects or other funded projects. As one interviewee summarized the situation: “Every project that we’ve done we’ve had a website built... Then the project ends, the website you have to keep paying for or you have to archive and if it gets archived after a couple of years the material disappears. It’s so frustrating.” (Producer interview, Education sector)

This is no better for government, in fact possibly worse. The survey data shows that many government organisations have little knowledge of what plans are in place to ensure ongoing access to public sector information, despite recent campaigns for open governments and policies to improve government management of information (OAIC [33]). A recent investigation into the withholding of government research in the UK was surprised to find that there is no easy way of finding out what research is commissioned by government or if it has ever been published (Sedley [30]). Despite the poor management and lack of strategies for dealing with link rot 55% of producing organisations agreed with the statement that ‘Providing long-term access to our online content is not an issue for my organisation’. It would seem that while many organisations are aware of the potential to lose content they are not concerned enough to do much about it. When asked why they don’t have a strategy, the most common reason given by around a third of producing organisations is that they hadn’t had the time or resources to deal with it. Another third either hadn’t thought about it or didn’t consider it important. The loss of online content resulting from poor resource management is seen as a serious issue by only a third (37%) of producers. A further 37% were somewhat concerned, while a quarter thought it was not that much of a problem. The policy cycle moves quickly and while many users find the loss of online content frustrating and time consuming, many organisations have no incentive to manage older content. So if it is not a priority for over half of all producers, who is going to provide long-term access to publicly funded policy research? It is interesting to compare this attitude to that of commercial publishers who have realized that their backlist of electronic books and journals is a potential asset that they can continue to sell or even resell to libraries and subscribers that had already purchased print copies (Bates [2]). This comes back to the issue of the financial model of public interest publishing which has public funding supporting production but does not adequately address access, discovery and management of publicly funded grey literature.

9.Improving production practices

From these survey results and interviews we can begin to see that grey literature operates at a scale and importance that needs to be taken seriously as part of the public policy debate and as a form of communication and knowledge production. Yet while this content is of great value, usually paid for through public funds, it is not produced in any systematic way that will ensure it is available to discover or access much beyond the media cycle in which it is intended to feature. Publicly funded research should include provision for the sustainable management of outputs and collection strategies. Expectations tied to funding create incentives for large-scale change that can be managed flexibly and with discretion at the individual and organisation level. For example grey literature should be integrated into future assessments of research impact and quality currently being reviewed by the Australian Research Council (ARC [11]). Similarly copyright reform to introduce fair use principles will support greater sharing and reuse of policy resources (Commission [29]).

With so many organisations producing material, evaluating the credibility of their work often requires knowledge of organisations in the field and their role and legitimacy. The task is often made harder because many organisations do not include adequate bibliographic information in their publications, and do not work with collecting services to improve discovery and long-term accessibility. As we have seen from the survey results most organisations manage their own production and publish directly online. Despite the use of some review systems and professional editors, professional publication standards are often overlooked. This is an issue that is raised constantly by those using or collecting grey literature. Clearly there is considerable scope for producers to improve standards. Simple steps would be to ensure basic bibliographic information is included in all their work, together with a clear statement of any reviewing process. Many organisations may simply be unaware that certain information is essential for users and collectors to be able to make an assessment of a document. Given this, it would be relatively easy to improve publication standards of grey literature with the development and adoption of clear publishing guides. By including essential bibliographic information – date of publication, authors, producing organisation, a copyright or creative commons statement, page numbers, and a web address – producing organisations could reduce use and collection costs and have a major impact on the accessibility and credibility of their work.

Producers can make sure their publications look professional and can be discovered by search engines and by readers, curated – by information services, clearing houses and libraries, evaluated – by anyone who wants to use it, and cited and measured by including basic bibliographic information on every resource published. A mnemonic to help remind authors and producers is T.A.P. D.A.N.C.E. This stands for Title and subtitle, Author, Producing organisation, Date of publication, Abstract or description, Numbers (and preferably title) on each page, Copyright or creative commons statement, and Electronic-location and/or Identifier e.g. a URL, DOI, Handle etc. A handy one page flyer version of this is available for anyone to download (See http://apo.org.au/node/42306). More collaboration and integrated systems between grey literature producing organisations, collecting organisations and others involved in providing and supporting public access to knowledge may help to develop an understanding of what is required for long term access and management. Governments as major producers, users and managers of information have a clear role to play but will also need the assistance of all stakeholders.

10.The economic value of organisation publishing

A key question for any research into grey literature is what is its value given most of it is made available for free. To address this question the project involved John Houghton to develop a methodology for estimating the economic value of grey literature production, based on reported time spent producing publications calculated to the national level based on the estimated population of grey literature producers discussed earlier. Houghton is well known for his work on estimating the value of data and of open access (see (Houghton [17]; Gruen [15])) and similar processes were applied to the case of grey literature. Australian respondents reported spending an average of 31% of their working time per week creating grey literature during the last 12 months (N=92). Similarly, Australian organisations reported spending an average of 12 hours per week creating grey literature during the last 12 months. That is a total of 1,122 hours for the 92 respondents. At average Australian weekly wages plus on-costs, average annual grey literature creator costs amount to around AU$29,385 per person per annum, or AU$377 million per annum across the survey respondents (US$285 million). If the population of grey literature producer organizations is 30,000, and their average staff numbers are similar to those of our survey respondents and their grey literature activities are, on average, one-third those of respondent organizations, then total national grey literature creation cost might amount to some AU$30 billion per annum (US$22 billion). Respondents reported their organization or department spends a total of AU$234 million per annum on projects that result in the production of grey literature, an average of AU$3.3 million per annum per respondent. On this basis total national grey literature related project spending could be around $33 billion per annum (US$25 billion). National R&D spending in Australia is AU$28 billion per annum so this seems plausible. Australian respondents reported generating almost $80 million per annum from the sale and/or distribution of grey literature, an average of almost $1 million per annum per respondent organization (N=80). Scaling reported revenues generated from the sale and distribution of grey literature suggests national revenue of $10 billion (US$7.5 billion). This implies a 32% cost recovery, with much grey literature made freely available.

It clear then that grey literature production represents a huge investment in money, time, resources and expertise and one that has been increasing over the last decade as tools and technologies have made production easier and therefore more achievable by organisations. This has been a major benefit for the community in being able to access information and evidence but without a corresponding investment in collection and management much of this benefit is not being realized. As Clifford Lynch put it recently, this is an advocacy challenge that the information sectors needs to recognize and address urgently:

“Moving beyond the scholarly record, there’s an enormous crisis in preserving and curating the broader cultural record that will be needed to support future scholarship: social media, news in the digital age, and so many more things: popular e-books for example, as well as the new digital “gray literature”. We don’t even have any good ways to measure how badly we are failing at this. The current copyright laws are going to be part of the problem here because they totally fail to recognize the broad societal need to preserve the digital cultural record as an essential and critical priority; dealing with this is going to be a very broad public policy discussion and advocacy challenge, not a narrow and technical series of legal arguments, and we need to be prepared to advance this discussion.” (Lynch [23])

11.Conclusion

If we are to have a more evidence-informed public sector in Australia or any other country, there needs to be greater recognition and long term support for the diverse range of data and publications that can be and are being produced for public policy and practice as well as investment in technical and managerial skills for producing, managing and providing access to this wealth of material. This requires a policy culture that is supportive of transparency and knowledge sharing and a willingness to invest in new forms of public knowledge infrastructure. Public policy is a highly contested space with diverse producers and sources of knowledge (Head [16]). Given the scale and significance of grey literature in public policy and the level of public investment, we need to ensure that it is produced to a standard that supports easy evaluation, correlation and analysis and that there is adequate investment in its management and collection so that it can be discovered and accessed today and into the future.