Affiliations: [a] Data Science and Engineering Department, Faculty of Informatics, Eötvös Loránd University, Budapest, Hungary | [b] Department of Digital Humanities, Faculty of Humanities, Eötvös Loránd University, Budapest, Hungary | [c] Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia
Note: [1] This paper is an extended and revised version of a paper presented at the International Workshop on Socio-Technical Aspects in Security.
Abstract: Web archives store born-digital documents, which are usually collected from the Internet by crawlers and stored in the Web Archive (WARC) format. The trustworthiness and integrity of web archives is still an open challenge, especially in the news portal domain, which face additional challenges of censorship even in democratic societies. The aim of this paper is to present a light-weight, blockchain-based solution for web archive validation, which would ensure that documents retrieved by crawlers are authentic for many years to come. We developed our archive validation solution as an extension and continuation of our work in web crawler development mainly targeting news portals. The system is designed as an overlay over a blockchain with a proof-of-stake (PoS) distributed consensus algorithm. PoS was chosen due to its lower ecological footprint compared to proof-of-work solutions (e.g. Bitcoin) and lower expected investment in computing infrastructure. We based our prototype on the open-source Nxt blockchain and implemented it in Python. The prototype was tested on web archive content crawled from Hungarian news portals at two different timestamps with more than 1 million articles in total. We concluded that the proposed solution is accessible, usable by different stakeholders to validate crawled content, deployable on cheap commodity hardware, tackles the archive integrity challenge and is capable to efficiently manage duplicate documents.
Keywords: Web archive, validation, blockchain, proof-of-stake, web crawling, censorship
