Affiliations: Development and Support Department, Division of
Macro-economic Statistics and Dissemination, Statistics Netherlands, The
Netherlands
Note: [] Corresponding author: Rutger Hoekstra, Development and Support
Department, Division of Macro-economic Statistics and Dissemination, Statistics
Netherlands, P.O.Box 24500, 2490 HA Den Haag, The Netherlands. E-mail:
[email protected]
Abstract: Summary: As the internet grows National Statistical Institutes (NSI)
are increasingly using data from the internet as a source for their statistics.
This paper describes the first experiences of Statistics Netherlands in the
automated collection of data from web pages. First, we explore the feasibility of this method of data collection
for NSI's. We conclude that from a technical point of view this type of data
collection is possible. Furthermore, the use of website data is probably legal.
It is also concluded, that methodological applicability will have to be
assessed on a case by case basis. Second, we investigate the desirability of this type of data
collection. On the basis of six case studies tentative conclusions are drawn.
Our results show that the cost of reprogramming the robots in case of a site
change and the initial investment in the manual process are the most important
factors. Furthermore, we conclude that efficiency gains are likely to occur due
to the learning-by-doing and scale effects. Finally, we conclude that the main
advantages of internetrobots are in applications with large amounts of data
(leading to increases in quality, frequency and speed). This will lead to new
statistics but also advance the methodological aspects of certain measurement
issues.
Keywords: Internetrobot, webcrawler, data collection, web scraping, web spidering
