The FAO Data Lab on statistical innovation and the use of big data for the production of international statistics

4.1Using “web scraped” information for agricultural statistics

Hand-in-Hand (HiH) is an evidence-based, country-led and country-owned initiative of the Food and Agriculture Organization of the United Nations (FAO) to accelerate the agricultural transformation and sustainable rural development to eradicate poverty (SDG 1) and end hunger and all forms of malnutrition (SDG 2). In doing so, it contributes to the attainment of all the other Sustainable Development Goals.44 The Initiative prioritises countries where national capacities and international support are most limited or where operational challenges, including natural or man-made crises, are greatest. It uses the most sophisticated tools available, including advanced geo-spatial modeling and analytics, to identify the biggest opportunities to raise incomes and reduce the inequities and vulnerabilities of the rural populations, who constitute the vast majority of the world’s poor. It also uses these tools to improve the targeting and tailoring of policy interventions, innovation, finance and investment, and institutional reform. For this, the Data Lab conducts agricultural production data collection by means of artificial intelligence in contexts where little data are available, typically in the HiH priority countries and at a sub-national level (different types of administrative divisions). This process allows the Data Lab to fill data gaps for the countries that are part of the HiH initiative; it also serves as a way to validate official data, creating a mechanism for data quality control and supporting governments to improve the quality of their work.

While the response rate to Agricultural Production Questionnaires (APQ) has shown an overall increase during recent years, as can be seen from Fig. 3, there are some areas for which it is well below 50% (Oceania) or consistently slightly below it (Africa).

Figure 3.

Average response rate to FAO’s Agricultural Production Questionnaire.

Figure 4.

Completeness indicator for Agricultural Production Questionnaires (see text for definition of “completeness”).

A similar picture can be obtained once we compute an APQ “completeness indicator”, calculated as the percentage of official data points over the total number of possible country/commodities combinations for which at least one official data point was ever provided by the country in one APQ.55 Results are reported in Fig. 4, and along with the previous Figure, shows that there is space to improve the statistical coverage.

Increasing the statistical coverage can be done mainly through two channels: there is a stronger collaboration with countries so that they produce and/or provide more data; or alternative data sources are used.66 Now, FAO has been collaborating closely with countries over the last few decades, and a possible outcome of recently renewed efforts may be the increased response rate reported in Fig. 3. However, the coverage can still be improved as shown in Fig. 4. Thus, this is the channel that is pursued by the Data Lab: obtaining data not included in the APQs by consulting sources available online.

The process for getting data from alternative online sources consists in “web scraping”77 the information by following these steps: 1) searching for information on the internet, mainly from National Statistical Offices or Ministries websites;88 2) extracting the data from its original source (e.g., PDF documents, Microsoft Excel files) and transforming it to a common format (a SQL table); 3) standardising the data, with respect to commodity names (converted to the Central Product Classification, CPC99) and administrative area names (converted to GADM1010); 4) comparing to FAOSTAT data; 5) saving consistent data to the agricultural production database or analysing discrepancies. Though this process seems relatively straightforward, it is indeed plagued with several difficulties. For instance, documents downloaded in step 2) are unfortunately never ready-to-save into a database: there are different types of formats with their own peculiarities; the most difficult cases are those where the original document is a scanned copy where information needs to get extracted by OCR (Optical Character Recognition). For this step, a series of heuristics is applied and documents are passed through scripts that use information extraction techniques to obtain as much information as possible in a way that it is easy to export it in a tabular format; besides this, documents are usually in local languages, thus they need to be automatically translated to English. Another time-consuming step regards the standardisation of commodity and administrative units; for this, we have developed a strategy that attempts to link names by a fuzzy matching strategy that uses pre-built regular expressions1111 for all commodities and region names. For commodities or regions with no matches, a classification expert from FAO is consulted to help assign the correct link.

As on 23 May 2022 the Data Lab has collected 385,259 data points referring to different administrative levels (national and/or subnational) of 25 countries1212 that include information on 592 different agricultural commodities that, after a standardisation process, are converted to 166 different commodities classified according to CPC.1313 The majority of data regards production and area harvested, which are the main variables of interest for the FAO agricultural domain, though during the collection also other types of information are collected (e.g., tree count, sown area, average yield, etc.).

Not all collected data, as presented in Table 2, is used to integrate the data obtained from questionnaires. Subnational data needs first to be aggregated at the national level as FAO’s production dataset contains information on country data, and commodities are aggregated at the CPC sub-classes level. Additionally, many of the data points were already obtained from the APQs; when this happens, the web scraped data is used to cross-check the data already available. Indeed, this has already been useful to find some discrepancies as, e.g., a country was reporting data for one of the two crop seasons (thus underreporting total production), and another one was not including data for a disputed region (though it was included in trade data). Once the proper aggregations are done, 11,159 data points are candidates for inclusion. Of these, 1,039 data points (9.3% of scraped data) have been used to fill data gaps in the agricultural production dataset. The remaining data points are still being analysed to check whether they are consistent with available data and/or they can be incorporated in the dataset.

4.2Creating vulnerability maps

Determining the living conditions of the inhabitants of a country at the sub-national level is of great importance for many purposes, including for policymaking, investment, urban planning, administration, risk management in hazardous areas as well as planning humanitarian intervention. To estimate living conditions, standard statistical methods make use of censuses and surveys organised at regular intervals in time. However, surveys and censuses are complex exercises that are time-consuming, labour intensive, and expensive [12]. Therefore, countries that are characterised by events that impact social and economic relations (like violent conflicts), often cannot carry out sample-based data collection that allows quantifying sub-national living conditions.

In recent years different approaches were proposed to estimate living conditions in data-poor regions. These methods make use of cheaper alternative sources of data that are available in a timelier manner, therefore helping to bridge existing data gaps. The most promising methods involve the use of remote sensing [13, 14], whereby a statistical model learns the relationship between remote sensing derived variables and a target variable [15]. A popular approach involves the use of satellite images from which specific features can be extracted and used to estimate sub-national living conditions and population levels. These features include among others, the number and density of buildings, the prevalence of shadows, the number of cars, the density and length of roads, different types of agriculture, roof materials, bodies of water, etc. For example, [13] aims at explaining poverty variations at the village level in developing countries such as Sri Lanka, and shows a strong correlation between predicted poverty and spatial imagery variables, indicating a 40% to 60% variation in village-level poverty as explained by the national model. [16] analyses rural India by means of a trained convolutional model using satellite imagery data on the roofing material, sources of lighting and drinking water, roads, farms, and bodies of water.

The Data Lab has developed and implemented a method that leverages these innovative methods for estimating living conditions at the sub-national level. Specifically, it makes use of deep learning models to estimate a country’s land cover from satellite images. This data is then combined with ancillary variables (for example night lights, meteorological data, atmospheric data etc.) to estimate population levels. Scraped socio-economic data, which varies by country, are then used to calculate through factor analysis, the vulnerability score of a region. Finally, this vulnerability score is projected on a map that helps to visually identify those areas in which interventions should be prioritised. Presenting these detailed estimates in maps is a powerful communication tool because the maps summarise living conditions for hundreds or even thousands of towns, villages, or urban neighbourhoods on a single page and in a visual format that is readily understandable by a wide audience. Moreover, such maps enhance the interpretation of the data by preserving the spatial relationships among different areas, something that simply is not possible in a tabular data format.

Figure 5.

Vulnerability maps for Yemen: Vulnerability score in the top panel, vulnerable population in the lower panel.

Figure 5 reports two vulnerability maps for Yemen: one shows a “vulnerability score” ranging from 0 to 100, to be used to provide an easy way to compare different regions; the other shows the amount of vulnerable people, and it thus indicates the magnitude of the problem in absolute terms.

This approach relies on two key aspects: the validity of the underlying model, as well as the degree of representativeness of these alternative data sources. In terms of validity, the model might face shortcomings if perturbing factors occur (such as economic or social shocks) that effectively change the context in which the model was trained. Additionally, if the model is too context-specific, the relations estimated from the model for a given country cannot be applied to another country, making the model useless (not generalisable). Having made these remarks, though, we believe that the methodology can foster a cheaper and timelier estimation of living conditions at sub-national level that can complement – and in some cases substitute –, censuses or surveys.

4.3Expanding knowledge on food loss and waste

SDG Target 12.3 states that “By 2030, halve per capita global food waste at the retail and consumer levels and reduce food losses along production and supply chains, including post-harvest losses.”1414 To measure and monitor food loss and waste (FLW) two indicators have been proposed: the Food Loss Index1515 and a Food Waste Index,1616 under FAO’s and the UN Environmental Program’s custodianship respectively. The Food Loss Index [17] requires data on the share of agricultural products that gets spilled, spoilt or otherwise lost, or incurs a reduction of quality and value during its process in the food supply chain before it reaches the retail stage (typically takes place at production, post-harvest, processing, and distribution stages in the food supply chain). The Food Waste Index requires information on food that completes the food supply chain and is available at the retail stage, of good quality and fit for consumption, but still does not get consumed because it is discarded, whether or not after it is left to spoil or expire (typically at retail and consumption stages in the food supply chain). These two types of data are scarce. To have an idea of its scarcity, and focusing only on food loss from now on, as on 24 May 2022 there is a percentage of official data points on this issue ranging from 7% that is obtained by questionnaires sent by FAO to countries (for the period 2015–2020). This means that the vast part of this phenomenon is not officially reported. However, there is a relatively good amount of literature on this topic that reports estimates of food loss for various products and stages of the value chain for one or more countries. On this premise, and with full commitment on calculating a Food Loss Index that is evidence-based, FAO has developed a semi-automated text-mining process to extract data on food loss from articles, reports, and grey literature.1717

The information extraction system consists of three main steps: 1) automated document collection and pre-processing; 2) modelled statistical assessment of the relevancy of documents; 3) guided extraction of data.

In the first step, we query different sources to obtain documents related to food loss by using generic loss-related keywords (e.g., “post-harvest food losses”, “post-harvest loss”, “food loss”). Scientific articles are obtained through the FAO library subscriptions. Other types of documents, such as working papers, conference proceedings, or technical notes, are harvested with several scripts that were designed to query specific websites such as the World Bank and World Food Program document repositories, among others. Once the documents are downloaded and saved locally, several data extraction steps are run to obtain metadata such as authors’ names, article’s title, date of publication, and language. Other useful information is also extracted or computed: a text summarisation routine based on Natural Language Processing (NLP) steps is performed to get a short summary of the document; keywords used in identifying loss factors for countries and commodities are retained.1818 All the extracted information is then stored into a NoSQL datastore (Solr)1919 to be used in later processing steps and to be displayed to the user into a user-friendly web interface.

The second main step of the process consists in checking if the document is likely to contain actual numbers of losses, i.e., checking its relevancy for the task. For this task, different machine learning classifiers were trained on a set of nearly 330 documents pre-validated by an expert (28% were considered relevant) by using as features different sets of word types and relations extracted from them2020 We considered 14 alternatives: AdaBoost, Decision trees, Gradient Boosting, Gaussian Naive Bayes, Gaussian Process Classification, K-Nearest Neighbors, Logistic regression, Multi-layer Perceptron, Perceptron, Quadratic Discriminant Analysis, Random Forests, SVM with SGD training, Support Vector Classification, Complement Naive Bayes. We compared results on different criteria (accuracy, recall, precision, Matthews correlation coefficient, area under the Receiver Operating Characteristic curve, and balanced F-score) by 10-fold crossvalidation on 67% of the complete sample, keeping the remaining 33% as a testing set. We decided to keep the Random Forests model because for most of the metrics it gave a better result for basically all criteria (and less variability across criteria). Each document is then passed through the implemented classifier, which returns a probability of it being relevant or not. This probability is stored into Solr and shown to the analyst in a web user interface – along with other details about the document – so that they can sort documents by decreasing degree of relevancy when they start the guided extraction.

The last step consists in a manual intervention by the analyst that needs to assemble and validate the information automatically retrieved in the previous steps. A copy of the document is shown to the analyst with the extracted information highlighted with different colours according to the typology of keywords (countries, commodities, percentages). The tool has some pre-compiled selection inputs with the information extracted from the text, so that the user can validate the extracted information and eventually add more details (e.g., food chain stage, sample size, methodology). When each set of information is saved, it is sent to a standard SQL database, which is linked to the documents stored in Solr by means of a key so that it is always possible to retrieve the document from which the information was taken.

Figure 6.

Data points by value chain stages contained in the food loss and waste dataset. “Other stages” refer to different stages of the value chain as grading, processing, wholesale, etc., which are recorded separately in the dataset, but the number of observations is less than 1,000 per stage.

Figure 7.

Interactive tool to visualise and extract information on food loss and waste. Available at https://www.fao.org/platform-food- loss-waste/flw-data/.

As of 28 May 2022, the published dataset is made of 35,284 data points extracted from several hundreds of different sources. It covers 161 countries and 267 commodities. While it contains data ranging from 1945 to 2022, the vast majority of figures are for relatively recent years: 84% of observations are from 2000 and after, while 44% are for 2010 and after. Besides information on aggregated losses (i.e., for the whole supply chain), it also contains data for different stages of the supply chain. Figure 6 reports some selected stages (those with a number of data points greater than 1,000), as well as “Other stages” for which there are less observations (e.g., grading, processing, wholesale).

The FLW dataset is disseminated to the public by FAO at https://www.fao.org/platform-food-loss-waste/flw-data/ so that it can be used by anyone who wishes to know more about food losses and waste. Data can be queried, downloaded, and plotted in an interactive and structured way. A screenshot of the interactive tool that displays the data is reported in Fig. 7.

All the data collected for the FLW dataset serve a twofold scope. On the one hand, the dataset wants to be an instrument for researchers and analysts to understand better a domain for which official data is scarce and unofficial data is sparse and not systematised, so that they can study the phenomenon and reach their conclusions. On the other hand, it is exploited by FAO to impute missing data for losses on primary products for all countries. While describing the details of this imputation process is out of scope for the current paper ([17] and a forthcoming FAO/ESS Working Paper), it may suffice to say that all the information available on losses (official and collected through the text-mining process described before), is used in a random effects model that considers as explanatory variables many factors that are deemed to have some effect on food losses (e.g., electricity prices, oil prices, fertilizer prices, weather, rainfall, agricultural investment); the resulting model is used to provide yearly estimates of losses for every country/commodity combination where data is not available. By aggregating official and imputed data, the magnitude of food loss has been found to be of around 14% at the global level [18].