Feasibility of nowcasting SDG indicators: A comprehensive survey

2.1SDG indicators

In 2012, as the target year for the MDGs approached, work began on the development of a post-2015 development agenda [7]. The result was the 2030 Development Agenda, adopted by the UN General Assembly on the 25th of September, 2015 [1]. In contrast to the eight goals of the MDGs, the 2030 Agenda outlined 17 goals, called the Sustainable Development Goals (SDGs), which are accompanied by a varying number of targets per goal, for a total of 169 targets [1]. Each target in turn has one or more indicators to aid in monitoring progress towards accomplishment of each target and goal, for a total of 231 indicators [2]. Each of these indicators in turn has a varying number of sub-indicators. The 2030 Agenda serves as a policy framework to help tackle such issues as the eradication of poverty, reducing inequality, and addressing climate change, among many others. For more information on specific targets and indicators, see UNSD [8] and Ritchie et al. [9].

Defining priorities and agreeing on a framework in 2015 was only part of the story, as a further two years were required to make the stated goals actionable by developing the targets and indicators for each goal [10]. Measurement is a vital aspect of the 2030 Agenda, both in terms of guiding and informing policy decisions at a national and international level, as well as in quantifying progress towards the goals. Developing indicators for such an expansive agenda with a diverse array of interconnections is no small task and highly dependent on the target and type of data available. As such, there are three tiers of SDG indicator:

In simpler terms, Tier 1 indicators are well defined and already being produced, Tier 2 indicators are defined, but not yet being produced, while Tier 3 indicators are still being defined. It should also be noted that even indicators classified as Tier 1 may still be plagued by issues of data coverage, availability, and/or timeliness. Assessment of progress towards the achievement of the 2030 Agenda relies heavily upon reliable, accurate, disaggregated, and timely indicators. Nowcasting, examined in the next section, has the potential to address the last of those characteristics.

2.2Nowcasting in the SDG context

The term “nowcast” itself is a portmanteau of “now” and “forecast”. Nowcasting as a term and discipline originated in meteorology in the 1980s [12] but began to appear in economic literature in the 2000s. Nowcasting in the economic sense, and in the sense relevant for SDG indicators, refers to the estimation of the current value of a target variable based on timelier data and information. The distinction with forecasting comes from the fact that estimations are produced for time periods that either have already concluded or that are currently running, as opposed to periods in the future.

The intuition or justification for nowcasting is best explained by way of example using gross domestic product (GDP). GDP is a frequently nowcast target variable due to three of its characteristics. First, GDP is often published with a significant lag due to the many data sources needed and the complex accounting and aggregation procedures necessary for its calculation. GDP figures for a given quarter or year are often published many months after the conclusion of the period, even though all economic activity measured in the eventual figure has already occurred. Second, GDP usually has a long publication history, meaning there exist sufficient observations to estimate a model on historical information. Finally, there exist numerous potential explanatory variables which are published on a much timelier basis which can be used as inputs for a nowcast. Series such as consumer price indices, industrial production indices, consumer and business confidence indices, and retail trade figures, among many others, are typically published with a significantly shorter time lag than GDP, so can be used to obtain an estimate of GDP well before final figures are made available. These characteristics, together with the salience and relevance of GDP as an indicator, have made GDP the target variable for many nowcasting applications and papers. A GDP nowcasting model could then be fit on historical data and fed the latest information of timelier indicators to obtain both an estimate of GDP months before final figures are published, as well as monitor GDP outlook during the period by rerunning the model on the latest data continuously. For examples of GDP nowcasting applications see Morgado et al. [13], Rossiter [14], or Bok et al. [15].

Nowcasting is relevant for SDG indicators because many face issues with timeliness. In order to successfully implement the 2030 Agenda for Sustainable Development, it is essential that policy makers have access to timely information as it relates to SDG indicators, a primary means of monitoring and evaluating progress and guiding policy interventions. The United Nations Statistics Division (UNSD) and UNCTAD have identified nowcasting as a key means of meeting this timeliness challenge [3]. The existing literature on nowcasting specifically as it relates to SDGs is sparse, but two notable works include Bierbaumer-Polly et al. [16], where a comprehensive nowcasting exercise of SDG indicators using dynamic factor models is performed for Austria, and Hughes et al. [17], where the International Futures forecasting system is used to nowcast many SDG-related indicators for more than 180 countries.

Nowcasting is however no panacea. It is only applicable for obtaining timelier estimates of an already produced SDG indicator. That restricts its application to Tier 1 indicators, where data are produced. For a given indicator to be suitable for nowcasting, a further two conditions need to be satisfied: the indicator needs a sufficiently long time series to be able to train a nowcasting model, and there need to exist sufficient related and timely explanatory variables. In order to assess the nowcasting feasibility of SDG indicators, these conditions were applied in carrying out the survey explained in greater detail in the next section. Complete survey results are available online at [survey link], with a visualization of results available in Appendix 1. The complete survey results will hereafter be referred to as the “full results table”.

Care should be taken if nowcasts are eventually adopted as advanced estimates of SDG indicators. Their status as data-based, quantitative estimates, liable to revision as the data outlook changes, should be clear to users, as well as when a figure has changed from a nowcast to its actual recorded value.

3.1Description of methodology

The first step in nowcasting an SDG indicator is determining whether it is even applicable to the case. That is, do the characteristics of the indicator fulfill the data requirements of nowcasting outlined in the previous section. This was the goal of this feasibility survey: to provide a comprehensive overview of every Tier 1 SDG indicator and their sub-indicators and their potential to be nowcast. The results of the survey could help statisticians and custodian agencies know at a glance whether their indicators have the potential to be nowcast and provide a springboard from which to launch their own investigations. There are no hard and fast rules for applying the earlier mentioned three conditions of nowcasting, which depend rather on the indicator. A one-month lag for an economic series may be considered a short lag, while for an epidemiological series it could be considered a long one, etc. Rather, each indicator needed to be examined individually and evaluated for nowcasting suitability on a holistic basis. It is also worth mentioning that some indicators may be composed of a combination of two or more series, for instance indicator 8.4.2 (Domestic material consumption, domestic material consumption per capita, and domestic material consumption per GDP). In these cases, the indicator’s publication delay may be due to one of its constituent series, and better results may be obtained from nowcasting just this series rather than the entire indicator itself.

As Tier 2 and 3 indicators lack the publication of any historical data, they could immediately be classified as not suitable for nowcasting due to the second condition. As such, they were excluded from the survey and do not appear in the full results table. Restricted to Tier 1 indicators, the survey was conducted in the following manner: the main sources for information on Tier 1 indicators were the SDG Indicators Metadata Repository and the Global SDG Indicator Database [8, 18]. The SDG Indicators Metadata Repository ideally includes information on data characteristics relevant to nowcasting feasibility. However, the type of information included in each SDG indicator metadata file tends to vary, despite ongoing work to standardize the contents. For instance, some files do not include any information on when the data are collected or released. Some metadata files are unfinished, have missing parts, or require updating or reviewing. The SDG Indicator Database displays data for each indicator but does not always reflect the data availability described for each indicator in the respective metadata file.

Custodian agency databases also usually provide access to data for their SDG indicators. Information from these databases was used as a direct source of indicator characteristics or used to validate metadata. Data availability in the SDG Indicator Database is usually up to date with the custodian agency databases but may also be vastly different in content. The survey combines data available in the two sources if the years do not completely overlap. Otherwise, the source with the longest time series provides the information in the full results table. The SDG Indicator Database is also limited to displaying annual data. Indicators with monthly or quarterly data, for instance, are only displayed as annual. Other database sources must be used to get information about these indicators. Data availability is not easily describable for some indicators and sub-indicators. In many cases the length of time series varies greatly by country or region. This is indicated in the full results table.

For an indicator to have Tier 1 classification, data must be available for over 50% of relevant countries. There are a few indicators that can variably be classified as Tier 1 or Tier 2, depending on sub-indicator, and some sub-indicators included appear to not meet the Tier 1 requirement stated above. The number of countries or territories covered by each indicator across all possible sources is difficult to confirm due to inaccurate metadata information and inconsistencies between data sources. Many SDG indicators are also not classified at the sub-indicator level, or existing classifications have changed over time without metadata updates. Some are broadly considered Tier 1 indicators despite having sub-indicators that may not meet the Tier 1 requirements, while others are classified as either Tier 1 or Tier 2 by sub-indicator, but without specifying which sub-indicators belong to which classification.

The amount of data available for a particular indicator can vary by country, location, or aggregate grouping, so the data availability described in the full results table generally focused on the data availability of a world or global aggregate, if available. When the global aggregate did not have enough data for nowcasting purposes or a global aggregate was not available for a particular indicator, for the purposes of this survey, a general summary of the relevant countries and other aggregations was used. Information on publication lags for each indicator often had to be inferred from the data that appeared to be available and the existing metadata information.

A further consideration for nowcasting feasibility is the existence of explanatory variables for a particular indicator. Each indicator or sub-indicator was given a score of “Highly likely,” “Likely,” or “Unlikely” for this area. An indicator got a label of “Highly likely” if explanatory variables would likely be easy to find. Many SDG indicators are macroeconomic variables, such as GDP, or fall under poverty, health, education, environment or ecological topics. Variables like these are frequently modelled and existence and availability of explanatory variables is well-documented in literature. An indicator gets a label of “Likely” if potential explanatory variables may not be closely related to the behavior of the indicator or had limited data available. For instance, variables related to Official Development Assistance (ODA) or certain government spending decisions were listed in this category, since values for these indicators are generally pre-determined by government decisions, although there are a variety of socioeconomic factors that may still contribute to initial spending decisions themselves. An indicator got a label of “Unlikely” if it may prove difficult to find explanatory variables. Appropriate explanatory variables may not exist for binary outcome variables that capture whether a country or territory has enacted a certain policy or legislation or joined a certain agreement or organization, as these decisions are generally not reversed or changed once decided upon. A regional or global aggregate of such an indicator, e.g., disclosing the number of countries or economies in a region adhering to a particular policy, legislation or agreement, etc., would be more suitable, if it exists. It should be noted that classifications for existence of explanatory variables in the survey should serve only as a starting point. Definitively determining whether explanatory variables exist and selecting them for an SDG indicator can only be done with extensive research and potentially modelling, which was not feasible for this survey due to the quantity of SDG indicators and time and resource constraints.

Finally, scores for overall nowcasting feasibility were determined by all gathered information on data availability, publication lags, and explanatory variables. Each SDG indicator and sub-indicator was given a score of “Highly likely,” “Likely,” or “Unlikely” for overall feasibility. Some indicators were determined to be less feasible for nowcasting due to the nature of the subject matter being unsuitable for nowcasting purposes, such as the previously mentioned binary outcome indicators and indicators related to spending and budgetary decisions, as well as election result indicators or upper parliament appointment results. Some indicators were considered unsuitable for nowcasting if they did not have enough data, generally around 10 data points at a minimum. Finally, if an indicator is published without significant data lags, nowcasting may not be relevant for the case. As mentioned previously, what constitutes a significant lag depends on the indicator.

3.2Survey results

Out of a total of 362 Tier 1 indicators and sub-indicators considered in the nowcasting feasibility survey, 150 received a classification of “Highly likely” for nowcasting feasibility, 87 received a classification of “Likely”, and 125 received a classification of “Unlikely.” The release of data for nearly all Tier 1 indicators and sub-indicators are accompanied by a lag. Most indicators are published at an annual frequency, with a publication lag of around one to three years. Publication lags may also differ by the organization, country, or region that provides the data. In these cases, the compilation of aggregate figures may depend on the “last” country or region to provide indicator estimates, so data publication lags are usually sufficient to warrant a nowcasting approach for both individual and aggregate figures alike. The existence of explanatory variables is unlikely to pose a problem for most indicators.

Some sub-indicators under the same indicator code measure identical concepts with different units, such as “Number of deaths and missing persons attributed to disasters” and “Number of deaths and missing persons attributed to disasters per 100,000 population” under Indicator 1.5.1. Usually, sub-indicators under the same indicator code share the same nowcasting feasibility as they share the same data availability, data release schedule, and generally surround related topics with similar sets of likely explanatory variables.

As a reference, the following sections will summarize the survey results by goal in a standardized manner, considering only Tier 1 indicators. They need only be read as interest dictates; overall survey summaries and conclusions are available in Section 5. For more detailed information on the results for a particular goal, indicator, or sub-indicator, see the full results table available online at [survey link].

4.1Indicator and data

The case study presented here will demonstrate how the information from the feasibility survey can be used to select promising SDG indicators for nowcasting and provide guidance on how the modelling exercise can then be performed. The selected indicator for the exercise was indicator 9.4.1, CO2 emissions per unit of value added [69], on the global level. As with most indicators, there exist many different regional and country-level aggregations, each with their own publication schedules and data availability characteristics. The global level was chosen as the aggregation of broadest interest. Examining the full survey results, we can see that 9.4.1 has data from 2000 onwards, thus satisfying the adequate series history requirement of nowcasting, is published on the SDG database with a multi-year lag, satisfying the requirement of an extended publication lag, and that there exist ample potential explanatory variables. These characteristics together make it a good candidate for nowcasting and for the case study.

Within the SDG indicator, the series EN_ATM_CO2GDP, “carbon dioxide emissions per unit of GDP (kilogrammes of CO2 per constant 2017 United States dollars)” was used as the nowcasting target variable [18]. Data for this series were obtained from the UN SDG database [18], where, at the time of writing in Autumn 2021, the latest global figures were available for 2000–2018 at an annual frequency. Data for the database in turn come from the International Energy Agency (IEA) [70]. It should be noted that while data available on the SDG database would imply a publication lag of more than two and a half years, timelier data may be available from the IEA directly. However, at the time of writing, timelier figures for the indicator are not available publicly. Any custodian agency, institution, or individual interested in nowcasting an SDG indicator should first make sure that timelier data are not available directly from the original data provider. For the purposes of this case study, where the goal is not to generate a nowcast of the indicator per se, but rather to illustrate and outline the modelling process, we can take the publicly available publication lag of two years as given.

With data for the target indicator in hand, the next step in the nowcasting process is identifying and gathering data for potential explanatory variables. The actual variables identified depend highly on which indicator is being nowcast. It is recommended to gather as many potentially related variables as possible. A process for selecting which variables go into the model in the end will be outlined in the next section. In the case of 9.4.1, there were two main components of the indicator: carbon emissions and economic activity, i.e., GDP. The data gathering process could then be guided by these two components.

Data on emissions were mainly gathered from two sources, Statista and the U.S. Energy Information Administration [71, 72]. Data from the former had a publication lag of between four and eight months on annual-frequency data, while data from the latter had a publication lag of three months on monthly-frequency data. Data relating to GDP and economic activity were drawn from many sources, but mainly from the OECD [73] and Eurostat [74]. Economic data had any of a monthly, quarterly, or yearly frequency. In the end, almost 30 emissions-related and more than 150 economy-related variables were gathered. Not all these variables were used in the final model, they rather served as a pool from which to train and test different models in the model selection process outlined in the next section. All series were finally transformed to period over period seasonally adjusted (if applicable) growth rates with the US Census Bureau’s X13-ARIMA-SEATS methodology [75].

4.2Methodology

Once the data have been gathered, three steps remain in the modeling process: selecting a modelling methodology, selecting which variables will go into the model, and selecting which hyperparameters to use for the chosen methodology. The last step depends on the type of model chosen, as some approaches do not have hyperparameters.

Nowcasting comes with its own set of challenges that any modelling approach must be able to handle. First, the model should be able to handle time series. Second, it should have some mechanism for dealing with mixed-frequency data. This refers to the fact that the variables in the model, be it the target variable or input variables, will not necessarily be recorded in the same frequency, for instance estimating a yearly variable using monthly and quarterly variables. Third, it should be robust to the differing publication schedules of its input variables, often called “ragged-edges”. The last challenge is the “curse of dimensionality”, where there may be few observations relative to the number of input variables, complicating the estimation of many classical econometric and statistical models.

Several different methodologies address these challenges and have been used successfully for nowcasting applications. Some of the most common include the dynamic factor model (DFM) [76, 77], mixed data sampling (MIDAS) [45, 78], mixed data sampling vector autoregression (VAR) [78], and Bayesian vector autoregression [79]. Hopp [80] examined an approach using long short-term memory artificial neural networks (LSTM).

No one approach is better than the others in all cases. Ideally, multiple would be tried in order to validate performance and increase the chances of obtaining a high-performing model. In practice, the methodology chosen will be influenced by other factors, such as which implementations are available in which programming languages, if any open-source options are available at all. It is primarily for this latter reason that the LSTM was chosen for this case study. In selecting a methodology, the nowcasting_benchmark open source repository is a good resource outlining the performance of all common nowcasting methodologies in nowcasting US GDP growth [81]. It additionally contains boilerplate code for each methodology which can be followed to perform one’s own nowcasting exercise.

Having settled on the LSTM for the nowcasting model, the next steps are selecting which variables will go into the model and which hyperparameters will be used. If the number of input variables gathered in the data collection phase is small, the former may not be necessary. In order to accomplish this, a performance metric must first be determined to compare different models to each other. In a regression application such as this, mean absolute error (MAE) and/or root-mean-square error (RMSE) are suitable. For a given model with given input variables and hyperparameters, its accuracy needs to be assessed via the performance metric. In order to ensure that a model is generalizable and not overfit, it should be assessed on data it was not trained on. A general rule of thumb is to train on 80 per cent of the data and test on 20 per cent. In this case, models were trained on data ranging from 2001 to 2011 and validated on data from 2012–2014 (the validation period). They were then trained on data ranging from 2001–2014 and tested on data from 2015–2018 (the test period).

The logic behind the validation set is its use as a way of selecting variables and hyperparameters. Selecting the best performing models based on validation performance then finally assessing them on the test set ensures that the model is not being overfit on the test set in terms of its variables and hyperparameters.

A final factor to keep in mind in nowcasting is performance before all data are available. This is commonly the case in nowcasting, and especially so if the nowcast is to be monitored over a period of time. To account for this, model performance was recorded on synthetic data vintages, or the artificial introduction of missing values to simulate the data as it would have appeared at different points in time. The vintages corresponded to the month of the target period, i.e., if the target period was 2020, the data as it would have appeared in December 2020, six months after the target period, and ten months after the target period, when the latest publishing input variable would be released. Publication lags for generating the synthetic data vintages were gathered from empirical observation from April 2020 to October 2021.

Now with a specified process of training and testing a particular model, many models could be tested to determine the best performing one. Variable selection and a small degree of hyperparameter tuning were carried out in the same step for the case study. Because there were far too many potential input variables to test all permutations, input variables were randomly sampled, run with a small selection of hyperparameters, and their performance recorded. This process was repeated for hundreds of random input variable samples. The three best performing models of these runs were then assessed with a more expansive set of hyperparameters. Finally, the best performing of these was selected for the final model. There are other approaches to variable selection than the process described here, which will not lead to the absolute best performing input variable and hyperparameter combination possible from the data. This best performing combination is impractical to find due to computational and time constraints. However, this approach is sure to find a relatively well-performing model from the space of all possible input variable and hyperparameter combinations. The LSTM library used for this analysis additionally enables the automatic selection of variables for a given model via its variable_selection function [82].

Results from the final selected model are presented in the next section.

4.3Results

The variables selected for the final model are listed in Table 1.

Figure 1 shows the model’s predictions for both the validation and test sets with full data compared with observed actuals. With these variables, full data is equivalent to about seven months after the period has ended, or July of the following year, when the variable with the longest lag is published. The blue line represents predictions on the validation set, so using a model trained with data from 2001 to only 2011. The red line represents predictions on the test set, with a model trained with data from 2001 to 2014. The model is remarkably accurate in the first two years of the validation set, 2012 and 2013, but struggles with the abnormally low observed 2014 level. The large drop in that year may have been due to abnormally warm winter weather in certain regions, reducing energy consumption, coupled with a reduction of coal use in China [83, 84]. This suggests that inclusion of variables related to weather and fossil fuel use or prices could improve the performance of the model. Overestimation of 2014’s value was common to all models, perhaps as the first year where carbon intensity of GDP levels began to decline at faster rates than previously observed in the ten years prior. However, the model was able to pick up on this faster declining trend between 2015 and 2017, as well as a relatively milder decline in 2018.

MAE and RMSE for the validation period were 0.011 and 0.024, respectively. In words, this means the model predicted year-over-year growth in the target variable that was 1.1 percentage points different from the actual over the validation period. The higher RMSE value shows how this particular performance metric punishes the larger error in 2014. MAE and RMSE for the test period were 0.006 and 0.007, respectively.

Figure 1.

Indicator 9.4.1, carbon intensity of GDP.

Figure 2.

Development of nowcasts over time.

Figure 2 shows the development of 2015 and 2020’s nowcasts over time, beginning in January of the target year, and ending in July of the following year. Predictions over time were gotten from running the model on synthetic data vintages based on the publication lags of the variables, or how the data would have looked at different points over the year. The y axis shows the nowcasted yearly growth rate, while the x axis shows the simulated date. Each point in the lines represents the nowcast for the yearly growth rate given the data that would have been available at that time. For 2015, we can observe a relatively monotonic development throughout the year, as the forecast was revised downwards as time went on. 2015 indeed registered the largest year over year decline in the target variable in the data; -3.3 per cent. In 2020, we can observe a development that changed directions over time, due to the volatile signals in the data owing to the COVID-19 pandemic. In the end, the model predicted reductions in line with the average for the period from 2014 to 2018, due to the fact that the pandemic affected both carbon emissions as well as economic activity. Figure 2 illustrates how nowcasting can be used to monitor the development of SDG indicators in real-time and gain insight to how various factors influence them.