Conflict forecasting and its limits

2.1.Experts

To the public, the most visible forecasters of political events are experts or pundits, whose opinions and predictions are typically relayed in the media. While they often provide valuable insights into a region or the relationship between two countries, there is no evidence that these insights have any predictive power. In fact, a study of predictions by 300 experts in politics and economics over the course of more than two decades revealed that the predictive performance of these experts on geopolitical events was on average no better than random guesses (Tetlock [55]). Experts who were successful in one period typically reverted to the mean in the next. Overall they performed no better than novices, and simple algorithms easily out-predicted them – a finding confirmed specifically in the context of conflict forecasting (Green and Armstrong [25]).

It is in fact not surprising that experts are poor predicters. First, they have an incentive to adopt a strong position, or else risk making dull predictions that are unlikely to receive media attention. Second, they rarely suffer the consequences of their failures. Correct forecasts are often hailed in the media, but wrong ones are promptly forgotten.

This is not to say, however, that experts, and more generally human forecasters, are useless. First, they may be more likely to offer dissonant opinions that ultimately contribute to forging the correct forecasts. The experts’ opinions, in other words, despite their poor predictive value, may contribute to a better performance by algorithms or ‘superforecasters’. But perhaps more importantly, a careful iterative selection of human predicters, for example via a tournament, has shown that they can produce consistently better forecasts than a wide range of competitors – including some of the most sophisticated algorithms (Tetlock and Gardner [56]).

2.2.Econometric approaches

Scholars and policy-makers have also relied on algorithms analyzing large amounts of data using various techniques, ranging from the more traditional regression techniques – typically logistic, due to the binary nature of the outcome – to more intricate random forest or neural network models (Rummel [41]; Brandt and Freeman [6]; Muchlinski et al. [34]; Hegre et al. [28]). In the fields of international conflict and civil war, the majority of this work has until recently relied on structural variables such as regime type, GDP, ethnicity, or terrain (Beck, King, and Zeng [2]; O’Brien [35]). Unfortunately, these variables are typically measured yearly and vary slowly or not at all (Gleditsch and Ward [20]). As a result, while they can identifying the situations that are conflict-prone, they generally fail to pinpoint which of them will actually break into war, and when (Chadefaux [14]).

Scholars have therefore increasingly relied on event data disaggregated at the temporal and spatial levels.33 This initially involved the human coding of large corpuses of journalistic and historical documents into various categories of international events. Advances in computational methods and in particular in text parsing have made it possible to automate these processes, and hence to analyze far larger and broader ‘live’ sources of information, and thereby to move from structural to short-term measures of tensions and other markers of conflict. TABARI (Textual Analysis by Augmented Replacement Instructions, Schrodt [44]), for example, uses the lead sentence of wire service reports (e.g. Reuters, Agence France press, etc.) to generate such event data. The World-Wide Integrated Crisis Early Warning System (ICEWS) is currently the most prominent of these event datasets (Boschee et al. [5]). Sponsored by the Defense Advanced Research Projects Agency in the United States, it provides a detailed database of political events at the sub-daily and subnational level (O’Brien [36]).

Approaches based on these datasets, and more generally on news-based sources, have scored some predictive successes (e.g. Schrodt and Gerner [46]; Weidmann and Ward [63]; Brandt, Freeman, and Schrodt [7]; Ward et al. [61]; Chadefaux [12]; Chiba and Gleditsch [15]). Yet their automatic approach to coding sources faces difficulties as well. In particular, scholars tend to rely on secondary sources such as news reports, which are neither comprehensive nor unbiased. Moreover, while temporal disaggregation has been relatively successful (there is generally little ambiguity about the timing of an attack or a speech), pinpointing the location of that event has proved to be much more challenging and error-prone (Weidmann [62]; Lee, Liu, and Ward [32]). Machine coding also still cannot analyze sentences with a complex structure, and typically ignores the connection between them.

Information derived from social media has also received increasing attention. Its advantage is obvious: the amount of data is considerable and transforms every user into a potential reporter – widely expanding the potential range of events covered. The amount of time that elapses between the event and the reporting is also greatly reduced, making real-time predictions possible in principle – an appealing proposition for policy-makers. Yet the challenges are also formidable. Most of what is written on social media is either irrelevant for the purposes of conflict forecasting – tweets, for example, are massively polluted by spam – or is deliberately false information planted by the government for purposes of manipulation (Spaiser et al. [52]).

2.3.Modeling war

Another approach to forecasting conflict relies on various forms of modeling. Game theory, in particular, has proven to be a core tool in the forecaster’s toolbox. An important advantage of game theory is that it forces the forecaster to model the interaction in great detail, and avoid omitting any relevant detail. Typically this involves two distinct processes. First, expert knowledge on a region or country is obtained to identify the relevant set of actors, together with their preferences and power. These elements are then incorporated in a model that simulates the interactions between these actors – for example a bargaining or coalition model. The model then returns a prediction about the likely outcome – whether it be the result of an election or a policy choice.

This approach has claimed a large number of successes (Gurr and Lichbach [26]), in particular when it comes to forecasting single events (Bueno de Mesquita [9]). Game theory software, for example, contributed to finding Osama bin Laden’s hiding location in Pakistan.44 Yet, the overall evidence is mixed (Green [23]; Goodwin [22]; Green [24]). Part of the reason may be that the assumptions underlying game theory are at odds with empirical observations of human behavior, as behavioral economics has made clear (Kahneman [30]). For example, game theory may not be “able to trap the subtleties of […] an international treaty bargaining session” (Shubik [49], p. 14).

Game theoretic models are also often limited by their complexity. They tend to require strong simplifying assumptions that limit their applicability and power. Agent-based models, which rely on simple rules but high computing power, can instead be used to model complex and emergent interactions. A drawback is that these models are typically so complex that our ability to draw inferences from them is limited. Yet if the point is to forecast, then agent-based models may be ideal in the sense that they allow us to run different scenarios and evaluate counterfactuals (Cederman [10]; Bhavnani et al. [4]).

2.4.Wise crowds?

Of the many approaches to forecasting conflict, the most promising venues may be those that rely on their combination. In particular, the aggregation and weighting of various models and opinions appears to significantly improve preditions. Thus ensemble forecasts, which combine the predictions of various models, often outperform those of any model in isolation, and have been used with success in conflict prediction (Montgomery, Hollenbach, and Ward [33]).

Tournaments are another way of aggregating estimates. Tetlock and Gardner [56], for example, show that the recursive selection of human forecasters – repeatedly pitting the best of them against each other in iterative rounds – yields predictions that outperform not only experts, but also forecasts relying on algorithms and large databases.

Markets are also valuable tools to aggregate predictions, because they convert the wisdom of crowds into predictive probabilities. Financial markets are particularly good because they combine the forecasts of actors who have a financial stake in making accurate predictions. Securities are traded in a way that reflects the investors’ beliefs about the probability of a certain event occurring. Large events such as wars are economically and financially costly, and market participants will therefore strive to anticipate them as early as possible and to react accordingly. For example, bonds are likely to be sold in anticipation of a war (Chadefaux [13]), as will be the stocks of industries most likely to be affected by it (Schneider and Troeger [43]). Political prediction markets function in the same way as their financial peers, but can create securities that focus on questions of interest to political researchers (Arrow et al. [1]). For example, the Defense Advanced Research Projects Agency (DARPA) created the Policy Analysis Market to test the feasibility of an online market in which participants would bet on geopolitical trends or specific events.55

3.1.Are conflicts clocks, clouds, or swans?

The question of predictability ultimately hinges on the underlying nature of conflict. In the words of Popper, does it more closely resemble the world of clouds – “highly irregular, disorderly, and more or less unpredictable” – or the one of clocks – “regular, orderly, and highly predictable in their behaviour” (Popper [37]). Unfortunately we do not yet know which we are facing. On most days, international and domestic interactions resemble a clock. Small deviations are corrected in a reversion to the mean, and the stochastic process of daily events and tensions that may emerge on a local or global level is trend-stationary. Yet there are also rare shocks that do not follow this clock-like pattern. These are, of course, the events of interest here – conflicts, coups, acts of terrorism – that may start a cascade and change the clock into a cloud and the trend-stationary time series into unit-root processes (Doran [17]).

There is yet a third possibility: that conflicts are neither clouds nor clocks, but black swans (Taleb [53]). Black swans are game-changing events with such low probability that they cannot be predicted (even though experts often claim to have found obvious warning signs for them ex post). Black swans are different from simple rare events. While rare events occur infrequently, their probability is not low conditional on the relevant set of variables. On the other hand, black swans have a low probability even conditional on other variables.

Where conflict processes should be located on the clock-cloud-swan continuum matters. With clocks, predictions are possible, whether they be point or probabilistic predictions.77 With clouds, the marginal cost of better predictions would be increasing, but we could at least learn about the aggregate distribution and data-generating process (e.g. Clauset, Young, and Gleditsch [16] on the frequency of terrorist events). With black swans, however, attempting to predict would be a fool’s game.

Several factors make it particularly challenging to predict conflict, and in fact may impose insuperable limits to our forecasts. First, our data are, almost by definition, prone to error and imprecision (e.g. Shellman, Hatfield, and Mills [47]). Part of it is due to poor measuring. But part of it is caused by strategic misrepresentation and concealing on the part of the relevant governments.

A second reason for the difficulty to predict conflicts is that their structure, and more generally the structure of international relations, are constantly evolving. The end of the cold war, for example, was largely unanticipated and challenged many of the structures and patterns that formed parts of the existing models. Even within a given conflict, the dynamic can dramatically change and necessitate a different model (e.g. the surge in Iraq, Bhavnani et al. [4]). The difficulty is that these changes are difficult to anticipate – they are often black swans themselves – such that our ability to forecast may be limited to the short-term. The long term, on the other hand, would be the result of too many compounding shocks to a point where predictions become futile. Ideally, our predictions would be able to accomodate these changes. This may require two-level predictions in which the structure itself is first predicted, and within that structure the short-term events would be forecasted with a different model. But of course this would compound the uncertainty about our model specification and data.

Another difficulty relates to the strategic nature of international relations and politics in general. First, actors are forward looking. They form their own predictions about the future, and act accordingly today. As a result, these predictions can affect their behavior today and invalidate these original predictions. As observers, then, we may have the right logic but end up not observing the phenomenon. Consider for example the problem of the onset of wars. If their contemporaries identify the underlying conditions as ripe for war, they may take additional steps to either prevent it, postpone its onset, or on the contrary speed it up, such that the initial predictions will be invalidated (e.g. Chadefaux [13] for empirical evidence of this pattern). The same logic also applies to the conduct and termination of war. The anticipations of forthcoming peace negotiations, for example, may lead certain actors – spoilers – to try harder to disrupt the peace process, thus reducing the prospects for peace (Kydd and Walter [31]).

Mixed strategies are another difficulty. States or domestic actors cannot always respond to the same situation in the same way, else their response becomes predictable and may be exploited by the adversary. Just like a tennisman will not always serve in the same place to prevent the opponent from anticipating his actions, leaders must vary their threats and responses to events. As a result, the same conditions and sequence of events may lead to different reactions, some potentially leading to wars whereas others do not. In such cases, probabilistic predictions remain possible, but point predictions are inherently impossible. While a large number of observations may bring us close to an estimate of the underlying probability distribution, the predictive value of our forecasts will be bound upward by a fundamental limit.

In fact, uncertainty itself may be necessary for the onset and continuation of war. Indeed, one of the central rationalist explanations for why bargaining might break down into war is incomplete information of at least one of the participants. As a result, “we cannot predict in individual cases whether states will go to war, because war is typically the consequence of variables that are unobservable ex ante, both to us as researchers and to the participants” (Gartzke [19], p. 567).

Conflicts and the processes leading to them may also be path-dependent. A small event may lead to a cascading effect and ultimately to war. Yet the same underlying structure could possibly have accomodated an alternate equilibrium in which peace prevailed. Self-reinforcing processes mean that international interactions may magnify the effects of chance. Looking back, we may be able to trace the explosion of a keg – conflict – to a single spark (though this is itself debatable, as evidenced by the unfaltering scholarship on the causes of WWI), but looking forward, we are unable to know which spark will ignite the keg. In the same way, seismologists understand the causes of earthquakes and are able to monitor seismological variations with high precision but still cannot predict their onset with much early warning.88

3.2.How can we know?

A core difficulty in determining whether we are dealing with clocks or black swans, and more generally in improving our forecasts, is that wars are rare events. Evaluating the predictive validity of a model may therefore require years or even decades. As a result, feedback is delayed and improvement slow. True long-term predictions such as Hegre et al. [27] (as opposed to forecasting exercises using cross-validation tests on past data) are thus particularly difficult to evaluate. However, new data disaggregated at the monthly and daily levels make it possible to test hypotheses and evaluate forecasts much more rapidly than before. Datasets such as ICEWS provide hundreds of events every day, such that forecasting competitions could be organized and evaluated within months.

The real difficulty, though, is not to know whether or not we predict well, but rather whether there are fundamental limits to our ability to predict. Knowing why we fail – because of poor models or the inherent impossibility to forecast – matters for several reasons. First, information about the limits of forecasting would help us decide which area of research is most promising and where to allocate our scarce research resources. Second, knowing these limits also matters because our scientific forecasts may in fact be harmful. Publishing forecasts of outcomes that are in fact unpredictable – and hence repeated failure to correctly predict – may lead to distrust of the public toward the scientific enterprise. Moreover, a good overall forecasting record may lead policy-makers to take into account the advice of the forecaster. Yet even a good overall forecasting record may mask systematically poor forecasts in a given area, due to the unpredictability of that area. If decision makers trust the overall record and follow the recommendation in that area, the results will potentially be disastrous. Knowing what is unpredictable would therefore avoid these costly mistakes.

We therefore need to study the limits of our own ability to know. At the theoretical level, we know that mixed strategies and incomplete information guarantee that the onset of war will never be perfectly predictable, as discussed above. Beyond theoretical arguments for the indeterminacy of international interactions, however, can we uncover empirically the limits of our ability to predict conflicts?

One approach is to simply evaluate the predictive performance of our best model, and assume that the errors are due to limits to the predictability of conflicts. This is the approach adopted by Song et al. [50], though in a different field. Most models forecasting the onset of conflict thus reach an area under the Receiver-Operating Characteristic (ROC) curve of approximately 80%, yet few manage to progress beyond that threshold. This may be the fundamental limit of predictability. The problem with this approach is that while failures to forecast may indeed be attributed to the inherent impossibility of forecasting conflict, nothing guarantees that they are not due to errors in our assumptions, models, or data.

A different approach consists in examining the marginal contribution of additional variables to predictive accuracy. From this, we can extrapolate the asymptotic limits of predictability from our current path of discovery. Chadefaux [13] thus shows that the size of marginal contribution of each additional variable is decreasing and quickly reaches zero, thus suggesting possible inherent limits to our capacity to predict the onset of conflict. But of course, there can never be any certainty that this path will not be upended by a new discovery, model, or technique.

Yet another method is to examine the past and evaluate the performance of past forecasters. Chadefaux [13], for example, shows that financial markets have underestimated the risk of conflict over the past 200 years, and that their forecasting ability has remained constant over time. This is particularly surprising, as financial market participants have a strong incentive to make correct forecast – and hence to improve on past performance. This inability to improve may be an indication of limitations inherent to the pattern leading to the onset of war, for example because of feed-forward effects.

Another approach consists in measuring our ability to forecast various conflict processes with different forecast lead times. If conflicts are cloud-like, then our ability to forecast should decrease exponentially as we make predictions farther out in time. If, on the other hand, they are clock-like, then we should expect the predictability to decline only linearly or as a function of the lead time.