An analysis of curling using a three-dimensional Markov model

2.1Data overview

We analyzed matches played in the Canadian Men’s Curling Championships (known for sponsorship reasons as the Labatt Brier, Nokia Brier, or Tim Hortons Brier) from 1998 to 2014. We restricted our analysis to the post-Olympics curling era; curling was first played at the Olympic Games in 1998. We believe that the addition of curling to the Olympics improved the quality of play (World Curling Federation, 2016) and, therefore, did not see value in comparing different eras of the sport. Box score information was retrieved from the Cassidys’ Curling Canada Stats Archive (Cassidy and Cassidy, 2015). The score information included the year of the match, round of the tournament, match location, teams competing, score in each end, final score, time remaining for each team, and which team started with the hammer in the first end. We used information from 10,933 ends reflecting 1,199 matches. Figure 2 summarizes all observed ends by end number and score differential.

Fig.2

Summary of Dataset, Showing Total Observed Ends by Score Differential and End.

We used these data to create a three-dimensional empirical state space tracking three variables. The empirical state space win probabilities were calculated from the frequencies of eventual victories given all observed games in a given state. We also built a three-dimensional Markov model to predict win probabilities assuming a homogeneous scoring distribution. The three state variables used in the models are the end, hammer state and score differential. Table 1 summarizes and explains each state variable.

2.2State space overview

To use Markov methodology to model the sport of curling, one must define all possible scenarios that could occur over the course of a game. All permutations from all possible variable values (shown in Table 2) collectively form the state space of the model, where each permutation is a possible state. For example, a particular state may be that the team in question is: down by 2 in the 8th end, and holds the hammer. That state can be represented as (x,e,h) = (–2, 8, 1). The objective of this analysis is to define the eventual win probability of all states in the relevant state space.

Due to the rules of curling, the possible transitions from each state to the next state are limited. The end of a state must always be greater than the end of a prior state by a value of 1. In other words, ends proceed sequentially throughout the game. Likewise, the hammer possession transitions are defined by the rules of curling, where the hammer possession of a state depends on whether that team held the hammer in the prior state and the end score recorded in that state. Finally, the score differential of a given state is equal to the score differential of the previous state plus the end score.

In order to use a Markov model to exhaustively estimate the win probability of every state in the state space, one must also know all possible state transitions and their associated probabilities. In curling this is simply all possible scores than can be recorded in a single end. From the data it is clear that the likelihood of scoring very much depends on whether a team has the last shot advantage (holding the hammer). The homogeneous model introduced assumes that state transitions are only a function of hammer possession and are thus independent of other parameters in the game such as end or score differential. The heterogeneous model later introduced assumes there is state dependence on the transition function; specifically, that teams play differently based on their current circumstances other than hammer possession. State dependence of the transition function becomes increasingly important in the 10th end, where the team with the higher score at the conclusion of the end wins.

For an example of this state dependence, one would assume that a team down by 2, in the 10th end with hammer advantage would play very differently than a team tied in the 1st end with hammer advantage. The difference in strategy and playstyle is not accounted for in the homogeneous model, but is accounted for in the heterogeneous model.

2.3Empirical state space methodology

With the data prepared, the first step in investigating the state space model required looking at empirical win percentages as a function of the variables described above. The total number of matches that passed through each discrete state were counted, as well as the eventual victors of those matches. The win probability was estimated as the total number of victors over the total number of matches that passed through a particular state.

2.4Markov methodology

The Markov model aims to develop the expected win probability of any curling team given the current state and all potential future transition states. The expected win probability for any state is denoted as: wp (x, e, h) where x is the score differential at the conclusion of end e and h equals 1 if the team holds the hammer.

2.4.6Estimating win probability in extra ends

If two teams are tied at the conclusion of 10 ends, extra ends are played. In the 11th end, there are two possible outcomes: one of the teams scores one or more points and wins the match, or the end is blanked and the match continues for a 12th end. By analogy, the possible outcomes for the 12th and subsequent ends are the same.

Thus, the win probability for the team holding the hammer in extra ends is:

(3)

wp(0,10,1)=P(Score in 10th)+P(Blank the 10th)×wp(0,11,1) wherewp(0,11,1)=P(Score in 11th)+P(Blank the 11th)×wp(0,12,1) and so on.

Because the strategy shouldn’t change in either “next score wins” situation, we can assume the probability distribution of scoring will be the same for all ends after the 10th when the score is tied. With that assumption, it can be inferred that the probability of the team with the hammer winning in extra ends is the sum of the resulting geometric sequence. From Fig. 3, P(y ≥ 1|1) = 68.23 % (the probability the hammer team scores one or more points in a given end) and P(y = 0|1) = 12.42 % (the probability of a blank end given hammer possession).

Fig.3

Homogeneous Probability Mass Function of Scoring.

Subsequently, we can calculate wp(0,10,1)=0.68231-0.1242=77.91% . This is the homogeneous win probability of a team tied in the 10th end with the hammer.

Likewise, the same logic can be used to estimate the heterogeneous win probability wp(0,10,1). From Fig. 4, P(y ≥ 1|0, 10, 1) = 79.97 % and P(y = 0|0, 10, 1) = 0 %.

Fig.4

Heterogeneous Probability Mass Function of Scoring of Select Groupings.

Subsequently, we can calculate wp(0,10,1)=0.79971-0.0=79.97% . Because there were no observed blanks in a tied game in the 10th end, the heterogeneous win probability with hammer advantage in this situation is the same as the probability for the hammer team scoring more than 1 point.

3.1Sensitivity analysis results

Before the model was finalized, a decision regarding model parameter k was required. Recall from Table 2 that parameter k represents the boundary condition for score differential at which the outcome of the game is assumed to be a foregone conclusion. As described earlier, Markovian win probabilities exhibit diminishing sensitivity to k. Multiple versions of the homogeneous Markov model were calculated and presented in Fig. 5 for 5 ≤ k ≤ 10.

Fig.5

Win Probabilities for Given States Assuming Different Model Parameters k.

As can be observed, most win probabilities are not sensitive to k across all model iterations. States which were the most sensitive to k include those early in the game and with lopsided score differentials.

Because all state specific win probabilities stabilized at k ≥ 8, it was concluded that this value would be sufficient and was thus selected for further analysis.

An alternative visualization of model sensitivity to k is presented in the appendix in Fig. 13.

3.2Empirical win probability

Before modelling win expectations, the likelihood of any team winning the match in any given state was estimated directly from the dataset. This was done by isolating all score differential and end combinations separately and tracking whether the hammer team in that scenario emerges as the eventual victor. The fraction of all observations in which the hammer team eventually wins is considered the empirical win probability for that state. The results are summarized in Table 3. While Table 3 describes the fraction of eventual victors for any given state, it does not count the total number of observations for each state. Blanks cells indicate no observations and thus, the empirical win probability could not be estimated.

3.3Markov model results

1. What does the Markov model say about win probabilities in the various states? How does the empirical win probability model compare to a Markov model?

Figure 6 below illustrates the results of the homogeneous Markov model (solid lines) against those of the empirical model (discrete points). The legend indicates the score differential, while the horizontal axis indicates the end in question. All win probabilities are presented from the perspective of the hammer team. Following a single line to the right would be the equivalent of tracking the Markov win probability of a team consistently blanking ends as the game approached its conclusion.

Fig.6

Homogeneous Markovian Win Probabilities at the Conclusion of Each End with Hammer Advantage (Modeled vs. Observed).

One insight from the homogeneous model is that it pays to start the match with the hammer: teams that hold the hammer at the beginning of a match have a 58% chance of winning. This result is consistent with the results reported by Park and Lee (2013) as well as Willoughby et al. (2001).

Although the homogeneous model generally reports the same win probability trends as the empirical model when the match is close, it consistently over-predicts the likelihood of comebacks compared to actual match data.

We subsequently extended our Markov analysis by incorporating a state specific scoring distribution that varied by end to arrive at the heterogeneous Markov Model. Figure 7 below plots the results of this new heterogeneous model (solid lines) against those of the empirical model (discrete points). The results of the heterogeneous model are largely consistent with those of the homogeneous model, with improved fit observed later in the match (e.g. for the 10th end). This is due to the fact that the heterogeneous scoring distribution aims to take into account the state-dependence of the scoring distribution, which becomes especially relevant in the 10th end due to proximate win conditions. For example, the shooting strategy of a team down by 2 with the hammer in the 10th end can be expected to be very different from that of the same team up by 1. Segregating and filtering the data set across these states yields different observed probabilities of scoring.

Fig.7

Heterogeneous Markovian Win Probabilities at the Beginning of Each End with Hammer Advantage.

In general, teams appear to play more defensively when ahead in later ends of a match, which reduces the likelihood of late-game comebacks. This is perhaps unsurprising, as this sort of behavior is common across many sports. It is worth noting, however, that even the heterogeneous model overestimates the likelihood of late comebacks for larger (>2) score differentials. A possible explanation for this over-prediction is that the heterogeneous model’s score distribution is not dependent on score differential until the 10th end or later, thus defensive behavior is not captured fully before the 10th end.

The results of both Markov models, in addition to empirical win probabilities, agree with the conclusions made by Kostuk and Willoughby (2004) who determined that it is preferable to go into the 10th end up by one without the hammer rather than down by one with the hammer assuming the probability of scoring one or more points with the hammer is greater than 0.5. Our scoring probability mass function estimates a 68% chance of scoring one or more points in an end when holding the hammer; our heterogeneous Markov model shows the win probability is 56% entering the 10th end up by one without the hammer and 44% when entering the 10th end down by one with the hammer.

Regarding the lower modelled likelihood of a comeback versus empirical results, we offer the following hypotheses to help explain the difference.

3.4Model uncertainty

Uncertainty was estimated in the model using standard bootstrapping analysis. Figures 6 and 7 contain error bars which were derived from the distributions accumulated via resampling and replacement. The span of the error bars represents a 95% confidence interval.

Predictably, the homogeneous model exhibits low uncertainty. The large dataset was not segregated as there was no state dependence on the probability of scoring. Thus, all 10,933 ends were resampled to produce new iterative scoring distributions, leading to minimal model variation. The average span of the 95% confidence interval is 0.55% across all states.

The heterogeneous model exhibits higher uncertainty. Because all 10th end states were grouped separately far fewer observations were recorded for each grouping. Therefore, bootstrapping yields higher variability which can be observed by the wider confidence intervals moving into the later ends.

3.5Data validation

We further validated our Markov analysis by plotting the heterogeneous model against empirical win probability implied by 2015 and 2016 Brier Tournament results. These years were not included in the data set used to develop the model: therefore, testing against this additional data would further validate our model. Similar to the section above, Tables 4 and 5 summarize the empirical win probabilities for each state with hammer advantage for the 2015 and 2016 tournament years.

Table 6

Expected concession behavior given win probability threshold of 2.57% with hammer advantage

Table 7

Expected concession behavior given win probability threshold of 2.57% without hammer advantage

Table 8

Observed number of concessions and associated state win probability

Figure 8 below show the results when the heterogeneous Markov Model (solid lines) are plotted against those of the empirical model for the 2015 and 2016 tournaments (discrete points).

Fig.8

Heterogeneous Markovian Win Probabilities at the Conclusion of Each End with Hammer Advantage. Empirical data shown for 2015 and 2016 tournament years (Modelled vs. Observed).

The results of this analysis generally agree with the initial validation in the above section. The heterogeneous model continues to overestimate the likelihood of late comebacks of score differentials greater than 2. The 2015 & 2016 Brier empirical win rates seem to indicate that the model over-estimates the likelihood of a comeback to an even larger extent compared to the 1998–2014 dataset. A possible explanation for this over-estimation is that teams have become more consistent and precise over the years, reducing variability of scoring and thus, reducing likelihood of a comeback. Because the model was derived from scoring probability of the 1998–2014 dataset, the hypothesized reduced variability of scoring in later years wouldn’t reflect in the model. Thus, the model would appear to over-predict comebacks to an even greater degree. This contribution would be cumulative with the aforementioned effect of the concession behavior, where concession behavior would be expected to reduce likelihood of comebacks.

In addition, the heterogeneous model overestimates the likelihood of comebacks during the middle of a match – specifically, between ends 4 and 6. There also appears to be high deviation in the empirical win probabilities around ends 5 and 8: the likelihood of a comeback decreases until the halfway point whereby comebacks become increasing likely until end 8 before decreasing again. This phenomenon is also observed in the initial data set to a lesser degree. This suggest the pronounced inflections in the 2015 and 2016 data may be due to additional noise from a smaller dataset or a peculiarity in these tournament years.

3.6Model implications and use cases

The Markov model allows us to graph the expected win probabilities for each team over the course of a match in a manner similar to those supplied on popular websites for other sports. Figure 9 provides an example of this win probability scoreboard for the championship match of the 2014 Brier between Team BC and Team Alberta.

Fig.9

Sample “Alternative Scoreboard” for the 2014 Brier Tournament Championship Match.

2. How can we use these win probabilities to help decide when a team holding the hammer should choose to blank the end?

We can see that the relative advantage of holding the hammer increases as the conclusion of the match approaches, most starkly at the conclusion of the 9th end. Teams are commonly faced with the following decision: score one point and give up the hammer in the subsequent end or blank the end to retain possession of the hammer in the subsequent end. Looking at win probabilities this decision can be modelled mathematically as:

If this inequality holds true, the team should choose to blank the end, assuming that teams are sufficiently skilled to blank the end with 100% certainty. Otherwise, the team should take the point, also assuming that the teams are sufficiently skilled to score one point with 100% certainty. If we check this inequality for every possible state of the match using the heterogeneous Markov model, Table 5 is produced.

The analysis indicates that prior to the seventh end, it is always preferable to blank the end, regardless of other factors.

However, there are two broad recommendations that are intriguing at first glance. First, the matrix suggests blanking in the 9th end if down by one or tied, which aligns with the general intuition that it is preferable to retain the hammer in the 10th end for close matches.

In the eighth end, the matrix suggests that it is preferable to score one rather than blank for score differentials between –2 and +1 (inclusive). This recommendation should generally agree with intuition since this will allow the team holding the hammer in the 8th end to regain the hammer for the 10th end in a close match. However, the marginal advantage of scoring one versus blanking in the 8th is minor and within the margin of error presented in the bootstrapping analysis.

Our recommendation for the 9th end is consistent with that of Willoughby and Kostuk (2005), which concludes that it is always preferable to blank in the 9th in order to retain the hammer. Willoughby et al. (2005) assumed an empirical scoring distribution conditional on score differential and end, while our analysis considers either a homogeneous scoring distribution as a function of only hammer possession, or a heterogeneous scoring distribution as a function of end, score differential (for the 10th end only) and hammer possession. The conclusion, however, is the same. Furthermore, the Markov model can use recursive logic to extend the strategy recommendation back to all possible states of the game and finds that this recommendation holds in most cases.

We can further generalize the strategy implications to a hypothetical curling match with infinite ends. In this situation, each team will want to maximize the expected value of their point differential. When a team with the hammer scores, the immediate point(s) associated with that score are realized, but the team loses the value associated with hammer possession. When the non-hammer team is scored upon, that team’s point differential is reduced but is partially compensated for with the benefit of subsequently controlling the hammer. Suppose v is the value of the hammer possession, r is the number of expected points scored per end with hammer possession, and p is the probability of maintaining hammer possession through the next end. We can see that the following equation must hold:

Solving for v, this reduces to ... 

Assuming the homogeneous scoring probability mass function illustrated in Fig. 3, we can see that:

Therefore, the value of holding the hammer is equal to 0.614.

When considering a situation where the team with hammer possession is deciding between shooting aggressively or blanking the end with complete certainty, then the following inequality must hold true to justify scoring over blanking an end:

We can see that given the value of the hammer of 0.614 points, the aggressive strategy must yield at least 1.23 points on average to be justified, or two points in practice. If the decision involves blanking versus scoring one point, blanking is always the optimal strategy in infinite curling, which is consistent with the recommendations presented in Table 5 for finite curling (with some minor exceptions).

3. When should teams concede? How does this compare to when teams actually concede in real life?

The heterogeneous Markov model assumes that a team has zero chance of winning if the deficit reaches eight points, with this assumption being supported by the sensitivity analysis presented earlier. Implicit in this assumption is the expectation that the teams play out regardless of either team’s win probability. However, concession behavior censors observed win likelihood in high point differential situations. None of the major authorities including the World Curling Federation, the U.S. Curling Association, and the Canadian Curling Association, have strict guidelines on when teams should concede.

To answer the first question we assume that there exists a threshold, w* such that if a team’s win probability dips below this threshold, the team should concede the match per curling etiquette. Summarizing all conceded matches in the 1998–2014 Brier Tournaments, we arrive at Figs. 10 and 11 illustrating the frequency of concessions by win probability at time of concession.

Fig.10

Observed number of concessions against conceding team’s win probability at time of concession when conceding team holds the hammer.

Fig.11

Observed number of concessions against conceding team’s win probability at time of concession when conceding team does not hold the hammer.

Intuitively, teams are observed conceding when their win probability is sufficiently low, with the mean win probability at time of concession being 2.28% when the team possessed the hammer and 3.58% when the team did not possess the hammer. Therefore, it appears that most teams have a psychological threshold for w* of around 2.57% averaged across both situations. Given this value of w* and the Markovian win probabilities, Tables 6 and 7 illustrate states in the game where one would expect to see the losing team concede.

This is compared to the observed concessions and the implied Markov win probabilities at the time of concession as represented in Table 8.

As illustrated, teams do not appear to concede above the 2.57% average threshold with the notable exception of the 9th end, where there were 53 observed concessions in a state where the win probability was above 15%, and 88 observed concessions with a win probability of 4%. Furthermore, teams who held the hammer but were down two points after the 9th end conceded 33 times, even though they still had a 15% chance of winning the match!

These exceptions may be partially explained by the fact that the Markov model currently does not take the clock into account. Prior to 2012, teams had 73 minutes to make all of their shots in a game. From 2012 forward, this rule was modified to provide teams with 38 minutes of “thinking time”. As the data did not differentiate situations where the clock had expired, these observations were not excluded. The win probability in a state where the team’s clock is near expired is likely lower than the probability predicted by the model thus, many of the concessions may still have been made adhering to the 2.57% average threshold mentioned earlier.

In addition, the win probability at time of concession only accounts for the win probability assuming the concession occurs before any stones are thrown. However, it is possible for a team to concede in the middle of the end, at which time they have a better assessment of the situation given additional conditional information which could affect win probability. For example, a team down by two, with the hammer, in the beginning of the 9th end is considered to have a win probability of 15%. However, if that end is proceeding poorly for that team, they may accurately assess that their win probability has decreased below 15%. If that team concedes before the end is concluded, no score will be posted and that game will appear to have ended before any stones were thrown in the 9th end.

In this situation, the modelled win probability will not incorporate the change in the situation which occurred during that incomplete end. Therefore, the team making the decision may have more information than what is available in the data, thus the analysis likely overestimates win probability at time of concession.

3.7Summary

In this study we used 18 years of tournament data from the Canadian Men’s Curling Championships to develop empirical as well as homogeneous and heterogeneous Markov models to analyze win probabilities in curling.

Both the homogeneous and heterogeneous Markov models suggest a lower likelihood of comeback versus empirical results, with a possible explanation being that concession behavior censors potential comeback wins, thus biasing observations towards fewer comebacks.

The heterogeneous model improves on the homogeneous model with a better fit line for states at later ends, due to state-specific scoring probabilities diverging from the average towards the end of the game as the proximate win conditions alter the optimal shooting strategy. However, larger uncertainty is also observed in the heterogeneous model, as fewer state-specific observations yield greater variability in the bootstrapping analysis.

The results of the models have afforded us new insights that can be applied to curling strategy. For example, when presented with the choice to score one point or blank the end, teams should always blank the end regardless of situation, prior to the 7th end. However, teams should only blank the 8th for score differentials between –2 and 1 (inclusive), and should attempt to score for other score differentials in the 8th end. Finally, if we assume that teams have a psychological concession threshold of a 2.57% win probability, our analysis would indicate that teams concede matches more frequently than they should, which could encourage teams to continue to play out matches they currently concede.

In addition, the win probability changes presented in table 5 allow one to formally evaluate the “blank vs score 1” strategic question. With minor exceptions, it is always favorable to blank in the first 9 ends. This largely agrees with the quantitative analysis of curling available in other literature. However, the degree of favorability depends on the state of the game, with the 8th end representing the exception where blanking isn’t expected to yield an advantage compared to the alternative strategy of scoring 1.

In general, extending the Markov analysis to all possible states yields interesting insights and allows for a more rigorous analysis of curling behavior and strategy.