Decision making for basketball clutch shots: A data driven approach

3.1Data preparation and preprocessing

At a given moment during clutch time, a team typically has eight to twelve available players (who are registered to the game and are not ejected or fouled out). We require a Clutch Shots Database, consisting of all the clutch shots that have been taken during the examined time period –in our case, the NBA seasons between 1996/97 and 2020/21. Specifically, each row corresponds to a clutch shot, and the columns describe its properties. In addition, each shot is labeled according to its outcome, i.e., ’successful’ or ’unsuccessful’.

The clutch shots database contains two main sets of player properties: pre-game and in-game. Pre-game properties are generated by processing all relevant plays of a player prior to the current game. These properties will assist in identifying how ’good’ a player is and particularly how he performs during clutch times. In-game properties are generated by processing the preceding plays of the game. These properties may indicate the player’s performance during the current game, focusing on his performance during the minutes preceding the shot, representing how hot the player is. To later examine player-selection policies, we also collect data for the players who did not take the clutch-shot. Each player in the team, at each clutch shot, is described using 34 player properties listed in Table 1.

The Clutch Shots Database is created by processing play-by-play tables for each game in the observed time period. We download the play-by-play table of each game using the NBA-API (2021) by PyPI (2021), which is freely available and connects to the official NBA stats website (NBA Advanced Stats, n.d.). An example for how to find a game (by a team, for example) and download its play-by-play table, is provided by Forbes (2018) in the NBA-API documentation. Each play-by-play table contains all the plays that occurred during a game, including shots, substitutions, turnovers, assists, rebounds, fouls, and more. We merge all play-by-play tables and process them to create the Clutch Shots Database described above. Obtaining the required data by processing play-by-play tables provides the flexibility of creating complex player properties. Also, it provides the ability to create in-game player properties, representing the players’ performance in the current game until the moment the shot was taken. Such properties are normally not available on external tables, which provide more accumulative data such as seasonal averages.

We split the Clutch Shots Database into three equally sized subsets: a train-set on which we train the lasso logistic regression, an inference-set on which we perform evaluation and comparison of policies, and finally a test-set on which we test the obtained ranking of policies. We term the second set as inference-set rather than validation-set as the latter is typically used in the context of hyper-parameter tuning, which is not the case here. Our decision is based on the trade-off between having more data available for training the regression model and having more data available for comparing different player-selection policies and assessing the obtained ranking performance on new data. Our primary goal is not necessarily to train the most accurate prediction model possible, as we show that even relatively simple models achieve a high success rate; we aim to perform accurate inference between policies and test it. Therefore, we allocate an equal number of samples to each set, ensuring that we have enough data to compare different player-selection policies and effectively evaluate the performance of our player ranking approach on new data. Alternative data division options are equally valid and likely to produce similar outcomes.

3.2Defining player-selection policies

At every given moment in a game, each player is characterized by a set of pre-game and in-game properties. A player-selection policy (policy) is defined as a set of guidelines for determining the preferred player to take the shot. Specifically, a policy receives as an input the current state - all the available players in the team and their properties - and provides the selection of a single player as an output. A policy can be based purely on pre-game properties, in-game properties, or a combination of both. By suggesting various policies and later evaluating and comparing them, we create a policy hierarchy.

First, we study player-selection policies based on a single property of each player. Such policies represent selecting a player based on a single property: for example, choosing the player who averages the highest PPG in his career. Another example is choosing the player who leads the team in rebounding in the current game.

Next, we introduce a policy based on a supervised learning model, considering multiple properties. We train a lasso logistic regression (Tibshirani, 1996) model for that purpose. The model’s features are the player properties describing each shot-taker while taking the shot, and the target is each shot’s outcome (i.e., 1 for a successful shot, 0 for a missed shot). Logistic regression is an explainable model that is easy to interpret; analyzing its coefficients may provide insights into the factors impacting clutch shots’ success. We use a lasso logistic regression over an unregularized logistic regression due to the large number of features, specifically 34 player properties. As future work might involve even more features, the lasso logistic regression is a suitable choice due to its ability to effectively zero out the coefficients of the least important features using regularization, which enhances the prediction’s accuracy, explainability and scalability. We utilize a classical 5-fold cross-validation to choose an optimal regularization parameter. The regression is trained on an independent subset of the data (train-set). We define a player-selection policy based on the regression’s predicted probability for each player to score.

Finally, we define a policy of randomly selecting a player to take the shot. While the policies described above are relatively simple, several policies achieve a high success rate, significantly greater than the overall clutch shots success rate. These results are discussed in the Analysis and Results Section. We emphasize that additional policies can be simply defined and considered within our framework.

3.3Policy comparison

For each player-selection policy, we examine shots taken according to the policy’s recommendation (i.e., the player who was selected by the policy is the player who took the shot). For example, for the ‘random selection’ policy, for each shot, an available player from the team is being randomly selected; if the selected player has in fact taken the shot, this shot would be included in the policy’s sample. Therefore, each policy is now paired with a set of shots containing successful and missed shots. The measure we use for the policies’ evaluation is the success proportion in the above set of shots associated with each policy, in an independent subset of the data, the inference-set.

Naturally, a policy hierarchy may be composed by ordering the policies in descending order of the success proportion of each policy. However, such ordering does not indicate whether this ranking is statistically significant. Therefore, we apply multiple hypothesis testing and compare each possible pair of policies, to determine which policies are better in a statistically significant manner.

The outcomes of different policies may be viewed as a binomial experiment, Bin (n, p), where n is the number of shots taken in accordance with the policy’s recommendation and p is the (unknown) success rate. To determine whether the difference in the success rates of two different policies is significant, we apply a variation of a t-test, as later discussed. We define H₀ as p₁ = p₂ and H₁ as p₁ ≠ p₂, where p_i is the success rate of the ith policy. In other words, the null hypothesis suggests that there is no difference in the success rates while the alternative claims the opposite. Notice that a standard t-test is defined for either disjoint sets or paired values. In our case, two policies may have overlapping samples in cases where both consider the same player. For example, suppose the current shot-taker is the player with the highest average PPG and the most average assists. In that case, the shot would belong to both policies - average career PPG and average career assists. Therefore, we consider the method suggested by Derrick et al. (2015), which enables computing p-values for partially overlapping samples.

In order to create a complete hierarchy of the policies, we compare all possible pairs of policies, using the inference-set. We also compare each policy to the entire set of clutch shots in the inference-set, to identity the policies that are better than the overall success rate. Since we preform multiple comparisons simultaneously, we apply a Bonferroni correction (Miller, 1981) to account for multiplicity.

Based on the obtained p-values, we form a hierarchy of player-selection policies. The hierarchy ranks the policies in descending order of the success proportion, and the p-values determine the statistical significance difference between policies.

3.4Evaluation of the ranking approach

We evaluate the ranking approach on an independent subset of the data (test-set). First, as a benchmark, we consider all the clutch shots in the test-set: the clutch shots taken in reality. Then, we re-examine the test-set and extract only the shots that our scheme recommended. These are the shots that were taken in accordance with our recommendation: shots taken by the highest-ranked policies. Finally, we compute the p-value representing the significance of the difference between the above sets of shots.

Notice that this comparison may be somewhat biased. Specifically, if our recommendation only considers the best clutch players and does not account for all the remaining shots, then we may trivially attain improved performance. Therefore, to have a valid comparison, we consider not only the highest-ranked player but the k highest-ranked players. Specifically, we consider the success rate of when one of the top three ranked available players in the team took the shot. This solution introduces a more realistic scenario, in which a shot is taken by one out of k suggested players (and not just the single best player).

3.5Model analysis

An additional goal of this research is to provide managerial insights regarding the game of basketball. We focus on the question of which factors are more impactful to clutch shots’ success: pre-game factors or in-game factors. We study the importance of player properties to clutch shots success rates using three different approaches:

First, we study the policies which achieve a significantly greater success rate than the overall clutch shots success rate. By analyzing this group of ’successful’ properties and their common characteristics, we conclude which types of player properties are more impactful to clutch shots success rate.

Second, we analyze the lasso regression coefficients that are used to determine the importance of each player-property to clutch shots success. The policy based on the regression’s predicted probability of each player to score achieves a high success rate, and therefore such analysis is valuable.

Third, we train two separate lasso logistic regressions. The first regression’s features are purely pre-game player properties, and the second regression’s features are purely in-game player properties. We define two player-selection policies, each based on a different regression model, and compare their performance. This comparison simulates comparing a decision made purely on pre-game data vs. purely on in-game data.

4.1Data preparation and preprocessing

For our analysis, we examine data from 25 NBA seasons spanning from 1995/96 to 2019/20. In total, there were 31,103 regular season and playoff games played during this period. Among these games, we focused on 15,146 games that were considered clutch, where the score margin was within five points in the final five minutes of the game. We extract each game’s data using the NBA-API (2021) provided by PyPI (2021), which allows access to the official NBA stats website (NBA Advanced Stats, n.d.). This data collection process enabled us to obtain information on 227,061 clutch shots taken throughout these games. For each shot we describe the shot-taker, at the moment the shot was taken, using 34 player properties, listed in Table 1.

Each player property is categorized as either a pre-game player property or an in-game player property. Many players have a little, if any, sample of clutch shots. To have a reliable sample, several clutch properties are collected using the entire 4th quarter or overtime, as described in Table 1 (instead of only the final five minutes). The constraint of score margin within five points is kept. Appendix A lists and describes in detail all the player properties we use in this study. Recall that we collect player properties for the shot-taker and all other teammates who were available when the shot was taken.

4.2Policies definition

For each player property in Table 1, we define a respective single-property policy, i.e., selecting the player having the best value in this property. For example, the policy based on the player property in-game points is to choose the player with the highest amount of points in the current game. In cases where several players are tied for the lead, the policy randomly selects one of the tied leading players. Further, we define a policy based on a lasso logistic regression model. The regression model we use was created by Pedregosa et al. (2011).

Naturally, when ranking the players in the team, it is not obvious that only the best performing players (under any policy) should be selected to take the clutch shots. In other words, perhaps the second-best points scorer should be chosen before the best rebounder. Therefore, for each policy described above (single-property policy or regression based), we further define two more complementary policies: A policy defined by selecting the player who is the second choice of the respective policy (for example, the player with the second-most points in the current game) and a policy defined by selecting the third player.

Overall, for the lasso logistic regression and each of the 34 single property policies we define three separate policies for the first, second and thirdly ranked players, resulting with a total of 105 policies. In addition, we examine a random selection policy.

4.3Policies evaluation

We evaluate the policies using the inference-set, consisting of 75,687 shots. In this subset, the overall clutch shots success rate is 41.37%, 31, 314 successful shots out of 75, 687 in total. Table 2 exemplifies the evaluation of the policy 1st player in-game points. This policy selects the player who scored the most points in the currently analyzed game to take the clutch shot.

Figure 2 shows the binomial confidence intervals of success rates of different policies in the inference-set. The x-axis represents the serial number of each policy, and a Wald 95% confidence interval for each policy’s success rate is evaluated. The red dashed line represents the binomial confidence interval for the overall clutch shots success rate in the inference-set. This figure demonstrates that while most player-selection policies achieve a success rate similar to the overall clutch shots success rate, certain policies achieve a greater success rate. The success rates of all 106 policies, as well as their usage rates (percentage of total shots), are presented in Appendix B.

Fig. 2

Binomial confidence intervals of estimated success rates of the considered policies.

We apply multiple hypothesis testing and compare each possible pair of policies to rank the policies and create a policy hierarchy. Further, we compare the set of shots paired with each policy to the entire set of shots in the inference-set to determine which policies achieve better results than the overall clutch shots success rate in a statistically significant manner. We compare 107 sets of shots (one set of shots for each policy of the 106 policies and one representing the overall success rate), hence we perform a total of (1072) tests. Therefore, using Bonferroni correction for the multiple comparisons problem with a confidence level of α=0.05, the p-value used is 0.05(1072)≅8.82E-06 . We conclude that 14 policies introduce a significantly greater success rate than the overall clutch shots success rate. Table 3 lists these policies and further demonstrates the internal ranking: which policies’ success rates are significantly greater than others. As we can see, the lasso logistic regression achieves the highest success rate, and several clutch-based policies achieve a high success rate.

Our real-time ranking of players is composed of the players selected by each policy in Table 3, sorted by each policy’s success rate in the inference-set. Figure 3 demonstrates an example of real-time players ranking from 2017/18 regular season: The Philadelphia 76ers were playing against Oklahoma-City Thunder and were in possession of the ball with 53 seconds left to play in the game, while the opponent led 94 –92. The top ranked player by our proposed hierarchy is Ben Simmons. He was also the player who took the shot in practice, and indeed he scored.

Fig. 3

Example of real-time players ranking from 2017, Philadelphia 76ers.

4.4Evaluation of the ranking approach

We evaluate the proposed ranking approach on an independent subset of the data, i.e., the test-set, consisting of 75,687 clutch shots. The overall success rate in this dataset is 41.23%. We evaluate the success rates when one of the top k ranked players (by our obtained policy hierarchy) takes the shot, for k = 1, 2, 3. The results are presented in Table 4. All considered cases achieve a significantly greater success rate than the overall rate in the test-set.

4.5Model analysis

We now study our models and provide managerial conclusions based on our findings. First, as presented in Table 3, 14 player-selection policies achieve a significantly greater success rates than the overall clutch shots rate. While two of these policies are based on the lasso logistic regression, 12 are single-property policies. Out of these 12 policies, nine are based on purely pre-game player properties, while only three are based on in-game player properties. Interestingly, the in-game player properties are in-game blocks, in-game rebounds and in-game fouls drawn, and are not based on the current game shooting performance. This analysis suggests that pre-game shooting performance is a better indicator for successful clutch shooting than in-game shooting performance. In-game aspects such as blocks, rebounds and fouls drawn may indicate a player’s level of focus and dominance in the current game or perhaps physical supremacy in the match-up against their defender.

Let us now study the logistic regression coefficients. The model is trained to predict clutch shots’ success probability, and the magnitude of coefficients assigned by the model to each feature may indicate its importance to clutch shots’ success. The model’s coefficients are presented in Table 5. This analysis further validates the conclusion that pre-game shooting performance is the best indicator for clutch shots success, as the feature with the most considerable coefficient magnitude is pre-game FG%. Other features have a relatively minor effect.

Moreover, we train a pure pre-game and a pure in-game regression models. We define two policies: selecting the player with the highest predicted probability to score by each of the above prediction models. We evaluate each policy on the inference-set and perform a t-test to determine which policy is superior. The results are presented in Table 6, which shows that the pure pre-game lasso logistic regression achieves a significantly greater success rate than the pure in-game model.

In addition to the above three analyses, we also use the described framework to perform ranking of players while distinguishing between two-point shots and three-point shots. We implement the strategy described in the Methods Section separately for the two disjoint sets of two-point shots and three-point shots in the Clutch Shots Database, resulting in a separate player ranking for each range. This may provide more detailed information for the coach to make informed decisions about player selection for different types of shots. In the analyzed sample, 71.77% of the shots are two-point shots and 28.23% are three-point shots. Our analysis shows that implementing our player selection strategy results in a significant increase in success rates for both two-point and three-point shots. Specifically, the success rate for two-point shots increases from 45.1% to 53.2%, while the success rate for three-point shots increases from 31.8% to 33.5%. Furthermore, we show that even when not the highest-ranked player shoots, but one of the top two or top three players are shooting, the difference is still significant. The results of our analysis, including the best policy for each range, are presented in Table 7. For two-point shots, the best policy is choosing the shooter according to a lasso logistic regression (trained separately on two-point shots), in which the feature with the largest coefficient is ‘Pre-game FG% ’. For three-point shots, the best policy is choosing the player with the highest Pre-game 3PT% in the previous 100 shots preceding the analyzed game.

All four analyses described above indicate that while several aspects of in-game player performance are indicative of clutch shots success, the pre-game performance is favorable. Specifically, pre-game shooting performance is the most impactful factor to clutch shots success.