You are viewing a javascript disabled version of the site. Please enable Javascript for this site to function properly.
Go to headerGo to navigationGo to searchGo to contentsGo to footer
In content section. Select this link to jump to navigation

Shrinkage estimation of NFL field goal success probabilities

Abstract

National Football League (NFL) kickers have displayed improvement in both range and accuracy in recent years. NFL management in turn has displayed a rather low tolerance for missed field goals, particularly in game-deciding situations. However, these actions may be a consequence of a perceived appreciable variability in NFL kicker ability. In this paper, we consider shrinkage estimation of NFL kicker field goal success probabilities. The idea derives from the literature on estimating batting averages in baseball, though the classic James-Stein shrinkage approaches there do not apply to independent binary field goal attempt trials. We study a variety of weighting schemes for shrinking model-based kicker-specific field goal success probability estimates to a league-wide estimate, as a function of distance. As part of the development, we briefly detail collecting NFL play-by-play data with the R statistical software package, identify the complementary log-log link function as preferable to the more commonly applied logit link function in a generalized linear model for field goal success, and demonstrate the desired variance-reduction, both in and out of sample, enjoyed by our proposed shrinkage estimators. We illustrate our methods by ranking NFL kickers from 1998 to 2014, analyzing individual kicker success at mid-range and long-range field goal attempts, and studying kicker ability over the last decade. Stadium effects are added to the model and found to be highly significant and to have an impact on the kicker rankings.

1Introduction

Given the high frequency of personnel transactions involving field goal kickers in the National Football League (NFL), there appears to be a perception of considerable variability among their abilities to carry out their duties, at least among general managers. In particular, when kickers miss field goals or extra points that are perceived to cost their team a game, they are sometimes terminated shortly thereafter, presumably reasoning that a replacement would have a better chance of success from the same distance. Consider the 2010 New Orleans Saints. In Week 3, Garrett Hartley missed a 29 yard field goal attempt in an overtime loss. He was replaced by veteran John Carney, who in turn missed a 29 yard field goal attempt in another loss two weeks later. Hartley then regained his job the following week. The Saints are not the only team with a quick trigger on in-season kicker personnel decisions. Table 1 summarizes the number of kickers who have been released, for reasons not listed as injury, over the 2000-2014 timeframe. On average, there are more than 3 such kickers released per season. These counts were assembled by inspection of season summaries of kickers found on and subsequent post-hoc inspection of various websites that track transactions.

Table 1

Number of mid-season performance-based kicker changes over time

Year2001-20032004-20062007-20092010-20122013-2015
Number players released121011610

Quantitative analyses (Morrison and Kalwani, 1993) have found surprisingly limited evidence of differences among kickers. In an investigation of weather and distance effects on kicking, Bilder and Loughin (1998) chooses not to model individual kicker effects, describing kickers as “interchangeable parts.” In a study of whether “icing the kicker” affects kickers (it does!), Berry and Wood (2004) also assumes that dependence of log-odds of success on distance to be the same for all kickers. If there are differences among kickers in the way they are affected by field goal attempts of increasing distance, these effects may be subtle enough to go undetected without large samples. Due to these subtleties, estimation of success probabilities for particular kickers can be a difficult task.

We consider the problem of estimating probabilities of field goal success using play-by-play data available on the web. As with analogous investigations in the literature, we develop generalized linear models which assume kicks to be independent Bernoulli trials. However, we show that the complementary log-log (CLL) link provides a better fit than the logistic regression models typically applied in prior work. In models that include kicker effects, estimates of field goal success for inexperienced kickers may have high sampling variance due to small samples. We investigate a number of shrinkage techniques that borrow information from field goal attempts by other kickers in an effort to reduce the sampling variance in these kicker-specific estimates.

The textbook by Morris and Rolph (1981) uses the problem of estimating field goal success probabilities after regression on distance (grouped into intervals with 10 yard widths) as an introduction to logistic regression. Berry and Berry (1985) develops a generalized linear model for field goal and extra point success probabilities using a customized link function that is derived from consideration of three factors: distance, left-right error, and the chance of a non-goalpost related error (a fumble or blocked kick). Other work investigating additional factors beyond distance that may be associated with field goal success probabilities, such as weather and psychological pressure, includes Bilder and Loughin (1998); Pasteur and Cunningham-Rhoads (2014); Berry and Wood (2004); Clark et al. (2013). All of these contributions are in agreement that the single most important factor in predicting field goal success is the distance of the attempt. Morrison and Kalwani (1993) fails to reject the hypothesis of equal success probability among all kickers. Pasteur and Cunningham-Rhoads (2014) uses cross-validation to select variables and finds that adding distance to an intercept-only model decreases a predictive mean squared error criterion by 14%, but that adding an additional 7 variables (including temperature, wind, kicker fatigue, defense quality, and whether the kick was in Denver) brings about a further reduction that is less than 2%. Since many failed extra point attempts are a consequence of poor snaps or holds, we restrict our attention to field goal attempts and do not consider extra point success probabilities or data in our analyses.

Our aim in this paper is not to select variables for the best model, nor to account for the various subtle (in comparison to distance) effects of weather or psychological pressure, but rather to demonstrate the potential advantages of shrinking regression-based estimators towards a central value with the goal of variance-reduction. Indeed, this work is motivated by the success of James-Stein shrinkage techniques (Efron and Morris, 1975; Brown, 2008) for the problem of predicting batting averages in baseball.

In Section 2, we describe our data collection procedure and consider data-driven selection of the link function in a generalized linear model for field goal success. In Section 3, we consider a variety of shrinkage estimators for the kicker and distance-specific success probabilities as functions of model parameters. In Section 4, we illustrate several applications of our shrinkage estimators, including a ranking of kickers and a discussion of long field goal attempts and stadium effects. We present a summary and conclude in Section 5.

2Data and statistical models

2.1Data collection: web scraping with R

Like most other work in the area, we assume kicks to be independent and model the dependence of success probability on explanatory variables using generalized linear models. The explanatory information we consider includes only the distance of the attempt and who the kicker is. Using the readLines() command and regular expressions functionality in the R statistical software package (R Core Team, 2014),, the distance of a field goal attempt is gleaned from the website with url. This website has complete play-by-play information going back as far as 1998. More specifically, the string “yard field goal" is matched and the kicker and distance occurring in the string before the match are recorded along with the outcome that follows the match. Every field goal attempted, even those on which a penalty occurred, but for which a result was given in the play-by-play transcript, are included in the dataset. The task loops over seasons 1998-2014, weeks within season (including the NFL playoffs), games within week, and finally plays within game.

The dataset contains 17, 104 attempts from more than K = 111 kickers. Kickers who made or missed all of their career attempts are excluded from consideration to avoid separability issues with maximum likelihood estimation. (Jaret Holmes played for three teams from 1999-2001, making all five of his field goal attempts and all four of his extra point attempts successfully. Danny Boyd was even better, making 5/5 field goals and 7/7 extra points for Jacksonville in 2002.)

Outcomes were dichotomized as “good” or “not good”, the latter event includes blocks and fumbled holds. Our reasoning is that the blame for failed attempts usually rightly falls on the kicker, but sometimes the snap, the hold or the blocking are at fault. The records online do not easily allow for determination of blame. An example of an entry that was scraped from the url (http://www.pro-football-reference.com/boxscores/199809060cin.htm). (a game from week 1 of the 1998-1999 season) appears in Fig. 1.

Fig.1

Play-by-play item containing relevant information on a field goal attempt.

Play-by-play item containing relevant information on a field goal attempt.

Occasionally, comparisons between total kick frequencies computed from play-by-play data with kicker career totals will reveal slight discrepancies, where certain plays are absent from the pro-football-reference play-by-play information. For example, in a 1998 Week 1 game of the New England Patriots at the Denver Broncos, Adam Vinatieri of the New England Patriots had a 37 yard field goal attempt during the second quarter. The pro-football-reference play-by-play account of the event is blank (see Fig. 2). We resorted to the site to attempt to rectify this particular record. Though we could not find links on the site to play-by-play information for games before 2001 (a situation in which automated screen-scraping is then difficult), we were able to find some urls, with links provided by a search engine.

Fig.2

Play-by-play item for a blocked kick by Adam Vinatieri in a 1998 Week 1 game that appeared as a blank on pro-football-reference.com

Play-by-play item for a blocked kick by Adam Vinatieri in a 1998 Week 1 game that appeared as a blank on pro-football-reference.com

2.2Choice of a link function

For notation, we let k = 1, …, K index the kickers (K = 111 in our application) and let j = 1, …, N (N = 17104) index all of the field goal attempts. For each attempt, we let binary outcome Oj = 1 if the kick is good, and Oj = 0 otherwise. The probability of success of attempt j is denoted πj. The general form for the log-likelihood function of all of the probability parameters {πj} for independent, dichotomous data with outcomes oj is given by

logL(π1,,πN|{oj})=j=1N{ojlog(πj)+(1-oj)log(1-πj)}.

Generalized linear models adopt a link function g, through which the probabilities πj are transformed to an expression that is linear in regression coefficients introduced to model the effects of available explanatory variables (and reduce the dimension of the parameter space). We define dj as the distance of the jth attempt and define indicator variables for kicker k as Xjk = I (jthattemptisbykicker k), for k = 1, …, K, with regression coefficients β1, …, βK. The probability parameters are now, after transformation by the link function g, linear in the regression coefficients,

g(πj)=β0+(β1Xj1+β2Xj2++βKXjK)×dj.

We refer to βk as a kicker-specific slope and can test the significance of estimated regression coefficients by comparison with a generalized linear model with only one slope, β, to accommodate all kickers,

g(πj)=β0+βdj.

In applications in which statistical regression models are developed for binary response variables, logistic regression seems to be the most popular choice (Pasteur and Cunningham-Rhoads, 2014; Morris and Rolph, 1981; Bilder and Loughin, 1998; Berry and Wood, 2004), perhaps because of its interpretability, particularly through odds ratios. However, several other link functions are easily explored using statistical software for generalized linear models. We introduce a superscript notation for probability functions associated with kickers that will simplify the shrinkage estimation exposition in Section 3. Let πk (d) denote the success probability function for kicker k at any distance d. We consider three choices for the link function: logistic, probit, and CLL. Let Φ and Φ-1 denote the cumulative distribution function and quantile function, respectively, of a random variable with standard normal distribution. Then the links and inverse links (probability functions) for these three candidates are given below:

logisticg(πj)=log(πj1-πj)πk(d)=1/{1+exp(-β0-βkd)}probitg(πj)=Φ-1(πj)πk(d)=Φ(β0+βkd)CLLg(πj)=log{-log(1-πj)}πk(d)=1-exp{-exp(β0+βkd)}.

The log-likelihood function of a specified link function uses the reparameterizations above. As an example the log-likelihood for the logistic link is given by

logL(β0,β1,,βK;{oj},{dj})=j=1N(oj{β0+(β1xj1++βKxjK)dj}-log[1+exp{β0+(β1xj1++βKxjK)dj}]).

Table 2 summarizes the maximized log-likelihood function when using the logit, probit, and CLL link functions for estimating field goal success probability in our data set. Tests comparing the nested models within a column of the table are all highly significant on 110 degrees of freedom (comparing models with K = 111 different slopes to a model with a single slope), indicating significant differences among the estimated slopes. With such a large sample, there does appear to be enough data to detect potentially subtle differences among kickers.

Table 2

Comparison of link functions: – 2 × maximized log-likelihood using full or restricted data set

Link function
Comp.
DataModelLogitProbitlog-log
AllCommon slope143821436514367
Kicker-specific slopes140811406214060
χ2 (df = 110) statistic301303307
Restricted toCommon slope118271182011815
25 ≤ d ≤ 50Kicker-specific slopes115471153511524
χ2 (df = 110) statistic302307312

Figure 3 presents the inverse link functions applied to the empirical success frequencies against distance. The lines overlaid on the plots are from maximum likelihood estimation of the single-slope model. Residuals computed as standardized differences between transformed success frequencies and the estimated link functions are plotted against distances in Fig. 4, providing an enhanced visualization of the fits. Link-specific lowess smoothers overlaying these plots suggest that the residuals from the CLL fit exhibit the least dependence on distance.

Fig.3

The fits provided by three inverse link functions. Plot (a) presents the three fits to the empirical frequency data: logit link fit solid/blue, probit link dashed/red, complementary log-log link dot-dashed/green. Plots (b), (c) and (d) present the fits after link transformation. All four plots have distance in yards as the horizontal axis.

The fits provided by three inverse link functions. Plot (a) presents the three fits to the empirical frequency data: logit link fit solid/blue, probit link dashed/red, complementary log-log link dot-dashed/green. Plots (b), (c) and (d) present the fits after link transformation. All four plots have distance in yards as the horizontal axis.
Fig.4

Residuals from three link functions in generalized linear models (logit as blue squares, probit as red triangles and complementary log-log as green circles). Residuals are differences from link-transformed empirical success frequencies and estimated link functions and have been normalized to have unit variance. Plotted against distances with at least 100 attempts, with Lowess smoothers overlaid using solid lines.

Residuals from three link functions in generalized linear models
(logit as blue squares, probit as red triangles and complementary log-log as green circles). Residuals are differences from link-transformed empirical success frequencies and estimated link functions and have been normalized to have unit variance. Plotted against distances with at least 100 attempts, with Lowess smoothers 
overlaid using solid lines.

In an analytical comparison of link functions, McCullagh et al. (1989) observes that for trials with intermediate success probability, 0.1 ≤ π ≤ .9 (the vast majority of field goal attempts), probit and logistic functions are nearly linearly related, so that discrimination between the fits of models using these two links is difficult. Additionally, for trials with success probability approaching one (which occur as field goal attempt distances decrease towards 17 yards), the CLL approaches one more slowly than either of the other two link functions. The CLL link can also differ (Dobson and Barnett, 2011) from logit and probit links for small values of π, which are often of greatest interest when assessing kickers. Close inspection of Fig. 3 and Fig. 4 seem to reflect these observations, with better agreement with empirical frequencies on short kicks than that provided by either logit or probit. Fitting these models using kicks at the extremes may be problematic in that misses at short distances are likely due to mishandled snaps and attempts at long distances may lead to selection bias since these attempts are probably only afforded to very good kickers. In light of these issues, log-likelihood ratios are also reported after restricting attention to kicks with distances between 25 and 50 yards. With this restriction, the CLL link achieves the maximum likelihood among the link functions for models with either a single slope or with kicker-specific slopes.

The statistic proposed by Hosmer and Lemeshow (2004) (HL) offers another way to assess goodness-of-fit in regression models for binary data with continuous explanatory variables. The HL statistic is a Pearson-style measure of goodness-of-fit whose sampling distribution under a correctly specified model can be approximated by a chi-square distribution. The HL procedure partitions all kicks into 10 groups of roughly equal size. (We accomplished this using the algorithm laid out in the documentation for PROC LOGISTIC within the SAS statistical software package (SAS Institute, 2011). The procedure then computes, for each group, observed (Oi) and expected (Ei) counts of successful kicks by averaging estimated probabilities within group, in order to compute the χ2 statistic, X:

πˆ¯i=1nij=1niπˆj,Ei=niπˆ¯i,andX=110(Oi-Ei)2Ei(1-πˆ¯i).

The summary statistics necessary for computation of the HL statistic are given in Table 3 for the logit and CLL links. The logistic regression model does not provide a good fit for the 10th group, where the estimated success probability is highest. The average estimated probability among these 1708 kicks is πˆ¯10=0.9692 for an estimated expected cell count of e10 = 1708 × 0.9692 = 1655.34. However, the observed count is o10 = 1676 successful kicks, resulting in a large contribution, (1676 - 1655.3) 2/{1655.3 (1 -0.9692)} =8.4, to the chi-square statistic.

Table 3

Summary statistics for Hosmer-Lemeshow assessment of the complementary log-log and logit link functions

Complementary log-log linkLogit link
FG MadeFG Made
Group ikicks, nioieiχ2kicks, nioieiχ2
11710883898.020.531711871842.321.92
2169911121097.460.54171211321104.611.91
3170912171213.650.03171512181241.271.58
4171213151310.010.08170513341336.550.02
5170613871392.390.11171013851425.987.09
6171114731478.970.18171214701500.635.06
7172015651558.600.28170915551555.200.00
8170916041606.430.06170916001598.850.01
9171316551653.440.04171316551635.255.26
10171516851685.860.06170816761655.348.37
sum171041389613894.831.89171041389613896.0031.23

Table 4 gives HL chi-square statistics andp-values on 8 degrees of freedom for each of the three link functions with either a single slope, or with kicker-specific slopes. The models with the CLL link function are the only ones that do not exhibit significant lack-of-fit. With such a large sample size (N = 17, 104 attempts), even subtle lack-of-fit is likely to be detected, as in the logistic (p = .0291,. 0657) and probit (p < .0001, p = .0001) models. Despite this significance, neither of the two provides a bad fit.

Table 4

Hosmer-Lemeshow statistics and p-values for two models, each using the three different link functions

Link function
DataModelLogitProbitComp log-log
AllCommon slope34.1(<0.0001)17.1 (0.0291)8.9 (0.3489)
Kicker-specific slopes31.2 (0.0001)14.7 (0.0657)1.9 (0.9843)
Restricted toCommon slope16.8(0.0329)11.6(0.1720)7.9(0.4463)
25 ≤ d ≤ 50Kicker-specific slopes32.3(0.0001)24.1(0.0022)14.8(0.0630)
Restricted toCommon slope19.4(0.0130)12.4(0.1326)7.8(0.4508)
25 ≤ d ≤ 55Kicker-specific slopes24.2(0.0021)16.3(0.0380)10.4(0.2383)

Lastly, a link function could be chosen on the basis of how well probabilities estimated using the link can predict successful and unsuccessful field goal attempts. A decision rule which predicts a successful field goal when the estimated success probability exceeds some threshold (πˆ>π0) has as its sensitivity and specificity the corresponding relative frequencies of successes among kicks which fall, respectively, above or below this threshold. A plot of sensitivity against one minus specificity is called the receiver operating characteristics (ROC) curve and the area under this curve (AUC) is a common way of assessing the capacity of a fitted model to predict successes and failures. The AUC for models with a single slope that are not kicker-specific will necessarily be the same for all three links, as the ordering of the estimated probabilities is determined only by the distance of the attempts. However, small differences in AUC exist among the three links when kicker-specific slopes are included in the model. This ordering is consistent with that implied by visual inspection of Fig. 3. The AUC values, all close to 0.75, are presented in Table 5.

Table 5

Area under the curve (AUC) for the three link functions

Link function
ModelLogitProbitComp log-log
Common slope0.7490
Kicker-specific slopes0.76400.7641.7646

Since the CLL link enjoys the smallest HL and largest AUC under cross-validation (as described in Section 3.1), computations under the other two link functions are excluded from presentation in the remainder of the paper.

3Shrinkage estimation

Model-based estimates of kicker-specific success probabilities have large variance except in cases where the number of observed attempts for a given kicker is very large. As an example, consider interval estimation of the probability of success when kicking a 45 yard field goal using the CLL link function. For Adam Vinatieri, who has 572 career attempts, the 95% interval estimate is (0.69, 0.78). However, the 95% interval estimate of 45 yard field goal success probability for Garrett Hartley, who has 115 career attempts, is (0.59, 0.80), more than twice as wide as the interval estimate for Vinatieri. Estimators with reduced variance may be constructed by taking a weighted average of the kicker-specific maximum-likelihood estimator (MLE) and one developed for the average of all kickers. That is, the variance of an estimator may be reduced by shrinking it towards another estimator which has much lower variance.

Efron and Morris (1975) and Brown (2008) employed this method, viewed from an empirical Bayes perspective, to predict end-of-season batting averages in Major League Baseball. In particular, the papers propose shrinking hitter-specific estimates based on a small number of at-bats towards the early season batting average among all hitters. The methods are based on the theory James and Stein (1961) develops for estimation of the mean of a multivariate normal distribution. That theory does not apply directly to estimation of success probabilities for independent binary trials, as in field goal attempts, but the idea of variance-reduction is worth exploring.

Without any shrinkage, the kicker-specific MLE of the success probability at a given distance d is given by

πˆk(d)=1-exp{-exp(βˆ0+βˆkd)}.

The estimator with complete shrinkage is simply the empirical frequency of all kicks taken by all kickers at that distance,

πˆEF(d)=j:dj=dojj:dj=d1.

These empirical frequencies are plotted in the top left graphic in Fig. 3.

3.1Proposed shrinkage estimators

For shrinkage estimators of the success probability at a given distance, π (d), we consider estimators that are weighted averages of the MLE and empirical frequency estimators defined above, πˆk(d) and πˆEF(d), respectively. These shrinkage estimators may be written in a general form using a weighting function ω:

πˆ(d)=ω·πˆk(d)+(1-ω)·πˆEF(d),
where the notation to indicate dependence of the probability estimate on the left-hand side on kicker k is suppressed for convenience. We consider three options for this weighting function: a constant, ω0; a function of the sample size n available for a given kicker; and a function characterized by a tuning parameter, say a, through which to optimize the weighting scheme.

The first shrinkage estimator we propose is simply the midpoint between the two components. We refer to this shrinkage estimate, obtained by setting ω = 1/2, as the midpoint estimator (denoted by superscript M):

πˆM(d)=12πˆK(d)+12πˆEF(d).

One approach towards constructing a shrinkage estimator that weights model-based kicker-specific estimates more heavily is simply to weight components in inverse proportion to their standard errors. This will upweight the MLE, πˆk(d) for veteran kickers for whom more information is available. For the MLE, we rely upon the GLIMMIX procedure of SAS to produce asymptotic standard errors to accompany the maximum likelihood estimator, denoted by SEk=SE(πˆk(d)). For the empirical frequency component πˆEF(d), we use SEEF=SE(πˆEF(d))=πˆEF(d)(1-πˆEF(d))/Nd where Nd denotes the total number of kicks attempted at distance d in the entire dataset. If the empirical frequency is 0, such as at d = 65, we used SEEF=0.5·0.5/Nd. The resulting inverse variance (IV) weighted shrinkage estimate can then be written in the general weighted average form by taking ω = SEEF/(SEk + SEEF):

πˆIV(d)=SEEFSEk+SEEFπˆk(d)+SEkSEk+SEEFπˆEF(d).

Note that because sample sizes differ, this weighting scheme can lead to different weights at different distances.

Another approach is to choose a weighting function for the MLE that is increasing in nk towards a maximum weight of one. Any cumulative distribution function (CDF) has this property. We choose the exponential distribution function, which for a suitably chosen rate parameter a, increases rapidly for small n with diminishing returns on large nk:

ω(n)=1-e-nk/a.

This choice leads to the exponentially weighted shrinkage estimator,

πˆexp(d)=(1-e-nk/a)πˆk(d)+e-nk/aπˆEF(d).

The quantity a in the exponential weights estimator can be viewed as a tuning parameter, whose value is “fit” using the entire dataset and whose performance is evaluated by cross-validation (see Section 3.2). For the a parameter, mean values 100 < a < 700 were considered, reasoning that the kick counts fell in the interval 4 < nk < 572. A search of values of a, to the nearest multiple of 10, achieves a Hosmer-Lemeshow (HL) statistic of HL = 9.32 at an optimized value of a = 670. Using the exponential CDF with mean a = 670 for a weight function, the corresponding weights on the model-based estimates for kickers with sample sizes of nk = 10,  50,  100,  200,  400 would be 0.02,  0.07,  0.14,  0.26,  0.45, respectively.

Figure 5 presents a plot of the weight function, ω for these different shrinkage estimators. The two separate curves provided for the IV weighted estimator corresponds to kicks at d = 45 and d = 55 yards. The empirical frequencies at these two distances were πˆEF(45)=353/484=0.73 and πˆEF=26/52=0.50, respectively. For these two distances, plots against sample size suggest that the model-based standard errors may be roughly approximated by 0.64/n and 0.54/n, respectively. Because of the dependence of the empirical frequency on sample size, with little information to go by for kicks as long as d = 55 yards, the IV weighted estimator places more weight on the first component for a kick at d = 55 yards than one at d = 45 yards. All of these curves, with vertical axis given on the right of the plot, are superimposed over a frequency histogram showing how the N = 17, 104 attempts are distributed across the K = 111 kickers. The frequencies of kickers with different experience levels is given on the left vertical axis.

Fig.5

Weight functions for shrinkage estimation, “exp” plots the weights based on the exponential distribution and IV plots the weights proportional to the inverse of the variances of the components.

Weight functions for shrinkage estimation, “exp” plots the weights based on the exponential distribution and IV plots the weights proportional to the inverse of the variances of the components.

The estimators of success probability can be naturally shrunk towards central values by means of a Bayesian generalized linear model with CLL link utilizing empirical Bayes estimation (see e.g.,Carlin and Louis (1997)). In such a model, as fit using the bayesglm function of the arm package in R (Gelman et al., 2014), the regression coefficients are assumed to be independent, normally distributed random variables with mean and variance characterized by hyper-parameters that are specified from data, in our case:

β0N(0,10)β1,,βKiidN(-0.05,0.02).

The hyperparameters for the intercept are the default choices for the bayesglm function and those for the kicker-specific slope were chosen from the sample mean and variance of maximum likelihood estimates from the frequentist generalized linear model fit.

Fig. 7 illustrates the reduction in variance due to shrinkage. For attempt distances, d = 30,  40,  50,  60 yards, random samples of 10 kicks were selected. The estimated success probability estimates are plotted using different colors, with estimation technique on the horizontal axis. Not only is the reduction in variation apparent, but the model-based estimates at 60 yards are shifted downward. The explanation for this shift is lack-of-fit of the model for very long attempts. Shrinkage towards the empirical frequency tends to alleviate bias caused by this poor fit.

Fig.7

Model-based maximum-likelihood and shrinkage (midpoint) estimates of success probability, colored according to attempt distance (30, 40, 50, and 60 yards plotted with black, red,green and blue lines, respectively).

Model-based maximum-likelihood and shrinkage (midpoint) estimates of success probability, colored according to attempt distance (30, 40, 50, and 60 yards plotted with black, red,green and blue lines, respectively).

3.2Assessment using cross-validation

While HL is useful for quantifying how well several fitted models align with observed data, it may not tell the entire story as to how well the fitted model would generalize to observations made in the future, or out-of-sample data. Cross-validation (CV) can be used to quantify the degree to which a model fitted to observed data generalizes to data not used to fit the model or to compare the capacity of several models to generalize. Pasteur and Cunningham-Rhoads (2014) uses five-fold CV to select weather, pressure, and fatigue variables for a multiple logistic regression model. In five-fold CV, the dataset is partitioned into five subsets of equal size. The first subset is held out of the process where the model is fit, and then the fitted model is used to estimate the probabilities of the events that were held out. HL is then computed only for the held-out subset. This step is repeated four more times, holding out each subset exactly once. As can be seen from inspection of the first two numerical columns of Table 6, neither the model-based estimate (CLL) nor the empirical frequency (EF) generalize well under 5-fold CV, in the sense that there is disagreement between the number of observed and expected successful kicks when attempts are put into the 10 groups for the HL statistic. Recall that the midpoint estimated probability for any kick is simply the midpoint between CLL and EF. While this shrinkage estimate has a greater HL than its CLL and EF components when the entire dataset is used both to fit the model and to compute the goodness-of-fit statistic, it shows better agreement between observed and expected successful kicks under CV. The CV procedure was repeated four times, using four random partitions of the dataset into five subsets. Each time, the midpoint estimate enjoys a better fit for the held-out data than either of its two components, as quantified by the HL statistic. The shrinkage estimators based on the exponential weights, with a = 660,  670 or 680 also performed better than either component by itself. In computation of these estimates, the step of tuning the a “parameter” was not repeated during cross-validation. Rather, estimates based on values of a = 660,  670,  680 (which did the best on the entire dataset) were computed for every observation, and only those results are reported here. If this weighting scheme is tolerable, then it appears to perform comparably to the simple midpoint shrinkage estimator. Among the six estimators reported in Table 6, there is no dominant choice when considering performance under cross-validation. One clear observation, however, is that the midpoint and exponential shrinkage estimators are generalizing to out-of-sample data better than either the CLL or EF components by themselves.

Table 6

Hosmer-Lemeshow statistics under 5-fold cross-validation for the estimators πˆk, πˆEF, πˆM, πˆexp (under different tuning parameters a), πˆIV, and the Bayes estimator

Shrinkage5-fold Cross Validation
  EstimatorAll Data1x2x3x4xCV avg
CLL1.8915.6020.1322.5922.1320.12
EF0.008.9625.2519.4513.1416.70
midpoint11.878.709.575.294.687.06
exp(a=660)9.836.9710.034.199.077.56
exp(a=670)9.327.0510.524.239.217.75
exp(a=680)9.937.1610.204.817.737.47
IV12.6916.3033.7019.1218.5021.90
Bayesian CLL19.0829.0310.7211.9417.7217.35

4Applications

4.1Ranking kickers

In addition to estimating the success probability for a given kicker from a given distance, the estimation methodology can be used to rank NFL kickers. Historically, comparisons among kickers in sports media have been made using the overall number (as on espn.go.com) or proportion of successful attempts (as on sports.yahoo.com or si.com) without regard to distance, as it not obvious how to incorporate distance and degree of difficulty based on summary statistics. Alternatively, Berry and Berry (1985) suggests averaging model-based kicker-specific estimates over the league-wide distribution of attempt distances. A histogram for these attempts is given in Fig. 8. The average distance over all 17, 104 attempts was 36.6 yards. Tables 7, 8, and 9 present a ranking of kickers under this summary statistic, using several different methods of estimation, along with the standard “overall” metric.

Fig.8

Frequency histogram of attempt distances.

Frequency histogram of attempt distances.
Table 7

Kicker ranking by shrinkage estimate over distribution of all kicks. The columns include the average attempt distance, actual success frequencies for kicks of less than 40 yards, 40-49 yards, and at least 50 yards, overall proportion of successful field goal attempts, ranking by overall success proportion, kicker-specific MLE (πˆ) and midpoint shrinkage estimate of success probability (πˆM)

Kickeravg attempt<4040 - 49≥50pctrank πˆ πˆM
Justin Tucker38.1164/6526/3015/210.9110.920.87
Dan Bailey38.7163/6635/3917/250.8830.900.86
Chandler Catanzaro38.2117/1910/112/30.8850.890.85
Kai Forbath36.9035/3822/252/40.8840.880.85
Blair Walsh39.0153/5719/2417/240.85170.880.84
Connor Barth38.5265/6942/5113/200.86110.870.84
Cody Parkey35.4224/264/64/40.8920.870.84
Patrick Murray41.089/116/75/60.8327.50.870.84
Dan Carpenter38.03111/11565/8220/340.85160.870.84
Steven Hauschka36.4595/10040/499/170.8780.860.84
Greg Zuerlein39.9643/4817/1913/220.82390.860.84
Robbie Gould36.55164/17374/10017/230.8690.860.84
Rob Bironas37.62147/15871/9224/350.85150.860.84
Jason Hanson38.31210/223108/14143/680.84230.860.83
John Kasay37.32198/20388/11432/630.84220.850.83
Stephen Gostkowski35.08185/20165/8514/180.8770.850.83
Josh Brown37.72185/19681/11135/540.83250.850.83
Matt Bryant35.91200/21473/9520/340.85120.850.83
Mike Vanderjagt35.85157/16771/9515/230.85130.840.83
Matt Stover34.96236/24797/1319/200.86100.840.83
Jeff Hall39.602/21/21/10.8054.50.840.82
Shayne Graham35.29197/21169/8915/300.85140.830.82
Sebastian Janikowski38.99222/239111/14848/880.80530.830.82
Phil Dawson35.65256/28085/11535/510.84190.830.82
Adam Vinatieri35.80322/354135/17326/450.84180.830.82
Jeff Wilkins37.35166/17767/10028/410.82380.830.82
Joe Nedney36.77154/16362/8817/310.83350.830.82
Ryan Longwell36.68228/24897/13324/400.83320.830.82
Josh Scobee38.15142/15374/10626/420.80500.830.82
Gary Anderson35.70105/11456/743/70.84200.820.82
Ryan Succop37.0388/9643/5812/200.82360.820.82
Shaun Suisham35.98142/15776/946/180.83300.820.82
Randy Bullock38.6634/3719/255/110.79570.820.82
Nate Kaeding35.46129/13551/7310/190.84210.820.82
Garrett Hartley35.1262/7328/346/80.83240.810.81
Jay Feely36.40232/25995/13019/340.82410.810.81
Nick Folk37.33128/14257/7819/340.80520.810.81
Table 8

Ranking by shrinkage estimate over distribution of all kicks, continued

Kickeravg attempt<4040 - 49≥50pctrank πˆ πˆM
Mason Crosby37.47155/16856/7623/500.80560.810.81
Jason Elam36.85200/21493/13026/420.83340.810.81
Nick Novak36.8181/9231/4112/200.81470.810.81
John Carney35.25213/23677/10811/180.83310.810.81
Mike Nugent36.42132/14557/7610/240.81430.810.81
Alex Henery36.0752/5722/293/80.82400.810.81
Matt Prater37.92107/11543/6625/350.81480.810.81
Cairo Santos35.1017/187/101/20.8327.50.810.81
David Akers36.34292/318110/15927/520.81450.810.81
Doug Brien36.9380/9048/617/170.80510.810.81
Rian Lindell36.09209/22675/11123/420.81490.800.81
Al Del Greco35.6148/5220/301/20.82370.800.81
Jeff Reed34.94169/18354/798/170.83330.800.81
Graham Gano37.2777/8940/5412/200.79590.800.81
Caleb Sturgis37.7236/3914/206/130.7861.50.800.81
Neil Rackers37.10183/20368/9526/530.79600.800.80
Jay Taylor34.673/41/11/10.8327.50.800.80
Olindo Mare35.54238/26789/11719/410.81420.800.80
Morten Andersen35.73136/15053/778/160.81460.790.80
David Buehler38.1513/168/114/60.76720.790.80
Mike Hollis35.8985/9836/517/110.8054.50.790.80
Clint Stitser30.756/71/10/00.8860.780.80
Paul Edinger38.4074/9249/6816/240.76730.780.80
Martin Gramatica37.20114/12837/6319/290.77630.780.80
Lawrence Tynes34.80153/17342/6011/210.81440.780.80
Norm Johnson37.0028/3217/221/60.77680.770.79
Kris Brown37.02167/19476/11118/350.77670.770.79
Todd Peterson35.56113/13143/598/170.79580.770.79
Pete Stoyanovich36.2136/4015/263/40.7764.50.770.79
Billy Cundiff36.39136/15652/739/290.76690.760.79
John Hall36.99128/14451/8410/230.75740.760.79
Brett Conway36.3049/5920/275/110.76700.760.78
Bill Gramatica35.6925/2710/152/60.77660.760.78
Cary Blanchard37.4429/3413/213/60.74780.750.78
Steve Christie35.25105/12135/5910/170.76710.740.78
Dave Rayner37.4142/5119/295/110.73810.740.78
Chris Jacke37.2725/286/111/50.73800.740.77
John Potter34.752/21/10/10.7575.50.730.77
Todd France33.785/52/40/00.7861.50.730.77
Brad Daluiso33.7640/4613/231/10.7764.50.720.77
Craig Hentrich48.502/22/31/50.50104.50.720.77
Cedric Oglesby30.004/51/10/00.8327.50.720.76
Table 9

Ranking by shrinkage estimate over distribution of all kicks, continued

Kickeravg attempt<4040 - 49≥50pctrank πˆ πˆM
Richie Cunningham35.0335/4312/181/40.74770.710.76
Jeff Chandler34.6016/186/100/20.73790.700.76
Wade Richey35.3658/6919/353/70.72820.700.75
Tim Seder35.9530/3614/230/30.71840.700.75
Aaron Elling36.7514/174/71/40.68900.680.75
Eddie Murray35.6512/144/80/10.70860.680.75
Doug Pelfrey35.9829/367/153/60.68880.670.74
Taylor Mehlhaff32.752/31/10/00.7575.50.670.74
Michael Husted35.9332/4414/211/40.68890.660.74
Brian Gowins35.833/31/20/10.6793.50.660.73
Jose Cortez34.0741/5011/241/10.71850.650.73
Steve Lindsey32.713/52/20/00.71830.650.73
Greg Davis36.6811/156/111/20.64970.640.73
Jeff Jaeger34.8519/232/92/20.68910.640.73
Jon Hilbert37.399/102/80/00.61990.630.72
Kris Heppner35.008/112/20/20.6793.50.630.72
Justin Medlock34.836/72/40/10.6793.50.630.72
Chris Boniol35.3019/246/141/20.65960.630.72
Brandon McManus34.857/82/30/20.69870.620.72
Jake Arians37.908/104/110/00.571020.610.71
Seth Marler35.8814/195/121/20.611000.590.70
James Tuthill35.067/102/51/10.63980.590.70
Ola Kimrin36.405/61/30/10.601010.590.70
Ricky Schmitt33.332/30/00/00.6793.50.590.70
Owen Pochman41.594/94/50/30.471060.580.69
Hayden Epstein35.335/60/20/10.561030.520.67
Nate Freese38.143/30/40/00.431070.450.63
Shane Andrus38.202/30/20/00.401080.450.63
Michael Koenen44.002/50/12/80.291110.450.63
Scott Blanton39.331/20/10/00.33109.50.420.62
Aaron Pettrey29.252/30/10/00.50104.50.320.56
Toby Gowin34.671/20/10/00.33109.50.300.56

For veteran kickers with a large number of kicks that are similar in distribution to that of the whole league, the effect of shrinkage is minimal. Adam Vinatieri has succeeded on 84% of 572 attempted kicks at an average distance of 35.8. His model-based estimate averaged over the distribution of all NFL kicks in 1998-2014 is 0.83 and the shrinkage estimate is only slightly different at 0.82. Conversely kickers like Patrick Murray and Greg Zuerlein have achieved very successful records so far in their young careers, hitting 20/24 (83%) and 73/89 (82%), at longer-than-average average distances of 41.1 yards and 40.0 yards respectively. Their model-based estimates averaged over the league-wide distribution of distances is more favorable (87% and 86%; note the bump up in rank from that based on the empirical frequency) but the shrinkage estimate brings these closer to more average kickers (84% for both kickers).

These tables suggest a ranking that places many current kickers among the best over the last 16 year period. Justin Tucker has been successful on 105 out of 116 attempts, with average distance d¯=38.1 for an overall mark of 91%. The kicker-specific, model-based estimate averaged over the league-wide distribution of attempt distances is 92%, but the midpoint technique shrinks this back to 87%. By any of the three methods, Tucker is ranked to be the most accurate among the K = 111 kickers underconsideration.

4.2Stadium effects

A topic of interest for many fans, writers and announcers is the effects of certain stadiums on the chances of making a field goal. The open end of Heinz Field receives particular attention for increased difficulty due to wind (Batista, 2002). Modeling fixed factorial effects for stadiums enables investigation of this issue. Though a likelihood ratio test finds evidence of stadium effects (χ2 = 112, p < 0.0001, df = 50), their inclusion does not lead to an improved fit as assessed by Hosmer-Lemeshow, where there is a greater discrepancy between observed and expected counts than the model based on kickers and distances alone (HL = 8.3with stadium effects and HL = 1.9 without stadium effects, from Table 3). Nevertheless, including these effects does help to identify the more difficult and more favorable stadiums in which to kick, though again the estimation may suffer from selectionbias.

Success probabilities for each stadium can be estimated by back-transforming the marginal mean on the CLL scale. These marginal means average equally over all kickers and correspond to the average distance of d¯=36.6 yards. These estimates provide a ranking of stadium difficulty and are given in Table 11 along with empirical game and field goal attempt counts. We make no attempt to shrink these estimates, rather, we simply restrict our attention to stadiums for which more than 100 attempts are available in our dataset. Table 11 also gives average distances at which coaches have allowed their kickers to attempt field goals, game counts and empirical successrates.

Table 11

Stadium effects and success frequencies. The columns include number of games, average distance, attempts per game, empirical success rates as counts and as a percentage and model-based success probability estimate (πˆ) and standard error

FranchiseStadiumgames d¯att/gmade/attpct πˆStd.err.
ColtsLucas Oil Stadium6237.43.9217/2440.890.880.03
LionsPontiac Silverdome3238.24.0109/1270.860.870.04
BroncosSports Authority Field at Mile High11937.93.7375/4420.850.840.02
VikingsMall of America Field13137.03.7408/4830.840.840.02
LionsFord Field10738.13.8344/4060.850.830.02
Giants/JetsMetLife Stadium8236.43.8270/3130.860.830.03
RavensM&T Bank Stadium14136.24.1494/5720.860.830.02
CowboysTexas Stadium9037.03.8267/3420.780.820.03
EaglesLincoln Financial Field10336.03.7324/3830.850.810.03
RamsThe Dome at America’s Center14038.63.9454/5510.820.800.02
SaintsMercedes-Benz Superdome13436.63.7410/4920.830.800.02
CowboysAT&T Stadium5137.63.8167/1960.850.790.04
TexansNRG Stadium10737.83.8327/4030.810.790.03
FalconsGeorgia Dome14136.43.7433/5250.820.790.02
PanthersBank of America Stadium14036.83.7425/5190.820.780.02
EaglesVeterans Stadium4436.53.8131/1670.780.780.04
SeahawksCenturyLink Field11436.63.8361/4330.830.780.03
CardinalsSun Devil Stadium6438.03.6173/2310.750.780.03
ChiefsArrowhead Stadium13836.73.9435/5430.800.770.02
ChargersQualcomm Stadium14135.93.7427/5250.810.770.02
BuccaneersRaymond James Stadium14037.33.7417/5150.810.770.02
DolphinsHard Rock Stadium14236.94.1470/5790.810.770.02
JaguarsEverBank Field11437.33.6323/4100.790.770.03
ColtsRCA Dome8736.04.0282/3450.820.760.03
49ersCandlestick Park13235.83.8403/5000.810.760.02
BengalsPaul Brown Stadium12336.03.8375/4620.810.740.03
PatriotsGillette Stadium12035.33.7364/4410.830.740.03
Giants/JetsGiants Stadium19934.93.7594/7400.800.740.02
RaidersOakland-Alameda County Coliseum14037.84.1441/5730.770.740.02
TitansNissan Stadium13236.83.8400/4980.800.740.03
BillsNew Era Field12935.33.9403/5040.800.740.03
PackersLambeau Field14636.43.8431/5480.790.730.03
CardinalsUniversity of Phoenix Stadium7736.84.0248/3110.800.730.03
PatriotsFoxboro Stadium3335.64.3111/1430.780.730.04
SteelersHeinz Field12335.93.6350/4450.790.720.03
BrownsFirstEnergy Stadium12835.13.8388/4820.800.710.03
RedskinsFedExField13836.33.7390/5140.760.710.02
BearsSoldier Field13435.93.7382/4950.770.690.03

The ranking in Table 11 seems to corroborate the anecdote that Heinz Field is a tough place to kick, as does the lower-than-average distance of field goal attempts. The new stadium in Denver, “Sports Authority Field at Mile High,” also has a reputation as a good place to kick and it does turn up high on this list. The estimates based on marginal means also tend to be lower than the empirical success rates. This observation may possibly be due to the fact that marginal means average equally over all kickers. Empirical success rates will tend to involve more attempts from high-frequency kickers, who tend to be better kickers.

4.3Big legs

The capacity to kick long-distance field goals is of particular interest to many football fans as well as NFL team management. One way to enrich the model to allow rankings among kickers to vary across distance is to expand the degree of the polynomial to quadratic. Using L and Q superscripts for the linear and quadratic kicker effects leads to the following representations of the generalized linearmodel:

log{-log(1-πj)}=β0+(β1Lxj1++βKLxjk)dj+(β1Qxj1++βKQxjk)dj2andlog{-log(1-πk(d)}=β0+βkLd+βkQd2.

For this investigation, attention is restricted to the K′ = 54 veteran kickers with at least 100 attempts in their careers, since small sample sizes present greater separability issues for fitting quadratic models. A comparison of the model with kicker-specific slopes nested within the quadratic model is not significant using this restricted dataset (χ2 = 63.3, df = 54, p = 0.181). This non-significance could possibly be explained by selection bias, as kickers with many attempts will all tend to be good enough to remain in the league. While not significant, there is a preponderance of kickers for whom the estimated quadratic coefficients have small p-values, suggesting that there are some kickers for whom quadratic coefficients are non-zero. In fact the average p-value for these 54 tests of the form βkQ=0 is 0.16 providing anecdotal evidence of curvature. For one further bit of evidence, a comparison of nested models based on the logit link function is weakly significant, (χ2 = 72.8, df = 54, p = 0.0450), though the fit is not as good as that provided bythe CLL.

Table 10 enables simultaneous inspection of kickers who have estimated success probabilities that rank highly (top 20) in at least one of the three distances, 35, 45 or 55 yards. Entries in the table are sorted by the success probability estimate at 55 yards, πˆk(55). Estimates at d = 35, 45 are also given along with ranks which enable identification of kickers who seem to have strengths at either medium or long distances (or both). While some kickers are accurate from any distance, for example current kickersBailey, Tucker, Walsh, and Barth, others have a decent shot at making long kicks but can be less accurate than is desired at short distances, for example Edinger and Hollis. Lastly are the kickers who rarely miss from intermediate distances, but are not estimated to have a big enough leg to have much more than a coin flip chance of success on long field goals, for example Gould, Kasay, and Stover. For comparison with observed data, the last three columns give empirical success frequencies after binning kicks into 10-yard intervals surrounding d = 35,  45,  55 yards. Clustering can be used to group kickers, based on their trivariate estimated success probability at the three distances, 35,45 and 55 yards, such that within groups, kickers are similar. The default hierarchical clustering algorithm implemented by SAS PROC CLUSTER suggests the existence of three clusters, and that cluster assignment is mostly determined by πˆ(55). The first 8 kickers in Table 9 form a “big leg” cluster, the last four kickers form the “accurate but no leg” cluster, and the remainder fall somewhere in between. The means of the estimated success probabilities for these three groups are plotted against yards in Fig. 6. This does suggest that there may be somewhat of a trade-off between a “big leg” and accuracy at closer ranges.

Fig.6

Three clusters of kickers using shrinkage estimates of success probabilities at three distances.

Three clusters of kickers using shrinkage estimates of success probabilities at three distances.
Table 10

Rankings of kickers among the top 20 at three different distances: 35, 45, and 55 yards. The columns include career span within the data set, ranks at each of the three distances, success probability shrinkage estimate at each of the three distances, and the empirical success probability at three distance ranges 30-39, 40-49, and 50-59 yards. Dashed lines separate clusters

careerranks πˆk(d)empirical success
Kickerspand = 35455535455530 - 3940 - 4950 - 59
Garrett Hartley08-14461110.850.750.6227/3528/346/8
Dan Bailey11-143220.910.800.6038/4035/3917/25
Justin Tucker12-141130.920.810.6032/3326/3014/19
Paul Edinger00-05533840.820.720.5935/5149/6816/24
Stephen Gostkowski06-1422850.880.760.5994/10565/8514/18
Blair Walsh12-149360.890.780.5830/3319/2417/23
Connor Barth08-147470.900.770.5733/3742/5113/20
Rob Bironas05-1315680.880.760.5775/8371/9223/31
Phil Dawson99-14311590.870.750.56113/13085/11535/48
Matt Bryant02-141913100.880.750.56101/11073/9519/33
Mike Hollis98-024937110.840.720.5543/5136/517/11
Matt Prater07-143319120.870.740.5552/5743/6624/34
Steven Hauschka08-1487130.900.760.5452/5540/499/15
Nick Folk07-144129140.860.730.5466/7457/7819/32
Jason Hanson98-12109150.890.760.54107/118108/14143/67
Graham Gano09-144332160.850.720.5431/4040/5412/19
Adam Vinatieri98-142618170.880.740.54150/178135/17326/45
Josh Brown03-141314180.890.750.5491/10081/11135/54
Shaun Suisham05-143222190.870.740.5366/7876/946/18
Jay Feely01-144030200.860.730.53121/14195/13018/32
Dan Carpenter08-1445210.900.770.5352/5565/8219/32
Jeff Wilkins98-072320220.880.740.5274/8267/10028/40
Mike Vanderjagt98-061216240.890.750.5269/7671/9515/22
Sebastian Janikowski00-141417270.880.740.52116/129111/14846/79
Jason Elam98-091821300.880.740.5194/10693/13025/39
Robbie Gould05-14510330.900.750.5181/9074/10017/22
Ryan Longwell98-112027340.880.730.51120/13597/13324/40
Shayne Graham01-141623360.880.740.5097/10669/8915/29
Joe Nedney98-101128440.890.730.4869/7762/8817/31
Nate Kaeding04-121731450.880.720.4850/5451/7310/19
John Kasay98-11212470.910.750.4882/8788/11432/60
Stover98-09633520.900.720.44101/11197/1319/20

Table 13 helps quantify the variance-reducing property of shrinkage estimation by comparison of the coefficient of variation among kickers with maximum likelihood (no shrinkage) estimates. The table gives results for both linear and quadratic fits, with diminished coefficient of variation for shrinkage estimates, particularly for shorter attempts.

Table 13

Summary statistics among estimated success probabilities for 54 veteran kickers

StatisticModelMethod404550556065
MeanLinearML0.800.710.620.540.450.38
shrinkage0.810.730.640.520.320.19
QuadraticML0.810.720.620.520.430.34
shrinkage0.810.740.640.510.300.17
Coef.LinearML5.938.2510.7713.4116.1318.87
Var.Shrinkage2.924.015.266.9011.5018.87
QuadraticML6.367.9410.6015.3122.6933.11
Shrinkage3.153.885.167.7715.8933.11

Another way to rank kickers is to consider big-legs and stadium effects and shrinkage estimation all together. Stadium effects can bias a ranking and by including them in the model, together with quadratic distance effects, one can attempt to rank kickers on an even footing. The model is then given by

log{-log(1-πj)}=β0+(β1Lxj1++βKLxjk)dj+(β1Qxj1++βKQxjk)dj2+β1Ssj1++βSSsjS
where β1S,,βSS and sj1, …, sjS denote stadium effects and indicator variables for kick j, respectively. The midpoint shrinkage strategy can again be used to rank kickers, resulting in Table 12. This ranking shuffles the kickers modestly from that in Table 11 and is more fair in the sense that it attempts to adjust for stadium effects. If these trivariate shrinkage estimates at d = 35, 45, 55 are again clustered, then only two clusters are recommended by the procedure. All of the members of the big-leg club are retained and three new kickers are admitted: Mike Hollis, Matt Bryant, and Phil Dawson. These three kickersprovide some interpretation of stadium effects. Estimates for both Hollis and Dawson are adjusted upward due to the fact that these kickers kicked for teams with tougher stadiums: Alltell Stadium and Cleveland Browns Stadium, respectively. Bryant kicked for teams in the Georgia Dome and Raymond James Stadium, both of which have nearly average stadium effects and his estimates are barely adjusted at all. He makes it into the big-leg club because the clustering criteria changed slightly with the adjustedestimates.

Table 12

Rankings of kickers among the top 20 at three different distances: 35, 45, and 55 yards, including adjustment for stadium effects.The columns include career span within the data set, ranks at each of the three distances, success probability shrinkage estimate at each of the three distances, and the empirical success probability at three distance ranges 30-39, 40-49, and 50-59 yards. Dashed lines separate clusters

careerranks πˆk(d)empirical success
Kickerspand = 35455535455530 - 3940 - 4950 - 59
Garrett Hartley08-14481510.850.750.6227/3528/346/8
Dan Bailey11-144220.910.800.6138/4035/3917/25
Stephen Gostkowski06-1423830.880.770.5994/10565/8514/18
Paul Edinger00-05533340.830.730.5935/5149/6816/24
Justin Tucker12-141150.920.810.5832/3326/3014/19
Connor Barth08-146460.900.780.5733/3742/5113/20
Rob Bironas05-1312670.890.770.5775/8371/9223/31
Blair Walsh12-148780.900.770.5730/3319/2417/23
Mike Hollis98-02412190.860.740.5643/5136/517/11
Matt Bryant02-142511100.880.760.56101/11073/9519/33
Phil Dawson99-141610110.890.760.55113/13085/11535/48
Graham Gano09-144331120.860.730.5531/4040/5412/19
Shaun Suisham05-142616130.880.750.5466/7876/946/18
Steven Hauschka08-1479140.900.760.5352/5540/499/15
Dan Carpenter08-1455150.910.770.5352/5565/8219/32
Jay Feely01-143625160.870.740.53121/14195/13018/32
Josh Brown03-142117170.880.750.5391/10081/11135/54
Nick Folk07-144538180.860.720.5366/7457/7819/32
Adam Vinatieri98-143123190.880.740.53150/178135/17326/45
Robbie Gould05-1423200.910.780.5281/9074/10017/22
Sebastian Janikowski00-141113210.890.760.52116/129111/14846/79
Jeff Reed02-101718240.880.740.5286/9454/798/16
Jason Hanson98-122220250.880.740.52107/118108/14143/67
Mike Vanderjagt98-06914270.900.750.5269/7671/9515/22
Shayne Graham01-141419340.890.740.5097/10669/8915/29
Ryan Longwell98-112030370.880.730.50120/13597/13324/40
Mike Nugent05-141932400.880.730.4965/7757/7610/24
Mason Crosby07-141528410.890.730.4977/8756/7623/49
Joe Nedney98-101022430.900.740.4969/7762/8817/31
Nate Kaeding04-121836460.880.730.4850/5451/7310/19
John Kasay98-11312480.910.760.4782/8788/11432/60
Matt Stover98-091345540.890.710.43101/11197/1319/2

4.4Non-stationarity

The analyses in this paper cover a wide time span. Kicker ability appears to be changing, especially in the last 10 years, with improving accuracy and experience among kickers (Clark et al., 2013). The frequencies of long field goal attempts are also increasing. Figure 9 shows the log of attempt frequencies against year, with separate plotting characters and colors for each distance, rounded to the nearest five yards. The log-scale is better for visualizing the increase in frequencies. In fact, the log of the frequency of kicks between 53 and 57 yards, denoted in the graph as “55”, and also of kicks between 48 and 52 yards, denoted as “50”, appear to be linearly increasing with season. The figure overlays lines using estimated coefficients from Poisson regressions of frequency on year, separate for each 5-yard distance interval, with the natural log-link function for the generalized linear model. The slopes are significant for distances of 50, 55, and 60 yards.

Fig.9

Log of attempt frequencies over season separated by distances rounded to the nearest five yards starting at 40 yards. Lines are linear regressions of log-transformed attempt frequencies on year.

Log of attempt frequencies over season separated by distances rounded to the nearest five yards starting at 40 yards. Lines are linear regressions of log-transformed attempt frequencies on year.

5Discussion

With the goal of variance-reduction akin to that achieved by Efron and Morris (1975) and Brown (2008) in the prediction of end-of-season batting averages, we presented a flexible framework for the construction of shrinkage estimators of field goal success probabilities. This framework allows for model-based kicker-specific maximum likelihood estimates to be balanced with model-free empirical frequencies aggregated over all kickers. For the model-based estimator, various link functions were assessed using maximum likelihood, Hosmer-Lemeshow statistics and restrictions on data motivated by possible selection bias. The complementary log-log function was selected by all criteria. Findings from several choices for the weight functions of the model-based estimate were presented, including constant, functional, or distribution-based weights, but our investigation was by no means exhaustive. We recommend the midpoint estimator which weights the two components equally on the basis of simplicity and generally good performance as measured by the Hosmer-Lemeshow statistic under cross validation. Interested readers are invited to give other weighting schemes a try. In particular, it seems that low-frequency kickers ought to be estimated using more shrinkage toward the empirical frequency. Since these kickers constitute a comparatively small fraction of the data set, they do not carry much weight as we assess the performance of shrinkage estimators. It is reasonable that in some personnel decisions, they should carry more weight, and a different shrinkage approach and method of assessment may be more appropriate.

Another limitation of the study has to do with failed attempts that are not the fault of the kicker, or perhaps partially the fault of the kicker. If a snap or hold is bad, but an attempt is made and missed, it may appear in a verbal recap, but appears in the play-by-play simply as “no good.” Our model does not account for this type of event. Excluding blocked kicks is also problematic as it is possible that longer attempts may be more likely to be blocked by lesser kickers, so that these failed attempts contain information about kicker ability.

Using the approach of Berry and Berry (1985) to average success probability estimates over the observed distribution of attempt distances, rankings for kickers were provided. Interestingly several current kickers came out on top. In addition to this overall ranking, distance was fixed at several lengths, d = 35,  45 and 55 yards, corresponding roughly to estimated success probabilities for elite (top 20) kickers in the range of 0.9, 0.8, and 0.6 respectively. These estimates are based on a version of the model that expands the degree of polynomial dependence on distance to be quadratic. Presentation of the bestkickers side-by-side enables investigation of kickers who have “big legs,” with a decent shot at long kicks, in comparison to those who are “automatic” at medium distances. An attempt to refine these rankings was made by adding stadium effects, which were found to be highly significant. The estimated marginal (averaged over kickers) means for stadiums with adequate sample sizes were back-transformed to give a ranking of the toughest venues in which to kick.

General managers have made kicker personnel changes with surprising frequency, perhaps overreacting after the sting of a loss which involved a missed field goal. Let us consider the case of Garrett Hartley in more detail. Hartley began his career with the New Orleans Saints in 2008 by making all 13 of his field goal attempts. The following year he made all but two field goal attempts, with the misses coming from 58 and 37 yards. As mentioned in Section 1, in 2010 Hartley missed a 29 yard attempt in overtime in a week 4 loss and the Saints chose to sign John Carney. Carney proceeded to make 5 out of 6 field goals for the two games he replaced Hartley, missing from 29 yards in week 5, the exact same yardage which precipitated Hartley’s demotion. So the following game, Carney was demoted in favor of Hartley and Hartley proceeded to make 16 of the 18 remaining kicks he attempted in the season. The frequency of personnel changes among kickers, averaging more than 3 mid-season cuts per year, is surprising in light of the small number of total kickers over the 16 year period of this study. Though kickers may lose their jobs, many find employment shortly after termination, with a small number moving on through free agency. Whether the high turnover is a result of misunderstanding of the sampling variability inherent in estimating success probabilities from a small number of binary trials, or due to reaction to fan hostility is unclear.

Over the last five years, there has been an increase in the distances at which field goals are attempted, most likely due to increases in kicker abilities. Attempting to model this increase could lead to improvements in the estimation of success probabilities and would be an excellent avenue for further study. Another potentially interesting topic that could be investigated would be the effect of age on elite kickers who have exhibited longevity in the league. Here it is implicitly assumed to be constant, but we may wonder if it is reasonable to expect the same degree of accuracy at all stages of a kicker’s career.

References

1 

Batista J. , 2002, Pro football; place-kickers at heinz field living in a swirl of trouble, New York Times.

2 

Berry D.A. and Berry T.D. , 1985 The probability of a field goal: Rating kickers, The American Statistician 39(2), 152–155.

3 

Berry S.M. and Wood C. , 2004, A statistician reads the sports pages: The cold-foot effect, Chance 17(4), 47–51.

4 

Bilder C.R. and Loughin T.M. , 1998, “it’s good!” an analysis of the probability of success for placekicks, Chance 11(2), 20–30.

5 

Brown L.D. , 2008, In-season prediction of batting averages: A field test of empirical bayes and bayes methodologies, The Annals of Applied Statistics, 113–152.

6 

Carlin B.P. and Louis T.A. , 1997, Bayes and empirical bayes methods for data analysis, Statistics and Computing 7(2), 153–154.

7 

Clark T.K. , Johson A.W. and Stimpson A.J. , 2013, Going for three: Predicting the likelihood of field goal success with logistic regression, in ‘Sloan Sports Analytics Conference’.

8 

Dobson A.J. and Barnett A. , 2011, An introduction to Generalized Linear Models, CRC press.

9 

Efron B. and Morris C. , 1975, Data analysis using stein’s estimator and its generalizations, Journal of the American Statistical Association 70(350), 311–319.

10 

Gelman A. , Su Y.-S. , Yajima M. , Hill J. , Pittau M.G. , Kerman J. and Zheng T. , 2014, arm: Data analysis using regression and multilevel/hierarchical models 2010, http://CRAN.R-project.org/package=arm. R package version pp. 1–3.

11 

Hosmer D.W. and Lemeshow S. , 2004, Applied Logistic Regression, John Wiley & Sons.

12 

James W. and Stein C. , 1961, Estimation with quadratic loss, in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability 1, 361–379.

13 

McCullagh P. , Nelder J.A. and McCullagh P. , 1989, Generalized Linear Models 2, Chapman and Hall London.

14 

Morris C.N. and Rolph J.E. , 1981, Introduction to Data Analysis and Statistical Inference, Prentice-Hall Englewood Cliffs Englewood Cliffs, NJ.

15 

Morrison D.G. and Kalwani M.U. , 1993, The best nfl field goal kickers: Are they lucky or good? Chance 6(3), 30–37.

16 

Pasteur R.D. and Cunningham-Rhoads K. , 2014, An expectation-based metric for nfl field goal kickers, Journal of Quantitative Analysis in Sports 10(1), 49–66.

17 

R Core Team, 2014, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/

18 

SAS Institute, (2011), SAS/IML 9.3 User’s Guide, SAS Institute.