Editors’ introduction: Spygate and sports law analytics
Abstract
The purpose of our editors’ introduction is three-fold. First, we provide a primer on sports law analytics for this special issue of the Journal of Sports Analytics. Second, we briefly summarize the papers accepted for inclusion in the special issue following double-blind peer review. Third, we present original research on one of the most vexing sports law analytics questions in the past decade— Is there evidence that the New England Patriots gained a competitive advantage from recording and analyzing opponents’ signals in contravention of NFL rules?
1Sports law analytics primer
We define sports law analytics as the application of parsimonious statistics to real-world legal issues in the sports industry. But sports law analytics is no panacea. Such work is, more often than not, far from definitive. Nevertheless, sports law analytics can be helpful in forensically finding statistical fingerprints in some sports legal issues. Sports law analytics can also complement more traditional investigatory techniques such as interviews, document review, interrogations, physical evidence collection, subpoenas, and polygraphs. In sum, sports law analytics is an additional tool in any investigator’s (complete) toolkit.
In a court of law, sports law analytics exists at the intersection of quantitative methods and formal rules of evidence. Indeed, “analytics are proving to be dispositive in high-stakes sports industry litigation.”1 Getting sports law analytics into the courtroom can sometimes be tricky, however.
Federal Rule of Evidence 702, coupled with the U.S. Supreme Court’s decision in Daubert v. Merrill Dow Pharmaceuticals, 2 provides the legal framework for admissible evidence in the quantitative realm. In relevant part, Rule 702 provides:
“If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case”.
2Special issue overview
Sports law analytics also exists outside the courtroom, as each paper in this special issue demonstrates. Matt Frankel wrote a non-quantitative legal essay about how sports analytics overlap with intellectual property law. Frankel touches on the FBI’s probe into allegations of possible trade secret theft involving two Major League Baseball teams. Rick Borghesi penned a short meta-analysis about point shaving in college basketball. With recent point shaving revelations involving the University of San Diego and the University of Toledo, the topic is germane.
Christian Deutscher analyzed recently-released NBA referee play-calling data and found no evidence of any late-game bias by the on-court officials. Deutscher’s null results are consistent with the NBA’s own internal evaluation of the topic. Jun Woo Kim wrote a short research note about precocity in the NFL, an inquiry that indirectly tested the league’s controversial minimum age rule. Former Ohio State University running back Maurice Clarett challenged the NFL’s age rule in court over a decade ago and one of the issues before the court was the rationale for the rule. Kim provides evidence that will likely be relevant the next time the NFL’s eligibility rule is legally challenged.
Three full-length articles round out the special issue. First, Kevin Hassett, Joseph Sullivan, and Stan Veuger completed a textured analysis of football air pressure issues at the foundation of on-going litigation pertaining to the NFL collective bargaining agreement, New England Patriots quarterback Tom Brady, the players union, and league executive Roger Goodell. Second, Michael Schuckers and Steve Argeris wrote about NHL team scouting and return on investment in draft selections. Third, Michael Palmer, M. Quinlan Duhon, and Brian Soebbing analyzed college deviance and its impact on NFL draft selection.
3Spygate: A case study in sports law analytics
3.1Introduction
No quasi-legal sports scandal unrelated to doping has lingered more the past decade than Spygate. From the scandal’s origins in September 2007 to recent in-depth investigative reports by both ESPN The Magazine 3 and Sports Illustrated,4 Spygate seemingly won’t go away. The scandal’s central question remains ripe for sports law analytics— To what extent, if at all, did the New England Patriots’ impermissible taping of opponents’ play calling signals affect game outcomes?
We attempt to answer the question here. We emphasize that our indirect detection methods have limitations.5 Most notably, such methods are not a complete substitute for direct investigatory measures by law enforcement or other personnel charged with the duty to monitor. Like other research methods, sports law analytics inevitably carries a risk of both false positives and false negatives. Forensic-leaning sports law analytics can, however, help shine a light on malfeasance when wrongdoers are trying to avoid detection and conceal their activity.6 In this way, such analytics can sometimes be used as a starting point for additional analysis and decision-making regarding the allocation of investigatory resources.
If the New England Patriots gained a competitive advantage from the long-running practice of taping their opponents— an act in clear violation of widely-known NFL rules7 — we assume that one way the practice would manifest itself would be in the second half of games, after the opponents’ signal system was confirmed by one or more code-breakers, and the game leverage was higher. If the spying was helpful, we reason, then the Patriots would win more games than expected due to relatively superior play(-calling) and audible selection in the second half.
To test this hypothesis, we created a halftime “win predictor” with a linear regression that weighed the halftime score and the betting spread of the game. This predictor estimated a team’s probability of winning the game at halftime. We compared the Patriots’ actual performance versus their model- based expectation for two different eras: (i) the 2000-2001 season through the 2006-2007 season and (ii) the 2007-2008 season through the 2014-2015 season. Resulting prima facie evidence indicated that the Patriots over-performed in both eras, winning roughly 17.46 games above the model’s expectation across the entire time span. The results were consistent with an effective clandestine recording and decoding signal system of the type the Patriots are known to have implemented and utilized. The data also suggested that after the Spygate scandal broke in September 2007, this effect was reduced on the road for New England, but remained consistent at home.
3.2Background and discussion
Under head coach Bill Belichick’s tenure from 2000-01 to 2014-15, the Patriots played 269 games8 and accumulated an overall win-loss record of 196-73,9 winning 72.86% of their games. During this time, the team won the AFC East division twelve times, won six AFC titles, and won four Super Bowls.10 This impressive resume is marred, however, by the Spygate revelations and subsequent punishment by theNFL.
On September 9, 2007, NFL security officials caught a Patriots videotape employee recording the New York Jets’ defensive signals during a road game. This was after New England was caught recording signals in an away game against the Green Bay Packers in 2006.11 Without first reviewing any of the videotapes or sign-stealing materials prior to having the videotapes and related materials destroyed,12 NFL commissioner Roger Goodell fined Belichick $500,000, fined the Patriots $250,000, and ordered the team to forfeit a first-round draft pick if they made the playoffs (or its second and third round draft picks if the team missed the playoffs).13
Goodell never disclosed the extent of the Patriots’ videotaping, but details were revealed in a September 28, 2015 ESPN The Magazine article by Don Van Natta and Seth Wickersham. According to Van Natta and Wickersham in ESPN The Magazine, “[t]he Patriots created a novel spying system that made the decoding [of opponents’ signals] more dependable.”14 The reporting of Van Natta and Wickersham was consistent with a June 5, 2008 Senate floor statement by former U.S. Senator Arlen Specter, a Republican from Pennsylvania, who suggested that the Patriots’ tape-recording was more extensive and serious than Goodell publicly indicated. On September 14, 2015, Sports Illustrated’s Greg Bishop, Michael Rosenberg, and Thayer Evans explained how pervasively other teams, coaches, and executives perceive possible cheating by the Patriots to be. The Sports Illustrated authors posited that: “[h]ard evidence is hard to come by, but suspicions about the wildly successful Bill Belichick won’t subside anytime soon.”
Why would New England risk sanctions from the NFL to record sideline signals? If these signals were accurately decoded, the Patriots could potentially react on-field to the signals sent by their opponents. The Patriots could know what plays their opponents were going to run on offense, or what defensive schemes or audibles were being used.15 Pointedly, revelations in the aforementioned ESPN The Magazine article by Van Natta and Wickersham indicated that investigators at Patriots’ headquarters found a library of scouting material containing videotapes of opponents’ signals with detailed notes matching signals to plays for many teams going back seven seasons.
Anecdotal evidence of such taping and code-breaking by the Patriots is equally bountiful. For example, as quoted in the aforementioned ESPN The Magazine article, former Pittsburgh Steelers player Hines Ward said of the Patriots in the 2002 AFC title game: “Oh they knew, they were calling our stuff out. They knew a lot of our calls. There’s no question some of their players were calling out some of our stuff.” After Pittsburgh lost the 2005 AFC title game to New England, a Pittsburgh defensive coach stated: “They knew our signals, so they knew when it went in what the coverage was and how to attack it. I had a couple of guys on my teams from New England, and they’ve told me those things.”
Other more recent examples of this perception exist too.16 The September 2015 Sports Illustrated article by Bishop, Rosenberg, and Thayer quoted a current player as follows: “‘You play the Patriots, and they know almost everything you’re doing and every defense that you’re in,’ [said] Chris Harris Jr., the [Denver] Broncos cornerback. ‘Which is crazy.”’ In a December 2, 2013 ESPN.com piece, Tania Ganguli wrote:
“Many of Houston’s defensive players left Sunday’s loss to the Patriots talking about the adjustments New England made to the Texans’ plan. In defensive end Antonio Smith’s estimation, the Patriots’ offensive adjustments were too good at some points. ‘Either teams are spying on us or scouting us,’ Smith told a group of reporters. … ‘I’m very suspicious,’ said Smith. ‘I just think it will be a big coincidence if that just happened by chance. I don’t know for sure, but I just know it was something that we practiced this week.’”
A hypothesis reasonably consistent with such observations is that New England could videotape signals sent during the first half, and have one or more code-breakers analyze the video at halftime.17 If the signals that were successfully de-coded at halftime did not change during the second half, then New England could exploit that.18 It could give the Patriots an advantage in the second half. However, from a game theory perspective, the more the Patriots exploited this hypothetical information, the greater the risk of discovery. While New England’s general goal is to win as many games as possible every season, the team’s specific goal each year, we assume, is to win the Super Bowl.19 Accordingly, if acting rationally given the risk of detection, the team would seemingly only exploit their taping in the most highly leveraged situations, and less often (or never) when the game outcome was not in question or the game itself was meaningless. In general, the leverage for a game is highest when the score is close, and as the end of the game nears.20
If New England were breaking opponents’ signal codes during halftime, what would one expect to see as a result? First, New England would seemingly gain an advantage in the second half. The team would win more often than one expects, given the halftime score and the relative strength of each team. Second, one would expect the advantage to be smaller on the road than at home. This is due to the difficulties in obtaining video on the road, where Patriots personnel do not have more permissive access to create the quality video feeds needed to tape and decode new signals vis-á-vis old signals already on file. Third, one would expect any advantage to be lower on the road after the Spygate scandal broke September 2007, working under the theory that the Patriots would no longer risk getting caught during away games, but they may continue to record signals at home where the risk of detection is likely lower.
During the 2000-01 to 2006-07 era, the Patriots outperformed the win predictor model by 7.95% per game, winning 9.78 games above expectation. They outperformed more at home (+6.43 games) versus the road (+3.35 games). During the 2007-08 to 2014-15 era, the Patriots were +7.61 games above expectation at home, but only +0.07 on the road.
How likely is a team to outperform the model at home by 14.04 games, which New England did in its 137 home games during a fifteen year span from 2000-01 to 2014-15? A simplified way to estimate this is to use a binomial distribution using the average projected win rate of all their games.21 During this period, the Patriots were projected to win 96.96 of the 137 home games. They actually won 111 home games. If one takes a binomial distribution of 137 home games played, and counts the frequency that New England would win 111 games or more, it happens with a frequency of p = 0.00424, or roughly one in 236 observations. This was rarer than any other team or coach measured during the time frame.
Notwithstanding documentation of taping in violation of NFL rules, what if Bill Belichick were simply an exceptional coach? Could these results be explained by brilliant halftime adjustments unaided by impermissible taping of opponents’ signals? If such halftime adjustments were the cause, one would expect to see similar outperformances at home and on the road. During Belichick’s entire career, his home performance was +14.04 wins out of 137 games, or +10.2% . His road performance was +3.42 out of 126 games, or +2.7% . The difference is 7.5% . If innocuous halftime adjustments were the sole cause of the Patriots’ superior performance, one would expect his home and road performance to be similar. If one limits the analysis to games after the Spygate scandal broke in September of 2007, the difference is even greater. From 2007-08 to 2014-15, the Patriots were +7.61 wins at home out of 75 games (+10.1%) versus +0.07 wins on the road out of 65 games, or +0.001% . Nearly the entire second-half advantage vanished on the road. This suggests that whatever was causing the Patriots to overachieve became more difficult to do on the road after the September 2007 Spygate revelations.
One next asks— Is there something special about the New England home field advantage? Could a larger than average home field advantage explain these results? If New England’s home field were more favorable than other teams, one might expect New England to do better at home, both in the first and second half relative to other teams. Similarly, if New England’s home field advantage were less, one would expect the Patriots to do worse, both in the first and second half, relative to other teams. The most obvious measure of home field advantage is by points scored versus points allowed. League wide, how much does the average team win by at home? If one adds all the points scored by home teams, subtracts all the points scored by visiting teams, and divides by the total games played, the home field advantage is worth about 2.65 points. We can use a team specific evaluation of home field advantage by looking at the difference of New England’s average margin of victory at home, minus their average margin of victory on the road, and dividing by two. This straightforward calculation suggests that New England’s home field advantage is only 1.71 points, less than the league average. If home-field advantage were the cause of New England over-performing at home, one would expect the point value to be higher, not lower. The Patriots lower than average home field advantage would make the Patriots’ second-half dominance even less likely.
3.3Aftermath and conclusion
According to Bishop, Rosenberg, and Thayer in Sports Illustrated, “[t]he effects of Spygate are still rippling through the NFL.” Within days of the Spygate scandal being revealed in September 2007, the NFL meted out punishment against both the New England Patriots and head coach Bill Belichick. But the controversy lingered far longer. A disgruntled New York Jets season ticket holder filed an unsuccessful federal lawsuit against the NFL, Patriots, and Belichick.22 U.S. Senator Arlen Specter called for a Congressional investigation.23 In his June 5, 2008 statement on the floor of the U.S. Senate, Sen. Spector concluded: “[t]he overwhelming evidence flatly contradicts [c]ommissioner Goodell’s assertions that there was little or no effect on the outcome of games.”24 And, most relevant to our inquiry here, others undertook a Spygate-focused analysis too.
Like us, author Bryan O’Leary opted “[t]o look for statistical footprints of the effectiveness of the Spygate system.” To do so, O’Leary consulted with statistician Miao Zang and analyzed three metrics: (i) home win/loss record; (ii) perfect home record; and (iii) record against the spread. On the first metric, Zang concluded: “[the Patriots] are nearly three standard deviations away from the average, an extreme outlier.” As to the second metric, Zang found “the event that the New England Patriots had five perfect home seasons from 2001 to 2011 is statistically highly unusual and deserves inspection.” With regard the third metric, “the Patriots are once again nearly three standard deviations away from the league average.”
The methodology, analysis, and results most similar us was completed by researcher Brian Burke in a series of 2007-08 Advanced Football Analytics blog posts. On September 15, 2007, Burke wrote, in relevant part:
“If Belichick’s Patriots exploited unfair advantages in stealing signs from opposing sidelines we would expect to see some sort of evidence that they won games ‘beyond their means.’ By means I am referring to the Patriots’ passing and running performance on offense and defense. By successfully exploiting stolen signs, we might expect the Patriots to choose to use that advantage on critical plays— 3rd downs in the 4th quarter for example. These critical plays would heavily ‘leverage’ performance on the field to be converted into wins. In other words, the Patriots would win more games than their on field stats would indicate. This is exactly what we see in the data. Year-in and year-out, Belichick’s Patriots have won about 2 more games than expected given their offensive and defensive efficiencies, including turnovers and penalties. No other modern team has even come close to the Patriots in consistently winning more games than their stats indicate. Could those extra wins be due to cheating?”
Like Burke, our analysis is consistent with New England gaining an advantage in the second half of games, possibly from recording signals and breaking teams’ sideline codes. Alternative explanations for New England’s performance were considered, including home field advantage and Belichick’s coaching acumen, but such factors seemed unlikely to cause the additional wins. The data also shows that New England may have gained an advantage from code-breaking opponents’ signals after Spygate when playing at home. However, given the limitations inherent in any “forensic sports law analytics” approach, we do not deem our research definitive absent a conclusive full-blown investigation by an independent third party.
Indeed, in a February 3, 2008 Advanced Football Analytics blog post, researcher Brian Burke specifically outlined how such an investigation could have been undertaken:
“The other point is that if the NFL really wanted to investigate these things, there is ample evidence in the NFL Films archive. There are probably hours upon hours of sideline film from just the Patriots’ Super Bowls alone, not to mention playoff games or regular season games. An honest investigation would have taken weeks, not the couple of days the NFL took before destroying the evidence.”
Such an investigation would seemingly be consistent with Roger Goodell’s statement, as memorialized in a January 31, 2008 letter responding to Sen. Specter, that: “I believe that I have no more significant responsibility than protecting the integrity of the game and promoting public confidence in the NFL … ”
We agree. And we offer that sports law analytics can be a useful tool to detect nefarious activity and, in turn, protect game integrity and promote public confidence.
Notes
1 Ryan M. Rodenberg & Tassos Kaburakis, Sports Law Analytics, ANALYTICS MAGAZINE, May/June 2011, p. 29-32.
2 Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993).
3 Don Van Natta Jr & Seth Wickersham, The Patriot Way, ESPN THE MAGAZINE, September 28, 2015, p. 13-38.
4 Greg Bishop, Michael Rosenberg & Thayer Evans, Fear and Loathing, SPORTS ILLUSTRATED, September 14, 2015, p. 32-39.
5 Dan Bernhardt & Steven Heston, Point Shaving in College Basketball: A Cautionary Tale for Forensic Economics, 48 ECONOMIC INQUIRY 14 (2010).
6 Eric Zitzewitz, Forensic Economics, 50 JOURNAL OF ECONOMIC LITERATURE 731 (2012).
7 In addition to being set forth in the NFL constitution and bylaws, NFL executive Ray Anderson sent a September 2006 memorandum to all league general managers and head coaches to remind the recipients of the NFL’s rule on the issue.
8 The total included 240 regular season games and 29 playoff games.
9 The Patriots’ regular season win-loss record was 175–65. The Patriots’ playoff win-loss record was 21–8.
10 Our analysis does not include the 2015-16 season. Likewise, our analysis does not include the Patriots’ six neutral site games (all Super Bowl games) between 2000-01 and 2014-15. As such, the resulting data set used in our analysis included 263 total Patriots games during the fifteen year time span (137 home games and 126 road games).
11 According to Bishop, Rosenberg, and Thayer in Sports Illustrated: “During the game one former Packers staffer says, the Patriots seemed to know Green Bay’s defensive calls from the outset. The Patriots won 35-0. ‘Whatever we called, they got us out of our base call every single play,’ the staffer says. ‘I’ve never seen anybody be able to do that before.”’
12 U.S. Senator Arlen Specter of Pennsylvania highlighted this aspect of the scandal in his June 5, 2008 statement on the floor of the U.S. Senate: “The [c]ommissioner’s stated reason for destroying the tapes lacks credibility. He said in his January 31, 2008 letter that ‘the tapes and the notes were destroyed by our office in order to ensure that they could not be used for any purpose going forward. Our goal was to ensure that the Patriots would not secure any possible competitive advantage as a result of the misconduct that had been identified.’ That objective could have been obtained by storing the tapes in a vault and they would have been preserved for future inspection if they need arose. The NFL would have avoided the inevitable smell of destroying evidence.”
13 In the course of doling out the Spygate-related punishment, Goodell wrote: “This episode represents a calculated and deliberate attempt to avoid longstanding rules designed to encourage fair play and promote honest competition on the playing field.” In a January 31, 2008 letter to Sen. Specter, Goodell wrote: “I am confident that neither the Patriots, nor any other NFL team, will engage in this conduct again.”
14 Indeed, Van Natta and Wickersham found that “[a]fter the [2007-08] season, Belichick would acknowledge that the Patriots taped a ‘significant number’ of games, and according to documents and sources, they recorded signals in at least 40 games during the Spygate era.”
15 As described by Bishop, Rosenberg, and Thayer in Sports Illustrated: “Certainly the advantage he stood to gain is significant. Trying to figure out signals with the naked eye is legal, and most teams try it. Doing it with video cameras is illegal because a team can rewind the tape and match signals with play calls. As a longtime NFL head coach tells SI, ‘If a good quarterback has that information, he can really use it. It’s way, way, way important.”’
16 Even dated examples of alleged impermissible taping by the Patriots are still discussed. Former St. Louis Rams quarterback Kurt Warner spoke of lingering doubts about a Super Bowl game versus the Patriots over a decade earlier in an April 13, 2013 CBSSports.com article by Mike Freeman with the headline: “Warner still wonders if destroyed Spygate tapes gave Pats an edge.”
17 Sen. Specter touched on this point during his investigation: “The NFL has not addressed the question as to whether the Patriots decoded signals during the game for later use in that game.” Sen. Specter cited May 14, 2008 statements made by former NFL player and current ESPN analyst Mark Schlereth to the New York Times in connection with this point: “Schlereth said that the breadth of information on the tapes— mainly, the coaches’ signals and the subsequent play— would be simple for someone to analyze during the game. There are enough plays in the first quarter, he said, to glean any team’s ‘staples,’ and a quick view of them could prove immediately helpful. ‘I don’t see them wasting time if they weren’t using it in that game,’ said Schlereth.” On November 24, 2015, ESPN’s Darren Rovell tweeted: “Pats have now won 80 straight home games when leading at halftime, an NFL record. Last game they lost in that situation came on 12/24/2000.”
18 In addition to head coach Bill Belichick, some mention has been made of another New England Patriots employee relevant to the discussion here— Ernie Adams. Wright Thompson, a senior writer for ESPN.com and ESPN The Magazine, profiled Adams in February 2008: “On game day, Adams wears a headset in the press box, a direct line to Belichick. Adams advises Belichick on which plays to challenge, and charts trends. ‘The one thing the Patriots do better than anyone else is they adjust and make halftime adjustments,’ [Hale] Sturges says. ‘Ernie Adams is the guy who does that.’ Are there other game-day duties? While it is commonly accepted that most teams try to steal signals, and New England was actually caught in the well-publicized Spygate incident, one former Patriots insider said a videotape of signals wouldn’t help the other 31 teams nearly as much because they wouldn’t have Ernie Adams there to quickly analyze and process the information.” Reporting by Van Natta and Wickersham for ESPN The Magazine in September 2015 revealed that “the Patriots told [NFL] officials they possessed eight tapes containing game footage, along with a half-inch thick stack of notes of signals and other scouting information belonging to Adams.”
19 This is not always the case. If a game is meaningless (e.g. New England has qualified for the playoffs and has home field advantage through the playoffs, or cannot change its playoff seed), one would not expect the team to risk discovery. The goal is to win Super Bowls, not accumulate regular season wins. For the purpose of this analysis though, all games were analyzed.
20 Sen. Specter cited May 13, 2008 statements made by former Miami Dolphins and Denver Broncos assistant coach Jim Bates to the Palm Beach Post on this point: “‘There’s only a certain number of plays that truly determine winning and losing,’ Bates said. ‘It might come down to five plays. Sometimes it’s just one play. A critical play at a critical time to move the sticks and get a first down, it definitely can change the outcome of a game.’ … ‘To know their personnel as soon as they do … it’s a tremendous advantage,’ Bates said.”
21 This approach raised the variance, and will overestimate the likelihood of an event occurring. If an exact p were made using the data, it would be even smaller.
22 Mayer v. Belichick, et al, 605 F.3d 223 (3d Cir. 2010).
23 Mike Fish, Specter: Goodell’s Spygate Explanations Don’t Pass Scrutiny, ESPN.COM, Feb. 15, 2008.
24 In his Senate floor statement, Sen. Specter highlighted Goodell’s comments on this point: “During his February 1, 2008 press conference, [c]commission (sic) Goodell stated, ‘I think it probably had a limited effect, if any effect, on the outcome of any game.’ Later, during that press conference, Goodell stated again, ‘I don’t believe it affected the outcome of any games.’ Commissioner Goodell’s effort to minimize the effect of the videotaping is categorically refuted by the persistent use of the sophisticated scheme which required a great deal of effort and produced remarkable results.”
