FIFA ranking: Evaluation and path forward

2.1Effect of weighting using game-importance: preliminary evaluation

In statistics, a conventional approach to performance evaluation is to rely on a metric, called a scoring function, which relates the result y_t to its prediction obtained from the estimates at hand (here, θ_t), (Gelman, Hwang, & Vehtari, 2014).

At this point we want to use only the elements that are clearly defined in the FIFA ranking and since the only explicit predictive element defined in the FIFA algorithm is the expected score (3), F(zt/s)=𝔼[yˇt|zt] , we will base the evaluation on the metric affected by the mean. Later we will abandon this simplistic approach.

Using the squared prediction error,

The MSE in (9) may be treated as an estimate of the expectation,

Therefore, by reducing the (absolute value of the) bias B (z_t, y_t), that is, by improving the calculation of the expected score F (z_t/s), should manifest itself in a lower value of the MSE, which is calculated as in (9).⁶

Using the MSE, we are now able to assess how the values of the importance parameters I_c (or alternatively, K and ξ_c) affect the expected value of the score (i.e., the estimate of the mean).

We find the coefficients K and/or ξ_c by minimizing the MSE (9) using the following alternate optimization which turned out to converge quickly (and be independent of the initialization)

The one-dimensional optimizations (11) and (12) only require one-dimensional line search (e.g., over a grid) and we preferred to avoid derivative-based methods which are not well suited to deal with the complicated functional relationship resulting from the recursive rating algorithm.

The results are shown in Table 2 and we observe the following:

Using a very simple MSE criterion derived from the definitions used by the FIFA algorithm, we obtain results that cast doubt on the optimality/utility of the game-importance parameters, I_c proposed by FIFA.

However, drawing conclusions at this point may be premature. For example, regarding K (which, after optimization should be much larger than 5), it is possible that the relatively short period of observation time (34 months) is not sufficient for small K to guarantee the sufficient convergence but may pay off in a long run, when smaller values of K will improve the performance after the convergence is reached. We will return to this issue in Section 5.1 when analyzing the interaction between the initialization of the algorithm and the scale s.

On the other hand, to address concerns about the weights ξ_c, the situation is quite different. Even after the convergence, the weights associated with different categories should affect the results in a meaningful way. To elucidate this point, we will take a more formal approach and, in Section 3, go back to the “drawing board” to derive the rating algorithm from the first principles.

Before that, however, we will evaluate the impact of the knockout/shootout rules.

2.2Effect of knockout/shootout rules

The basic algorithmic Equations (1)–(2) guarantee that the teams “exchange” the rating points so that their total stays constant, i.e., ∑m=1Mθt,m=∑m=1Mθt+1,m : this is a well-known property of the Elo rating algorithm, (Langville & Meyer, 2012, Ch. 5). On the other hand, by applying the knockout/shootout rules, we always obtain a new δ_t,m that is equal to or greater than the original one in (2). Thus, the shootout/knockout rules are a source of “inflation” in the rating. In fact, in the analyzed period, the total number of points increased by 2099 (with the initial total being 254680) and this was due to 24 games with the shootout rule (but not the knockout), 90 games with the knockout rule (but not the shootout), and 30 games where both rules were applied.

In the absence of known mathematical principles from which the knockout/shootout rules are derived, our initial hypothesis is that the knockout rule is a heuristics introduced to compensate for the increased value of I_c in the advanced stages of competitions. To test this hypothesis, we will proceed by removing the shootout or/and knockout rules from the algorithm and observe the impact of such removals on the MSE.

The results are shown in the last three columns of Table 2 and we conclude that the prediction capacity of the algorithm is negligibly affected by the shootout rule and it slightly but still notably deteriorates if the knockout rule is removed.⁸Thus, our hypothesis is not supported by the results, even if the argument in favor of using the knockout rule is very weak, as we will also see later in Section 5.2.

To obtain an intuitive understanding of how the removal of shootout/knockout rules affects the results, Table 3 compares the ranking obtained using the FIFA algorithm (first column) with the rating resulting from the modified algorithm in which we (i) eliminate the shootout rule (second column), (ii) eliminate the knockout rule (third column), as well as (iii) eliminate both rules (fourth column).

The Spearman correlation coefficient, ρ (Myers & Well, 2003, Ch. 18.5.3), which quantifies the difference in the rankings of all teams (perfect agreement yields ρ = 1.0) indicates that the changes in the ranking are similar to what was obtained by observing the MSE: comparing the shootout and the knockout rules, the latter affect the results more significantly.

We also show the rankings of the top teams where the changes are not major and the most notable is the switch of ranks between Belgium (BEL) and France (FRA), which can be attributed to a different number of times the teams benefitted from the knockout rules. Indeed, by analyzing the results of the games, we observed that in the original ranking, BEL benefited four times from the knockout rule for a total of 85 points (which would be lost without the rule (6)), while FRA benefited only once, gaining 14 points.⁹

Although the knockout rule provides a slight but notable improvement from the prediction point of view, we may still debate whether this heuristics is fair and desirable.

In particular, we note that the points-preserving knockout rule partially ignores the direct comparison between the teams. For example, games in which BEL was not penalized (for losing in the knockout stages) were played against FRA (twice). Thus, despite direct evidence indicating that FRA was able to beat BEL, the knockout rule preserved the points earned by BEL in other games.

In fact, such situations are not surprising and, indeed, the top teams are likely to make it to the final stages of the important competitions and then play against each other in the games where knockout rules are applicable (in case of BEL’s games: World Cup 2018, Euro 2020, and UEFA Nations League 2021). Although these games will provide direct comparison results, the current knockout rule will preserve the points of the losing team.

3.1Davidson model

The rating now depends on the choice of the likelihood function L (z ; y) and here we opt for the Davidson model, (Davidson, 1970), being a particular case of the multinomial model used also in Egidi and Torelli (2021).

Using (21)–(23) in (19), with straightforward algebra we obtain (see Appendix A and (Szczecinski & Djebbi, 2020, Sec. 3.1))

Therefore, the SG algorithm (20) becomes

It is easy to see that for η = 0 and κ = 0 (i.e., when L (z ; D) =0 and the draws are ignored) we have F₀ (z) = F (z) which is simply a logistic function as in the Elo algorithm. Furthermore, for η = 0 and κ = 2 we obtain F₂ (z) = F (z/2) and thus (27) is again equivalent to the Elo rating algorithm, but with the doubled scale value. These observations explain our preference for the model: it leads to a simple algorithm which generalizes the Elo/FIFA rating algorithm, (Szczecinski & Djebbi, 2020).

Although we conclude that the FIFA rating algorithm may be seen as the instance of the maximum weighted likelihood estimation, this is, of course, a “reverse-engineered” hypothesis because the FIFA document, (FIFA, 2018), does not mention any remotely similar concept.

3.2Regularized batch rating

While our goal is to obtain the on-line rating algorithms where the skills at time t + 1 are calculated from the observations up to time t, we will, for a moment, ignore this on-line rating aspect and rather focus on the evaluation of the model and the optimization criterion that underlie the algorithms.

We thus concentrate on the original problem defined in (16) for the entire set of data, and, in this way, we (i) will not need to remove a significant portion of the data (meant to eliminate the initialization effects during evaluation, see (9)) and (ii) eliminate the limitation of the SG optimization where, by fine-tuning the adaptation step K, the estimation error is traded off against the convergence speed.

We start by noting that the problem (16) is, in general, ill-posed: since the solution depends only on the differences between the skills, z_t, all solutions θˆ and θˆ+θo1 are equivalent because the differences z_t are independent from “origin” value θ_o. To remove this ambiguity, we may regularize the problem as

Under the model (21)–(23), the regularized batch-optimization problem (28) is useful to resolve another difficulty. Namely, if there is a team m having registered only wins, i.e., when ∀i_t = m,  y_t = H and ∀j_t = m,  y_t = A, then (16) cannot be solved (or rather, θˆm→∞ ) because J (θ) does not limit the value of θ_m. Such a solution not only is unattainable numerically but is, in fact, meaningless, and the regularization (28) settles this issue.¹² footnote:ranking.info.

The estimated skills θˆ now depend on the weights ξ_c, on the regularization parameter α, and on the model parameters η and κ. If not known, all of these parameters must be optimized.

Regarding the optimization criterion, we recall that the FIFA algorithm only specified the expected score, so the quadratic error (8) allowed us to evaluate the algorithm and stay within the boundaries of its definitions. Now, however, with the explicit skills-outcome model, we may go beyond this limitation and will use the prediction metrics known in machine learning such as the (negated) log-score, (Gelman et al., 2014)

Furthermore, thanks to the batch-rating, we are able to consider the entire data set in the performance evaluation by averaging the scoring functions (30) or (31) over all games

In simple words, for given parameters (α, κ, η, ξ_c), we find the skills θˆ∖t from all, but the t-th game [this is (34)–(35)], and then use them to predict the results y_t; we repeat it for all t∈T , summing the scores obtained. This is the well-known leave-one-out (LOO) cross-validation strategy (Hastie et al., 2009, Sec. 2.9), (Duda et al., 2001, Ch. 9.6.2): no data is discarded when calculating metrics (32)–(33) and this comes with the price of having to find θˆ∖t for all t∈T . To reduce the computational load, we opt here for the approximate leave-one-out (ALO) cross-validation (Rad & Maleki, 2020) based on the local quadratic approximation of the optimization function defined for all data. Details are given in Appendix B.

Although both the average log-score (32) and the accuracy (33) can now be optimized with respect to α, κ, η, and/or ξ_c, we only optimize the log-score whose optimal value is denoted as LS_opt; the resulting accuracy, ACC will also be shown.¹³. It is, of course, possible to optimize the log-score with respect to any subset of parameters while keeping the rest constant;

we will do to this to determine how useful it is to optimize only some parameters.

Again, we use alternate minimization similar to the one shown in (11)–(12): LS was minimized with respect to one parameter at a time: α, κ, η, or ξ_c, until no improvement was observed. This simple strategy led to the minimum LS_opt which turned out to be independent of various starting points we used and, although we cannot prove the solution to be global, in all our observations the log-score functions seemed to be unimodal.

Optimized α, κ, η, ξ_c are shown in Table 4 and indicate that

Here, it is interesting to compare the parameters we found by optimization with the simplified formulas proposed in Szczecinski and Djebbi (2020, [Sec. 3.2])

The parameter η^hfa predicted by (36) is practically equal to the one obtained by optimization. And while the parameters κ^hfa and κ^neut. are slightly different from the one predicted by (37), using them in the rating, we obtained LS_opt = 0.864 (see last line in Table 4), which is still notably better than using the conventional FIFA rating. This is interesting because finding the parameters η and κ from the frequencies of the games not only avoids optimization, but also provides a simple-to-verify origin of the values used in the algorithm.

4.1MOV via weighting

For context, in Table 5 we show the number of games depending on the value of the MOV variable d. While, in principle, it is possible to use directly d, it is customary to consider their absolute value, |d|.

The Elo/FIFA algorithms (27) can be easily modified as follows, to take into account the MOV variable:

Integrating the MOV-weight into the online rating defined in (27) (by replacing, therein Kξ_ct with Kξ_ctζ_vt) yields the Davidson-MOV algorithm.

For example, (eloratings.net, 2020) uses

To elucidate how useful such heuristics are, we note that the problem is very similar to the importance weighting we analyzed before; the difference lies in the fact that the weighting now depends on the product ξ_cζ_d. Therefore, we may reuse our optimization strategy to find the optimal weights for games with different values of |d|.

To this end, we discretize |d| into V + 1 MOV-categories, v = 0, …, V and we use a very simple mapping v = |d| for v < V and v = V ⇔ |d| ≥ V. For example, with V = 2, ζ₀ weights the draws (|d|=0), ζ₁ weights the games with one goal difference (|d|=1) and ζ₂ weights the games with more than one goal difference (|d|≥2).

Breaking with the predefined functional relationship as shown in (41) we are more general than the latter, e.g., treating the cases |d|=0 and |d|=1 separately. This makes sense since these events are not only the most frequent ones (covering, respectively, 23% and 36% of the total; see Table 5), but also correspond to the events of draw and win/loss treated differently by the algorithm.

On the other hand, we are also less general due to the merging of events |d| ≥ V, although this effect will decrease with V, simply because there will be very few observations, as may be understood from Table 5. For example, with V = 4, the weighting ζ₄ will be the same for the events with |d|=4 and |d|>4 but the latter make only about 5% of the total.

We again consider the game categories defined in Table 1 and thus we now solve the following problem:

Parameters ξ_c, ζ_v, η, κ, and α will be optimized again using the ALO approach we described in Section 3.2, that is, by minimizing the log-score criterion (32) using an alternate optimization similar to that defined in (11)–(12). The results shown in Table 6 allow us to conclude that:

4.2MOV via modeling

The MOV modelling consists in defining a formal relationship between the skills θ_t and the observed MOV variable d_t, where a simple approach proposed in Karlis and Ntzoufras (2008) relies on a modeling of the goal difference using the Skellam distribution

The model (43) is a particular case of a more general form shown in Karlis and Ntzoufras (2008), which models the offensive and the defensive skills. Here, however, we are interested in rating and thus one skill per team should be used. As noted in Ley et al. (2019) and in Lasek and Gagolewski (2020) this offers a sufficient prediction capacity avoiding the problem of overparameterization due to the doubled number of skills.

Using (45) in (43), the following log-likelihood is obtained:

The derivative of (46) is given by

The batch rating then consists in solving the following problem:

To calculate the log-score, we have to calculate the probabilities m^ls (z_t ; A) = - log Pr{d_t < 0} (away-win) and m^ls (z_t ; H) = Pr{d_t > 0} (home-win). Since the closed-form formulas do not exist, we use truncated sums

The results shown in Table 7 indicate that, with this very simple approach (only two parameters of the model which must be optimized), we are able to improve over the MOV-weighting strategy and this should be attributed to the use of a formal skills-outcome model. The price to pay for the improvement lies in abandoning the legacy of the Elo algorithm.

Moreover, possible implementation issues may arise since the expected score (49) is theoretically unbounded. Thus, whether the improvement of the log-score from LS = 0.849 (in the MOV-weighting, see Table 6) to LS = 0.843 in the Skellam MOV model is worth the change and the implementation risks is at least debatable.

5.1Scale adjustment

The scale is obviously irrelevant in batch optimization, and the on-line update can also be written in a scale-invariant manner by dividing (20) by s:

However, in the FIFA ranking, a non-zero initialization θ₀ was determined in advance (see footnote 3) so θ0′ is not scale-invariant. Thus, given the initialization at hand, the question is how to determine the scale. We do not know any clear answer, but an insight into finding the useful scale value may be gained assuming that the initialization corresponds to the “optimal” solution, e.g., θˆ obtained in batch optimization with a given scale s₀.

It is easy to see that using s > s₀ will force the algorithm to change significantly θ_t (attainable with large values of the adaptation step, K); the same will happen for s < s₀ because the optimal estimates θ_t will have to be scaled down.

Since scaling the skills up/down changes their empirical moments, we suggest choosing the scale s in a moment-preserving manner. To this end, we define the empirical standard deviation of the skills

In fact, the initialization used by FIFA yields σ₀ = 220 and, after running the original FIFA algorithm, we obtain σ_T = 252.

Changing the scale s, we will obtain different σ_T so the idea is to run the algorithms for different values of the scale s (e.g., for multiples of 50) and to choose the one that produces a standard deviation σ_T ≈ σ₀. In practice we might do it using historical data before the new rating is deployed.

In this manner we found s = 150 to be suitable for the Davidson algorithm: we obtained σ_T ≈ 220 when κ = 2 and σ_T ≈ 210 for κ = 1.

This indicates that the scale s = 600 was too large for the FIFA rating. This can be seen by comparing the result of the FIFA rating with ξ_c = 1 (in Table 8a) to the results of the Davidson algorithm (with η = 0 and κ = 2). Both algorithms are essentially the same (although FIFA uses the shootout/knockout rules, which have rather small impact on performance) and the main difference resides in the scale. Since the Davidson algorithm (η = 0, κ = 2) with the scale s = 150 is equivalent to the FIFA algorithm with the scale s = 300, this latter scale value would ensure a better performance of the FIFA rating. However, this effect appears only due to the limited observation time we have at our disposal and will vanish after a sufficiently large number of games.

Similarly, for the Davidson-MOV algorithm, using s = 200, and for different values of V we obtained σ_T ≈ 220, while using the scale s = 300 in the Skellam algorithms yields σ_T = 225.

5.2Evaluation of the algorithms

To evaluate the SG algorithms, we used the same methodology we applied to make a preliminary evaluation of the FIFA rating in Section 2.1. That is, we used the first (approximate) half of the observation period for initialization and the second half is used to calculate the performance metrics. The difference from Section 2.1 is that now we use the log-score and the accuracy metrics

We consider the original and modified FIFA algorithm (Table 8a), the Davidson algorithm (Table 8b), the Davidson-MOV algorithm (Table 8c), and the Skellam algorithm (Table 8d).

In all cases, but in the original FIFA algorithm, we ignore the game-category weighting (i.e., we use ξ_c ≡ 1) because, as we have already shown, its effect is negligible. This is clearly shown in the first row of Table 8a where we see that, using the FIFA weighting, we obtain worse results than when the weighting is ignored. This is essentially the same result as the one we have shown in Table 2 but we repeat it here to show the log-score metric which we could not calculate without first introducing the Davidson model underlying the FIFA algorithm.

In Table 8a we also show the log-score LS_∖so&ko obtained applying the FIFA algorithm but removing the shootout and knockout rules. We observe that the knockout rule improves the prediction in terms of the log-score which confirms our preliminary evaluation in Section 2.2 based on the MSE. Nevertheless, the improvement is minor, and a much more important decrease of the log-score may be obtained by changing the model as indicated by the remaining results. In particular

We present in Table 9 the rating obtained for the top teams through new rating algorithms. Of course, due to the different scales that we used, the skills obtained with different algorithms cannot be compared directly.

We emphasize that the quality of these rankings (i.e., ordered skills) cannot be assessed because there is no reference order of the teams to which the shown rankings can be compared. The only tool we have to assess their validity is to calculate the performance criteria as we did in Table 8 using the log-score.

Noting that even rather mild differences between the Davidson and the Davidson-MOV algorithms alter the final order/ranking, Table 9 should be treated as a cautionary illustration that the ranking/order of the team can be very easily changed by relatively benign modifications of the rating algorithm. With that caveat, the different algorithms based on different models consistently put the same group at the top of the list. In fact, the algorithms are rather consensual regarding the current (as of March 31, 2002) official top team, BRA which, or remains on the top of the list, or has skills within fraction of percentage of top team’s rating. On the other hand, BEL’s second position in the official ranking (see Table 3) is much more questionable. While the second spot is preserved with the optimized FIFA-like algorithm (the first column of Table 9 where we use K = 55 and ξ_c ≡ 1, but the knockout and shootout rules are kept), the new algorithms consistently demote BEL to the fifth and lower position.

6.1Recommendations

Given the analysis and the observations we made, if the FIFA rating is to be changed, the following steps are recommended:

6.2Further work

The analysis we carried out in this work should not be considered exhaustive by any means and was meant (i) to provide an understanding of the current FIFA rating and (ii) to propose the simplest and yet meaningful modifications of the current algorithm.

Our recommendations regarding further work on the improvement of the rating are the following: