The biggest off-season story in the NRL was the transfers of Cooper Cronk from Melbourne to Sydney and then Mitchell Pearce from Sydney to Newcastle. From the Roosters’ perspective, for two players likely on similar pay packets, how did the Roosters decide one was better than the other? Then I wondered if it were possible to work out a way of judging value for money in player trades. It’s big in baseball, so why not rugby league? This led me to develop StatScore and Win Shares as ways to numerically evaluate rugby league players.
Approximate Value
Hurling People Now developed an approximate value system for rating AFL players last year which looked really interesting. It was derived from the similarly named system from Pro Football Reference, which evaluates NFL players. These systems use player statistics to generate a number that represents the player’s value. Approximately.
At the time, I had no idea what to make of it or how I could derive a similar system for rugby league. I let it gestate for around nine months and then it sort of clicked into place.
I call it StatScore because I am not good at names.
StatScore
StatScore is a big dumb number that evaluates individual player performances across a season. Specifically, it looks at their contributions to statistics that are correlated with winning games. The bigger the contribution, the better the rating and the more likely the team with that player is to win games.
We begin by assigning a player their role. If they typically play in a jersey with one, two, three, four or five on it, they’re considered a back. Anyone in six, seven or nine is a spine player. The rest of the team – eight, ten, eleven, twelve, thirteen – are considered forwards.
The NRL had (see “Development” below) a comprehensive list player statistics on their website. Of those, eleven correlate to wins. That is, the more of them you rack up, the more likely your team is to win games. Those eleven stats are: tries (T), all runs (AR), all running metres (ARM), line breaks (LB), try assists (TA), tackles (TK), tackle breaks (TB), kicks (K), line break assists (LBA), kick metres (KM) and kick return metres (KR).
Some of these, like ARM, correlate better than others, like TK. Forwards tend to have more tackles and running metres, backs more line breaks and tries and spine players more line break assists and kick metres. It’s important to classify players so we can get a proper sense of their contribution to their team’s victories.
I plotted each of the stats at a team level (x-axis) against the number of wins for each team (y-axis) and put a trendline through it. The graph above shows tries scored by a team versus the team wins that season for seasons 2014 through 2017. The gradient on the trendline (underlined in blue) is multiplied by the individual player’s statistic value and then weighted proportionally to the trendline’s correlation (highlighted in yellow), i.e. better correlation, higher weighting (remember, 1 is perfectly correlated and 0 is not correlated at all).
For example, Mitchell Pearce scored nine tries in 2014. By the graph, this is worth 1.647. We repeat for all eleven stats, weighting them according to their correlation and sum up for the player’s total production.
The total production for each player is divided by the average production for that role in that season and then multiplied by 100. In effect, rather than being an approximate value system, StatScore is more about value over average.
Crazily enough, the system works. When comparing a team’s total StatScore to their wins for the season, we get a 0.73 correlation. No single player statistic has a correlation this high. The closest is ARM which has a correlation of 0.64. We also get similar correlations between total team StatScore and Pythagorean expectation.
Trivia
- Seven or eight players on each team are responsible for over half the team’s total StatScore (let’s call them the “engine”), meaning the other twenty guys on the roster are basically ballast.
- Roger Tuivasa-Scheck has the highest single season StatScore of 433 in 2015. He and Lachlan Coote that year are well ahead ahead of the next best players.
- Ben Hunt has the highest career StatScore. Remember that this only covers the last four seasons. If Thurston hadn’t spent most of the 2017 season on the side lines and had played at his usual level, he would be the career leader. If StatScore extended back twenty years, Thurston would still probably be at the top.
- In four seasons, the premiership winning team has always had the highest StatScore, mostly because they play the most games and tend to win a substantial portion. The lowest premiership-winning team StatScore was 3278 (2016 Sharks) and the lowest spine StatScore was 579 (also 2016 Sharks). The 2017 Storm have the highest team total at 3425 and the 2015 Cowboys the best spine at 653.
- Unsurprisingly, the 2016 Knights have the lowest team StatScore of 2183.
Limitations
Any player evaluation system will have issues because nothing is perfect. Firstly, StatScore is not a precise measure. It is a big, dumb number. A player rated 350 and another rated 310 are basically at the same level but a 300 player is better than a 200 who is better than a 100 player. Some common sense is required.
StatScore loves kicking halves, fullbacks and star prop forwards. Typically, these are the players we see with the top StatScores at the end of each season. A reason for this is the nature of some statistics. ARM is relatively evenly distributed across the league as demonstrated by its comparatively low Gini coefficient (0 is perfectly equally distributed, 100 is perfectly unequally distributed). ARM’s coefficient is typically in the low to mid 40s. Other statistics, like LBA, are unevenly distributed with a coefficient in the mid 70s, meaning a few players hoard the statistics and the overwhelming majority don’t register one at all. When the stats are distributed unevenly, the outliers are more extreme and find it easier to rack up rating points when compared to the average.
The best example of this are the kicking stats (K, KM, KR). Kicking stats have Gini coefficients well into the 80s and kick return is usually in the mid 90s. Typically, a team will only have one or two kicking players who are able to pick up rating points in this area, meaning that the rest of the league is at a handicap. This is one reason why StatScore is angled more to value over average than a standard approximate value apparoach. There’s an argument that kicking stats should be excluded but they correlate well with winning and we shouldn’t ignore the contribution made by this skill.
We don’t get a good measure of defensive contribution through StatScore. Tries conceded correlates excellently with winning percentages but it’s hard to assign a try conceded, or even a part of, to an individual player. Conversely, missed tackles and errors don’t correlate well with winning at all and tackles only at a low level.
The other main issue should be obvious: StatScore doesn’t love Cam Smith. The reason is that we only work with a limited range of statistics and there’s no way to easily capture things like defensive organisation, leadership and diplomacy with referees. In this sense, Smith’s best qualities are not captured in his own stats but they likely make his teammates’ stats a lot higher than they otherwise would be. All hookers suffer from this on StatScore but Smith is the most noticeable: his career ranking is 47th while former teammate Cronk is 2nd. This is one reason to have Win Shares to go with StatScore – making a smaller StatScore contribution on a team that wins lots of games compensates for the shortcomings of the statistics themselves.
Win Shares
Bill James, one of the earliest pioneers in the sport’s analytics world, put out a book called Win Shares in 2002. He used extremely complicated mathematics to assign shares of a baseball team’s wins to individual players. In contrast to the perhaps better known wins above replacement (WAR), James’ Win Shares was a process of accounting. Players can’t be assigned more wins than the team had actually won.
I prefer to take a much more simplistic approach. A player’s StatScore is divided by the team’s total to work out their contribution. Top players will be around the 8% to 10% mark, higher if the player in question is a star on a poor team. We then multiply that contribution by the number of wins the team actually accrued during the regular season for Actual Win Share (AWS) or by the number of wins predicted by Pythagorean expectations for Pythagoras Win Share (PWS). I’ll leave it up to you to decide which is better but I will typically quote AWS, as I did in the sidebar, if not otherwise specified.
Top players can have win shares in the ballpark 1.5 to 2 wins per season. Ben Hunt has averaged 1.61 over the last four seasons. Over time, that becomes an impressive contribution to a twenty-four game season.
Further Developments
Development is somewhat stymied at the moment. The NRL website used to publish everyone’s stats. Now with the new ‘digital’ website, you can only access the top five players on each stat. I’m hoping this is resolved before we get too far into the season, otherwise StatScore is going nowhere. Alternatively, if someone has access to the data, I will pay a (small) subscription fee if necessary.
My plan was to develop ‘ProScore’, which would approach a WAR type metric. The applications for that would be massive. It would be easy to calculate a value for money approach to player recruitment (is Cronk or Pearce better value for money, which is what we wanted to know in the first place) or charting the expected path of a player’s career (see CARMELO).
Actually, scratch paying a subscription fee. Someone should be paying me to develop this further. Haven’t you seen Moneyball?
For now though, its one step at a time.
Pythago NRL 