On the inherent bias in the 2011 and 2012 Super Rugby tournament format

  
 Fifteen regional rugby union teams compete for the Super Rugby title – five each from Australia, South Africa and New Zealand. In 2011, significant changes were made to the tournament format and these were continued in 2012. The new changes created an inherent bias, favouring teams that are strong relative to their conference, regardless of their overall strength. It is shown using logic and numerical results from simulations that teams that are weaker are able to unfairly progress to higher positions on the competition log at the end of the ‘regular’ season. The bias arises as a result of teams playing more matches against teams within their conference than other teams within the competition, and therefore the final log position represents a team’s strength relative to their own conference more than their overall strength. In the face of this evidence the tournament format should be altered to re-align with the primary goal of determining the best team in the competition. Should the tournament organisers choose not to change the tournament format, it would not be unreasonable for them to face criticism for failing to implement measures to ensure fairness to each team within the competition. 


COMMENTARY
In 2011, significant format changes were made to the Super Rugby, rugby union tournament.These changes created an inherent bias favouring teams within a particular country.This commentary shows the existence of the bias using logic and simulation and then makes recommendations for changes to the tournament format.
Fifteen regional teams compete for the Super Rugby title -five each from Australia, South Africa and New Zealand (representing the current number of teams and the distribution).From 1996 to 2010 the tournament consisted of a 'regular season' stage, with every team facing every other team once, and a 'finals' stage, consisting of a knock-out tournament between the top four teams.The round robin ensured that the final log position at the end of the regular season represented the average performance of a team relative to all the other teams, with no one team being unequally weighted in the determination of the final log position.
The regular season format first implemented in 2011 and continued in 2012, sees each team playing every other team from the same country (referred to as a conference) twice, and only four of the five teams from each of the other countries, resulting in 16 matches per team (see the official SANZAR website for a detailed description of the 2011/2012 tournament format: http://www.sanzarrugby.com). 1 The 2011/2012 format therefore ensures that each of the other teams within one's home conference is represented twice among one's 16 matches, but a team from another conference only has an 80% chance of being represented once.Each match is weighted evenly in the determination of log points and therefore the final log position is strongly weighted towards the relative ranking of a team within their own country.
Given the above, it is now possible that a team which is weak, but stronger than the other teams within their own conference, can top the log, being ranked higher than teams better than they are.The most extreme example of a tournament format displaying such bias would be to have teams pair off and play 16 matches against only one other team.In this example the final log points would only represent a team's ranking relative to their partner team, and not fairly represent their ranking relative to the other teams.Teams paired in competitive bouts would be disadvantaged, and teams paired off against teams that are relatively weaker, regardless of their own absolute strength, would be advantaged.
The bias within the 2011/2012 format can be demonstrated using simulation by assuming relative rankings for each team.If a weaker team can consistently finish the simulated log above a stronger team, then the tournament format can be said to create bias.The first step is to create three sets of five teams and assign objective rankings, 1 through 15, with the team ranked 1 being objectively the best.
For the simulation 'Conference 1' has teams with rankings 1, 4, 6, 8 and 10, 'Conference 2' has teams with rankings 2, 5, 7, 9 and 11, and 'Conference 3' has teams with rankings 3, 12, 13, 14 and 15.This arbitrary arrangement shows that Conference 3 is more polarised in terms of rankings than the other two conferences, which show identical relative ranking distributions.
Assuming no bonus points, which may exacerbate the situation, the simulation shows that the third ranked team from Conference 3 will top the log roughly 35% of the time, finish second 33% of the time, and only finish its deserved third position the remaining 32% of the time.In contrast to this result, a single round robin shows the third ranked team coming first, second and third roughly 8%, 20% and 72% of the time, respectively, a much more appropriate spread.A double round robin, with every team playing every team twice, once at home and once away, ranks every team correctly every time.
Each of these results is based on the outcomes of 10 000 simulated tournaments, and in each iteration which team had home advantage in each fixture was varied.In the case of the 2011/2012 format, which of the four teams from a foreign conference was played by a particular team was also varied.It was ensured throughout that every team played the same number of home and away matches.These chance-

MRC/UCT Research Unit for Exercise Science and Sports Medicine, Department of Human Biology, University of Cape Town David Karpul, MSc Electrical Engineering
Corresponding author: D Karpul (david.karpul@uct.ac.za).

Abstract
Fifteen regional rugby union teams compete for the Super Rugby title -five each from Australia, South Africa and New Zealand.In 2011, significant changes were made to the tournament format and these were continued in 2012.The new changes created an inherent bias, favouring teams that are strong relative to their conference, regardless of their overall strength.It is shown using logic and numerical results from simulations that teams that are weaker are able to unfairly progress to higher positions on the competition log at the end of the 'regular' season.The bias arises as a result of teams playing more matches against teams within their conference than other teams within the competition, and therefore the final log position represents a team's strength relative to their own conference more than their overall strength.In the face of this evidence the tournament format should be altered to re-align with the primary goal of determining the best team in the competition.Should the tournament organisers choose not to change the tournament format, it would not be unreasonable for them to face criticism for failing to implement measures to ensure fairness to each team within the competition.S Afr J SM 2012;24(2):67-68.

On the inherent bias in the 2011 and 2012 Super Rugby tournament format
David Karpul related changes in the format for each simulation account for the variability seen in the abovementioned results.
One can include several types of match outcome variability in the model, allowing weaker teams to beat stronger teams occasionally.As it is of interest what happens on average, most types of match variability are undesirable.Despite this, home advantage has been shown in the literature to play a role in match outcomes, 2,3 and therefore the model includes teams being able to beat another team ranked five places higher if playing at home.For example, a team ranked ninth can beat a team ranked fourth if playing at home.The effects of travelling to a foreign country have been shown not to affect performance and therefore the travel schedule of the tournament was not taken into account. 4his particular spread of rankings, and extent of home advantage, is arbitrary and only serves to demonstrate that relative ranking within a conference unfairly affects the overall log position at the end of the regular season.It is not reasonable to expect that each of the real world conferences would have identical relative rankings, and therefore bias is inherent in the current format.
In the face of this evidence, the tournament format should be altered to re-align with the primary goal of determining the best team in the competition.The bias can be resolved in one of two ways: the tournament can return to the original round robin format, or, the weighting towards intra-conference matches on the log can be remedied without changing the match format.This can be done either by weighting intra-conference matches half as much as interconference matches in terms of points or by using intra-conference matches to rank a team within a conference only and not relative to the other conference teams, essentially creating a group pool stage to the tournament.
While it has been shown that bias exists, it has not been shown to which team or teams the bias benefits.From the perspective of the tournament organisers the simple establishment of the existence of bias should suffice to recommend changes to the tournament structure.Should the organisers choose not to change the tournament format, it would not be unreasonable for them to face criticism for failing to implement measures to ensure fairness to each team within the tournament.