CFC Rating statistics for the period 2006-2010 for Victoria, Vancouver area, and Ottawa

This document presents some statistics from the CFC rating database with the aim of exploring the issue of what effect, if any, several changes to the CFC rating system have made. Specifically, in 2007, some rating points were given to individuals based on prior activity and also, subsequently, a participation and a bonus point system were installed in order to counteract alleged deflation. Quite incidentally, some other interesting observations from the data are also possible.

The motivation for undertaking this study was the observation that a) a participation point system will systematically inflate active regions compared to less active regions, b) an observation by me that people I considered to be my peers in Ottawa seemed to be going up in rating as part of a general inflation but no such trend seemed apparent in BC c) no assessment of the impact of participation points has been done or looks to be forthcoming.

The study considers the rating change for individuals who were active recently from their ratings 4 years ago in 2006. Three regions were chosen for review, Victoria (as I live there), Ottawa (as a region which has a large number of tournaments each year) and the Vancouver area (as the Victoria sample is quite small). As it is desired to try and isolate rating level changes as opposed to playing strength changes, juniors were excluded.

Summary of Results:

The median rating change for active, adult players from 2006 to 2010 varies between the regions studied. The numbers by region are:

· Victoria: a median rating change of +2 points

· Vancouver and area: a median rating change of +52 points (perhaps lower, depending on the data used)

· Ottawa: a median rating change of +102 points

These numbers appear to be statistically significant and show that different rating pools are diverging under the current rating system. This is clearly a matter of concern. Although this study was motivated by the hypothesis that participation points would drive regional differences there are other possibilities including a) a correlation of learning with activity – more active regions could legitimately raise average and median skill levels b) some kind of dependence on the vastly different levels of activity in the CFC rating of juniors in these regions c) there could be systemic bias in the selection of the data (e.g. not as good identification of juniors in one sample or other selection errors).

I would not say that this study conclusively shows a systemic regional rating bias but rather that it highlights some probability of a systemic regional bias. An improvement to this study would be to repeat over a larger sample and if birthdates were available, to use those as a criteria for exclusion. It might be advisable to exclude young adults as well as juniors.

Other observations arising from the data are:

· Something like 80%-90% of CFC life members are not active. Although some are involved instead in organization, I suspect most have simply disengaged from the CFC or are perhaps even dead. I doubt that the CFC has managed to reliably track the status of CFC life members. Consequently, the CFC membership level (see http://www.chess.ca/MemStats.shtml ), if one is looking to that number as an indication of activity level and/or popularity or as a comparison to historical levels, overstates itself by about 300 people. In addition, governors are awarded on per membership level and considering these life members are largely inactive, perhaps they should be excluded from that calculation.

· There are hugely different numbers of junior events between regions. For BC, there were 746 people who are listed as either being recently active or current members compared to the official number of ~250 CFC members in BC. The difference is largely due to junior events which are CFC rated but do not require CFC membership. In Ottawa, by contrast, there are essentially zero CFC events of this type (presumably, they are CMA events instead). This results in two further remarks:

o If the BC pattern is true for other regions, it is probable that the bulk of the rating data entry labour is for junior events which bring in no CFC membership dues and only $0.50 per player as rating fee.

o Although the contact between the adult rating pool and this junior activity is minimal, it is possible that this has an effect on local adult rating levels and hence results in systemic differences between regions where these junior events are CFC rated and regions where they are not.

Detailed Procedure and Methodology:

The links on the CFC site at http://www.chess.ca/ratings.shtml give files for each province listing current members (including life members) and those who have recently played using tournament memberships. From these files, a list of players for each of the cities Victoria, Ottawa (Ottawa only, does not include Orleans, Kanata, etc.), and Vancouver (including North Vancouver, West Vancouver, Surrey and Burnaby) was compiled. In order to make the size of the file manageable, for Vancouver, all those with ratings <1000 were dropped (essentially all juniors). For each name, the current rating was compared to the 2006 rating. If the player was not active in both the last year and in 2006, that name was dropped as were those whose rating was <1000 in 2006. Further analysis was done by marking those believed to be juniors, ascertained by inspecting their tournament record for participation in junior tournaments [as birthdates are not available]. The motivation for removing juniors is to try and remove as much as possible rating changes due to changes in strength. It is possible that some people were not properly identified as juniors in this treatment.

Detailed Results:

a) Victoria

Sample size before cull of players not active in both time periods: 39 (of which 6 are life members)

Sample size after cull: 11 (of which 0 are life members)

Sample size after removing known juniors: 7

	all players after cull	no juniors
mean rating change	36	1
st dev of rating change	118	57
st dev of estimated mean	37	23
median rating change	37	2

A histogram of the rating differences is given below as number of people in each bin versus bin. The bin size is 50 (e.g. the bin 25 contains all those with rating difference from 0 to 50)

For adult players, the null hypothesis of zero rating change would seem to be confirmed. Clearly, the juniors involved have mostly improved although interestingly enough, the biggest loser is also a junior.

b) Ottawa

Sample size before cull of players not active in both time periods: 127 (of which 28 are life members)

Sample size after cull: 39 (of which 3 are life members)

Sample size after removing known juniors: 31

	all players after cull	no juniors
mean rating change	191	109
st dev of rating change	230	120
st dev of estimated mean	37	19
median rating change	143	102

Histogram of people vs. rating change:

The sample with juniors in it is strongly skewed, presumably due to the juniors improving. Removing the juniors obtains a distribution that is more plausibly bell shaped although it still has some skew. Whether one used the mean or the median as a measure of central tendency, it seems that ratings in Ottawa have gone up by ~100 points (both the median and the mean) and this is clearly statistically significant (about 5 standard deviations).

The results for Vancouver will show strong skew which is attributed to the performance improvement of a few individuals. Looking at the data here, the skew is not so evident but surely those in the rating change bin 350-400 have shown some actual skill improvement. Removing those two from the sample does not change the conclusion of ~100 rating point shift statistically different from 0 however (in this case average =91, median =93, st dev of mean =19).

c) Vancouver (including North Vancouver, West Vancouver, Burnaby, Surrey)

Sample size before cull of players not active in both time periods: 139 (of which 23 are life members)

Sample size after cull: 53 (of which 5 are life members)

Sample size after removing known juniors: 25

	all players after cull	no juniors
mean rating change	212	91
st dev of rating change	253	138
st dev of estimated mean	35	28
median rating change	143	52

Again, the data with juniors is strongly skewed (and interestingly, both largest gains and largest losses are juniors). Even the data with no juniors has a skew (those adults with rating gains >200). Knowing the names of those people, I would characterize most of them as young adults and that this rating change probably incorporates some level of actual improvement.

Given the skew in the adult data, the median at 52 is probably the best measure of central tendency and is not quite statistically significant (1.86 standard deviations). If the 5 people responsible for the skew are removed from the sample, the mean and median go down even further (in this case average =31, median =22, st dev. of mean =15).

Discussion:

The data suggests that there has been a systemic differential rating increase between the regions studied from 2006 to 2010. Those conclusions are not definitive as my choice of data set and methodology (using the median) can be attacked. In addition, the Victoria data set is really too small to put too much faith in that set of results. In addition, it is clearly difficult to separate out drift in the ratings from actual improvements in skill.

Assuming that there is indeed a differential increase between the regions, explanations for this could be:

· my original hypothesis (participation points selectively inflates rating pools with greater activity)

· some interaction between the large junior CFC rating pool in BC and the adult rating pool which does not exist in Ottawa.

· Some actual differential change in skill (as caused perhaps by more activity)

· some bias in selecting the data set and in identifying and removing juniors from that data set.

That the biggest rating point losers are also juniors is surprising and perhaps warrants further investigation.