Ultimate - Hat Tournaments - Why do we suck at rating ourselves?

Background:
For a while now, I've spent some (very small amount of) my time helping to make teams for hat tournaments in Israel. Lately it was for the latest Ligat Cova (Israel's monthly hat tournament, put on by an awesome bunch of volunteers), and the annual Succot hat tourmanent, but I've done it for several ligot cova in the past.


The first couple of times I did it (I think the first was 2009's Holylanders Hat), I (or we - in that case, Tomer and I) did it manually. Take all the players we have, derive from their self-rating and what we know about them a rating we're more likely to trust, and started throwing people into teams. Several hours later (without any exaggeration, and anyone else who's done it before can attest) we were finished, with what was largely a 'do-it-by-feel' process. Being somewhat of a nerd, I'm not a huge fan of making shit up as I go - I'd like to work in a somewhat organized fashion, with a process that's at least somewhat mathematically justified.


This is going to be a two-part post - in this first part, I'll tackle the issue of rating players - and why the ratings we receive from players are, in my mind, essentially useless. In the next part, I'll write a bit about the work I (with help from several guys in the community) have been doing to make this process smarter, easier, and less time-consuming.


Claim:
The data we receive from the players about themselves (we ask them to rate their experience, disc skills, and speed) correlate very weakly to our ratings of them, making their ratings effectively useless at helping us build balanced teams. Also, I have graphs on my side, so I must by right.

Data:

I worked with the data from the latest ligat cova - 51 players who signed up. A few more signed up, but obviously disregarded the ranking area in the signup sheet (veteran players ranking themselves as 1's (out of 5) across the board). More data is available - the succot tournament, and many past ligot cova - but I'm not sure it's necessary.

Method:

I plotted our (tournament organizers, the closest we believe have to a true rating- since it's all subjective in the end) rating scores against the players' self-rated numbers. The first shot I took is plotting against the sum of all three self-rated parameters (again, we're talking about experience, disc skills, and speed), and this came out:

Even without the numbers, it's easy to see the plot is all over the place. The numbers only help support the claim. If you've never taken statistics before, try the Wikipedia article on the coefficient of determination (R^2). Math majors, feel free to correct me: in simple terms, R^2 tells us how much variation in the Y (dependent) variable can be explained by the X (independent) variable. In our case, R^2 = 0.2366 means that only 23.66% of the variation in the Y variable (our ratings) can be explained by variation in the X variable (the players' ratings). It also means our correlation coefficient, R, is the square root of that - R = 0.4864. If you want to show correlation, you're usually looking for 0.9 or more. Essentially, there is no strong correlation between the players' rating of themselves and our ratings of them. Also in Hebrew:

אם לא למדתם סטטיסטיקה, וזה נשמע כמו סינית (ואם כן, ואתם רוצים להסביר את זה יותר טוב, אני אשמח): R^2 עוזר להבין כמה מהשינויים במשתנה שאנחנו היינו רוצים לחזות (במקרה הזה, את הדירוג שלנו) אפשר להסביר דרך המשתנה שהיינו רוצים שיחזה (במקרה הזה, הדירוג העצמי של השחקנים) הערך שקיבלנו אומר שרק 23.66% מהשינוי בדירוג שלנו אפשר להסביר על ידיד הדירוג של המשתמשים. זה גם נותן לנו שהקורלציה היא השורש של המספר הזה - או R = 0.4864. אם מנסים להוכיח משהו סטטיסטית, בדרך כלל מחפשים קורלציה של 0.9 ומעלה. יש קורלציה מאוד חלשה בין הדירוג העצמי של השחקנים לדירוג שלנו שלהם.

I then tried another method. We classify the players (in our ratings) to Cutters (whose speed is more important), Handlers (whose disc skills are more critical), and Girls (whom there are few, so we don't split internally). With that in mind, I tried to create a different predicting statistic:

• For cutters, the sum of their experience and speed
• For handlers, the sum of their experience and disc skills
• For girls, 2/3rds of the sum of all three (to keep the number on the same 2-10 scale)

The results, unfortunately, were not better:

In fact, the statistics are even weaker = R^2 = 0.2141, which gives R = 0.4627. 

Conclusion:

If you got here, congratulations, and thank you :) I have three ideas as to why this happens, but I haven't thought much about trying to prove or disprove them:

• People don't really care. For most players, this isn't their first or second time filling up the form. They probably think we don't even look at what they fill up (which I just claimed we shouldn't, so maybe they're right :)). As a result, they don't really bother to fill it up seriously, and make up numbers that sound about right. That's even after removing the people who left them all at 1's, who obviously weren't trying. Frankly, my dear, they don't give a damn.
• Most players' standards are different than ours. Two quick examples, one in each direction. Someone who only plays with a few of his buddies every couple of weeks decides to join us. Compared to them, he's the fastest, and thinks his throws are hot shit. He can even throw a flick, once our of every two or three times. He would probably overrate himself. Conversely, some guy who was around the middle of his D1 college team doesn't think of himself as a star, but would probably be one of the best players at the tournament. He would probably underrate himself.
• Our rankings aren't as good as we think they are. Quite possible - but they're the best we have. 

How do we fix it? I have some ideas, but this is already long enough. I'll throw them in with the second part of this post.

If you've read this far (and even if you just skipped to the conclusion) - I'd love your thoughts. Anything wrong with my math? What do you think we can and should (if at all) do?

-Guy