Rating Reform [Yes. Another Request.]

[betiko]

Basically, i said all i have to say as i like the system the way it is. People want to change a system instead of changing the way they rate...

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Rate people 5s if you enjoyed playing them, don t rate if you have not much to say, and rate 4 and bellow for people you didn t like.

Just give me 1 name, 1 player that is over/under rated. The rates give a very good idea of the opposition and i know i shouldn t play a multiplayer game with a 4.3 as he will be a jackass.

[homes32]

Basically, i said all i have to say as i like the system the way it is. People want to change a system instead of changing the way they rate...

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Rate people 5s if you enjoyed playing them, don t rate if you have not much to say, and rate 4 and bellow for people you didn t like.

Just give me 1 name, 1 player that is over/under rated. The rates give a very good idea of the opposition and i know i shouldn t play a multiplayer game with a 4.3 as he will be a jackass.

That is your opinion and interpretation. It is not however how the site instructions say the system should be used. In fact it is completely opposite. There should be no "figuring it out" as you suggest. A rating system should be simple and concise, leaving little room for people to misuse, misinterpret or question the results.

As for names, this is not the time or the place for such a discussion. As the ratings stand they are subjective to each players opinion and such discussion has little to no value in enhancing or destabilizing this suggestion.

[betiko]

Basically, i said all i have to say as i like the system the way it is. People want to change a system instead of changing the way they rate...

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Rate people 5s if you enjoyed playing them, don t rate if you have not much to say, and rate 4 and bellow for people you didn t like.

Just give me 1 name, 1 player that is over/under rated. The rates give a very good idea of the opposition and i know i shouldn t play a multiplayer game with a 4.3 as he will be a jackass.

That is your opinion and interpretation. It is not however how the site instructions say the system should be used. In fact it is completely opposite. There should be no "figuring it out" as you suggest. A rating system should be simple and concise, leaving little room for people to misuse, misinterpret or question the results.

As for names, this is not the time or the place for such a discussion. As the ratings stand they are subjective to each players opinion and such discussion has little to no value in enhancing or destabilizing this suggestion.

Well, no. You aren t making any sense. You are basically telling me that a self regulated system after years of ratings and thousands of people rating you isn t accurate?

I'm a 4.7, i am good sports most of the time but i can have my outbursts in game chat. So i get 5s most of the time, 1s when i had a fight in the chat. I know lots of people never say anything bad in game chat. They are 5.0 or 4.9. The whole thing is very straight forward.
I know that being a dick sometimes I m at 4.7, a 4.8 almost never complains. When I see a 4.6 I know the guy has often a bad attitude. 4.5 will generally be a bad opponent, and bellow that they should be avoided at all costs.

YOu just rate accordingly!!!!! You don t need a master in science to understand how to rate people in order to match the site s standards. I understand if a noob rates me 3s, but if you ve been here for years and you rate 3s someone who hasn t done anything wrong to you then you simply worried about you ability to be minimally crafty.

[macbone]

A Top Tags section is a great suggestion. I'd definitely support that.

Changing to a thumbs up/thumbs down system is fine with me, but I don't really see what difference it's going to make, especially if it's not converted to a percentile system. I don't spend a ton of time looking at ratings unless I'm checking out a potential team teacher, and even for games, I have no problem playing any player, but I can get a decent idea of what to expect from someone rated 4.5 as opposed to 4.8.

Another thing that's not clearly defined is "Average." Average on Conquer Club, or average compared to everyone I've played in world domination type games with chance elements? Most people who find their way here and play a good game are above the level of the average casual dice roller. =)

[Metsfanmax]

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Which is why we should have automatic neutral ratings by default. The problem is that if everyone did in fact rate according to their opinion, then the average rating would indeed be around 3. Since the typical rating that a person leaves is straight 5s, that means they think basically every person they play is significantly above average. Which is obviously a lie. I don't know why anyone would condone such blatant lying among basically everyone who participates in that system.

[macbone]

There's really not much difference in terms of player skill in 4.7-5.0. The 4.7 player probably plays more speeders. There's probably not much separating players ranked general and up, but their ratings range from 4.6 to 5.0, and there are only a couple of 5.0s.

Master Fenrir and uckuki are two of the sexiest players I know in team play, but they don't have perfect 5.0s. Kaskavel is Conqueror, and he only has a 4.8.

Ratings are more about player perception than anything else. To best judge a player's ability, you have to look at things like win-loss records and qualities of wins. Even the scoreboard and rank don't tell the whole story.

Perhaps moving to a percentile-based rating would be better, though, with a simple thumbs up/thumbs down, and a percentage calculated based on that.

Yeah, it is true that there are several different ways of using the rating scale. Some people actually do use it as a thumbs up/meh/thumbs down, with 5 being thumbs up, 3 being meh, 1 being thumbs down. Other people try to use the scale as it's described. Ah, this person was an average player, so 3 stars. This player did a good job, but they didn't win, so 4 stars. This player played masterfully and won, so 5 stars. And some people use the rating system out of spite. Kaskavel had a 1-star rating from a 4.6-rated cadet recently. Really? The cadet thinks the conqueror is a 1-star player? Huh.

[IcePack]

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Which is why we should have automatic neutral ratings by default. The problem is that if everyone did in fact rate according to their opinion, then the average rating would indeed be around 3. Since the typical rating that a person leaves is straight 5s, that means they think basically every person they play is significantly above average. Which is obviously a lie. I don't know why anyone would condone such blatant lying among basically everyone who participates in that system.

So instead of everyone being 4.6 - 5.0, everyone will be 2.8 - 3.2 and do nothing to really show the real "ratings", it'll just input a shit load of 3's into the rating system and make it just as (or more) worthless then it currently is, dragging everyone closer to 3.0 over time due to lazy raters.

[macbone]

I simply don't rate everyone I play. It wouldn't be fair to leave them automatic 3s for players who don't take the time to consider the ratings they're going to leave. Some of the best players I know quietly go about getting the job done and never drawing attention to themselves. I do try to leave a rating for a player who wins a game if I haven't rated them before, but if a player wins an 8-player Standard game and I forget to rate them, I disagree that the player should receive an automatic 3-star rating.

[Metsfanmax]

If you give 3s, you are basically giving a bad rate to your opponent. If he was really average, you don t give any ratings, simple as that, i guess you can figure it out by yourself why.

Which is why we should have automatic neutral ratings by default. The problem is that if everyone did in fact rate according to their opinion, then the average rating would indeed be around 3. Since the typical rating that a person leaves is straight 5s, that means they think basically every person they play is significantly above average. Which is obviously a lie. I don't know why anyone would condone such blatant lying among basically everyone who participates in that system.

So instead of everyone being 4.6 - 5.0, everyone will be 2.8 - 3.2 ... dragging everyone closer to 3.0 over time due to lazy raters.**

Yes, exactly. The whole point of having an average is that most people should be relatively close to the average. I expect this will look something like a normal distribution. If you're really bad, you'll be significantly below 3.0 because a lot of people will rate you poorly, and if you're really a standout player, you'll be closer to 4.0 or maybe even above it. But the majority do not stand out -- by definition. A corollary to the problem with the current system is that it results in people giving false ratings. As betiko pointed out, if you wanted to give a 3.0 rating, you're stuck with two options -- either doing nothing, or going along with the fold and rating 5.0. The problem with rating them a 5 when you don't think they're a 5 should be obvious*. The problem with abstaining from rating them is that you're still reinforcing the system -- by not actually bringing them closer to "average," you're artificially boosting their rating above what it would have been if you told the truth. And then this makes it more likely that the next person to rate them faces the same dilemma. So even though the width of the rating scale may not change that much, the accuracy of the ratings for any individual player should go up substantially.

There's plenty of room in the other thread for discussion about this -- I originally just wanted to point out that no one has attempted to put forth a method that could deal with this ratings inflation. The same situation would occur in this system as well -- lots of people would give most people thumbs-up, and then if you gave a person a thumbs-down, you'd be seen as a jerk. Except in this system it's even more exaggerated, because there's no wiggle room. At least in the current system, you can rate someone straight 4s and it mostly doesn't raise an eyebrow. Now if you want to affect their rating, you'd have to give them a thumbs down, without any modification.

*I think one problem here is the lack of specificity in ratings. If most people's ratings are between 4.5 and 5.0, why is there no way to rate them anything other than a 4 or 5? This problem is even worse with the thumbs-up/thumbs-down scale. It would be a very interesting test to see what would happen if we removed the star rating system, and instructed people to type a rating between 1.0 and 5.0 into a text box.

**I've addressed the "lazy rater" problem in the other thread as well, but to get into that would derail this topic too much.

[macbone]

There's no way to rate someone as a 4.7 or 4.8 because that level of granularity really isn't necessary, especially for a casual site. A Likert scale 1-5 rating system is widely accepted in survey research, and I don't think CC requires more than that. Movie reviewers use 4 stars or 5 stars, which does create a bit more confusion, but readers can process it. Understanding grades on essays can be more difficult, especially when they're not supported by some kind of rubric, but if a rubric is given that clearly differentiates between scores, it's easier to understand marks. My paper got an 85, but yours got an 86? Why? Oh, ok, I got 1 mark less on grammar than you did on the grading rubric.

Please, CC, never introduce marking rubrics to the rating system. =)

Edit: I know of no survey that processes unanswered responses as neutral responses, and no evaluations that do the same. I don't always rate my Amazon sellers (gasp!), and they don't automatically get some kind of neutral response. I don't see why CC should be the exception to this.

Edit the second: Here's an idea. How about we change the rating system page to show the rating given when a star is clicked? So when 5 stars is clicked, "Excellent" shows up next to it, and for 4, "Good" is displayed next to it. Perhaps that would make us think more about whether a player is Excellent or merely Good (or Average, Poor, or Abysmal).

3rd edit: Ah, there's a hover text already. Never mind.

Edit D: Would changing the scale from "Bad, Below Average, Average, Above Average, Excellent" to "Terrible, Poor, Average, Good, Excellent" make a difference in ratings? No? OK, then, carry on. =)

[Metsfanmax]

There's no way to rate someone as a 4.7 or 4.8 because that level of granularity really isn't necessary, especially for a casual site. A Likert scale 1-5 rating system is widely accepted in survey research, and I don't think CC requires more than that. Movie reviewers use 4 stars or 5 stars, which does create a bit more confusion, but readers can process it. Understanding grades on essays can be more difficult, especially when they're not supported by some kind of rubric, but if a rubric is given that clearly differentiates between scores, it's easier to understand marks. My paper got an 85, but yours got an 86? Why? Oh, ok, I got 1 mark less on grammar than you did on the grading rubric.

It's very difficult to draw analogies from surveys to community rating systems. In a survey the responses are anonymous, which removes the pressure to give inflated positive responses (yes, of course surveys still have bias, but this kind of bias is much larger when the "survey" responses are public). The reason why people are so inclined to give 4s or 5s is that everyone can see if you don't follow the community trend, so there's peer pressure to change your rating. That never happens in surveys because you don't know what everyone else is responding until after the survey is over.

Edit: I know of no survey that processes unanswered responses as neutral responses, and no evaluations that do the same. I don't always rate my Amazon sellers (gasp!), and they don't automatically get some kind of neutral response. I don't see why CC should be the exception to this.

Yes, and Amazon's system is similarly lacking in information. There are people who are generally reliable and have lots of "A+++++++++ would buy again" ratings, and there are people who have lots of negatives or neutrals. This is much closer to a binary because either a person can be counted on to deliver the product, or they can't be. That's all Amazon's system is meant to tell you. For our rating system we want much different information because there's no binary like "is this person fun to play with" -- or else there wouldn't be a distinction between 4.6 and 4.8, which there probably is.

The fact that we're not aware of anyone having done this doesn't mean it's a bad idea. I think it would be a very interesting experiment, and there's not much to lose.

Edit D: Would changing the scale from "Bad, Below Average, Average, Above Average, Excellent" to "Terrible, Poor, Average, Good, Excellent" make a difference in ratings? No? OK, then, carry on. =)

It might. Temporos' suggestion in Submitted included moving the scale from -2 to +2 instead of 1 to 5. I think we should do that even if we don't do the automatic neutral ratings.

[macbone]

I'm still slightly against this suggestion. On a scale of 1 to 5, with a 1 being completely disagree, and a 5 being agree, I'm at a 2, disagree somewhat. =)

The current system already offers like/don't like in the form of 5 stars and 1 star, and it offers more granularity. If we rate someone a good player with a 4, we're saying the player isn't among the best on the site, but they still are better than average. If people cry about getting 3 stars now, changing this to a thumbs-down isn't going to change this.