The BG Rating Scale is a bit odd

[Talen Lee]

I've always had the 1-10 scale gripe.

I agree. It's a bad idea - especially because people don't necessarily like the same things. The point of a review should be to cover the material reviewed and give an impression. Trying to then tie that up with a score at the end is a really bad idea. I have already voiced my opinion, however, that the 0-1-3-10 scale is just as bad at trying to express a complex opinion in a cute fashion. Just let people listen to the review. Summarise at the end. Fuck scores.

[veekie]

At least it's a useful number here. I'd know intuitively what the rating means.

[Josh]

I've always had the 1-10 scale gripe.

I agree. It's a bad idea - especially because people don't necessarily like the same things. The point of a review should be to cover the material reviewed and give an impression. Trying to then tie that up with a score at the end is a really bad idea. I have already voiced my opinion, however, that the 0-1-3-10 scale is just as bad at trying to express a complex opinion in a cute fashion. Just let people listen to the review. Summarise at the end. Fuck scores.

I understand that you don't like the system.  That does not mean it does not work for others, actually it does not even mean it does not work for you.

Out of the BG's I am certainly the least talented in the field of education.  And I do lots of work training.  An example of something I would do at work is, take a piece of equipment and refine the procedure for it.  Then I have to train someone, often someone with poor English skills, and of course it needs to be done today.

This stuff works.  This discussion even reinforces that it works.  If you don't like it, ignore it.

That all being said we have made a decision to revamp the system anyway, so keep up the discussion.

[Talen Lee]

I don't think you can prove that it 'works' any moreso than people listening to the rest of the review works. It's like video game ratings, which are the emptiest thing in the world.

[Josh]

I don't think you can prove that it 'works' any moreso than people listening to the rest of the review works. It's like video game ratings, which are the emptiest thing in the world.

When a review comes down to a single person the decision is binary Yes or No.  Am I interested or am I not? 
The intent of any review is to introduce a game that someone has not considered or to ward them away from a game they thought they were going to like.

So does this system work?  Yes.  That is a fact.  People have said "I listened to your review and played game X or I was going to buy a game but stopped, because of your review."  So that is not even something debatable, the system works. 

The question is "how well?" "is there a better system?"

I think we can do better.  I also think that no one else is doing as well.  If you think anyone else out there is doing a good job with reviews, let me know (then I can go learn/steal from them).

[Talen Lee]

So does this system work?  Yes.  That is a fact.  People have said "I listened to your review and played game X or I was going to buy a game but stopped, because of your review."  So that is not even something debatable, the system works.

The process of making reviews works. But unless people have come to you and said 'I had no idea whether or not I should buy this game or not, even listening to the review, but thanks to rating system you use for your games, I was able to make my decision despite my uncertainty.'

The rating system, I would think, is superfluous to the success of the review.

[The87]

I have already voiced my opinion, however, that the 0-1-3-10 scale is just as bad at trying to express a complex opinion in a cute fashion. Just let people listen to the review. Summarise at the end. Fuck scores.

I disagree. I think that the 0-1-3-10 scale can more accurately express a complex opinion because it gives a range. I think it also opens up the floor for more meaningful debate. For example, given a 1-10 rating scale, is an argument between a 7 and an 8 a worthwhile argument? I would say that it probably is not. However, using the 0-1-3-10 scale, one person giving a game a 3 and another person giving it a 1 clearly shows substantial difference in opinion; a difference that could more easily result in a worthwhile debate.

[Talen Lee]

It expresses no range. It expresses audiences. A 3 is not better than a 10, nor is it worse. It's for different consumers.

[Josh]

The process of making reviews works. But unless people have come to you and said 'I had no idea whether or not I should buy this game or not, even listening to the review, but thanks to rating system you use for your games, I was able to make my decision despite my uncertainty.'

The rating system, I would think, is superfluous to the success of the review.

There is a concept in critical thinking called "convergence."  In general I can drop a few words from a paragraph and you will get my meaning, but the clarity is lower.  If 100 people read the original 98 will understand.  Drop a few words and 90 will understand.   Keep dropping words and more people don't understand.  So yeah you can drop stuff, but you do loose out on infrormation.

Also the scale gives people an ability to compare games to each other.

It expresses no range. It expresses audiences. A 3 is not better than a 10, nor is it worse. It's for different consumers.

OK.  Better is your word.

Books are not rated by better-ness they are rated by usefulness.  Let me give an example, I own a bunch of GURPS books, I love GURPS books, the system is laughably bad.  But the research that goes into these books is great.  In fact if you were going to go out and buy three books for your cyberpunk game I would recommend "Transhuman Space."  If you are only going to buy one, the book would be useless, because you need a system.  To the guy who buys 3 and 10 for a cyberpunk game, buy transhuman space, you will not regret it.

The ideas are good, the writing is good, the science is good(for an RPG, don't get me started) it has more good than bad in a sense. 

Though it is worth noting that if you view gaming as a business selling more books is better, so appealing to more people is better. 

And selling people books that are inappropriate for them is unethical, and that's less-better.

[Talen Lee]

Why not just say 'The system is bad, but the research is good,' and let the people who are looking for good systems look elsewhere? Why must you append a numerical value on it? You might as well have the system be Blueberry-Dishwasher-Windchime-Seacup?

[Josh]

Why not just say 'The system is bad, but the research is good,' and let the people who are looking for good systems look elsewhere? Why must you append a numerical value on it? You might as well have the system be Blueberry-Dishwasher-Windchime-Seacup?

How Good is Good?  How Bad is Bad?

The reason that the system is not good-medium-bad is because that's arbitrary, from the other direction.  What does good mean in practice?

And we do say whats good and bad about a book as well, not in exclusion of.

[Talen Lee]

So you're saying that because a simple summary can be ambiguous, you're fine with using an extra-ambiguous simple summary, because it works for you?

[Josh]

So you're saying that because a simple summary can be ambiguous, you're fine with using an extra-ambiguous simple summary, because it works for you?

No.  I'm saying...  actually I have no idea what you thought was that.

We give a summary and a rating as well as key elements of the game and setting. 

[The87]

It expresses no range. It expresses audiences. A 3 is not better than a 10, nor is it worse.

It does express range. The scale is built to express range.

A score of 3 means that given 3 books purchased (per year, I believe), this should be one of them. In other words, this book sits in the RANGE of first-third preference in the eyes of the reviewers. 10 means given 10 books purchased, this should be one of them (first - tenth preference). These are qualitative explanations for the 0,1,3,10 rating scale.

Given a scale of 1-10, what does a 4 mean? Does a rating of 4 have any concrete, qualitative meaning?

[Talen Lee]

You seem to think I'm arguing in favour of a simple numeric scale. I'm not.

I don't think you need a rating system. At all. I think you should review the book, tell people what it made you think, what elements in it are good, what elements are bad, and numbers can go fuck 'emselves, because you've exchanged one generally-meaningless number for another generally-meaningless number.

[The87]

You seem to think I'm arguing in favour of a simple numeric scale. I'm not.

Fair enough.

I was under the impression that you were in favor of a generic scale (1-10 or 1-5.. whatever).

So, apologies for the mistake. However, you did say that you thought their rating system was "as bad" as a typical 1-10 scale. And on that point, I still disagree.

The 0-1-3-10 scale gives both a qualitative and a quantitative summary of the game at hand.

[Josh]

You seem to think I'm arguing in favour of a simple numeric scale. I'm not.

I don't think you need a rating system. At all. I think you should review the book, tell people what it made you think, what elements in it are good, what elements are bad, and numbers can go fuck 'emselves, because you've exchanged one generally-meaningless number for another generally-meaningless number.

First the supposition that the number is meaningless has been disproven several times over now.  With everything given on the subject the burden of proof has shifted, you would need to demonstrate how the number is arbitrary.

Second, you run into my new mantra "people are different."

When I train people at work I: Explain the process, demonstrate it, walk them through it, let them try on their own and then give them corrections.  No one needs all of these steps but different people learn differently. 

You don't like the number, ok.  Does its inclusion hurt you or render the opinion lesser? 

Many, many people have commented that they like the rating system.  A small number dislike it. 

[Talen Lee]

Alright, back.

If you're going to hold the idea of vox populi as the pure motivation behind using the system that you are using as you are using it, especially when you take into account, your, individually, Josh's perspective that people come in the flavours of agreeing with me and misinformed and ignorant, then lean on the authority of your own ability as a trainer in an unrelated position, I do not think that your argument is strong.

If 'more people like it' is a good metric for you to use as a reason to keep the system, then you should really shut the fuck up about Burning Wheel (because it's less popular than D&D 4E), and you should also start singing the praises of WOD (because it's still the second biggest elephant in the pen).

Obviously, you don't and you won't and it's a stupid rationale. My objection to the system is that any numerical rating system operates on a very flawed idea that you can quantify opinion. Not every opinion is the same. A game that's a 3 for me might be a 10 for you. What about if I have a high threshold for fan canon? What if I like a particular art style? All of this plays into the problem of a unified scale for an unified thing.

Have an opinion, say it, and don't try to tie it all up at the end with a rating system. People can make use of it doesn't seem to be as good an analogy to me.

Look, I guess it boils down to this: The reviews of material I've gotten from Yahtzee and Penny Arcade have been miles better than the reviews I've got from here, IGN, or even my beloved PC Format. If you can't articulate your opinion in a useful, meaningful way for your listeners or readers without a numeric system there is a problem deeper than a numeric system, and no system is really going to do anything more than push dirt over that problem.

Oh, and yes, they make the reviews lesser for me, because they both increase your laziness (we don't have to explain ourselves too well, because the number at the end will do that for us, such as demonstrated by your terrible set of reviews ending in Bullshit, Fullashit, Crockashit and Totalshit rating system or whatever the fuck it was), because they require qualifications (if you're going to do a steampunk game, this book is a 1!), and because it's like having an interesting scientific paper at the end decide to talk seriously in favour of flat-earth theories.

[Cam_Banks]

I would tend to agree with Talen Lee on the nature of reviews and rating scales. I would much rather read somebody's opinion of a game, have them explain coherently why they liked it and why they didn't, even if it's entirely subjective, because that way you'd build up a sort of review profile. If you know that you played and liked Game X and Josh the Reviewer liked Game X also, you would then think "maybe I will give Game Y a try" if Josh then says he liked Game Y.

I don't see any point nor possibility of a review without reviewer bias or subjectivity. But then I don't look for those, since I don't believe they exist. I tend to look very suspiciously at people who claim their reviews are NOT subjective or biased in some way, because that implies that if I liked the game and they didn't, I'm some sort of moron.

Some of my favorite reviews are the Zero Punctuation reviews for video games by Yahtzee, precisely because he doesn't give a rating scale, he doesn't make claims of being objective or without bias, and he's hilarious. I don't agree with him on all his points, but at least I don't think I'm being judged for that.

Cheers,
Cam

[Josh]

If you're going to hold the idea of vox populi as the pure motivation behind using the system that you are using as you are using it, especially when you take into account, your, individually, Josh's perspective that people come in the flavours of agreeing with me and misinformed and ignorant, then lean on the authority of your own ability as a trainer in an unrelated position, I do not think that your argument is strong.

I do nothing of the sort.  I presented that datum as a response that negated another theoretical argument.

I don't see any point nor possibility of a review without reviewer bias or subjectivity. But then I don't look for those, since I don't believe they exist. I tend to look very suspiciously at people who claim their reviews are NOT subjective or biased in some way, because that implies that if I liked the game and they didn't, I'm some sort of moron.