Author Topic: The BG Rating Scale is a bit odd  (Read 12515 times)

0 Members and 1 Guest are viewing this topic.

okeefe

  • Ring-Tailed Lemur
  • **
  • Posts: 15
The BG Rating Scale is a bit odd
« on: January 04, 2009, 08:58:10 PM »
Quote
Our rating scale is:
  • 1= If you are only going to buy 1 book, get this one.
  • 3= If you are going to buy 3 books so you have a “main game” plus a fun pickup, this is the book for you
  • 10= If you are going to get 10 books for a bit more variety, get this one
  • 0= Don’t buy it.

The rating scale, while understandable, doesn't make a lot of sense.  One is the best rating, increasing numbers indicate worse content, yet zero is the worst.  But mathematically 0 < 1 < 3 < 10.

Intuitively, 0 means to me "you don't need to buy it, you should already have it!"  And a Very Large Number or Infinity -- or perhaps a "Library of Congress" -- rating implies that you should buy every other book except this one, which would be the worst rating.  (Negative numbers were mentioned on at least one episode.  I suppose that could mean that you should buy copies to give out as gifts.)

Does the rating system rub anyone else the wrong way?

Cam_Banks

  • Curious George
  • ****
  • Posts: 325
    • Margaret Weis Productions
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #1 on: January 05, 2009, 12:26:36 AM »
Well, I'm rarely in a position to have the funds to pick up three game books at one time, let alone ten. So a scale that informs me whether or not I should spend money on the book relative to the number of books I buy isn't helpful. It's also no use in cases where I might ask for the book as a gift, or when the book is a core book vs. a supplement, or whatever.

But, I don't really think the BG folks consider themselves reviewers, so much as trying to tell you what a game is about and whether it succeeds in its stated purpose. I think. Regardless, no, I don't really bother with the scale.

Cheers,
Cam
Managing Editor & Community Manager | Margaret Weis Productions

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #2 on: January 05, 2009, 04:49:34 AM »
Quote
Our rating scale is:
  • 1= If you are only going to buy 1 book, get this one.
  • 3= If you are going to buy 3 books so you have a “main game” plus a fun pickup, this is the book for you
  • 10= If you are going to get 10 books for a bit more variety, get this one
  • 0= Don’t buy it.

The rating scale, while understandable, doesn't make a lot of sense.  One is the best rating, increasing numbers indicate worse content, yet zero is the worst.  But mathematically 0 < 1 < 3 < 10.

Intuitively, 0 means to me "you don't need to buy it, you should already have it!"  And a Very Large Number or Infinity -- or perhaps a "Library of Congress" -- rating implies that you should buy every other book except this one, which would be the worst rating.  (Negative numbers were mentioned on at least one episode.  I suppose that could mean that you should buy copies to give out as gifts.)

Does the rating system rub anyone else the wrong way?

The rating system is the reverse of what most people expect.  We describe the person who would buy the book rather than try to gauge the arbitrary quality of the book. 

One of the other aspects is the qualifiers, we describe the type of book, as well as the numerical rating.  That is something very important.

The system is different, however it is also the best rating system ever devised.  If you have any input on how to make it better we would love to hear it.



Ennies Nominees - Best Podcast 2009

Wordman

  • Ring-Tailed Lemur
  • **
  • Posts: 70
    • Asteroid
Re: The BG Rating Scale is a bit odd
« Reply #3 on: January 05, 2009, 05:57:01 AM »
« Last Edit: February 27, 2009, 08:04:37 AM by Wordman »
Ctrl ]

Shoggoth

  • That monkey with the orange ass cheeks
  • ****
  • Posts: 246
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #4 on: January 05, 2009, 07:01:14 AM »
Every rating system is arbitrary, so I'm pretty OK with it.  They very carefully explain what the ratings mean every time they review a game, which is good, and it was clear to me what it meant the first time I heard it.

Besides, by creating a system that moves away from the concept of mathematically relative ratings, the BGs are proving to us all that they are MAVERICKS.

MAVERICKS.

I think we can all respect that.
Still came that eldritch, mocking cry - "Tekeli-li! Tekeli-li!" and at last we remembered that the demoniac Shoggoths...had no voice save the imitated accents of their bygone masters.

Meg

  • Message Board Extraordinaire
  • Brilliant Gameologist
  • Man in Gorilla Suit
  • *
  • Posts: 2069
  • Are you rapier than me?
    • Brilliant Gameologists
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #5 on: January 05, 2009, 07:07:57 AM »
I really like our rating scale- the only part I'd like to clarify is the time frame.  So Cam, when you say "10 books at a time"- that's not what we mean.  We mean "10 games this year" but I'm not sure if a year is a proper timeframe.  I think the first step would be to survey how many games (RPG's) people buy in a year and that would inform the timeframe a bit better.
All of my updates are on twitter! 

This is my angry voice.  Text written in red, by me, is  an official moderator "suggestion"

Want to meet me or the other Gameologists?  Check out where we'll be on the Conventions, Meetups and Events board!

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #6 on: January 05, 2009, 07:46:46 AM »
Every rating system is arbitrary, so I'm pretty OK with it. 

Except when they indicate something.  Stars or 1-10 are absolutely meaningless.  Our system says what kind of person are you?  And then lets you know if you should consider buying this book or takes a book you were thinking about and warns you to consider not buying it. 

The purpose of a review is to either indicate a book that you might want to purchase, or warn you not to buy a book you were considering.  And so our system actually fulfills that. QED.

It might be argued that the system is too granular.  But there are really very few games once you categorize them. 

Ennies Nominees - Best Podcast 2009

Shoggoth

  • That monkey with the orange ass cheeks
  • ****
  • Posts: 246
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #7 on: January 05, 2009, 06:31:55 PM »
Every rating system is arbitrary, so I'm pretty OK with it. 

Except when they indicate something.  Stars or 1-10 are absolutely meaningless.  Our system says what kind of person are you?  And then lets you know if you should consider buying this book or takes a book you were thinking about and warns you to consider not buying it. 

The purpose of a review is to either indicate a book that you might want to purchase, or warn you not to buy a book you were considering.  And so our system actually fulfills that. QED.

It might be argued that the system is too granular.  But there are really very few games once you categorize them. 

Every rating system is arbitrary, and they all indicate something.  It's not mutually exclusive.

1-10 is arbitrary, and it indicates that '10' is better than '9' is better than '3'.

0-1-3-10 is arbitrary.  As has been pointed out by others, the number of books and the timeframe in which they are bought are different for everybody.  You can't pick numbers that are universal, so you picked 4 that are different and indicate a scale of none, one, more, many.  You could have done 0-2-5-12, and it would have been just as meaningful.  That is the definition of arbitrary.

I like the system.  I think it's good.  But let's not pretend that you've reinvented the ancient art of rating things.
Still came that eldritch, mocking cry - "Tekeli-li! Tekeli-li!" and at last we remembered that the demoniac Shoggoths...had no voice save the imitated accents of their bygone masters.

Cam_Banks

  • Curious George
  • ****
  • Posts: 325
    • Margaret Weis Productions
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #8 on: January 06, 2009, 05:11:36 PM »
I like the system.  I think it's good.  But let's not pretend that you've reinvented the ancient art of rating things.

Or that it's the best rating system ever devised.

I am willing to accept that it's the best rating system ever devised by Josh, Meg, and Zeke however.

Cheers,
Cam
Managing Editor & Community Manager | Margaret Weis Productions

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #9 on: January 06, 2009, 08:31:09 PM »
[socrates]
So if this rateing thing is so easy and basic, it should be pretty easy to show me some good examples.

To clarify, we are not looking at the review itself, just the system of output.
[/socrates]
Ennies Nominees - Best Podcast 2009

AfterCrescent

  • Honorary Moderator
  • Organ Grinder
  • *
  • Posts: 4220
  • Here After
Re: The BG Rating Scale is a bit odd
« Reply #10 on: January 06, 2009, 08:47:30 PM »
I really like our rating scale- the only part I'd like to clarify is the time frame.  So Cam, when you say "10 books at a time"- that's not what we mean.  We mean "10 games this year" but I'm not sure if a year is a proper timeframe.  I think the first step would be to survey how many games (RPG's) people buy in a year and that would inform the timeframe a bit better.
I don't think the time frame actually matters for your scaling efforts.

I mean if I have the money to buy one book now or ten books now, the system you've given makes sense. Likewise if I am looking at buying only one book this year, or I buy a book a month, so I'll get 10+ this year, the system still works. It's probably simpler in the long run for the viewer to define their own time frame.
The cake is a lie.
Need to play table top? Get your game on at:
Brilliant Gameologists' PbP Forum. Do it, you know you want to.
The 3.5 Cleric Handbook
The 13th Guard - An alternate history campaign idea.
Clerics just wake up one morning and decide they need to kick ass, and it needs to be kicked NOW. ~veekie

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #11 on: January 06, 2009, 10:25:51 PM »
AC is correct about the mathamatics and self definition aspects.

There are two extra categories that we don't mention, because they are not useful to advise.

The 'buy no books' and the 'buy every book' guys. 

I think the reasons we don't try to advise those guys is obvious.
Ennies Nominees - Best Podcast 2009

Shoggoth

  • That monkey with the orange ass cheeks
  • ****
  • Posts: 246
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #12 on: January 07, 2009, 03:22:33 AM »
[socrates]
So if this rateing thing is so easy and basic, it should be pretty easy to show me some good examples.

To clarify, we are not looking at the review itself, just the system of output.
[/socrates]

I'm not actually objecting to your rating system.  I think it's clever.  I really like it, and it actually manages to include a little bit of context into the rating itself, which most reviews have to do separately.

I'm just commenting on your self-righteous indignation that someone has actually had the temerity to question the amazingness of your system.  It entertains me to poke you and see how you will respond.
Still came that eldritch, mocking cry - "Tekeli-li! Tekeli-li!" and at last we remembered that the demoniac Shoggoths...had no voice save the imitated accents of their bygone masters.

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #13 on: January 07, 2009, 04:04:07 AM »
[socrates]
So if this rateing thing is so easy and basic, it should be pretty easy to show me some good examples.

To clarify, we are not looking at the review itself, just the system of output.
[/socrates]

I'm not actually objecting to your rating system.  I think it's clever.  I really like it, and it actually manages to include a little bit of context into the rating itself, which most reviews have to do separately.

I'm just commenting on your self-righteous indignation that someone has actually had the temerity to question the amazingness of your system.  It entertains me to poke you and see how you will respond.

Good Point.

I feel the need then to point out that okeefe is completely correct here.  It is confusing to go from 1 to 3 to 10 to 0.  And the scale is something that may rub people the wrong way.  We may discuss the genesis of the system at some point(it might explain why), and it might get retooled(but we are loathe to change it for a number of reasons).

My indignation stems from everything said after okeefe.  So okeefe, sorry for the misunderstanding, you are completely correct.  And your point of view is something we take seriously into consideration.

Ennies Nominees - Best Podcast 2009

Talen Lee

  • Bi-Curious George
  • ****
  • Posts: 447
  • Forum Ninja
Re: The BG Rating Scale is a bit odd
« Reply #14 on: January 07, 2009, 05:27:03 AM »
I think it's obscure for obscurity's sake.

Zeke

  • Brilliant Gameologist
  • Bi-Curious George
  • *
  • Posts: 540
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #15 on: January 07, 2009, 05:58:32 PM »
We did not make the scale wierd just to be wierd.  We  have learned along the way that it confuses people. At the time Josh came up with it, we thought it answered the questions we thought a rating scale should answer. It's obscure more out of negligence than by design.

Wordman

  • Ring-Tailed Lemur
  • **
  • Posts: 70
    • Asteroid
Re: The BG Rating Scale is a bit odd
« Reply #16 on: January 07, 2009, 07:06:23 PM »
« Last Edit: February 27, 2009, 08:05:19 AM by Wordman »
Ctrl ]

Josh

  • Brilliant Gameologist
  • Grape ape
  • *
  • Posts: 1835
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #17 on: January 08, 2009, 05:35:30 AM »
OK, here's the deal.

The scale originally was designed to rate books in particular Science Fiction.  The key moment was when I gave Zeke a book and said "here read this" and Meg said "I need a book.  Can I read if first?"  And I said "no the book is not good enough for you." 

You see meg reads about 10 or so fun books a year (and about 7000 books for work).  The book I was giving Zeke had this one funny minor character but Zeke reads 100 or more books a year.  That means that he is interested in books that are of a much lower quality than Meg. 

As counter intuitive as it seems, the more prolific reader is less interested in quality than the lesser reader.  And they are willing to get much less out of the books they read.

So we translate the system over, looking at the types of people who buy gaming books.  And we rescale the numbers.

So we have:
0 - This person will never under any conditions buy a new book
1 - this person is interested in one game, learning one new game, brings one game back from a con or plays only one game at a time
3 - this person is interested in three games, learning three new games brings three games back from a con or or plays three games at a time.
10 - this person is interested in ten games, learning ten new games brings, ten games back from a con or or plays ten games at a time.
ALL - this person buys every game, picks up new games constantly, wants every game, plays all the games they can.

So note that there is no bad rating.  In trying to simplify things for the podcast we realized that 0 and all are pointless to talk to.  One of them buys everything anyway and the other never buys.  So to avoid confusion we drop them.

We are about to publish a review and are taking a look at the Serenity game.  A game with absolutely no redeeming value.  It does not even have good pictures.  I never considered that such a bad book would ever exist.  So I say new category and call it, being a math nerd, the empty set.  And just before we record the first review Meg and Zeke suddenly go "wait, empty set, what the hell is that?"  So minutes before we record, we change it to zero.  (my second choice was null)

In retrospect, bad choice.

Ennies Nominees - Best Podcast 2009

Cam_Banks

  • Curious George
  • ****
  • Posts: 325
    • Margaret Weis Productions
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #18 on: January 08, 2009, 08:17:48 PM »
We are about to publish a review and are taking a look at the Serenity game.  A game with absolutely no redeeming value.  It does not even have good pictures.  I never considered that such a bad book would ever exist.

Comedy gold!

We need to add a quote to the back of Serenity for the next printing: "Leads Josh to bad choices."

Cheers,
Cam
Managing Editor & Community Manager | Margaret Weis Productions

The87

  • Ring-Tailed Lemur
  • **
  • Posts: 35
    • TrapCast
    • Email
Re: The BG Rating Scale is a bit odd
« Reply #19 on: January 08, 2009, 08:27:14 PM »
Personally, I think this rating system is great.

I've always had the 1-10 scale gripe. What differentiates a 5 from a 6 or a 4? It's too subjective. 1-10 scales are better used in situations with larger sample sizes instead of a single, arbitrary rating.

0,1,3,10 makes sense to me because it grants ranges.

0: fuck this game
1: buy this thing now
3: definitely give it a shot at your earliest convenience
10: keep it in the back of your mind if you see it for a good price when your bored of your other games

On the other hand, a rating system like this would not fair so well in terms of building an accurate distribution. But that isn't the point. The point is for the BG hosts to suggest/not suggest the game at hand.