# Review inflation on ENworld



## tleilaxu (Oct 19, 2002)

* Read all statements of X should be Y as "IMHO X should be Y" *

Yesterday I couldn't get into the boards so I went through and started reading reviews for d20 products. This got me thinking about another thread (http://enworld.cyberstreet.com/showthread.php?s=&threadid=27439) in which Avalanche Press complains about getting stinky reviews. All other aspects of that issue ASIDE, I became interested in whether or not Avalanches reviews on ENworld are in fact favorable.
_
Morrus says: Of course, this is made funnier by the fact that said publisher has received two 4/5 scores from EN World staff reviewers, a 3/5, along with two 2/5 scores. Average score = 3/5, which does not constitute,as they put it, "disturbingly unfavorable reviews
our products received up to, and including XXXXXX". _

So I went through the three official reviewers, Kushner, Simon and Psion. (heh, Simon and Psion, anyone remember Simon and Simon?)

Scrolling down quickly I found that out of Psions 100+ reviews not a -single- one warranted a "1 - atrocious" rating. Kushner as well has not nailed anyone with a 1 rating. In Simon's 100+ reviews there are only about 3 or 4 atrocious ratings. 

(i can't get back to the reviews page right now, but when I do i'll give a more statistically accurate rundown). In addition 2's are much more rare and 5s and 4s. So -in effect-, even though Avalanche's rating is 3.00, this is actually _not_ average as is stated, but is actually 1 step away from being put in the same catagory as those companies who released one bad .pdf and got slammed. 

Now, in my opinion, you should ideally be able to add up all the scores from all the products, divide by number of reviews and get 3.00 . So when I get access to the reviews page that is what I am going to do. My guess is that the average is much closer to 3.66 than 3.00

"What does this matter?" you ask? Well it doesn't, but I still find the situation interesting. This is in no way to impinge on the text of the reviews (which I find excellent by all reviewers, they have firmly convinced my of quintessential wizard and some green ronin stuff!)

Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent. 
Why, time and again will people say a movie is "nothing special" and then rank it 7 on a scale of 1=10? A 7 should be a pretty darn good movie. (my theory, people are mentally affected by the way grades work; 7=70 to 79% which was a C in my high school)

So here is what I think: if there are only 3-4 atrocious ranks, there should only be 3-4 amazing ranks. that would surely make them stick out of the crowd much more. As it is, the rankings for publishers are absolutely clogged between 3 and 4.5 while the low numbers are virtually untouched. Spread it out a bit.

Opinions, thoughts, rants, flames?


----------



## Dragongirl (Oct 19, 2002)

My opinion would be to get rid of the ratings altogether and let the reviews stand on their own.  Because :

1) This would stop people (if any have done this) from posting a short review just to pad the ratings.

2) A number rating is too subjective.  For instance, some people deduct points or fractions thereof for artwork, which other people might not care about.  Some people just want good crunchy or crunchy and story.  Unless you had separate ratings for various parts of a review (artwork, grammar, crunchy, story, etc).

2) IMO a 1-5 rating does not give enough leeway if ratings were wanted.

All I can think of at the moment.


----------



## Crothian (Oct 19, 2002)

I've done 19 products so far.  THe numbers break down as:

5/5 : 2
4/5 : 11
3/5 : 5
2/5 : 1
1/5 : 0

I have reviewed two products I got for free, everything else I bought.  I try to only buy good products, so that's why so many of the books I review get a 4.  Also, it's more enjoyible for me to review a book I like verse a book I don't like.  I think those tweo reason are why you see more higher reviews then lower.


----------



## Umbran (Oct 19, 2002)

tleilaxu said:
			
		

> *Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent. *




There's more than one way to grade.  One way to say "The average will be a C".  The other is to say, "The basic acceptable grade is a C".  These compose two different grading philosophies.

What you want - the average being C - is a mathematical construction that you can't easily get unless you have all the scores together before you actually assign the final grade.  You grade on some other numeric scale, find what the avverage is, and call that a C.  If you're doing one grade at a time, and adding them to a pile, it's not realistic to look for this.  Especially if general quality rises over time.  I guess you could have the computer constantly recomputing what's "average", but that's a pain, because you have to say, "averaged with respect to what?"  Averaged with respect to all reviews of all products?  With respect to all reviews of this one product?  With respect to all reviews by this reviewer?

On top of that, it isn't actually _useful_ in this context.  In order to find out what the "average is C" really means, you have to go through and read all the reviews.  What happens if you apply the "C=average" to a whole lot of products that are actually really, really good?  You get a misleading result.  

The "C is the basic acceptable" grade is more user-friendly,   easier for the reader to interpret.  Reviews are not math test grades.  Leave 'em be.


----------



## tleilaxu (Oct 19, 2002)

I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others.


----------



## Crothian (Oct 19, 2002)

tleilaxu said:
			
		

> *I'm not suggesting that we somehow fit all new reviews into a mathematical model, I am just suggesting that in no way does "3" equate to a average rating relative to other products and that reviewers shouldn't hesitate to really nail a product if it isn't as good as others. *




I think reviewers do nail products they think are bad.  I also don't think you can really speak in general terms like this and get any concrete discussion going.  All the reviewers use different criteria when reviewing an item.  There really isn't even a uniform amount of info in the review.  I've see some reviews that barely give any real info and it's mostly their opinion.  Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code.  The standard of reviews has increased, I think it should be applied to all of them.


----------



## tleilaxu (Oct 19, 2002)

Here is a suggestion that I just thought of that I think could make the ratings more accurate. Just include an extra column to the right of the star rating with the percentile rank of a product. (For example: Tomes and Tunnels has a 3.7 star average, which ranks it in the 61st percentile of all products reviewed on ENworld. (meaning it has a higher score than 61% of the products) us.imdb.org has some system for including number of votes/reviews into this to add weight (IE if 10 people say 4.5 it means more than if 2 people say 5.0)

What do you all think of this suggestion? I know that the admins have other stuff to worry about right now but this kind of addition wouldn't take to long to html in. Heck, I could do it on a spreadsheet and email you the results if you wanted. It could even be made to auto-update the numbers when new data is added.


----------



## Mr Fidgit (Oct 19, 2002)

Crothian said:
			
		

> *Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code.*



which code, Crothian?

if you're going to institute some sort of review standards (i.e. a code), i'm with dragongirl, toss the 1-5 rating system


----------



## tleilaxu (Oct 19, 2002)

Crothian said:
			
		

> *
> 
> I think reviewers do nail products they think are bad.  I also don't think you can really speak in general terms like this and get any concrete discussion going.  All the reviewers use different criteria when reviewing an item.  There really isn't even a uniform amount of info in the review.  I've see some reviews that barely give any real info and it's mostly their opinion.  Personally, I think it would be far more productive to get a few people to go back through all the reviews and start deleting the ones that are not up to code.  The standard of reviews has increased, I think it should be applied to all of them. *




Well, my intention is not to generalize, I purposely went through and read the reviews for products that got 1, because you can learn as much about what -not- to do as the opposite. My intention is not to discuss this, but merely about the proponderance <sp?> of 4s and 5s and the general lack of 1s and 2s. If you scroll down a page and see twenty 5s and 4s, ten 3s, and one 2 guess what: The 2 -is- atrocious, at least from someone looking at the page and trying to decide what to buy. I wouldn't touch any products that got a 2 on ENworld because they are so rare they -must- be bad to warrant that rating.

I think your suggestion for culling empty reviews is one that deserves some attention. I read more than one review that basically said 'woopie this product is awesome end of story'.

edit:
BTW Crothian, your average rating is 3.74


----------



## Crothian (Oct 19, 2002)

tleilaxu said:
			
		

> *
> BTW Crothian, your average rating is 3.74*




It'll drop.  The next review I'm doing is a 3/5.  After that I'm going to do some of my favorite companies books that I haven't reviewed yet.  They are my least favorite of their books, so will probably be a pair of 3's and a single 2 in there.  

A product would have to be pretty darn bad for me to give it a 1.  Just like a product has to be darn amazing to get a 5.  If I ever get around to doing the 3 core books, they will all be 4's.  There fine books, but not 5's in my so very humble opinion.


----------



## Umbran (Oct 19, 2002)

There is no "code" for reviews to my knowledge.  And I looked for one to start writing a review myself.  You want to propose one, Crothian?

I don't know if dropping the 1-5 rating system is really a great idea.  Sure, it's hardly precise, but people aren't looking for precision from such a thing.  They want a quick guideline.


----------



## Crothian (Oct 19, 2002)

Umbran said:
			
		

> *There is no "code" for reviews to my knowledge.  And I looked for one to start writing a review myself.  You want to propose one, Crothian?
> 
> I don't know if dropping the 1-5 rating system is really a great idea.  Sure, it's hardly precise, but people aren't looking for precision from such a thing.  They want a quick guideline. *




If a code was instituted it would be up to the 3 staff reviewers to set.  I'm not going to step on their toes as they have a much better understanding for the review process then little old me.  However, if they wanted suggestions I'm sure I could come uyp with a few.  

And the rating system isn't going anywhere.  It's coded into the program and I don't think anyone wants to try to alter or try to remove it.  I'm not sure they can.

Edit: the closest we have to a code is the review FAQ.  Please read it before posting as it has information you do need to know.


----------



## JoeGKushner (Oct 19, 2002)

For me, there are some things I try to keep in mind.

1. Does it do what it sets out to?

2. How's the layout & art?

3. How's the editing?

4. How's the text to page ratio?  I despise companies that put one page for table of contents, another page for credits, another two pages for OGL.  One page for toc and credits, another page for OGL.  That' it!

5. If the product doesn't suit my taste, will it others?

6. How expensive is the product?

7. Are there other products in the same field?

I don't think I've had a 1 product yet and some products, maybe should've been higher.  Wrath & Rage for DMs is a 4 but players, a 2.

I also don't love the 1-5 system, but I'm stuck with it here and on Gaming Frontiers.  I think 1-10 would probably be better.


----------



## Blacksway (Oct 19, 2002)

Purely for statistical freaks out there...

Psion: 139 reviews, avg: 3.69
Simon: 167 reviws, avg: 3.50
Joe: 30 reviews, avg: 3.63

but as lots of people have been saying there are many reasons why this could be (like prefering to review products you've bought and liked...)


----------



## Darkness (Oct 19, 2002)

Blacksway said:
			
		

> *Purely for statistical freaks out there...*



Hmm... As you said, these reviews don't differentiate between "bought and liked" and "got sent a copy to review." Is it possible - from the database - to discern which are which? Otherwise, these statistics don't help much...
(Not that I think that this entire topic is necessary at all, but since you guys are discussing it...)


----------



## tleilaxu (Oct 19, 2002)

I just want to reiterate that I think the_ reviewers _are doing a fine job for ENworld, but there is nothing wrong with trying to improve the_ system_.

To infinity and beyond


----------



## Darkness (Oct 19, 2002)

tleilaxu said:
			
		

> *I just want to reiterate that I think the reviewers are doing a fine job for ENworld, but there is nothing wrong with trying to improve the system.
> 
> To infinity and beyond *



Absolutely.
For example, I'd prefer a 1-10 scale to the current 1-5 one.


----------



## Umbran (Oct 19, 2002)

tleilaxu said:
			
		

> *Here is a suggestion that I just thought of that I think could make the ratings more accurate.  *




Ooh, my buttons are getting pushed 

A thing like this does not increase _accuracy_.  It increases _precision_.  There's a difference.

Accuracy talks about how correct the number is, how well it reflects reality.  Precision, in essence, talks about how many decimal places the measurement can realistically include.

A laser range finder can take measurements very precisely.  It can always give you a number out to many decimal places.  Your measurement may still be inaccurate if you use this very precise tool incorrectly (by saying you measured the width of a field when you actually measured it's length, for example).

There is no point to taking a measurement precisely if you aren't going to be very accurate anyway.  That's the situation we have here.  The rating system is subjective (since it's rating an opinion, and opinions are subjective).  There's no "code" for how you give a rating.  Unless everyones giving ratings in exactly the same way, for the same reasons, you're not getting a very accurate measurement.  There's little point in going for high precision on it.

In other words, the ratings system is by it's nature very vague.  Giving more info on the ratings is then not terribly useful.  It actually then makes sense to keep the ratings vague, rather than giving percentile scores, or going to a 1 to 10 system.  You actually have a greater chance of getting consistent accurate measurements if you give people less freedom of "false precision".

Also, how many ratings do most products get?  Percentile scores are not particularly good if the number of scores is less than (you guessed it) 100 or so.  If only 5 people rate a product, a percentile score is pretty meaningless.  If there are many such products, and you weight for how many people actually rated the product, you now reduce many of the rating to statistical irrelevance.  If you're throwing many of them away, what's the point of doing a percentile score in the first place?

In the end, it comes down to this - what do users want to get out of reviews?  They want to know how good the product is.  Percentile socres don't actually tell you that.  They are a bit of statistical minutae that only tells you something about the products if their quality actually follows a normal distribution.

(edit: My typing and grammar are often rather inaccurate )


----------



## Crothian (Oct 19, 2002)

Umbran said:
			
		

> *
> There's no "code" for how you give a rating.  Unless everyones giving ratings in exactly the same way, for the same reasons, you're not getting a very accurate measurement.  There's little point in going for high precision on it.
> *




When I say code, I'm talking more about some minimium requirements for the review.  Sort of like a word length of a couple hundred words.   There are many reviews of about 4 lines or so, and I haven't read any of that length that get the job done.


----------



## EricNoah (Oct 19, 2002)

Reviewers don't review "everything" -- they get to pick and choose what they review.  Maybe they stay away from the stinkers, whether consciously not.  I certainly would rather spend my time getting to know a good or average product than a bad one.  

When I look through the product lists, I find that not only do I avoid products with low scores -- I also avoid products that have been out for a while and have not garnered even one review.  To me that says a) people aren't buying it or b) people didn't care for it.


----------



## tleilaxu (Oct 19, 2002)

Well, all that I am proposing for actual changes is to add a percentile ranking. For example, if Manual of the Planes gets a better rating than 98% of products reviewed on ENworld then I pretty much know it is a must have. We all have limited amounts of money here and the only game store in HK shrink wraps the books! I need to economize.

Now, the main negative to this suggestion is that it would add some work for the admins. That is the biggest barrier to it happening AFAI can tell. Does anyone object to having a percentile ranking for any other reason?

Umbran: I bow to your somantic arguements, but for me this is a casual medium of communication, meaning

I am lazy with puncuation, grammer and word use. I'm not writing my doctoral thesis, just Idally discussing something that doesn't matter all that much in the 'grand scheme of things'*

*fnord


----------



## tleilaxu (Oct 19, 2002)

EricNoah said:
			
		

> *
> 
> When I look through the product lists, I find that not only do I avoid products with low scores -- I also avoid products that have been out for a while and have not garnered even one review.  To me that says a) people aren't buying it or b) people didn't care for it. *




Eric: is there any list of titles that have never been reviewed? A wanted list for products no one has commented on?


----------



## Mathew_Freeman (Oct 19, 2002)

I go by the Empire Film magazine system for rating things.
5*) A movie masterpiece, like LotR, Star Wars Ep4...films that are generally agreed to be incredible pieces of work.
4*) A very good film. Almost everyone should go and see it, but for some reason it isn't quite a masterpiece.
3*) It's worth seeing, if you like the genre, or one of the actors, and don't mind a few clunky bits. Not a dud, but not a waste of money.
2*) Only go and see it if you really like the genre, actor etc etc
1*) No one should see this film. It is awful.

This scale of things has served me fairly well so far, but it does tend towards 3 or 4 star reviews, mostly because in my case I'm a nice person that can enjoy most things. However, if it is a 1* film it gets mercilessly hacked to death.


----------



## Umbran (Oct 19, 2002)

Crothian said:
			
		

> *
> 
> When I say code, I'm talking more about some minimium requirements for the review.  Sort of like a word length of a couple hundred words.   There are many reviews of about 4 lines or so, and I haven't read any of that length that get the job done. *




Oops. Sorry, my bad.  I used the word "code" to refer to two different things.  One was a code for writing a review - meaning the full text of the review, the other a code for giving the numeric rating.  Sorry for the confusion.

I don't think there _should_ be a code for the numeric ratings.  That measure is best left as user defined.  I think that's the way it's most useful.


----------



## Umbran (Oct 19, 2002)

tleilaxu said:
			
		

> *... Does anyone object to having a percentile ranking for any other reason?
> 
> Umbran: I bow to your somantic arguements, but for me this is a casual medium of communication, meaning*




Well, amid all the formal argument, there's a simple, casual meaning.  A percentile ranking doesn't mean much, and can actually be misleading in this context.  

While perhaps being in the 95th percentile means it's a "must have", depending on the shape of the distribution, you can still have "must haves" down in the 70th percentile. You can wind up giving really good products a bad reputation.  

I think that constitutes a good argument against percentile ratings.


----------



## Crothian (Oct 19, 2002)

I don't think you can ever have a number rating system that means the same thing to everyone.  My 3 might not be your 3, and this guys 3 might be different from both your and mine.  Having a scale from 1-10, percentage, or stars isn't going to change that.  I think most people who use the reviews realize that reading the review is the best way to get actual information.  That's why I think we need to trime the reviews and cull the ones that just don't cut it.  I like how the reviews are done by the fans for the fans, but I think the standards need to be raised.


----------



## Krug (Oct 19, 2002)

What's the point of looking for a mathematical average? For the staff reviewers, they get a lot of product sent by companies. Undoubtedly most companies will send good or excellent products, and as the reviewers do more reviews, the companies will try to send products they are confident the reviewers will like. As long as I read the review and get the information I want, I'm satisfied.


----------



## Umbran (Oct 20, 2002)

Krug said:
			
		

> *What's the point of looking for a mathematical average? For the staff reviewers, they get a lot of product sent by companies. Undoubtedly most companies will send good or excellent products...*




Not all the reviews are done from free copy, and not all reviews are done by the staff reviewers.  People can and do freely review things they have bought.


----------



## Dragongirl (Oct 20, 2002)

Well I don't think we are going to see any major changes in how the reviews are graded.  

1) Too many reviews in there now to go back and change to a new system.
2) Last time I talked to Liquide about it, he was finding it hard to get the time to tinker with the code (programming).  Or had timing conflicts with Morrus since the the whole review section would have to come down to be tinkered with.


----------



## Dragongirl (Oct 20, 2002)

Tallarn said:
			
		

> *I go by the Empire Film magazine system for rating things.
> 5*) A movie . . . *



Hey, I like the the way you put that!


----------



## Psion (Oct 22, 2002)

tleilaxu said:
			
		

> *So I went through the three official reviewers, Kushner, Simon and Psion. (heh, Simon and Psion, anyone remember Simon and Simon?)
> 
> Scrolling down quickly I found that out of Psions 100+ reviews not a -single- one warranted a "1 - atrocious" rating. Kushner as well has not nailed anyone with a 1 rating. In Simon's 100+ reviews there are only about 3 or 4 atrocious ratings.
> 
> ...




Allow me to thoroughly disagree. There are a few important things you are missing.

*1)* First, nothing about the ENWorld rating system says that is is a ranking system that is strictly relative to one's peers. The only thing that says what the numbers mean is the short description on the review page:

*1 - Appalling
2 - Poor
3 - Average
4 - Good
5 - Superb*

Note that those ratings _*don't*_ say

*1 - Bottom 20 percentile
2 - 20th-40th percentile*
etc.

Nothing I have reviewed strikes me as "appalling." Nothing. That is not to say that there is not anything appalling out there. This leads me to point number 2.
*2)* You assume that all of the staff reivewers have reviewed a fair cross section of the material available out there. This is not the case.

There are a few factors at work here.

First, while some publishers do contact me, for the most part I solicit the publishers that I would like to see review material from. I wont solicit companies whose material I consider poor to begin with. As a result, I typically get material that is above the d20 average to begin with. I am in large showing you reviews of the companies that I consider the best d20 publishers.

Second, when publishers are rated badly, they tend to not want to send you review material anymore. There is not a company I have given a "2" to that sends me material anymore. Many publishers take it pretty hard when I give them a "3".
*3)* Third is the moving average. Material has significantly improved in quality over the last 2 years. But I try not to shift my standards as time goes by. When a person comes to the board and looks at ratings, those ratings have to stand up irrespective of when the review was posted.

*4)* The primary content of the reviews page is the reviews, not the numbers. Anyone can put up a poll of what people thought of product X or Y, but in the end, those are just subjective measures. The numbers don't tell you how the reviewer came to that conclusion, and the reviewer's criteria may be irrelevant for your puproses. The best way to determine what the value of a product is is to you is to...

(_*drum roll...*_)

*READ THE REVIEW!*


----------



## Psion (Oct 22, 2002)

> *Reviewers don't review "everything" -- they get to pick and choose what they review.  Maybe they stay away from the stinkers, whether consciously not. *




That would be "consciously". I have seen a few stinkers, but I certainly didn't squander my money on it, and if the companies responsible solicited me to do reviews, I would politely decline. I make enough that I can buy most RPG products with an hour or two's wages. Writing, editing, and submitting review take AT LEAST two hours (for a large product, more like four or six). Reviewing poor products is annoying to me and only adds a product that I didn't want in the first place to my collection. The motivation is just not there for me to do lots of reviews of bad products.

I do plan on doing a review of a stinker I picked up durring the ENnies (though still a 2, not a 1) but gratis items submitted for reviews come first.


----------



## WizarDru (Oct 22, 2002)

Tallarn said:
			
		

> *5*) A movie masterpiece, like LotR, Star Wars Ep4...films that are generally agreed to be incredible pieces of work.*




I would hasten to point out that the review of some films changes over time.  Star Wars (I refuse to call it episode 4, "A New Hope" or anything else) did not recieve universal praise when it first came out.  That it rose to cultural touchstone has affected reviews of it since, but it received it's share of bad reviews when it was new.


That said, I'd prefer a rating system that incoporates several categories, with or without a 'master' score.  An example is how some magazines review computer games: they rate them based on three to five categories, such as graphics, sound, replayability and so on.  

I don't much care for going to 1-10, as I think it will just skew the scores to a different point, without much benefit.  And technically, you have that now, if you include 1/2 stars.  Ultimately, the score should only be an invitation to read the review, and find out what the reasons behind the score were.  No simple number or collection of numbers is going to tell the story effectively enough.


----------



## HellHound (Oct 22, 2002)

*Re: Re: Review inflation on ENworld*



			
				Psion said:
			
		

> *
> Second, when publishers are rated badly, they tend to not want to send you review material anymore. There is not a company I have given a "2" to that sends me material anymore. Many publishers take it pretty hard when I give them a "3".
> *




Wow... harsh.

I know you have never given one of my products a "2" yet, but another staff reviewer has, but I certianly still have him in my distribution list (hint hint, Simon Collins, you know who you are... ). The best thing a company that gets a bad review can do is work hard to get a better review. That's why I still send out promotional copies even to people who have panned my owrk in the past. 

But there ARE exceptions. For example, we won't send out promotional copies of our material to several review sites after the Kid's Coloring Book fiasco. My two children wrote a book, paid to have a limited print run done *out of their own pockets*, and brought said book to GenCon. They gave away copies to their favourite game designers, and sold the rest. A lot of reviewers came up and asked for free copies to review, and they all got one. In fact, enough were given away that the girls lost money on the print run. 

That was August 8th - 11th and there are still NO reviews of this book up anywhere. One review site "lost" the copy they were given, and none of the others have been bothered to post these reviews.

It's one thing when a complimetary review copy is volunteered, but when a reviewer SOLICITS a review copy, especially from a 7 year old child, *they had damn well better actually review it*.

Wow... did I mention I am bitter about this? If these same two children hadn't have been part of the award-winning Portable Hole team, they would have walked away from the whole GenCon thing with a significant dislike for the whole game publishing scene.


----------



## Psion (Oct 22, 2002)

Dextra was saying something about this that had her up in arms on a thread on the d20 publishers forum. At the time, she didn't specify what the product or situation was, but now I understand. Yes, I would agree that sounds a bit aggravating.


----------



## Conaill (Oct 22, 2002)

I don't think we need to change the ratings system for reviews, but I do agree with tleilaxu that it would be useful to add a percentile to the score.

Likewise, I think we could add a percentile to indicate how this particular review compares to other reviews by the same reviewer. Yes, some reviewers give higher marks than others. One person's "2" may be another's "4".

No need to actually change any of the ratings or overhaul the whole system, mind you! Just a little more info on how a product compares to others and how a review compares to others by the same reviewer...


----------



## Henry (Oct 22, 2002)

First, Jason, I am terribly sorry to hear about this chain of events. Your kids definitely deserve better than that. I don't know if the publishers in question didn't feel that they book was worth reviewing, because of the age of the authors, of if there were other circumstances, but I am very disappointed to hear of the shoddy treatment. Frankly, it sucks.

Second, about the reviews: I think Eric and Psion put it best: You already HAVE your pool of potential one's and two's out there - all those products that people have consciously or not avoided. Frankly, the purpose of a review is to tell another person to either seek out or avoid a book that in your personal experience affected you. 

Most ENWorld Reviewers PAY for their books; only a select few house reviewers are on distribution lists of any appreciable size. The rest are amateur reviewers who have to go buy their books and then review them. all of the reviews I have posted to ENWorld (so far only 5 or 6, the products I have bought in the past year), I have given 3's or 4's to, because if they weren't good or interested me, I wouldn't have bought them. This shows up in my reviews, for people to see if the material will interest them. 

Ironically, the only review copy I EVER received for free, I had to post its review on RPGnet, because the review logins here were terrible at the time, and I couldn't log in to post a review if my life depended on it! Fortunately, the problems have been resolved, but the review was posted, nonetheless. If you ever read this, Hyrum Savage, I'm sorry it took so long to post a review at all!


----------



## Tsyr (Oct 23, 2002)

> Here is a question for the rest of you: Why does this always seem to happen, in grades, movie reviews, etcetera. A "C" should be the average grade in classes, but often B is the average. If someone gets an A in a class they should be in at least the top 10 if not 5 percent.




I'll explain to you very simply why that doesn't work: It doesn't actualy measure the persons (Or, in this case, the products) ability in and of itself.

As an example. Say you had a class that gave math tests, each one with 100 problems, 10 times a semester. Assume you have 50 students. 10 always get all the questions right. 10 always miss 1. 10 always miss 2. 10 always miss 3. And ten always miss four. (We are assuming this is an advanced class, I guess). 

Now, individualy, if someone said "I got 96% on all my tests this year!", you would think they did pretty well, right? I sure would.

But by your model, they would all fail and have to repeat the class. Despite having done good work. Simply because there was someone slightly better than them.

Now, granted, that's a fabricated example. But I do think the point it examines is valid. I mean, if you were to assume that only a "D" grade would let you pass a class, by nessesity, in your system, 40 percent of every class MUST fail, regardless of how well they did. 

Or, to shift back to reviews, if you reviewed ten really really good products... lets say... Wheel of Time, FRCS, Rokugan, Manual of the Planes, Monsternomicon, Kalamar players guide, Oriental Adventures, Traps and Treachery, the EQ RPG, and Spycraft... Of those, 2 MUST, no matter how good they are (assuming you are still going on a hard 1-5 scale) earn a 1. Likewise, if you reviewed 10 horrible products, two MUST earn a 5. This system is deceptive and doesn't really tell you anything. For example, if I see that Psion gave the Manual of the Planes a 1, I might think he hated it, when in fact he simply loved it less than other stuff. 

That's not really useful... it forces me to read the entire review to understand how he felt about it. And while I do make it a point of reading the reviews, sometimes I just want a quick baseline... which your system would not give me.


----------



## Ghostwind (Oct 23, 2002)

*Review ratings*



> *The primary content of the reviews page is the reviews, not the numbers. Anyone can put up a poll of what people thought of product X or Y, but in the end, those are just subjective measures. The numbers don't tell you how the reviewer came to that conclusion, and the reviewer's criteria may be irrelevant for your puproses. The best way to determine what the value of a product is is to you is to...
> 
> READ THE REVIEW! *




Well said, Psion. The numeric rating system should never be considered the definitive grade of a review, but rather a subjective measure that both complements the text of the review and provides a means of comparing similar prodcuts.  For example, if I was looking for a city book and was trying to decide between Freeport: CoA and Skraag, I could look at the review scores as a comparison. If Freeport was consistently rated high and Skraag wasn't rated as high, I would be inclined to lean towards Freeport especially after reading the reviews. That's not to say that Skraag was a bad book (it wasn't), but rather the scores can serve to guide me when looking at similar products. In the end it all boils down to one thing for every single consumer, "Can I use this in my campaign?" If the answer is yes, then reviews of a product can certainly sway a person to buy it. But if the answer is no, then it doesn't matter if the review got a 1 or a 5 because it probably will not be purchased. Period.


----------



## 2WS-Steve (Oct 23, 2002)

One nice thing that has happened recently is that we now have reviewers with track records and websites with editorial policies regarding their reviews. I'll pick up a movie in a video store if it got a thumbs-up from Ebert because I've done so in the past and seldom been disappointed. Likewise, if I notice a reviewer assigns similar grades as I would then I'll give a product I otherwise might have passed over a try if the reviewer scores it well.

The main way to build this faith is for a reviewer to have a large number of reviews, a consistent rating system, and good write informative reviews. 

A reviewer that gets a stamp of approval from a respected entity, such as becoming a staff or affiliate reviewer here or becoming a print reviewer for Dragon magazine also makes me more confident in their opinion. This process of editorial review of internet reviewers is new; many websites still seem to let just anyone post reviews and that decreases the value of those reviews (at least to me). I still look at those reviews but I don't treat them any differently than what someone might post to a message board; unless they've got a strong track record *and* I know about that record.


----------



## Crothian (Oct 23, 2002)

2WS-Steve said:
			
		

> *A reviewer that gets a stamp of approval from a respected entity, such as becoming a staff or affiliate reviewer here or becoming a print reviewer for Dragon magazine also makes me more confident in their opinion. This process of editorial review of internet reviewers is new; many websites still seem to let just anyone post reviews and that decreases the value of those reviews (at least to me). I still look at those reviews but I don't treat them any differently than what someone might post to a message board; unless they've got a strong track record *and* I know about that record. *




There are a few good fan reviewers we have here.  Plus, since many of us post on these message boards as well as review, you get a greater feel for them, more so then just through their reviews.


----------



## ced1106 (Oct 24, 2002)

tleilaxu said:
			
		

> *So here is what I think: if there are only 3-4 atrocious ranks, there should only be 3-4 amazing ranks. that would surely make them stick out of the crowd much more. As it is, the rankings for publishers are absolutely clogged between 3 and 4.5 while the low numbers are virtually untouched. Spread it out a bit.
> 
> Opinions, thoughts, rants, flames? *




Heh. Just yesterday, I was rebutting with the words, "EnWorld rating inflation" on another message board.

Myself, I rate product versus all other products I've experienced (and I've rifled through 20 years of it!), not just the products I've reviewed (as an average rating of 3 on a review site would imply). Likewise (we yammered this on RPGnet before), a "volunteer" reviewer (ie. reviewing product he purchased and didn't receive comp), would, more likely than not, only bother to review products they feel are good but not noticed, (although some would also slam overly hyped stuff as well). As for the staff reviewer who's reviewed a hundred or so comparable products, I **would** expect an average review of, well, average. I'm wrong, of course, since even the solicitation of product introduces a preference for good product. (Heck, that's what **I** do.)


Cedric.
aka. Washu! ^O^


----------

