# A rant about D20 Magazine Rack



## JacktheRabbit

I like to go into the review section on ocassion and just tool around. I use it often tohelp  determine my purchases. I also go by suggestions from friends or from reading the book myself when I get it.

Well what a surprise.

D20 Magazine Rack has flooded the reviews again. What a surprise 21 reviews by D20 MR and what is the range? 19 of them got 4's and two of them got 3's.

This got me curious and I did some quick checking.

D20 MR has done 185 reviews so far.

117 of them have been 4's.

33 of them have been 3's

32 of them have been 5's

3 of them were 2's


What does this tell me? Absolutely nothing. The reviews give equal ratings to products that I have found (and my friends have found) to be nowhere close to the rating that D20 MR has given out.

To make things worse D20 MR is not one person but several people who all post reviews under the same name. So now I cannot quickly check reviews and base them on what I know about the reviewer.

Can someone PLEASE create unique logins for each D20 MR reviewer so we do not get blasted by huge swaths of reviews that tell very little and rank almost every product they touch (149 out of 185) as being good or better.

My apologies for this rant.


----------



## blackshirt5

I've gotta second this one, Doc.

They give good reviews to a lot of things that are, frankly...sugar-blasted cat crap.


----------



## JacktheRabbit

blackshirt5 said:
			
		

> *I've gotta second this one, Doc.
> 
> They give good reviews to a lot of things that are, frankly...sugar-blasted cat crap. *




It is also obvious that different editors play favorites in their reviews.


----------



## blackshirt5

Playing Favorites?  Yeah, Just a Skosh.

What do we like?  Answer: Anything remotely D20.

Do you think that maybe they had that one reasonable guy, who used to say "but it's a garbage book!" and they just ignored him and shooed him away and belittled him until they drove him insane, and now he carries around a Rifts PHB, stroking it, petting it, muttering, "my precious, yessss"?


----------



## Iron_Chef

*that site is a total scam!!!*

I went to the site looking for reviews but all I found were quickie press releases that told me absolutely nothing about the alleged quality of a given product that the publisher didn't want me to know. 

d20 Magazine Rack is worse than biased... it's *useless!* 

Down with the duplicitous corporate complicity of d20 Magazine Rack! 

Up with ENWorld!


----------



## DanMcS

And to make it worse, they're just using it as an excuse to spam for their website.  I despise that.


----------



## JeffB

The thing that really turned me off was when one of the D20MR reviewers complained of the Kingdoms of Kalamar Atlas  being too specific about Kalamar...HUH? 

I just totally ignore D20 MR reviews now.


----------



## JacktheRabbit

JeffB said:
			
		

> *The thing that really turned me off was when one of the D20MR reviewers complained of the Kingdoms of Kalamar Atlas  being too specific about Kalamar...HUH?
> 
> I just totally ignore D20 MR reviews now. *





I thought you were joking so I checked. Sadly you are not joking.


----------



## Olgar Shiverstone

Let me start off by saying that I'm biased, since I write reviews for d20 Magazine Rack.  (And I'm respondind to a potential troll, but I'm wearing my nomex undies.)

As I and others have posted on this board many times before, the d20 Magazine Rack has a different rating system for reviews than EN World.  Instead of a 5-star score, we use a set of five scores plus an overall grade.  That rating system does not translate well to EN World's system.  Most reviews will translate as "4" scores.  If you want to get the full details you have to read the review.

Incidentally, if you want to know exactly how d20 Magazine Rack does it's reviews, you can go here, where the exact review requirements are spelled out.  Those are the procedures d20MR reviewers are expected to follow -- and I'm not aware of any other gaming review site that has so clearly -- and publicly -- posted their standards.

For the poster that mentioned that all they see are press releases, try *clicking* on the title, so you can get to the page with the full review, including the numeric score.  You can search for old reviews/products with the search function at the upper right, or use the "Critic's Corner", "Fast Tracks", and Gamer's Corner" links for long products, short products, and non-d20 games respectively.

I can't overstate enough that if all you're doing is judging a product or review by its score (whether d20MR score or EN World star rating), you are doing neither the reviewer or the product justice.  Read the entire review.

I can't speak for all of the (8-10) staff reviewers at d20 MR, but if you have issue with any of my reviews, or think my reviews show a bias, then I encourage you to post comments to the review at d20MR's site, or on d20 MR's boards.


----------



## blackshirt5

Olgar Shiverstone said-
As I and others have posted on this board many times before, the d20 Magazine Rack has a different rating system for reviews than EN World. Instead of a 5-star score, we use a set of five scores plus an overall grade. That rating system does not translate well to EN World's system. Most reviews will translate as "4" scores. If you want to get the full details you have to read the review.

Me-
Then, why do you insist on bringing them over to ENWorld, if you've already got them on another site, and you yourself say that the ratings system doesn't translate well to the ENWorld one?


----------



## JacktheRabbit

Olgar Shiverstone said:
			
		

> *Let me start off by saying that I'm biased, since I write reviews for d20 Magazine Rack.  (And I'm respondind to a potential troll, but I'm wearing my nomex undies.)
> 
> <snip>
> 
> I can't speak for all of the (8-10) staff reviewers at d20 MR, but if you have issue with any of my reviews, or think my reviews show a bias, then I encourage you to post comments to the review at d20MR's site, or on d20 MR's boards. *




I do not have issue nor do I give a rats behind about reviews on D20's site. My issue is with reviews transplanted to ENWorld that are often shallow, inaccurate, and or biased.

You post the reviews on this forum then you open yourself up to comments on this forum. We shouldnt have to go to your playground to complain about the plethora of poor reviews.


BTW. This may seem obvious to me but maybe it isnt so I should give it to you as a suggestion.

IF YOUR REVIEW PROCESS DOESNT TRANSLATE WELL INTO ENWORLDS REVIEW SYSTEM THEN MAYBE YOU SHOULDNT PORT THE REVIEW OVER INTO ENWORLD.

You are doing no one any favors at all by posting grades that by your own admission are not accurate.


----------



## Olgar Shiverstone

blackshirt5 said:
			
		

> *Me-
> Then, why do you insist on bringing them over to ENWorld, if you've already got them on another site, and you yourself say that the ratings system doesn't translate well to the ENWorld one? *




Not my policy -- I don't run the site; I just write the reviews -- hopefully Ghostwind will be along to clarify.

As I understand it, it's a deal worked out with Morrus to increase the number and variety of reviews an EN World.  I fail to see how having a greater variety of reviews is a bad thing.

If you don't like/agree with the quality of a review, fine -- you can certainly post comments related to the review here, though if you want it to have some impact, making your concerns visible at d20 MR is a more effective way to go about it.

Reviewers are volunteers; d20 MR is providing you with reviews for free at multiple locations, and provides 8 free d20 e-zines to boot.  You don't have to be grateful, but if you want your concerns addressed, you'll have a greater impact providing specific details in a more appropriate forum.


----------



## drnuncheon

Olgar Shiverstone said:
			
		

> *As I and others have posted on this board many times before, the d20 Magazine Rack has a different rating system for reviews than EN World.  Instead of a 5-star score, we use a set of five scores plus an overall grade.  That rating system does not translate well to EN World's system.  Most reviews will translate as "4" scores. *




Perhaps you should look into a different method of translating your ratings to the ENWorld rating, then.

Edit: Having read the review scores, I see no reason why they would not translate well into the ENWorld system.  One glaring problem I see is that guidelines are not given for what constitutes a '2' or a '1'!

J


----------



## blackshirt5

Olgar Shiverstone said:
			
		

> *
> As I understand it, it's a deal worked out with Morrus to increase the number and variety of reviews an EN World.  I fail to see how having a greater variety of reviews is a bad thing.
> *




When your reviews don't make any sense, and you make EVERYTHING out to be good, it's a bad thing.


----------



## shouit

Okay, I am going to put my two cents here...

First of all, I read reviews from all different sites to get their opinions, but to be honest, I have never seen a review which said the product totally sucked at least not in a numerical value.  Even here, I have very rarely seen either Simon or Psion give a review numerical value less then a 3, even though in the review they say it wasnt very good.  Heck, I even read a review Psion did, cannot remember the one exactly, where he basically said the product blew chuncks, but still gave it a 2 rating.. Numerical values on something like a review, is very difficult to do.  I listen to Psions reviews for the most part, actually reading them, not quick peek at the number.  

Second, Olgar has a point, if you are going to bash something, which I may or may not agree with, at least do it where it can be heard by the people that do it.  It does no good to talk about an error on another web site then the one that the error is on.

Anyways, sorry, just wanted to drop my two cents in..


----------



## JeffB

While I'm not really that concerned and/or irrate about the whole thing, I have to agree that if the reviews don't translate well to the ENWORLD scoring system then either they should stop being put up on Enworld, or there should be some kind of "disclaimer" at the beginning of each D20 MR review posted at ENworld. That just makes common sense, if you are not using the same guidelines, does it not?

That being said, anyone who complains about a campaign world sourcebook being too specific about the campaign world should really not be writing reviews.


That Darn Living Greyhawk GAz..it's just got too much...too much GREYHAWK in it....So I have to give it a 3 instead of a 5....


----------



## Olgar Shiverstone

JeffB said:
			
		

> *While I'm not really that concerned and/or irrate about the whole thing, I have to agree that if the reviews don't translate well to the ENWORLD scoring system then either they should stop being put up on Enworld, or there should be some kind of "disclaimer" at the beginning of each D20 MR review posted at ENworld. That just makes common sense, if you are not using the same guidelines, does it not?*




That's a good suggestion.  There is currently a disclaimer with a link to the scores at d20MR, but it's at the bottom.



> *
> That being said, anyone who complains about a campaign world sourcebook being too specific about the campaign world should really not be writing reviews.
> *




Perhaps -- the quote did seem a bit off -- but consider your _entire_ audience.  The gamers here at EN World are pretty astute, but you may get people looking at a product for the first time who have absolutely no idea what the product is.  Maybe they've never heard of Kalamar.  You've got to try and write a review that covers a wide variety of expectations from novice gamers to d20 experts -- which isn't an easy thing to do.  If I'm shopping for stuff to port to my own game world, knowing that I'm going to have a hard time of it because too many of the mechanics are inextricably linked to the original setting's assumptions might be something I'd want to know.


----------



## Flexor the Mighty!

I have been pretty much ignoring thier reviews as I can nearly always guess what they will score something w/o even knowing what the product is.


----------



## HellHound

I have to agree with Olgar in just about every respect.

Yes it is true that the review scores posted by d20 Magazine Rack are high compared to the baseline, but it has been stated repeatedly that the LEAST important aspect of a review is the star-rating. 

The reviews themselves are long and comprehensive, far from the "Spam" that they are being described as in this thread.

Additional reviews, especially comprehensive ones, are a boon to this review database, not a problem. If you don't like the style of the D20MR reviews, then don't read them. Just like posts to these message boards... if you don't like them, no one is forcing you to read them.


----------



## JacktheRabbit

HellHound said:
			
		

> *I have to agree with Olgar in just about every respect.
> 
> Yes it is true that the review scores posted by d20 Magazine Rack are high compared to the baseline, but it has been stated repeatedly that the LEAST important aspect of a review is the star-rating.
> 
> The reviews themselves are long and comprehensive, far from the "Spam" that they are being described as in this thread.
> 
> Additional reviews, especially comprehensive ones, are a boon to this review database, not a problem. If you don't like the style of the D20MR reviews, then don't read them. Just like posts to these message boards... if you don't like them, no one is forcing you to read them. *




Those scores do though affect a companies overal rating and can affect the top 20 D20 list which people use to find what is considered the best product out there.


----------



## Dragongirl

DocMoriartty said:
			
		

> *Those scores do though affect a companies overal rating and can affect the top 20 D20 list which people use to find what is considered the best product out there. *



Well if they give everyone basically the same score then it won't affect the top 20 since their reviews would pretty much cancel each other out and it will still depend on other reveiws.


----------



## Ghostwind

I usually try to stay out of potentially trolling insults, but when it involves an attack on myself, my site, or my staff, then you can bet I will get involved. 

Now to set the record straight for everyone to understand, when Morrus and I agreed to include the d20 MR as an affiliate reviewer for EN World, one of the significant reasons behind it was to help him build up his database of reviews. Morrus and I discussed the difference in my grading system versus the more restrictive one here at EN World. We came to the conclusion that it could probably still work out since the average 5 pt grading system (5-A, 4-B, 3-C, 2-D, 1-F) would come close to my grading system. We both had concerns but felt they would work themselves out. To my knowledge, the d20 MR is the only website that has an open and public posting of exactly how our reviews are scored. This has been done so that there is no misunderstanding or misconception on our grading system. All a person has to do is sit down and read it if they have a question.

At the bottom of every review that is posted on these boards is a line that directs readers who want to see our scoring back to the original d20 MR site. Furthermore, at the top of every review (except for ones that are over eight months old) there is the name of the reviewer himself. Why is this a problem? To address Doc's issue, if you don't want to read a certain person's reviews then go to the main d20MR site where you can easily see the name. I am not about to create a new account each time I have a staff change since I am the one who posts and hand codes every review that is posted here. My staff sends all reviews to me where they are then posted to my site and then eventually here. To go to all of the extra work of logging in with separate accounts just to keep one or two people happy isn't going to happen.

Another issue that has been brought up is that we are nothing more than the mouthpieces of the industry because of the scores posted here. That's utter bull crap! I have said this time and time again and will likely say it even more in the future. *Read the review, not just look at the score! A review represents the opinion of that person about that product.* Not every reviewer will score a product the same way, even among my own staff.

JeffB, your biggest beef is with the review of the Kalamar Atlas that I wrote. Yes, I cracked on it for being specific, however I marked it down for the retail cost vs. its overall useability. It's damn expensive for a book of maps, plain and simple. While we are talking about this, I find it very interesting that, except for the rarest of occasion, the only time comments are left are when folks really disagree with a reviewer (regardless of whether they are my staff's or not). If a review is scored a 3 or lower, you can almost make a bet that someone will be there to chastise the reviewer (especially if the publisher is well liked). You don't see that with reviews that score higher unless it represents a contrast to the bulk of the other reviews of that product.

Personally, I do not care if you do not like the reviews that my staff faithfully volunteer their own time to write. If you don't like them, don't read them. It's that simple. No one is twisting your arm and forcing you to sit there and read it. But I never want to hear my staff being referred to in a derogatory manner because you may disagree with their findings. They work hard and I am very appreciative of that fact. Everyone has their own style of gaming and that fact will surface in the way reviews are written. You may disagree with it, but do so in a polite manner rather than coming across as an inconsiderate whiner complaining about the merits of the numeric score. Unlike some reviewers on the Net, my staff tries to not only look at a product without bias but they also actively look for the good qualities and how they may apply in an overall bigger picture. If a product is bad, they will state so. But they will also say why it is bad, which is more than I can say for a lot of other reviewers.

I also want to note that at no point during the tenure of d20 MR's affiliation with EN World has Morrus ever contacted me to voice a concern or problem regarding our reviews or the scoring of them. Don't you think that if he felt it was a problem, he would have contacted me long before now? If Morrus feels it is a huge issue, then I would have absolutely no problem with discontinuing our affiliate relationship and having him delete all 185 reviews. I'm sure that would please several of you, but I also ask who's next? Do you single out Gamewyrd or trancejeremy or Crothian because you start to disagree with their scores. The EN World reviews are an open review forum where anyone can post a review. Yes, there are times when a moderator steps in to ask for a modification or even deletes it, but those are rare. You cannot be a community and then start trying to exclude people because you don't like the scores they give. To do so defeats the spirit of what EN World stands for.

Don't feel a company is worthy of its average rating? Write a few reviews yourself giving them the scores you feel they deserve. Being a reviewer is much harder than most folks realize when you take into account how much time is spent writing one. Some feel that our presence on these review boards may unfairly skew the top 20 list or a company's rating. Since both scores come from an average, our reviews will really have little effect unless no one else reviews those products.

Now if anyone takes offense at this rebuttal to the charges leveled at me or my site, feel free to take it off-forum and email me privately. I stand behind my review staff and the reviews they write and will continue to do so. If you don't like them, don't read them.


----------



## Gizzard

> Those scores do though affect a companies overal rating and can affect the top 20 D20 list which people use to find what is considered the best product out there.




At this point the Top 20 list is overgrown and increasingly useless.  

Since it never gets culled, products with small and fanatically loyal followings tend to rise to the top over time.  I've complained about this before - is Call of Cthulhu the "best" game d20 game system, or is it just that all 6 people who bought it wrote positive reviews of it here on the boards?  

I'm not knocking CoC; it probably is a fine game.   But I think its wrong to suggest that some reviewers aren't welcome because they'll mess-up this already messed-up List system.  If anything, I think even a few extra reviews could balance out the fanboy driven "5 reviews, all are 5.0" syndrome.


----------



## JeffB

Ghost,
As I said, I'm not really concerned. Also that particular review (the KoK Atlas) is to my recollection the ONLY time I have made a comment about an Enworld review, because in that case I felt it was pretty obvious that knocking a campaign setting book for it's campaign setting content was just plain silly.

So yeah that's my beef. I think it's poor reviewing. But I really don't care one way or the other. Like I said that was enough to make me just ignore the D20 MR reviews which you stated we should do, anyways.  So we are in agreement. I don't hold any grudges or malice about it. Nor were any of my comments "personal". I was not calling for that review to be removed in those postings like some other folks were (in fact I stated the opposite) nor was I downgrading the entire D20MR staff in this one, simply stating that in that particular review, I felt the product was poorly judged. I'm all for fair reviews, but common sense tells me in that review, it was not "fair". We will just have to agree to disagree on that matter.


----------



## Psion

JeffB said:
			
		

> *While I'm not really that concerned and/or irrate about the whole thing, I have to agree that if the reviews don't translate well to the ENWORLD scoring system then either they should stop being put up on Enworld*




Eh. While I agree that d20 MR's ratings are a bit generous, I don't agree with this notion and think that some people bank WAY too much on the scoring system and don't think it should be the basis of removal of some creidible and insightful reviews.

That, perhaps, d20 MR needs to be split into its component reviewers is a bit more of a pressable point though.


----------



## Olive

i just read the KOK Atlas review over atd20MR, not over at d20reviews. I think it's a fair comment actually, when translated into the d20MR category of playability. 

It's pretty much only usable for kalamar, which limits it's usefullness for non-Kalamar reviers. Fair enough call to me... especially given that it says outright it's a beautiful and well made book.


----------



## Olive

just a wee suggestion, maybe d20MR should calculate the final score in two ways: one taking OGM into consideration, and one NOT taking OGM into consideration. That would be interesting, and more useful to me.


----------



## Crothian

I like their reviews and I think it will be a big loss for EN World if they got pulled.  Ghostwind, you and your guys do good work and I like that you cross post them to EN World.  I can understand people's frustration, but to me its frustratyion in the 1-5 rating system, a system that most people think should be mostly ignored at least until one reads the review.


----------



## Morrus

Wow - what a cuffuffle!

The D20 Mag Rack reviews are well-written and complete, and I feel that they certainly add to the review collection as a whole.  That said, looking at it, I do agree that perhaps we may need to have a look at the two scoring systems and see if we can't figure something out.

Sometime in the next few months, I intend to revamp EN World's scoring system (certianly not anything as complex as D20 Mag Rack's, but less coarse than the current 1-5 rating).  This will be a while, though, as there are a lot of other complicated technical things going on in the background at the moment which you'll all find out about as the months progress.

For the moment, I'd ask you to put this issue aside; I'll talk to Ghostwind at some point once I get to playing with EN World' scoring system, and see what can be done.

For the record, though - please try to concentrate on the reviews themselves rather than the scores.  The scores are only a *very* rough guide to overall impressions.

One suggestion that I have, which seems obvious to me.  Ghostwind, instead of trying to "translate" between two incompatible scoring systems, why not score each review according to the system where the review is found?  So, on your own site, it will have one score, and at EN World it will have another.  That seems to make sense to me.


----------



## Psion

Olive said:
			
		

> *i just read the KOK Atlas review over atd20MR, not over at d20reviews. I think it's a fair comment actually, when translated into the d20MR category of playability. *




No, I really don't agree. I find the idea that a setting specific supplement (especially a maps book, for goodness sake!) should be required to be useful outside of the setting for which it was intended ludicrous. It may be worth _mentioning_, but should not play into the final score.


----------



## Henry

Why the heck do people value a rating score over the actual review, anyway? The review tells you the "meat and potatoes" of the actual product anyway, and one can usually tell from what a good reviewer says whether the product will be useful to you. If the review itself is honest and factual, then one shouldn't be surprised with the contents of the d20 product itself.

Speaking of which, I need to get into this "review game" myself, but I consistently have trouble logging in on the reviews section itself.


----------



## JacktheRabbit

Psion said:
			
		

> *
> 
> Eh. While I agree that d20 MR's ratings are a bit generous, I don't agree with this notion and think that some people bank WAY too much on the scoring system and don't think it should be the basis of removal of some creidible and insightful reviews.
> 
> That, perhaps, d20 MR needs to be split into its component reviewers is a bit more of a pressable point though. *





I have posted MANY times as comments to individual reviews that D20 MR needs to break their reviews into different logins. I have been ignored every time and now when I bring it up I am accused of being a troll.

Gee, wonder why I dont want to go to THEIR forum to make my complaints.


----------



## JacktheRabbit

Henry said:
			
		

> *Why the heck do people value a rating score over the actual review, anyway?  *




Simple.

How many people want to wade through every review reading it carefully to determine if that product might be what they want.

I read some reviews and when I get a feel for that reviewer I can make some rough estimates based on seeing what rating they give a product. 

D20 MR kills this by giving a vast majority of products a "4" while at the same time lumping a dozen or so reviewers (all of course with their own biases and preferences) under the same review name.


----------



## JacktheRabbit

Ghostwind said:
			
		

> *I usually try to stay out of potentially trolling insults, but when it involves an attack on myself, my site, or my staff, then you can bet I will get involved.
> 
> Don't feel a company is worthy of its average rating? Write a few reviews yourself giving them the scores you feel they deserve. Being a reviewer is much harder than most folks realize when you take into account how much time is spent writing one. Some feel that our presence on these review boards may unfairly skew the top 20 list or a company's rating. Since both scores come from an average, our reviews will really have little effect unless no one else reviews those products.
> *




First you call me a troll.

Then you suggest people do something that is very specifically AGAINST the rules of product reviews. The rules very clearly state that a review is not to be put up if you are only doing it to offset what you think is a poor review from somene else.


----------



## Ghostwind

DocMoriartty said:
			
		

> *I read some reviews and when I get a feel for that reviewer I can make some rough estimates based on seeing what rating they give a product.
> 
> D20 MR kills this by giving a vast majority of products a "4" while at the same time lumping a dozen or so reviewers (all of course with their own biases and preferences) under the same review name. *




The name of the reviewer is the first thing thing on the review itself. Are you saying that merely hitting the "Back" button after seeing a name of a person you don't like is too much effort? 



> *First you call me a troll.
> 
> Then you suggest people do something that is very specifically AGAINST the rules of product reviews. The rules very clearly state that a review is not to be put up if you are only doing it to offset what you think is a poor review from somene else.*




First off, I said "potentially trolling". I did not call anyone a troll. Secondly, when the title of the thread is "a rant about D20 Magazine Rack" it is a reasonable assumption that unkind things may be said. Thirdly, I am not suggesting you write a review to offset the score, I am saying that if you honestly feel it is a bad review you are more than welcome to post your own with the score YOU feel it deserves. I have seen on more than one occasion where certain individuals scream for the deletion of a review just because it doesn't meet their own conception of the quality of the product. EN World reviews are a community forum. You don't like the style of the reviews being presented? Then encourage more members to get involved and write reviews of their own. After all, the more reviews that are posted from a diverse selection of members, the better sampling you have of a product's overall quality.


----------



## Wulf Ratbane

Henry said:
			
		

> *If the review itself is honest and factual, then one shouldn't be surprised with the contents of the d20 product itself.*




Agreed!

Now if only all of the professional reviewers were attentive and diligent about their fact checking...  


Wulf


----------



## Henry

DocMoriartty said:
			
		

> *How many people want to wade through every review reading it carefully to determine if that product might be what they want.
> *




Only those who want to be absolutely certain that the product they buy is one that they will use.

In my experience, no reviewer hits a mark 100 percent of the time, especially in their estimation of a product. The most important ability of a reviewer is to factually report the content that is there, and how useful said content would be for not only their style of campaign, but others' as well. I've never seen a score system, not even d20MR's, that can accurately reflect the usefulness of a product. However, seeing a reviewer take the time for explicit descritpion of the types of material included gives the reader the knowledge they need for a product.

A score of "4" or "2" or "5" does not automatically tell me that a product focuses on warrior-type rpestige classes, or that the classes in question are very high-powered compared to the classes in the core rules; or that the publisher's new rules on warfare are very skimpy and do not give you enough to fully stat out an army and play engagements. Only be reading the review itself does the reader get a true feel for the product on a case-by-case basis. It does not even take reading pages of reviews: Just reading one or two fully detailed reviews can give the reader the opposing opinions needed to determine if such a product is right for him or her.


----------



## Darkness

Henry & others:

If the number ratings don't count for much, and can be freely ignored (or randomly assigned) by reviewers and readers, why use them at all?


----------



## JacktheRabbit

Wulf Ratbane said:
			
		

> *
> 
> Agreed!
> 
> Now if only all of the professional reviewers were attentive and diligent about their fact checking...
> 
> 
> Wulf *




Yeah, then maybe we won't see books on dwarves get plinked in reviews because they are too dwarvish.


----------



## Wulf Ratbane

DocMoriartty said:
			
		

> *
> 
> Yeah, then maybe we won't see books on dwarves get plinked in reviews because they are too dwarvish. *




Heh... Well, that's not what I'm talkin about but, yeah... I suppose naming the series DWARVES, HALF-ORCS, ELVES, etc. could be a bit misleading to the buyer...

Seriously though, I think (I hope) that after three books, most of the reviewers are on board with what the series is all about. 

I just hate to see a reviewer pull a fact to support a particular criticism, _and then proceed to get the facts wrong_. Drives me up the wall.


Wulf


----------



## blackshirt5

DocMoriartty said:
			
		

> *
> 
> Yeah, then maybe we won't see books on dwarves get plinked in reviews because they are too dwarvish. *




PLEASE tell me you're joking Doc.


----------



## JeffB

Psion said:
			
		

> *
> 
> ....I find the idea that a setting specific supplement (especially a maps book, for goodness sake!) should be required to be useful outside of the setting for which it was intended ludicrous. It may be worth mentioning, but should not play into the final score. *




Exactly what I was trying to convey Alan, thanks for doing it much better than I did.


----------



## JacktheRabbit

blackshirt5 said:
			
		

> *
> 
> PLEASE tell me you're joking Doc. *




Not at all. Go read the review of Heroes of High Favor: Dwarves from Bad Axe Games as done by D20 MR.

The book is knocked for being too typical. I guess the reviewer would have prefered Aquatic Dwarves and other pulled out of your butt ideas. 

Or maybe the book just came out too close to the same time Helms and Hammers by GR came out for the reviewers taste. That book since it is from GR of course had to rate a 5 from D20 MR.


----------



## Darkness

DocMoriartty said:
			
		

> *Or maybe the book just came out too close to the same time Helms and Hammers by GR came out for the reviewers taste. That book since it is from GR of course had to rate a 5 from D20 MR. *



Was it the same D20 MR reviewer for both books?


----------



## Ghostwind

I'll save everyone the trouble of searching for the reviews:

Heroes of High Favor: Dwarves

Hammer & Helm


----------



## JacktheRabbit

Darkness said:
			
		

> *Was it the same D20 MR reviewer for both books? *





Contrary to what D20 MR has said those two reviews (HOHF and H&H) are reviews that do NOT list who the specific review is.

The links provided to D20 MR' sites reviews show that they are two different reviewers.

Of course it interesting that he did not post up links to the reviews here on ENWorld. Those are the reviews we are talking about.


----------



## blackshirt5

It was the same reviewer.


----------



## JacktheRabbit

blackshirt5 said:
			
		

> *It was the same reviewer. *




Yep and a very poor example of how to review a product he uses one product as an example of what the other product should have been.

Very poor review standards if you ask me. Each and every review MUST be independent of other products.

Anything else is just beign a fanboy.


----------



## Wulf Ratbane

Ghostwind said:
			
		

> *I'll save everyone the trouble of searching for the reviews:
> 
> Heroes of High Favor: Dwarves
> 
> Hammer & Helm *




Thanks for the link, Steve. I hadn't actually read the other review from your site.

I have never hidden the fact that I take issue with your rather subjective review of Dwarves. I think any reasonably honest look at your review would find it extremely light on supporting information and facts. 

Opinions such as "I can’t help feeling that there was a rush to get this book done quickly..." seem particularly out of place, to me, for a professional review. I would expect that to be backed up with facts: specific examples of poor editing, poor layout, poor game balance, etc. I need facts such as those in order to improve my work.

But, honestly, man to man here (not publisher to reviewer, just two gamers talkin here), that specific review reads like a hack job. 

I happily note that your review at the rack differs in at least one significant way from the review you posted at ENworld.

Wulf


----------



## JacktheRabbit

> Overall, Hammer & Helm is a fantastic book that *breathes new life into some old stereotypes*. All of the game mechanics and rules are balanced and well written. A great deal of thought and effort went into this book and it shows. I will be very interested to see if Paradigm’s forthcoming giants book is equally well done. If you *love dwarves and are looking for something new*, then by all means buy this book.





Ok, I cannot refrain from commenting.

This right here shows the problem with the "reviews".

Dwarves do not exist. They are a fictional creation in DnD pretty much copied straight out of Tolkien. A dwarf is a dwarf because he is short, carries an axe, and has a bad attitude.

Anything else is just a short humanoid. If you love dwarves but want something new then run a halfling or a gnome. A dwarf is a dwarf because of those old "sterortypes".


----------



## blackshirt5

I somewhat agree.  While I think that it's appropriate to compare and contrast products(for example, if I were to review the Quintessential Paladin once I get it, I'd surely draw comparisons between QP and Defenders of the Faith, because they're both Paladin splats).  What I don't think is appropriate is saying things like "This is what Heroes of High Favor: Dwarves should have been" is appropriate in the Hammer and Helm review.

Also, I still contend that lowering something's score(such as the 3.5 both books got because they're "primarily restricted to dwarves") when reviewing a race or class splatbook is wrong.


----------



## Psion

Darkness said:
			
		

> *If the number ratings don't count for much, and can be freely ignored (or randomly assigned) by reviewers and readers, why use them at all? *




I see it as a bell curve sort of thing. Different people use different criteria for evaluating a product. Sometimes those criteria are not credible, and I try to weed out such instances, but you are still going to see some disagreement about how good a given product is.

You take several such products, you sort of have a ballpark that gives you SOME indication of the quality of the product, but we quite simply don't have enough reviews to make that number rigorous. But it a fair quick guide as can be had.

However, my point is that regardless of the aggregate opinion, the value to a specific reader may vary wildly because the readers have their only criteria and needs. The only way to truly tell if a product will be useful to YOU specifically is by reading the review and getting an idea of what is in there and if the reviewers criteria match yours.

So, in short, the numbers are not totally worthless, but their use is limited and there comes a point that the effort taken to try to make it more accurate will not really acheive much given the inherent statistical limits to how much they can convey. If you want a measure more accurate than what the numbers can convey, you are well advised to read the review and (possibly) indentify reviewers who pay attention to the factors YOU think are important. (Which is why I could see supporting the idea of splitting up the d20 MR into the actual reviewers.)


----------



## blackshirt5

Psion, do you think a poll would be in order, on whether or not D20 MR should split up into it's different reviewers, for the reader's benefits?  I think we should let the community decide.


----------



## Ghostwind

Doc, if you were paying attention to my initial reply, I did say that there are older reviews that do not carry a name. Those were done by me. 

You will also note in both postings of HoHF, that I put a note stating I modifed the score after talking to Ben (Wulf) Durbin about some factual errors I made in calculations and his intent behind the series. I still maintain that the overall book could have been much better. However, I have been pleased with improvements in the series and felt that Elves and Half-Orcs were quite good and have even told Ben this on a couple of occasions. How many times does it need to be said that a review represents an opinion? There are many occasions where I disagree with the reviews of others (especially regarding my own works, which is natural) but I still respect the right of that reviewer to have his opinion regardless of the outcome.

You are digging up a review that was done nearly one year ago to illustrate a point that, for some reason, my site is not fit to have reviews posted on EN World. Whether it is me, personally that you have a dislike for, or my site in general, I couldn't care less. We'll both agree that neither of us will ever see eye to eye regarding reviews.

The fact remains that Morrus runs the site and relies on his moderators to help police it if the reviews don't meet with the standards he has established. Until ownership of the site changes or Morrus and I agree to discontinue our arrangement, the d20 MR will continue to maintain a cross posting presence on EN World. If that is unacceptable, then take it up with Morrus.


----------



## Psion

blackshirt5 said:
			
		

> *Also, I still contend that lowering something's score(such as the 3.5 both books got because they're "primarily restricted to dwarves") when reviewing a race or class splatbook is wrong. *




You present an interesting conundrum, one which I am not willing to say is right one way or the other.

I agree that if you are writing a book about Dwarves, then requiring the book to be applicable beyond dwarves may be a bit much to ask.

That said, if it is, does it not increase the utility of the product if it is? For example, _Plot & Poison_ has alternative magic item creation and power component rules I can use anywhere; would that not be more useful than a book of Drow alone?

My personal take is "give points for broad support, but don't dock for lack of it." But I don't really object to one method or the other when considering if a review should stand.


----------



## Psion

blackshirt5 said:
			
		

> *Psion, do you think a poll would be in order, on whether or not D20 MR should split up into it's different reviewers, for the reader's benefits?*




(shrug)
I don't know that it's that big of a deal that it needs to go up for a poll. I imagine (and I am totally guessing here, as I have not discussed the issue with Morrus and Mr. Creech) is that we will do it if it is practical (segregating the reviews will take time, possibly requiring volunteers), otherwise not.


----------



## Ghostwind

Since the HOHF review is causing such an issue, I ask the moderators to delete it.

Secondly, there will be no need for a poll concerning different logins because I will not do it. The website is the affiliate reviewer, not each individual member of my staff. The name is posted at the top of "nearly" every review (with the exception of the older ones). There is no reason why a person cannot simply hit the back button if he or she sees it is written by someone that is disliked.

The amount of work that is spent on the d20 MR is considerable, as is the time I spend cross-posting reviews. To have to continually switch accounts when posting reviews will be too time consuming. I believe having the name of the reviewer at the top of the page should be sufficient. Other than a very small handful of people (with Doc being at the forefront), no one has openly had an issue with this.


----------



## Wulf Ratbane

Ghostwind said:
			
		

> *I still maintain that the overall book could have been much better.*




A fair criticism, and I have worked to improve each book since, based in large part on specific feedback from reviews. 

I think it is fair to say that a publisher should be open to specific criticism from a professional reviewer-- but a professional reviewer should be open to criticism from the public he serves, too.

Your review of _DWARVES_, specifically, spoke volumes more about you as a reviewer than it did about the book you were reviewing. Your reviews have grown since then, as I hope the books in my series have.

Just to comment on the larger issue of the thread, I don't have a problem with the way d20MR reviews are posted here. I'd resist any additional work on Steve's part (splitting up the reviews) that drove his reviews away and resulted in fewer ENworld reviews overall. 

The number system in general, I can do without-- I always read the whole review. But I have to admit-- the number system draws my eye to certain reviews. I like to scan for those rare 5's (and yes, those equally rare 3's or lower) to see what warranted the shift from the ubiquitous 4 rating. 

Wulf


----------



## Ghostwind

Wulf-

Drop me an email, the address I have for you isn't valid apparently.

Thanks.


----------



## whtknt

*D20 M.R. reviews*

I am a d20 M.R. reviewer. Each week, I donate (dare I say, generously donate?) several hours of my free time to review the latest d20 products. Often this time investment cuts into personal time with my wife, my lunch hour, or time I could spend working on other things. I do not get paid for this and I am not a professional reviewer. I provide this time willingly because I'd like to think that I am helping people make informed decisions about the latest d20 products.

When I review a product, I try to do so with a critical eye. I do not allow the publisher or the author's credentials to sway my thinking. I try to provide the best and most objective review possible, though if I really don't like something, I'll tell you that. It just so happens that a lot of the products that I've been reviewing lately have been high-quality. A review is someone's opinion. Yours may vary greatly from mine. I tend to disagree with about 75% of everything that Ebert and Roper have to say, but that doesn't make them bad reviewers.

I assumed that because I wasn't getting any feedback here, that my reviews were being received fondly. Apparently, it's because people have stopped reading any review from d20 M.R. If you disagree with me, post it, either here or at the d20 Magazine Rack and I will respond. I'll tell you why I feel the way I do about something. I don't promise you'll like the answer, but I will address your concerns. If I don't hear from you, I assume that I'm doing a good job.

Finally, we come to the bottom line. If you think you can do a better job, then why don't you give up *your* free time to produce a review? It's not as easy as everyone seems to think. Reviewing is a big investment of time and effort, not to mention money. When you just look at the score and offhandedly discard the review, you're doing yourself and the reviewer a grave injustice.


----------



## Piratecat

Steve, Wulf's address on the email link below his post is the correct one.


----------



## Ghostwind

Thanks, PC. Got it corrected in my address book.


----------



## Wulf Ratbane

Ghostwind said:
			
		

> *Thanks, PC. Got it corrected in my address book. *




But if you want to reach me during the work day-- indeed, before tomorrow afternoon really, since I'll be out late tonite playing in Piratecat's game! -- then I recommend using the ENworld private message system.


Wulf


----------



## Olgar Shiverstone

I appreciate hearing the support from Morrus, Psion, Wulf, and the others who have expressed it.  It's nice to have our work appreciated some.

A note on HOHF reviews:

I have to admit to being a big fan of D&D archetypes/stereotypes; I enjoy the HOHF books, and I do own them all so far.  

That said I do think that it's important to include in the review the fact that they are based on the stereotype, since someone down the road will be reading HOHF: Halflings as his first book in the series and may not know that.  EN Worlders are pretty astute and may all know what assumptions underlie Wulf's books, but I do think a review has to be written from a point of view that doesn't assume that the reader knows everything about the book, its philosophy, or its publisher.  

Some people don't want gaming books that play to type (I'm not one of them, but they're out there), and they ought to be able to pick up a review and know whether or not that's the case.

I've noted in at least one of my reviews that what we call a "Critical Hit" to one reader might be called a "Critical Miss" by another; the reviewer has to make a call and that falls to his own opinion and judgment.  Someone's bound to disagree -- but I hope we get it right more often than not.


----------



## Olgar Shiverstone

*Re: D20 M.R. reviews*



			
				whtknt said:
			
		

> *When you just look at the score and offhandedly discard the review, you're doing yourself and the reviewer a grave injustice. *




Worse, you're doing the writer and publisher an injustice too.

I find myself buying more and a greater variety of d20 products since I have writing reviews than I did before.  Go figure.


----------



## Yeoman

I'll freely admit I generally go by the number rating, simply because I don't have time to go into the details of each and every product review. I'll take a glance at the numbers to see what others are thinking and then take a look at the product for myself, and decide for myself then.


----------



## diaglo

*Re: Re: D20 M.R. reviews*



			
				Olgar Shiverstone said:
			
		

> *I find myself buying more and a greater variety of d20 products since I have writing reviews than I did before.  Go figure. *




you could just borrow mine. or listen to my endless review/rant about what's wrong with the youth of today.


----------



## Henry

Darkness said:
			
		

> *If the number ratings don't count for much, and can be freely ignored (or randomly assigned) by reviewers and readers, why use them at all? *




Bingo!


----------



## Dragongirl

Henry said:
			
		

> *Bingo! *



 

IMO As long as you keep letting people assign a score to reviews that will encourage "some" people to do padding reviews or fanboys.  I think it would be much better to get rid of the numbers, get rid of the top 20 lists and let the actual reviews stand for themselves.

_steps down from soapbox_


----------



## trancejeremy

> IMO As long as you keep letting people assign a score to reviews that will encourage "some" people to do padding reviews or fanboys. I think it would be much better to get rid of the numbers, get rid of the top 20 lists and let the actual reviews stand for themselves.




I don't think the average person is the problem (or even the "fanboy"). I've looked over the average ratings of reviewers, and most range in the 3s, if they've done more than a few reviews. It's only the d20 Magazine Racks people that always give good ratings. They're the problem, IMHO.

They say they have all these guidelines, but in practice, it almost always results in a 4 when translated over here...

Which is also why I think splitting up the d20 Magazine racks into individual reviewers is pointless, because the net result is they all give 4s.

Personally, I think their reviews should be dumped, and simply post a link to the d20 Magazine reviews site, because the way it works now doesn't really benefit anyone. The poor way their scores translate over makes the d20 Magazine people look bad (unfairly), it gives d20 buyers the wrong impression, and it reflects badly on ENworld, giving it even more of a reputation as fanboy central.

Really, I think what's needed here is getting more average gamers to write reviews, by giving them an incentive. Maybe have a contest for people who write reviews - winner gets a free Natural d20 PDF (also would give you more reviews of those products, which presumably help sell more of them...)


----------



## The Sigil

blackshirt5 said:
			
		

> *I somewhat agree.  While I think that it's appropriate to compare and contrast products(for example, if I were to review the Quintessential Paladin once I get it, I'd surely draw comparisons between QP and Defenders of the Faith, because they're both Paladin splats).  What I don't think is appropriate is saying things like "This is what Heroes of High Favor: Dwarves should have been" is appropriate in the Hammer and Helm review.
> 
> Also, I still contend that lowering something's score(such as the 3.5 both books got because they're "primarily restricted to dwarves") when reviewing a race or class splatbook is wrong. *



Agree on both points.

IMO, products can and SHOULD be compared to other products, with the following factors weighted in (in approximately this order):

1.) How much they overlap on subject matter.

2.) How close their release dates were to one another.  Later products should have "learned" from mistakes in prior products - including those put out by other publishers.

3.) Price/size

For example, Hammer & Helm, the Quintessential Dwarf, and HoHF: Dwarves all came out at almost exactly the same time.  That means I have every right to compare them, bearing in mind that HoHF is shorter and cheaper than the other two, and so should be able to have less total "stuff" in it without being penalized.  H&H and QD, being nearly identical in size and price, should easily be comparable.  That doesn't mean I should look for something in H&H and try to find the same thing in QD to see whose is "better."  It means I should try to compare the total breadth of options offered, in diversity and utility, and compare that breadth - and if there ARE areas where they happen to try to do the same thing, only then should I try to figure out which is "better" - but keep that within the confines of "assuming I use all the other rules in the book it came from only."  IOW, I can't say, "mix and match X, Y, and Z from QD with A, B, and C from H&H."  It does mean I can say, "X in QD was done as B in H&H and I liked X (or B) better."

However, I should take more care when comparing, say, Path of Faith to Defenders of the Faith.  The price is somewhat comparable, but the fact that Path of Faith had an extra year or so to "sit back and watch" the evolution of the d20 world means I should hold the material therein to a higher standard than DoF.  PoF should have learned from DoF's mistakes.

Similarly, I should not hold the Creature Collection (first edition, not revised edition) to the same standard as the Fiend Folio... because the FF has had three years to "learn from the mistakes" of the CC and other works.

I should not really compare Alchemy & Herbalists with Darwin's World as the subject matter is different.

Lowering a score based on lack of overall applicability should only be done when it's not fairly clear that the book is directed at one specific niche.  While I shouldn't penalize the Quint. Paladin for not giving me options for evil characters, I might well penalize, say, "Bob's Super Guide to Prestige Classes Various and Sundry - Suitable for Everyone" for having almost no PrCs for non-magic-users and zero PrCs that a character of NG alignment could take (an extreme example, to be sure LOL).

--The Sigil


----------



## whtknt

*Did you ever stop to think...*

Has it occured to anyone that if all you're reading is the number, the problem isn't with us, it's with you? I do this out of my love for the hobby and the faint hope that someone, somewhere finds my thoughts helpful. Since all you read are the numbers, obviously, that hope is wasted here. You all seem to think it's easy to write a good review, but have you tried it?

A review is an opinion and I try to offer the best and least biased opinion that I can. I don't bash something because it's a Wizards product, nor do I offer praise to a product just because it's from Bastion Press (a company I've grown much fonder of in recent months). I try to present a good picture of the product while picking out the good and the bad points. Sometimes I do better than others. That's because some weeks, I just don't have as much time. See, I work a full-time job, support a family, and try to fit in one night a week for recreation.

You all talk real big, but when I don't see feedback, I tend to assume that everything is alright. If you disagree with me, present a logical argument to my review in the space provided. I'll respond and I'll support my reasoning or else I'll offer an apology if you spotted something I missed. You can make the reviews better by simply putting your two cent's worth in. But instead of doing that, you'd rather openly and publicly flame us.


----------



## GuardianLurker

Ghostwind, and all the other d20 MR reviewers - despite the guff you've been getting, I've found your reviews generally helpful (when I read them). There have been a few times when your reviews have been the only ones available for a recent product I was interested in, and in those cases, I always received enough information to make my purchasing decision.

And really, what more can you ask for from a review than that?

However ....


> *
> 
> D20 MR has done 185 reviews so far.
> 
> 117 of them have been 4's.
> 
> 33 of them have been 3's
> 
> 32 of them have been 5's
> 
> 3 of them were 2's
> 
> *




After reading d20 MR's reviewing standards, I feel fairly safe in saying this is due to a compressed and inflated scoring scheme that actually centers around the 4 point mark. From d20 MR's standards page (which is full of very useful advice, but did NOT have a readily apparent link from the most recent Airships review) :



> _
> Scoring (as it applies to the d20 Magazine Rack)
> Open Game Content: The scoring on this is a little subjective but use this as a guide...
> *5.0 - The entire contents except for graphic elements are ogc.
> *4.5-4.0 - Most of the contents except for an occasional section of single chapter are ogc.
> *3.5-3.75 - Only the game mechanics themselves are ogc. Spell names, monster names, class names, etc. are closed.
> *<3.0 - There is very little game mechanics present (less than 5% of the book).
> *0.0 - No open content (Wizards of the Coast, Kenzer, Arthaus, and other licensed settings from Wotc).
> 
> d20 Compliance:
> *5.0 - Everything follows proper game mechanics including skill points, feats, game balance and other actual rule functions.
> *4.0 - There are a few minor errors (less than 10) or only one major error.
> *3.0 - A significant number of minor errors or few major errors exist.
> *<3.0 - The book has serious game mechanics and rules issues.
> 
> Originality:
> -This is a judgment call. Most scores will fall into the 4.0-5.0 category unless it is something that has been done time and time again (such as a new version of a Star Trek game).
> 
> Playability:
> *5.0 - Useable by everyone, player and GM.
> *4.0 - Primarily either player or GM, but still somewhat useable by the other.
> *3.0 - Either player only or GM only. Or devoted solely to one particular class or race.
> *<3.0 - Very restrictive as to who can gain the most use from it.
> 
> Value for the dollar:
> -Again, a judgment call.
> 
> *5.0 - Good value for all.
> *4.0 - Reasonable investment but better if on sale.
> *3.0 - Only purchase at full price if you truly want or need it, otherwise discount sales only.
> *<3.0 - You have to really want this book, because it's overpriced for what it offers. _




I think a better system might be:
<table>
<tr>
<th>Category\Rating</th>
<th>0.0</th>
<th>1.0</th>
<th>2.0</th>
<th>3.0</th>
<th>4.0</th>
<th>5.0</th>
</tr>
<tr>
<td>OGC</td>
<td>No open content (Wizards of the Coast, Kenzer, Arthaus, and other licensed settings from Wotc).</td>
<td>There is very little game mechanics present (less than 25% of the book).</td>
<td>Large portions of the book, especially the most innovative and/or useful parts, are NOT OGC.</td>
<td>Only the game mechanics themselves are ogc. Spell names, monster names, class names, etc. are closed. (This is currently the deFacto standard in the industry.)</td>
<td>Most of the contents except for an occasional section of single chapter are ogc.</td>
<td>The entire contents except for graphic elements are ogc.</td>
</tr>
<tr>
<td>d20 Compliance</td>
<td>Is not a d20 product.</td>
<td>Is for an ancestor of the d20 gaming system (1st or 2nd Edition ADnD).</td>
<td>The book has serious game mechanics and rules issues that actually interfere with game play.</td>
<td>The book has serious game mechanics and rules issues, but these do not interfere with gameplay, or are easily compensated for.</td>
<td>There are a few minor errors (less than 10) or only one major error.</td>
<td>Everything follows proper game mechanics including skill points, feats, game balance and other actual rule functions.</td>
</tr>
<tr>
<td>Originality</td>
<td>Not applicable</td>
<td>An unupdated reprint of a previously issued product.</td>
<td>An updated version of a previously issued product. E.g. A 3e revision of the Basic DnD module B1</td>
<td>A new product that uses all the standard tropes and cliches of the genre in an unimaginative manner</td>
<td>A new product that uses some of the standard tropes and cliches of the genre in an imaginative manner</td>
<td>A new product that uses few, if any, of the standard tropes and cliches of the genre.</td>
</tr>
<tr>
<td>Playability</td>
<td>Not applicable</td>
<td>The product is extremely setting-specific, or is otherwise very difficult to incorporate into a game. It requires extensive adapation before use.</td>
<td>The product is setting-specific, or is otherwise difficult to incorporate into a game. It may require extensive adapation before use.</td>
<td>The product can be incorporated into any game with a reasonable amount of preparation.</td>
<td>The product can be incorporated into any game with a minimum of preparation. Most "drop-in" adventures will fall here.</td>
<td>The product can be incorporated seamlessly into any game with no preparation at all.</td>
</tr>
<tr>
<td>Value for the Dollar</td>
<td>Not applicable</td>
<td>You have to really want this book, because it's overpriced for what it offers.</td>
<td>Only purchase at full price if you truly want or need it, otherwise discount sales only.</td>
<td>Reasonable investment but better if on sale.</td>
<td>Good value for all.</td>
<td>Well-worth the price.</td>
</tr>
</table>


----------



## Dragongirl

*Re: Re: A rant about D20 Magazine Rack*

Attempting to make that visible.



			
				GuardianLurker said:
			
		

> *I think a better system might be:<table><tr><td><font color="#FFE078">Category\Rating</font></td><td><font color="#FFE078">0.0</font></td><td><font color="#FFE078">1.0</font></td><td><font color="#FFE078">2.0</font></td><td><font color="#FFE078">3.0</font></td><td><font color="#FFE078">4.0</font></td><td><font color="#FFE078">5.0</font></td></tr><tr><td><font color="#FFE078">OGC</font></td><td><font color="#FFE078">No open content (Wizards of the Coast, Kenzer, Arthaus, and other licensed settings from Wotc).</font></td>
> <td><font color="#FFE078">There is very little game mechanics present (less than 25% of the book).</font></td>
> <td><font color="#FFE078">Large portions of the book, especially the most innovative and/or useful parts, are NOT OGC.</font></td>
> <td><font color="#FFE078">Only the game mechanics themselves are ogc. Spell names, monster names, class names, etc. are closed. (This is currently the deFacto standard in the industry.)</font></td>
> <td><font color="#FFE078">Most of the contents except for an occasional section of single chapter are ogc.</font></td>
> <td><font color="#FFE078">The entire contents except for graphic elements are ogc.</font></td>
> </tr>
> <tr>
> <td><font color="#FFE078">d20 Compliance</font></td>
> <td><font color="#FFE078">Is not a d20 product.</font></td>
> <td><font color="#FFE078">Is for an ancestor of the d20 gaming system (1st or 2nd Edition ADnD).</font></td>
> <td><font color="#FFE078">The book has serious game mechanics and rules issues that actually interfere with game play.</font></td>
> <td><font color="#FFE078">The book has serious game mechanics and rules issues, but these do not interfere with gameplay, or are easily compensated for.</font></td>
> <td><font color="#FFE078">There are a few minor errors (less than 10) or only one major error.</font></td>
> <td><font color="#FFE078">Everything follows proper game mechanics including skill points, feats, game balance and other actual rule functions.</font></td>
> </tr>
> <tr>
> <td><font color="#FFE078">Originality</font></td>
> <td><font color="#FFE078">Not applicable</font></td>
> <td><font color="#FFE078">An unupdated reprint of a previously issued product.</font></td>
> <td><font color="#FFE078">An updated version of a previously issued product. E.g. A 3e revision of the Basic DnD module B1</font></td>
> <td><font color="#FFE078">A new product that uses all the standard tropes and cliches of the genre in an unimaginative manner</font></td>
> <td><font color="#FFE078">A new product that uses some of the standard tropes and cliches of the genre in an imaginative manner</font></td>
> <td><font color="#FFE078">A new product that uses few, if any, of the standard tropes and cliches of the genre.</font></td>
> </tr>
> <tr>
> <td><font color="#FFE078">Playability</font></td>
> <td><font color="#FFE078">Not applicable</font></td>
> <td><font color="#FFE078">The product is extremely setting-specific, or is otherwise very difficult to incorporate into a game. It requires extensive adapation before use.</font></td>
> <td><font color="#FFE078">The product is setting-specific, or is otherwise difficult to incorporate into a game. It may require extensive adapation before use.</font></td>
> <td><font color="#FFE078">The product can be incorporated into any game with a reasonable amount of preparation.</font></td>
> <td><font color="#FFE078">The product can be incorporated into any game with a minimum of preparation. Most "drop-in" adventures will fall here.</font></td>
> <td><font color="#FFE078">The product can be incorporated seamlessly into any game with no preparation at all.</font></td>
> </tr>
> <tr>
> <td><font color="#FFE078">Value for the Dollar</font></td>
> <td><font color="#FFE078">Not applicable</font></td>
> <td><font color="#FFE078">You have to really want this book, because it's overpriced for what it offers.</font></td>
> <td><font color="#FFE078">Only purchase at full price if you truly want or need it, otherwise discount sales only.</font></td>
> <td><font color="#FFE078">Reasonable investment but better if on sale.</font></td>
> <td><font color="#FFE078">Good value for all.</font></td>
> <td><font color="#FFE078">Well-worth the price.</font></td>
> </tr>
> </table> *


----------



## Ghostwind

GuardianLurker, you offer some good suggestions and I will consider them. Thank you for the input.


----------



## GuardianLurker

Dragongirl, Ghostwind : Thanks.


----------



## Morrus

Yeah, I see the problem.  A D20MR score of 3 is a BAD score, equivalent to a 2 (or maybe a 1) at EN World.  Their score range is realistically from 3-5, not from 1-5.

Which is absolutely fine.  The problem is you can't just move that score over to another place which uses a different system.  To take the example to the extreme (and, yes, I know it's silly), what if EN World used a % score?  You wouldn't still give it a score of 4 then - that'd be daft.

So the very simple solution is this - when D20MR posts a review at EN World, it uses EN World's scoring system, not D20MR's.  On their own site, the same review uses D20 MR's own scoring system.

Thus, a score of 3 translates to a 1 or 2; a score of 4 translates to a 3; a score of 5 translates to a 4 or 5.  They don't translate directly across.

The system shown above is essentially a reworking of D20MR's system into a 1-5 scale instead of a 3-5 scale, which would translate across to EN World just fine.  Well, nearly - that above is 0-5, so it's slightly different.


----------



## citizengames

*Standards*

D20 Magazine Rack is one of the few zines that have adopted standards for their reviewers. Every review must meet these standards in order to be published. They were nice enough to share their standards with the manufacturers and I was impressed at the lengths that they went to in order to provide a fair and balanced review. 

Regards,

Rob Stone
President
Citizen Games LLC


----------



## TempesTMR

Well I must say this came out of the blue!

I seldom come to the ENworld forums. Not that they are bad, actually pretty fun, but when you get this many people, insicurities pop-up, and they are better just avoided. Although I can always find something i'm looking for, good job there!

But then things like this pop up. Kinda odd really. You guys really don't seam to notice how hard it can actually be for reviewers to do a good job. And you are not even paying attention to what the reviews say (which is everything) you look at the score. Which is totally pointless. People like numbers, which is why they are there, but they are not the review, what is typed out, and described is the review. If you don't have the time to read them, then I'm sorry, but your complaints here seem useless on my stand point.

Reviewers are in a no-win situation. If they give low scores, they are flamed. High scores and they are biased. Average scores, and people complain cause there are to many average scores. This makes no sense to me, but preach your own I guess.

Morrus said it best above. The d20MR scores do not translate the best. And apperantly to please you all, they should. I'm not sure why, it must be peoples fascination of numbers, but you should read the review, that is the heart of everything. From most of what I have seen, I think we have one of the best scoring techniques available. By just looking at the score of each section you can get a good idea of what the book is, compared to a single number. But with 5 different area's, it's almost a gaurantee most will end up with an average score, that's just the way it works out. Somebody mentioned the whole "bell curve" above.

So I will leave it with a few simple idea's. Most importantly read the reviews, not the numbers, that is obvious. If you don't like the reviews, don't read them. And if you think you can do better, then start writing reviews, but don't get mad if we call you biased for high scores, or wrong on low scores. Maybe even be upset if most of them are average.

And to open a thread, on another site, flaming a completely different site, without even mentioning it to the site you are flaming, asking questions, or e-mailing somebody? That is just, well, ...rude to put it nicely.


----------



## Morrus

TempesTMR said:
			
		

> *Morrus said it best above. The d20MR scores do not translate the best. And apperantly to please you all, they should. I'm not sure why, it must be peoples fascination of numbers, but you should read the review, that is the heart of everything. From most of what I have seen, I think we have one of the best scoring techniques available. By just looking at the score of each section you can get a good idea of what the book is, compared to a single number. But with 5 different area's, it's almost a gaurantee most will end up with an average score, that's just the way it works out. Somebody mentioned the whole "bell curve" above.
> 
> So I will leave it with a few simple idea's. Most importantly read the reviews, not the numbers, that is obvious. If you don't like the reviews, don't read them. And if you think you can do better, then start writing reviews, but don't get mad if we call you biased for high scores, or wrong on low scores. Maybe even be upset if most of them are average. *




Sorta.  I agree that people shouldn't put too much weight on the scores.  However, given that people do notice them (and yes, I'm sure they read the reviews too - this hyperbole about just looking at scores and not reading reviews is exaggeration in the extreme), and given that the "fix" is so easy and effortless, why not adopt it?  

Keep the same scoring system at D20MR but when posting a review to EN World, use EN World's scoring system.  That way it all makes sense wherever you see the review, and it's so easy to do, and keeps everyone happy 

Anyway, this thread is starting to get hostile.  I think it's run it's course.  I'll leave it open in case someone has something new and constructive to add, but I'm keeping an eye on it.


----------



## Khur

*U Can Ree-VU 2*

I think everyone is taking this way too seriously. (Did _I_ just say that?) Let's lighten up.



			
				DocMoriartty said:
			
		

> *Yep and a very poor example of how to review a product he uses one product as an example of what the other product should have been.
> 
> Very poor review standards if you ask me. Each and every review MUST be independent of other products.
> 
> Anything else is just beign a fanboy. *



 I'm not defending the specific review with my comments, but do you mean to say anyone comparing one product on the same subject to another product on the same subject is a fanboy? Strange. I, for one, want to know which product on a specific subject is the better for my limited gaming budget. In a non-gaming example, isn't it pertinent to know which sports car is better if one can only afford one or the other? If one wants an instruction manual on how to build a radio, isn't it important to know which manual best suits the project? If one wants to build a custom dwarf and toss him into play, isn't it good to know which book serves better? Of course, one has to be careful about comparing apples to apples, or noting significant differences between items (like price, page count, and so on). It is valid to compare similar products, and valuable.

The major problem with comparing _Hammer & Helm_ to _Heroes of High Favor: Dwarves_ is that one book is so much bigger and more expensive than the other. It could be argued that you just can't make a _Hammer & Helm_ out of such a small book as _HoHF: Dwarves_.




			
				DocMoriartty said:
			
		

> *Dwarves do not exist. They are a fictional creation in DnD pretty much copied straight out of Tolkien. A dwarf is a dwarf because he is short, carries an axe, and has a bad attitude.
> 
> Anything else is just a short humanoid. If you love dwarves but want something new then run a halfling or a gnome. A dwarf is a dwarf because of those old "sterortypes". *



 Just for fun here, but dwarves do exist--they are persons suffering from a genetic disorder called dwarfism. They have unusually short torsos and limbs and are distinct from midgets in that they are usually atypically proportioned. Midgets are also wee folk, but tend to be proportional for their size.

As for the fictional creation, D&D probably did lift them from Tolkien, but Tolkien lifted them from Scandinavian myth, a subject on which he was quite extensively studied. I'd also like to note that some Tolkien dwarves neither carried axes nor had bad attitudes.  Scandinavian dwarves, on the other hand, tend to get stoned by sunlight.

Your second statement is a bit strange. Do you mean to say there's no room for innovation even in a stereotype? I say those who want to really play the stereotype should play Discworld dwarves! Now there's a living stereotype (and that Terry Pratchett is a hoot!).

All seriousness aside, though,  a dwarf is not a dwarf because of stereotypes. That's like saying all Americans are the same. There's plenty of room for innovation, cultural exploration, and hairy, muscular lumps of crunchy goodness in the dwarf race, as Jesse Decker and Benjamin Durbin proved in both their works. Are all the dwarves played in your game exactly the same? If so, it sounds _extremely_ boring.

Perfected dwarven roleplaying:







Belonging to the online comic PVP Online, of which I am a fan, but not the artist! Therefore it's (c) 2003 Scott R. Kurtz!

Anyway, I see no reason for the reviews to change. A look at many of them shows they're in line with the general opinion on this site. If that's true, what's the problem?


----------



## JacktheRabbit

TempesTMR said:
			
		

> *Well I must say this came out of the blue!
> 
> 
> And to open a thread, on another site, flaming a completely different site, without even mentioning it to the site you are flaming, asking questions, or e-mailing somebody? That is just, well, ...rude to put it nicely. *




Rude would be jumping in on a topic you do not know anything about and saying something as stupid as this.

The reviews are here on ENWorld. The scores granted are on ENWorld.

Why would I go open a topic at D20 MR about a complaint I have about something they are doing at EN World? That is just dumb. I make the complaint HERE because HERE is where it needs to be fixed. I do not frequent D20 MR's site so I really do not care what they do over there. I do frequent this site on a regular basis and it is content here on this site that made me start this discussion.


----------



## diaglo

TempesTMR said:
			
		

> *...insicurities pop-up...pop up...seam...seem...apperantly...peoples...gaurantee*




please tell me you review your reviews before you post them.




> * If you don't like the reviews, don't read them.*



  how do i do this?


----------



## Ghostwind

Alright, I've had some sleep and time to think without my emotions interfering (stupid things - they tend to get in the way). The biggest problem I see is the perception of exactly what the numeric score represents. To me, it's a matter of straight scale. 5 is an A, 4 is a B, 3 is a C, 2 is a D, and 1 equates an F. Reading some of Psion's reviews (which likely is the standard everyone is judging us against), that seems to back up this line of thinking (Feats C+ =3, Sanctuary C+ =3, Beyond the Walls B- =4, Hallowed Might B =4, Necromancer's Legacy A- =5, and so forth). Previously, I had been trying to combine both the final d20 MR numeric score and the letter grade our reviewer assigns. Sometimes it is an easy match (i.e. both equal a B), other times it is not (score indicates a C, but reviewer gives it a B).

What I will do is strictly use the letter grade the reviewer assigns since that represents the common ground of both systems. What does that mean for scores here on EN World? It means you will continue to see 4's and 5's, most likely. Face it people, the level of quality is definitely better than it was even one year ago. Books are going to rate higher because the product is better. 

So now you have to ask yourselves, is the issue here really the so-called "bias" that my staff exhibits by awarding higher scores or is it because the thought of seeing scores that award quality is too much because it somehow makes the review database less legitimate? It is entirely possible to give a book a good overall score and yet be critical of one or two points. The individuality of every person (including reviewers) means that the interpretation and assessment of the products are going to vary. Just because I like Airships doesn't mean you will. Likewise, just because you like HOHF: Dwarves doesn't mean I will or do. 

As I said before, maybe those dissatisfied with what they are seeing in terms of the scores should make a concerted effort to get more people involved in review writing. If 25 people write a review of _Feats_ by AEG and all but one find it good enough to rate a 4, then there is a undeniable concensus that it is a good book. However, if one person rates it a 4 while everyone else gives it a 3 or lower, then that one person sees something in the book that stands out for him that others do not. The majority would indicate that the book is fair in quality, but that one person sees it as a good product. 

The review is an opinion of that product as written by the reviewer himself. Some reviews are more critical than others and likewise, some reviewers are more critical than others. This is universal regardless of where the review may be posted or read. This is why it is so important to read the review itself rather than making a snap judgment based upon the score of the product. I ask everyone who has openly criticized myself and my staff about the reviews here, have you actually read all 185 reviews or are you simply looking at the score and judging us because of that? I will be the first to say that not every review is Pulitzer material, especially some of the older ones from a year or so ago. But I'll also say that my current staff represents some of the best reviewers around.

Going back to Doc's very first post, I wonder if the issue is the number of reviews that get posted at once? 


> *D20 Magazine Rack has flooded the reviews again. What a surprise 21 reviews by D20 MR and what is the range? 19 of them got 4's and two of them got 3's.
> 
> This got me curious and I did some quick checking.
> 
> D20 MR has done 185 reviews so far.
> 
> 117 of them have been 4's.
> 
> 33 of them have been 3's
> 
> 32 of them have been 5's
> 
> 3 of them were 2's
> 
> What does this tell me? Absolutely nothing. The reviews give equal ratings to products that I have found (and my friends have found) to be nowhere close to the rating that D20 MR has given out.*



If you go into the database and look at the Full Publisher List that breaks things down by product and then lists the reviewers and the scores of that particular product, for the most part, our reviews are in line with everyone else's.

For example:
*Undead by AEG (Simon 3, d20MR 3, Psion 4)
*Toolbox by AEG (Psion 4, d20MR 4)
*Oathbound by Bastion Press (Simon 3, d20MR 4, Gamewyrd 5)
*Path of Magic by FFG (d20MR 5, Simon 4, Sigil 4, Crothian 4, shadeus 5)
*Wrath & Rage by Green Ronin (Illuminati 5, d20MR 4, Simon 3, Madfox 4, trancejeremy 4, Psion 4, Gamewyrd 4, JoeGKushner 3)
*Fury in the Wastelands by Kenzer (d20MR 4, Psion 4, Simon 4, JoeGKushner 4, outlaw 5)
*Broadsides! by L.I. (d20MR 4, Psion 4, Simon 4, voadam 4)
*Beyond the Walls by Mystic Eye (d20MR 4, Psion 4, Simon 3, Gamewyrd 3)
*Book of Vile Darkness by Wotc (Gamewyrd 3, d20MR 4, Sigil 1, Henry@home 4, Isida 4, Crothian 4)

Now I ask, how are we unfairly biased in our scoring? It looks like we are fairly close to others opinions on most products. Look at the big picture of all reviews of that product before saying we are nothing more than the "mouthpieces of the industry".

Finally, if you disagree with a review because you don't believe it represents the score YOU think it should have, is it fair to ostracize the reviewer into not writing anymore reviews? It happens quite a bit around here from what I have seen.  New reviews are being posted by people who are at least making the attempt and they are often targeted for deletion because they come across as either "fanboy" or too short. Why not create a quick little "form comment letter" that can be stuck in the comments pointing out some of the essential points a review should include such as an assessment of both good and bad points? You could also set a minimum word count both in the letter and on the d20reviews main page. Both would give a sense of trying to be helpful rather than "this isn't a review and we should delete it immediately." Review writing isn't easy and not everyone can do it successfully on the first few attempts. Work with them rather than smacking them down.


----------



## Psion

> *To me, it's a matter of straight scale. 5 is an A, 4 is a B, 3 is a C, 2 is a D, and 1 equates an F. Reading some of Psion's reviews (which likely is the standard everyone is judging us against), that seems to back up this line of thinking (Feats C+ =3, Sanctuary C+ =3, Beyond the Walls B- =4, Hallowed Might B =4, Necromancer's Legacy A- =5, and so forth).*




Heh... never would have guessed that my little rating comment come under such scrutiny. 

At any rate, yes, that is how I see the 1-5 scale. Some people seem to be under the impression that the statistical average of scores on ENWorld should be 3, and that is _just not practical_. Only the harshest of professors in college that I knew would inflict a quota on grades handed out.

By showing the equivalence to the standard grading system, I hope to relate to a scale people are more familiar with; some people think that a 3 is a "bad" or "failing" review; I see it as a passing grade. Some people howl angrily if you give a product that they like a 3. I resist such characterization because I need _room to be discerning_.

However, that is sort of a secondary effect. The primary reason that I threw some + and - grades scores in is that a lot of times, I waffle about what to give a product, and I want the audience to be able to see that.


----------



## Morrus

DocMoriartty said:
			
		

> *
> 
> Rude would be jumping in on a topic you do not know anything about and saying something as stupid as this.*




Did you read what I said earlier?  If not, go back and read it now.  

Somehow I knew it would be you who couldn't resist having the last word.  Keep it nice, please.


----------



## JacktheRabbit

Morrus said:
			
		

> *
> 
> Did you read what I said earlier?  If not, go back and read it now.
> 
> Somehow I knew it would be you who couldn't resist having the last word.  Keep it nice, please. *




Sorry, but after about the 5th time I get told that I am being rude or trolling for having this discussion here (where the damn reviews I have a problem with are located to begin with) I start to lose patience.

I don't give a rats behind what D20 MR does on its own site. My problem is their porting over reviews and not bothering to create a proper rating for the product.


----------



## Khur

*Sweet!*



			
				Ghostwind said:
			
		

> *Alright, I've had some sleep and time to think without my emotions....*



 Ah, the sweet voice of sanity and reasoned analysis. Ambrosia! This one post did more to diffuse the problem here than any other three posts, especially that bit about the grading being on par with everyone else's. Excellent!

Well, now that that's cleared up....


----------



## Ghostwind

Psion said:
			
		

> *Heh... never would have guessed that my little rating comment come under such scrutiny.
> 
> At any rate, yes, that is how I see the 1-5 scale. Some people seem to be under the impression that the statistical average of scores on ENWorld should be 3, and that is just not practical. *




We are in agreement on this then. For example, the numeric score of a 4 may represent a letter score ranging from a B+ to a B-, thus providing the leeway to be flexible.



> *By showing the equivalence to the standard grading system, I hope to relate to a scale people are more familiar with; some people think that a 3 is a "bad" or "failing" review; I see it as a passing grade. Some people howl angrily if you give a product that they like a 3. I resist such characterization because I need room to be discerning.*




Perhaps the inherent problem lies with the words attached to each score. The score of 3 indicating "average" creates a more unfavorable image than the word "good", yet both can mean the same under the right circumstances. Maybe the words associated with each score could be modified to something along the lines of "5-superb, 4-excellent, 3-satisfactory, 2-problematic, 1-appalling"? Or perhaps the words could be replaced with the simple letter grade associated with that score: "5-A, 4-B, 3-C, 2-D, 1-F"?

Then has been more than one discussion about how the current EN World review scoring system has its problems and perhaps something as simple as a slight change in labeling may shift the whole perceptive balance about what the scores represent.


----------



## TempesTMR

diaglo said:
			
		

> *
> 
> please tell me you review your reviews before you post them.*




I have written like maybe 3 reviews . .and that's what editors are for! Unless you meant the post, then no, I did not review it very well obviously! 

At three in the morning I'm about whooped, and could have cared less if I spell the same word 12 different ways. lol



> how do i do this?




Hmmm, I forget. I'll get back with you though! Promise . .


----------



## whtknt

*And the light goes on...*

I begin to see, now. I've offered a viable solution, which no one seems willing to comment on. It would seem to me that the people who have a problem with our reviews want to argue, not resolve the issue.


----------



## Piratecat

I think it's time to close this. Let the subject sit for a week or two, please, before it gets brought up again.


----------

