# Redesigning the Reviews Page



## Morrus

Erithtotl has kindly volunteered to help redo the reviews page.  I plan to start from scratch, and to combine both the reviews page and the d20 System Guide into one page.  Both use the same info, and having them separate is kind of redundant.  

This is being started about now, and is taking place over the next week or two.  Feel free to post your opinions, requests, issues, aches, pains and relationship troubles here.

The new page will be smoother, faster, and less buggy than the current one.

We'll try to accomodate everything that you suggest.  Remember - this is a new design from the ground up, so suggesting alterations/amendments to the current page is rather pointless.  

Oh - in response to the question that will no doubt be asked at least several times despite my answering it here - no, the current data will _not_ be lost.


----------



## Erithtotl

*Hello!*

Hey this is Erithtotl (Ian, actually  ).  I'm thrilled to be helping out the best D&D site on the web.

The reviews will be the first thing we tackle.  The priority is getting it running smoother, faster, and reliably.  We may also do some quick interface tweaks to make it more useful.  I'm thinking of putting in a simple search box, and a product by alphabet, and maybe some filters, so you can be more specific in what you want to look at.  I'm totally open for suggestions.  Some ideas might be used later while others might go in the first draft.

I don't intend to stop with reviews, and we'll slowly make the whole site a smoother, more integrated, easy to use system.

So let the suggestions fly.  Try to split up any comments into reviews specific and other future enhancements.  Also, let me know what you think are the absolute most important things about the reviews section and the way it works.

Ian


----------



## Morrus

Here are my initial thoughts:

1) A list of upcoming products sorted by date.

2) Breakdowns of products by 'type' - adventure, rulebook etc.

3) Replace the top ten lists with a single top twenty list?  Add a top ten publisher list?

4) Initial screen for each publisher more reminiscent of the current d20 Guide than the current Reviews page.  One page per publisher, click on products for more detailed info, image etc. 

5) Amend the calculation for a product's score with some obsure Einteinian formula which also doubles as a theory for time-travel.


----------



## Tuerny

Two things.
1.  A list of the top ten rated publishers.
2.  For the authors of a review to be e-mailed when they get a comment.

That is all.


----------



## Psion

Okay, where do I begin?

I have been musing over how reviews are done, and here are a few of my thoughts.

1) the existing page is too hard to browse. About the best key the visitors have to browse is the top ten guides. Othen than that, they have to know what product they are looking for.

How to help this, I'm not sure, but maybe you could, in addition to the top 10, botton 10, top 10 D20, top 10 adventures, you could make:

a) last 10 -- or last 20 -- products released (or last two months, whatever)
a) top 10 non-advetures, top 10 free products, top 10 downloads, etc. Maybe a top 10 gallery.

2) The reviewer list is too long to be on every page. Perhaps the top 10 reviewer should be on every page, and then you can jump to a page that says "browse by reviewer, etc."

3) While the recent move to cutting off anything below 7 is understandable, it goes too far. There are many good products that just don't get reviewed as much. In the past, I recomended cutting out the top and bottom review... this won't work. This won't work for the same reason... there are a lot of products that just don't get enough reviews. Anyway, what I am thinking now is that the top review and bottom review should be weighted by 50%. It doesn't matter if you only have one or two reviews since the weighting comes out in the wash. We can still hold them out of being listed until they get 3 scores, but 7 is too much. 5 at the very lowest.

4) I think I have suggested some weighting schemes in the past. First, Morrus expressed that the staff reviewers could be weighted heavier. Second, one of the major problems that cause abberent review ratings is someone doing their first review ranting and/or raving.

So, what I think is that you can do this: Weight the reviewer by a factor equal to the square root of their number of reviews, with some "cap", which I would say should be 16. This would give a weight of 4. Give the staff reviewers a weight of 5.

Another possibility is to do as above, but subtract one from the final weight. This can give a weight of 0 for someone who has only done one review. Ah, well... 

5) Another threshold you might consider is weighting or making a cutoff for short reviews to the effect of "everybody has said it better, yadda yadda..." Not sure about that one.

Those are my initial thoughts.


----------



## CRGreathouse

Morrus said:
			
		

> *5) Amend the calculation for a product's score with some obsure Einteinian formula which also doubles as a theory for time-travel. *




LOL!  I didn't think our search for the perfect formula was that complex...  the geometric mean of the product of the man and the median, or Sqr(Mean*Median).

It decreases the effect of a single extreme vote, while still letting the abberent vote count for something. Example: 

Votes: 3, 3, 3, 4, 4, 4, 4, 4, 5, 5 
Old method: 3.90 
New method: 3.95 

Votes: 1, 3, 3, 3, 4, 4, 4, 4, 4, 5, 5 (same, with an extra vote of 1) 
Old method: 3.64 
New method: 3.81 

When the votes are close (as in the first example), the two methods have similar results, but when one stray vote is included, this proposed method minimizes its effect on the total score. This works both ways - a stray high vote is also minimized. 

All of this is transparent to the user; the new method is still an average, though not a "typical" average. The results are in the same range as the original vote.

This could be combined with Psion's weighting - though I'd go with Min(Sqr(Reviews),4) rather than Min(Sqr(Reviews)-1,4); staff reviews could have a weight of 5.


----------



## Morrus

Two more things which occur to me:

1) Links to errata, enhancements, previews etc. for each product.

2) Allow publisher access to add those links?  Would they use it?


----------



## NeghVar

Let me know if you want revised graphics...like a true transparent gif version of the larger WotC logo...

Later!


----------



## Suzerain

***


----------



## Suzerain

***


----------



## Suzerain

***


----------



## Suzerain

***


----------



## trancejeremy

It would be nice to give half a score, like 3.5 instead of having to pick either a 3 or 4.


I agree about 7 being the minimum is too big. Not many non-WOTC products have that many, with the exception of some of Green Ronin's stuff, and SSS/Necromancer.  


I would also be nice to be able to add a product to the review database. I mean, it seems to be missing a lot, and with more being added all the time, it would be quite a job for someone to keep it up to date. Especially for stuff that won't get reviews, or only 1. 

It might be best to restrict it to someone who has written 10 reviews (or so many kb worth), so it's not abused. So no one adds something like the Quintessential Pimp.

As for a weighting system, maybe you could let people vote on it. I always trust Psion's reviews, for instance, so I would give them a lot of weight. But some others, well, I don't give them much, if any.


----------



## Monte At Home

*Top Ten Lists*

I think that you should either adopt a system like CRGreathouse
 suggests, or you should drop the top and bottom 10 lists altogether. Currently, they are deceptive because they don't include all the products. Before, they were deceptive because people could (and by appearances, were) spiking ratings with good or bad reviews just to alter the top 10 lists.

I wouldn't weight reviewers differently I don't think. I certainly wouldn't base it on number of reviews done--it encourages people do a lot of reviews, but not necessarily good reviews.

You could (although this would be complex) have people rate how useful they felt various reviews were, and then weight reviews based on previous reviewer ratings, but I don't think that's necessary.

Mostly, I think that if you're going to have a site where anyone can come along and review any product, you should treat all the reviewers the same.

I like very much that the recent reviews are listed, and that you can click on publishers.


----------



## Morrus

*Re: Review the Reviews!*



			
				Suzerain said:
			
		

> *Another idea could be to allow each review to be a seperate message board thread.  Along these lines, users could read a review and also see comments listed about the review on the same page.  In this way, users can talk about the reviews themselves, and this could allow the users to have a renewed focus on their d20 reviews.  Its an interactive feature, and I could see it being used with great frequency.
> 
> People like message boards to boot, though with all the problems that message boards tend to have, this may not be a viable option.
> 
> Food for thought.
> 
> Robert *




Already does that.


----------



## Morrus

trancejeremy said:
			
		

> *It would be nice to give half a score, like 3.5 instead of having to pick either a 3 or 4.*




Agreed.




> *I would also be nice to be able to add a product to the review database. I mean, it seems to be missing a lot, and with more being added all the time, it would be quite a job for someone to keep it up to date. Especially for stuff that won't get reviews, or only 1.
> 
> It might be best to restrict it to someone who has written 10 reviews (or so many kb worth), so it's not abused. So no one adds something like the Quintessential Pimp.*




I shudder to think what we'd end up with. ..

"players handBook" by "wizards", "A bok for plying dnd". "Oh, sorry, I didn't realise what you meant by an ISBN number, I thought I was supposed to enter my favourite page number there."


----------



## EricNoah

I have a thought or two about the d20 guide...

I'd like to see colorcoding or some other visual aid to indicate when an item is...

available
not yet published but has a set release date
cancelled
out of print
is not available and has no set release date
comes out periodically (like a magazine, etc.)

Each publisher entry should have what's already in the guide (publisher name, general publisher/product description, website, fan sites, a picture of an upcoming or well-known product from that company, and lists of products with release dates.  

However what's not really codified quite yet is a system of marking products by different types (adventure, adventure anthology, rulebook, campaign setting, campaign accessory, monster book, stand-alone d20 game, etc.)  -- people will want to search by those types -- and products should be able to be put into multiple categories.  The product format (printed softcover, printed hardcover, download, packaged software, etc.) should also be indicated.  And there should be a place to link to the publisher's webpage for each product.  And then that's also where there could be links to the D20 Reviews for each product.

Each product might also be part of a "product line" like Penumbra or Freeport, etc.  

We don't have much author info for products listed currently, that could be another thing people might want to search for (all of Monte Cook's products across different publishers).

And finally the review system should include not only reviews by us folks but a way to add links to other websites' reviews.

And maybe a very unobtrusive link to where the product can be purchased at EN World's online store.


----------



## Erithtotl

*Good feedback*

So there are some great ideas.  And some not so great ideas, but thats part of the process.

Some things I think make sense so far:

I will set up a few layers of permissions:

1 - Unregistered User - Can only read reviews
2 - Registered User - Can read and write reviews, plus comment on products
3 - Asst.  Administrator - Can submit products for the product list
(but must be approved by an administrator)
4 - Administrator / Editor - Full rights to system

That way we can manually designate those who can add products to the list, but they won't actually appear in the list until they are approved by Morrus (or whoever else he designates as an Administrator)

I also like the idea of categorizing the products, like supplement, periodical, core rulebook, module, etc.
I'll have to think about a way of handling periodicals, it would be cool to be able to group, say, all Dungeon magazines into one area, so that we could have issue by issue reviews and an overall rating for the periodical.

I think clearly the front page of reviews needs to be simplified.  Listing top lists in various categories, the newest reviews, newest products added, and maybe top reviews would be good, and then you could select any category to get a full listing.

Perhaps the top ten lists could be split into two parts:  Products with over X amount of reviews, and products with less than X amount of reviews.

I also plan to add some kind of Alphabet pick list for the different categories, and a way of browsing through products.

A simple text search will also be added (likely to be very simple, for example search by name 'handbook' would get back all products with the word handbook appearing in the title.

Another feature might be if you select a reviewer, not only can you get back all their reviews, but you could get back a list of all comments for the different products.

Understand that most of this is not difficult in itself, but we may phase features in piece by piece, as I can't do everything at once.

Anyway, I'm excited by all the feedback.  Keep it coming.

Erithtotl/Ian


----------



## Psion

*Making my case*



			
				Monte At Home said:
			
		

> *Before, they were deceptive because people could (and by appearances, were) spiking ratings with good or bad reviews just to alter the top 10 lists.
> 
> I wouldn't weight reviewers differently I don't think. I certainly wouldn't base it on number of reviews done--it encourages people do a lot of reviews, but not necessarily good reviews.*




_If you had seen what my eyes have seen..._

Ahem... I really think I need to make my case here. I think that if you don't do something like weight the reviews, the average system suffers.

There have been times that people have posted a review and someone has decided that they disagree with the ratings and are sign up just so they can contradict the review they don't like, and they assign it the most extreme score that they can (a 1 or a 5). This compromises the fairness of the system, because the "reactive" score give is not out of an honest assessment of what they think the product is worth, but an attempt to annul the review that they didn't agree with and to bump the average in the direction they want to. I have seen this multiple times.

I understand what you are getting at by saying people can write more reviews but not good reviews. However, that is why I suggested using the root of the number of reviews as a baseline. You get little payoff by just writing 2 or 3 extra reviews just to get your point across about one, so as a result it would discourage this sort of reactive behavior... or at least lessen its effects. If more products got more reviews, it might not matter, since it would all average out. But there are really not that many reviews on a lot of products, so I think some artifical compensation is warranted.

Okay, on another note... I recommened earlier that you could half the weight of the highest and lowest review. I think Charles' idea of using the root(mean*median) acheives much of the same thing but would be easier to program since you don't have to worry about the sticky issue of how it related to the weighting system I recommended above.



> *
> You could (although this would be complex) have people rate how useful they felt various reviews were, and then weight reviews based on previous reviewer ratings, but I don't think that's necessary.*




I think number of reviews is a simpler metric, and will be sufficient to address the syndrome that I am worried about.


----------



## Psion

> *
> I will set up a few layers of permissions:
> 
> 1 - Unregistered User - Can only read reviews
> 2 - Registered User - Can read and write reviews, plus comment on products
> 3 - Asst. Administrator - Can submit products for the product list
> (but must be approved by an administrator)
> 4 - Administrator / Editor - Full rights to system
> *




Actually, it is sort of that way now... I can put products up if I want review one that isn't up yet. But formalizing it might make it easier to get more people involved in the process.


----------



## War Golem

My essential question is - if it is even possible, is it "bad form" to simply review a product by giving it a score, rather than a full-blown written review?

I have never reviewed anything here at EN World, so I am not intimately familiar with the procedure for doing so. Nonetheless, I certainly have a lot of d20 material, and would be happy to score a lot of it to even out the average rating and bump up the number of reviews per product (and thus hopefully help out other board members with their purchases)... but there's simply no way I have the time to write actual formal, written reviews of all this stuff.

Please advise, Morrus or Psion or anyone who deals with the EN World reviews.

Thanks.


----------



## Morrus

War Golem said:
			
		

> *My essential question is - if it is even possible, is it "bad form" to simply review a product by giving it a score, rather than a full-blown written review?
> 
> I have never reviewed anything here at EN World, so I am not intimately familiar with the procedure for doing so. Nonetheless, I certainly have a lot of d20 material, and would be happy to score a lot of it to even out the average rating and bump up the number of reviews per product (and thus hopefully help out other board members with their purchases)... but there's simply no way I have the time to write actual formal, written reviews of all this stuff.
> 
> Please advise, Morrus or Psion or anyone who deals with the EN World reviews.
> 
> Thanks. *




It's discouraged.  A simple score is not very useful to people - they need to know _why_ you gave it that score.  Requiring an actual review also encourages people to be more thoughtful about the score they give.  

When the review page was orginally created in the early months of 2001, it was possible to just rate a product without scoring it.  The feature was immediately and widely abused and was quickly removed.   I have no intention of putting it back.


----------



## Erithtotl

*more*

I have no problem with using the root(mean*median) method.  It seems like this would make sense for the most part.  I was also thinking of converting the ratings to a 1-10 scale instead of the 1-5 that we have now.  I'll have to run this by Morrus but what it will in-effect do is allow for halves (3.5 = 7).  I'll just throw this out as a possibility:  What about rating multiple features about a product, and building in some weights to those ratings.  For example:

Original Content
Presentation
Rules accuracy
Your take

And then average the 4 results, either straight up, or some kind of weighting system we come up with later?

The only thing about not allowing the option of rating without reviewing, is couldn't people just put in nonsense for their review as an opportunity to rate it?  I kinda like the idea of forcing a review, but there's only so much we can do.  Another option is allow people to actually rate the reviews.  If there are more than x number of ratings (say, 5) for a review, we then use that rating as a weight on the review rating.  The idea is that if many people disagree strongly with a review, they can rate it poorly, eventually reducing its effect on the overall average product rating.  It sounds a little complex but it shouldn't be too hard to implement, and it might solve several problems.

Ian


----------



## KDLadage

*some thoughts...*



> I have no problem with using the root(mean*median) method.  It seems like this would make sense for the most part.




I agree. I think it would solve a great number of issues with the reviews.



> I was also thinking of converting the ratings to a 1-10 scale instead of the 1-5 that we have now.  I'll have to run this by Morrus but what it will in-effect do is allow for halves (3.5 = 7).




Should not be too much of a leap of faith to do that one. I would let all of those that have reviews in the datbase know so that they can go and modify thier review scores to the +/- 1 values if need be.



> I'll just throw this out as a possibility:  What about rating multiple features about a product, and building in some weights to those ratings.  For example:
> 
> Original Content
> Presentation
> Rules accuracy
> Your take
> 
> And then average the 4 results, either straight up, or some kind of weighting system we come up with later?




I would say no.

I would say no for several reasons: many people will place widely varying weights on each catagory you rate the product on. Leave the rating as a single score; let the review tell the tale.



> The only thing about not allowing the option of rating without reviewing, is couldn't people just put in nonsense for their review as an opportunity to rate it?  I kinda like the idea of forcing a review, but there's only so much we can do.




I would not worry about it. As it is, we have nearly 800 reviews, and nobody has really abused the system that I can tell.



> Another option is allow people to actually rate the reviews.  If there are more than x number of ratings (say, 5) for a review, we then use that rating as a weight on the review rating.  The idea is that if many people disagree strongly with a review, they can rate it poorly, eventually reducing its effect on the overall average product rating.  It sounds a little complex but it shouldn't be too hard to implement, and it might solve several problems.




I think it causes others, however.

The only reviews I would give greater weight to would be those of the "official" reviewers. I do not say this because they are "official" butbecause they are the most well thought out, well written and well recieved reviews in the database, for the most part.


----------



## corvus

1) Add a simple search on the main page. Have it match on any of title, author, or publisher by default, but allow user's to change the default with a pop-up. See how Amazon.com does it.

2) Use tabs to navigate between the main areas of the reviews section. Again, see Amazon.com.

3) Don't use colored text. It doesn't work for everyone, i.e. those with vision problems. Currently there is use of red on "red" in the reviews pages. That's hard to read for everyone, I imagine.

4) Don't mess with font type or size. Allow user's to control that with their browsers.

5) Simplifly the main page. There's just too much stuff there now which makes it hard to find what you want.

6) On the publisher pages, make the product's title a link to that product's reviews.


Also it would be nice if the link to the reviews page was more prominent on the enworld.org top page, like where it currently says:

D&D/d20 System News & Reviews

at the top. Reviews could link to *gasp* the reviews page.


----------



## War Golem

Morrus said:
			
		

> *
> 
> It's discouraged.  A simple score is not very useful to people - they need to know why you gave it that score.  Requiring an actual review also encourages people to be more thoughtful about the score they give.
> 
> When the review page was orginally created in the early months of 2001, it was possible to just rate a product without scoring it.  The feature was immediately and widely abused and was quickly removed.   I have no intention of putting it back.  *





Aya, aye captain. No reviews without body text!

Thanks for the edification.


----------



## CRGreathouse

*Re: more*



			
				Erithtotl said:
			
		

> *I was also thinking of converting the ratings to a 1-10 scale instead of the 1-5 that we have now.  I'll have to run this by Morrus but what it will in-effect do is allow for halves (3.5 = 7).*




From what I'v read, rating from 1-10 is too much for most people to do accurately - 1-7 has been called the "ideal" system, as it gives enough choice, but not too much.  I stress that I don't know this to be true, though it makes sense; I'm just bringing it up for discussion.

As for the weighting system, I'll explain a little more.  Let's say a person is willing to "work the system" with an unpopular product s/he likes.  The person wants it to be rated a 4, but the average is now 2.  Obviously, the person would rate it a 5, bringing it closer to 4 than if s/he had rated it a 4.

With the median, there's no reason to overshoot - 5 won't being it any closer than 4.

Some would object to the median, though, because it means that extreme reviews count for nothing, and that most or all reviews will have a whole-number rating.  Thus, the geomean of the two methods provides an excellent solution to the problem.


----------



## Suzerain

***


----------



## bramadan

I am not a big fan of median but agree that the idea of having outliers count for 50% is probably decent (if something like that is to be done). I personaly like the 7 reviews+ rule as it keeps the top lists to established classics. What would be a good thing to have to offset this is "top 10 new releases" list that would require only 2 or 3 reviews but products could stay on it only month or two after they are released. 
I also second 1-10 scale, I do feel there is noticable quality difference between some stuff I have to give same mark to.
Merging WotC and d20 list is also a good idea.

Also on a personal note, Morus or Erithtotl, if you could merge my reviews under Bramadan and Bojan (bramadan) it would be great. I have lost the pasword for the old account and have started the new one a while ago but would like to have them all in one place (especialy if the number of reviews is going to count for anything). I mailed Morus about this a while ago but he was probably too busy....
Thanks a lot.


----------



## Clint

I'm voting for a smaller page template.

My biggest problem with the reviews page is that the server seems to be swamped at many points during the day.

Instead of listing all the companies and reviewers and the Top 10's on every page, simplify the design. I don't look at that information every page view, but I download all of it, every time, and it brings the size of the page way, way up. (to ~120KB, not counting graphics)

I'd much rather have a list of three simple links to cover those three topics. The left bar could look like this:

link:Main Page
---
Staff reviews by:
link:Simon Collins
linksion
---
link:Top 10 Lists
---
link:Search by Publisher
link:Search by Reviewer
Search by Product Name:
[form]

I saved the front page locally and cut out the sections I mentioned. You could quickly trim the page down to 20KB if you replace those sections with links. Easier on the server, easier on the users. 

Keep the "most recent reviews" expanded on the page, though.

Also, a slim/no-graphics option (stored in a cookie) would be pretty sweet.

Good luck, I can't wait to see the results!

-Clint


----------



## drnuncheon

*Weighting reviews/reviewers*

Here's my suggestion:  allow people leaving comments about a review to rate the reviewer on the same scale, in a "how helpful was this review" box.

This makes the entire thing basically self-correcting.  If a person puts out a bunch of crummy reviews that amount to nothing more than "I like this product" or "This product sucked" then their opinion will be marked down and count for less.

If they put out a series of decent, well-thought-out reviews that genuinely help people decide on whether or not to purchase the product, they will get marked as 'more helpful' and the weight of their opinion will rise.


J


----------



## drnuncheon

This is in response to some suggestions by Charles and Psion:

I ran some numbers on the idea of minimizing aberrant ratings by using sqrt(mean*median) and I don't like what I see, especially when dealing with extrme values.  Using this method actually _increases_ the effect a single aberrant review has on a product.

Consider this set of ratings: 5, 5, 5, 5, 1

Using the arithmetical mean (that's "average" for folks who don't speak statistics) you get a rating of *4.2*.

The median (halfway between the lowest & highest points) is *3*.

The sqrt(mean * median) is *3.54*.  The single dissenting review has pulled the score down even lower under this method than it does under the current one.

Heck, if you had _99_ 5s for a product, and some joker put in a 1, the rating would go from 5 to 3.85 (presumably rounded to 4), where just using the average would give you 4.96 (presumably rounded to 5) - that means that 1 single review out of 100 pulls the score down by more than a point!  I don't think that makes for an accurate portrayal.

Now, this may have less effect where the scores are less extreme, but the effect is there - putting in a markedly lower (or higher) score drastically alters the median, and since you're giving the median as much weight as the mean in the final score, you're greatly multiplying the effect of that single review.

J


----------



## Warchild

Well, i don't know jack crap about medians and mean and such.
I like the current scoring system. It seems more honest to me.
As for the "spikers" several people have mentioned, i have this to say:
If i see several 5's and a 1, I know that there was a fruit-loop rating thrown and i can ignore it. What i actually do is take a look at the reviewer's history and judge whether i should ignore the guy or not. Thats what that whole set up is for isn't it??
I don't need ENWorld or a Einsteinian math system to make up my mind for me. Let me think for myself, please.

As for weighting reviewers, what a terrible idea. What you want is MORE people doing reviews, thats how you'll get the right averages. But if some people's reviews become weighted, then most people that might submit a review, might not when they find that their opinions count for less. Some of us that are writing reviews will stop. i will. I have nothing against Alan and Simon(do you have a reviewer hidden away somewhere called Theodore?), but no one's review should weigh in more than anyone else. That way lies madness and conspiracy theories!! 

I like the Rating system as it is. It forces people to make real choices, is the product good enough or isn't it? The rating system of 1 to 5 carries more weight and impact than 1-10 or even 1-7. I think a larger scale only creates dodges for people wanting to be liked or create joy-joy fuzzy feelings, instead of just giving the product a 2, if it darn well deserves a 2!! If a product isn't a 4 then its a 3. Stick to your guns or get out of the armory!!

Call me a hardcase if you will, but thats how i see it.


----------



## Furn_Darkside

corvus said:
			
		

> *
> 3) Don't use colored text. It doesn't work for everyone, i.e. those with vision problems. Currently there is use of red on "red" in the reviews pages. That's hard to read for everyone, I imagine.
> 
> 4) Don't mess with font type or size. Allow user's to control that with their browsers.
> *




Salutations,

First- let me state how much I appreciate the review pages- they are a great resource.

However, I agree with Corvus-  especially with the current reviews being hard to read. Grey text on a black background is tiring on the eyes for the longer reviews.

Thanks for the hard work.

FD


----------



## KDLadage

*...a quick lesson in statistics...*



			
				drnuncheon said:
			
		

> *I ran some numbers on the idea of minimizing aberrant ratings by using sqrt(mean*median) and I don't like what I see, especially when dealing with extrme values.  Using this method actually increases the effect a single aberrant review has on a product.
> 
> Consider this set of ratings: 5, 5, 5, 5, 1
> 
> Using the arithmetical mean (that's "average" for folks who don't speak statistics) you get a rating of 4.2.
> 
> The median (halfway between the lowest & highest points) is 3.*



*

Incorrect.

In this case the mean is 4.2 (5+5+5+5+1=21; 21/5=4.2)

The median is the central value of a sorted set. With 5 values, sort them and select the 3rd value -- in this case 5.

Thus 4.2 x 5 = 21; sqrt(21) = 4.58*


----------



## Nail

*"Newbie" take*

Just so y'all know, I haven't read a darn thing in this thread.  So think of this as just "extra data", rather than unique suggestions.

I jus' discovered the reviews page a week or so ago, as only recently have I decided to squeeze out some o' th' hard-earned cash.  What I wanted on a review page was:

  - Five star ranking system, sortable by company or catagory, with rankings only given to those products with many (read: more than, say, 10) reviews.  Heck, all I need is the number, so if people jus' wanna put a rating down an' be done w' it,...well that'd be fine.  Suggestion: develop an easier "vote" system to collect more voters, so people don't feel they have to write a Magi's tome to review somethin'.

- Li'l snippets of some of the written reviews on a summary page.  Ya got that.  Good, but how about some way to sort them?  Like: "Hey, give me all the reviews that had a 5 star rating for this product.  Good.  Now give me all the reviews of people that hated it.  Excellent...."

-Links from the reviews to a few places that sell the product, plus the publishers web site.

That's it, really.  Anything else is gravy.

-Nail


----------



## drnuncheon

*Re: ...a quick lesson in statistics...*



			
				KDLadage said:
			
		

> *
> 
> Incorrect.
> 
> In this case the mean is 4.2 (5+5+5+5+1=21; 21/5=4.2)
> 
> The median is the central value of a sorted set. With 5 values, sort them and select the 3rd value -- in this case 5.
> 
> Thus 4.2 x 5 = 21; sqrt(21) = 4.58 *




Whup, my bad.  You are right.  This makes a lot more sense than what I thought you were doing.

Carry on, then.

J


----------



## drnuncheon

Warchild said:
			
		

> *As for weighting reviewers, what a terrible idea. What you want is MORE people doing reviews, thats how you'll get the right averages. But if some people's reviews become weighted, then most people that might submit a review, might not when they find that their opinions count for less.*




Note that their opinions would only count for less if most people on the server thought their opinions should count for less.  (Then again, my rating idea isn't based on the number of reviews like some others are.  Just because you write a lot of reviews doesn't mean they're good - I'd rather judge quality, not quantity.)

All it's doing is exactly what you do when you see a reviewer and say "Oh, I know them, they never think things through when they review" - it's just reflecting that in the scoing.

Otherwise you have John "I giv this a 5 cuz it has nekkid girl in it" Smith's rating counting exactly the same as anyone who gives a serious, worthwhile review.

Of course, the ideal would be one like Alexandria Digital Literature(www.alexlit.com) has - it matches your ratings up with the ratings of everyone else in the DB to predict how much you personally will like a product.  Dunno if Morrus wants to do anything that complex though!

J


----------



## Moon_Goddess

Morrus know's I only have one request.

Make the text size resizable,  I never use the reviews on the site as it stands now becuase I would have to change my screen resolution, to do so.     Control- MouseWheel, is the greatest thing to happen in computing in years.   Please don't turn it off.


----------



## Psion

*Re: more*



			
				Erithtotl said:
			
		

> *
> Original Content
> Presentation
> Rules accuracy
> Your take
> 
> And then average the 4 results, either straight up, or some kind of weighting system we come up with later?
> *




I'm not so sure how productive that would be. Although it would be nice to think that we can get some reviewers to review using similar criteria, you really can't, and most people aren't going to have a firm enough view of a product to make a meaningful assessment of each quality. When I am trying to rate a product, in my head I give it a +, 0, or - in 5 areas*, 2-3 is a 4 and 4+ is a 5... but even I think my opinion isn't so niggly I could put a 1-10 number on 4 qualities.

However, I agree that there have been many times I had wished I could assign half points while just using one number.

* - in case you care, those are presentation, quantitative value, system fidelity, ideas, usable material.


----------



## Warchild

I'm more worried about the fact that i just saw a review get put up 3 times for the same product!!! And it counted!!!!!!

How do you fix a server-screw up like that and more importantly...has it happened before?????

That not good.


----------



## Blacksad

1) edit function

2)spellchecker


----------



## poilbrun

There is one thing I'd really *love*: the ability to search only for PDF products...

I'm a great fan of pdf products, and if I could have a listing with reviews of the books that have been published like this, it would really be great!


----------



## EricNoah

DarwinofMind said:
			
		

> *Control- MouseWheel, is the greatest thing to happen in computing in years.    *




WOW.  I just learned something new.  That rules!!


----------



## KDLadage

How is the redesign coming? Just wondering what suggestions from this thread will get incorporated in the first go-round.


----------



## Morrus

The coding is underway.  May take a few weeks, though.  Here's the list I put togther from this thread (I includined things I like the look of and though were feasible):

1) A list of upcoming products sorted by date. 

2) Breakdowns of products by 'type' - adventure, rulebook etc. 

3) Replace the top ten lists with a single top twenty list? Add a top ten publisher list? 

4) Initial screen for each publisher more reminiscent of the current d20 Guide than the current Reviews page. One page per publisher, click on products for more detailed info, image, reviews etc. 

5) For the authors of a review to be e-mailed when they get a comment. 

6) Las 10 -- or last 20 -- products released (or last two months, whatever).

7) Rework the overall rating calculation (somehow).

8) Ability to give half a score, like 3.5 instead of having to pick either a 3 or 4. 

9) Colour coding or some other visual aid to indicate when an item is... available, not yet published but has a set release date, cancelled, out of print, is not available and has no set release date, comes out periodically (like a magazine, etc.), overdue. 

10) Author info for each product.

11) Link to buy product at the online store.

12) Something to eliminate 4-line reviews.  A mimimum word count, or a 'rate this review' feature?

Note that not all of these will necessarily make it, but that's what we're aiming for.


----------



## KDLadage

> 1) A list of upcoming products sorted by date.



Good one. I am looking forward to this.



> 2) Breakdowns of products by 'type' - adventure, rulebook etc.



Another good one I am looking forward too...



> 3) Replace the top ten lists with a single top twenty list? Add a top ten publisher list?



Given the number of products in teh database, rather than sticking to a single number, how about the *TOP 10%* listing -- thus, if you have 300 items in the database, it rates the top 30... This would keep things dynamic and not exclude too many good products. I would say the same thing for the publishers listing as well.



> 4) Initial screen for each publisher more reminiscent of the current d20 Guide than the current Reviews page. One page per publisher, click on products for more detailed info, image, reviews etc.



Not sure here; but it could be a good thing.



> 5) For the authors of a review to be e-mailed when they get a comment.



YES!



> 6) Last 10 -- or last 20 -- products released (or last two months, whatever).



Go with time.



> 7) Rework the overall rating calculation (somehow).



I personally think that the *SQRT(MEAN x MEDIAN)* is teh way to go. Gives an honest rating, does not allow a single outlier to kill a product (or boost it too much).



> 8) Ability to give half a score, like 3.5 instead of having to pick either a 3 or 4.



I guess if you are going to do this, I would say double the ratings and go with a 1-10 scale; half point ratings seem _artificial_ to me. Not sure why, when the overall effect is the same.



> 9) Colour coding or some other visual aid to indicate when an item is... available, not yet published but has a set release date, cancelled, out of print, is not available and has no set release date, comes out periodically (like a magazine, etc.), overdue.



OK.



> 10) Author info for each product.



Cool.



> 11) Link to buy product at the online store.



Sure.



> 12) Something to eliminate 4-line reviews.  A mimimum word count, or a 'rate this review' feature?



Not sure here. When you open reviews to everyone, you open them to 4-line reviews. It is simply something we will have to deal with. Anything like a rate-the-review system creates more problems than it solves, in my opinion.



> Note that not all of these will necessarily make it, but that's what we're aiming for.



Fully understood.


----------



## CRGreathouse

I like the idea of either half-point ratings, because it makes the sqr(mean*median) formula more meaningful - when the top 7 products all have the same median, it's not very infomative.

Why not 1-10 ratings?  Likely so the existing ratings could stay...


----------



## Morrus

CRGreathouse said:
			
		

> *
> 
> Why not 1-10 ratings?  Likely so the existing ratings could stay... *




Exactly.  Doubling the ratings won't work - all the 5s should not necessarily be 10s, and all the 1s should not necessarily be 2s.  All the 3s (average) should not necessarily by 6s (above average).


----------



## MythandLore

Morrus said:
			
		

> *
> 
> Exactly.  Doubling the ratings won't work - all the 5s should not necessarily be 10s, and all the 1s should not necessarily be 2s.  All the 3s (average) should not necessarily by 6s (above average). *



Good point.
One persons 3/5 could be 7/10 but to another person it might mean 5/10.


----------



## CRGreathouse

3/5 -> 6/10 or 5/10, not 7/10.

Sorry, I just had to do it...


----------



## Staffan

One thing I saw mentioned in passing in a post in this thread, but that bears repeating again, is: NEVER use "hard" font sizes. By hard font sizes I mean specifying that the font should be 12 points or whatever - the review pages use 7 or 8 points, which is WAY too small (especially when viewed on 1280x1024). Please keep the review texts at size "0" - the default size.


----------



## Monte At Home

*A little late, I know...*

... but I've come to the conclusion that the top 10 and bottom 10 lists aren't worth the trouble. As long as someone (like say, Psion) can put all the effort into writing out a detailed, well-thought out 1K to 2K review on a product and someone else can shotgun out a dozen or so "I thought this was cool" or "I thought this sucked" reviews and they all get weighed equally, the top ten list isn't worth the electrons they use up on the screen.


----------



## Henry

I do sympathize with Monte on this: I read reviews to find out WHY a product was liked. Even if the rating is bad, it doesn't affect my opinion of the product until I read the actual review - 

UNLESS the product is one I have no opinion about or interest in the first place, in which case I tend to gloss over products that get bad ratings and take a look at the reviews if the product got a good one.


----------

