# What Games People Are Talking About: A Pie Chart



## Ahnehnois

The amount of 3e discussion continues to amaze me. Even as someone who still plays 3e, I sometimes wonder what else there is to discuss at this point.


----------



## Lindeloef

[MENTION=17106]Ahnehnois[/MENTION]
You can always discuss the topic of fighters vs spellcasters 

@topic
Nice, thanks [MENTION=1]Morrus[/MENTION]. Do the 13% for Shadowrun come from the "RPG of the Moment" or whatever it was called Subforum that once existed? Or was that more than 90 days ago?


----------



## Olgar Shiverstone

I guess TheOneRing is doomed ... it doesn't even crack the 1% list.


----------



## Nytmare

Somebody out there must have done a The Orc and the Pie Chart joke by now, right?


----------



## coz

I wonder if there is some danger in publishing explicit results like this to game the system, for instance, while delighted that the Best RPG ever(TM) (Star Wars D6) is still being talked about, it's percentage seems awful low, could that not inspire someone to go out and mention their favorite system as often as possible online in order to get its' numbers up? For instance: hiding multiple mentions of a particular game in the header or footer of a webpage (or using multiple shill accounts to add many comments to a blog post/forum thread/subreddit with their favorite system within it). I suppose the danger isn't that regular gamers will do this , but that a designer or the marketing departments at a larger publisher seeing that their game doesn't have "enough" mindshare here and going out and directly gaming the results to make their game look much more popular than it is.

Also, a few questions about your sources, do you include reddit, it has many many subreddits for specific games, tho many are fairly dead in terms of activity compared to the main r/rpg sub, if not why? Do you include the GenCon forums (they're independent of any publisher I think and plenty of games are discussed on there), if not, why? And why not include the non-independent forums in the results, I've seen plenty of discussion of other systems on the Paizo boards for instance (and they have a no edition war policy to boot). For instance if your algorithm could be tweaked to look at paizo's forum and exclude mentions of pathfinder or look at wotc's forum and exclude D&D/gamma world, etc?

Finally, regarding your formatting, please use colors that are easier to distinguish from one another in future charts, even after zooming the image it was difficult to tell which green for instance referred to Shadowrun, which to firefly, etc.


----------



## haakon1

Interesting stuff, Morrus.


----------



## Morrus

coz said:


> I wonder if there is some danger in publishing explicit results like this to game the system, for instance, while delighted that the Best RPG ever(TM) (Star Wars D6) is still being talked about, it's percentage seems awful low, could that not inspire someone to go out and mention their favorite system as often as possible online in order to get its' numbers up?




And talking about Star Wars is dangerous? I think that would be a good thing. More people talking about games! 



> sources, do you include reddit, it has many many subreddits for specific games, tho many are fairly dead in terms of activity compared to the main r/rpg sub, if not why? Do you include the GenCon forums (they're independent of any publisher I think and plenty of games are discussed on there), if not, why? And why not include the non-independent forums in the results, I've seen plenty of discussion of other systems on the Paizo boards for instance (and they have a no edition war policy to boot). For instance if your algorithm could be tweaked to look at paizo's forum and exclude mentions of pathfinder or look at wotc's forum and exclude D&D/gamma world, etc?




The current list of sample sources is on the hot games web page:

http://www.enworld.org/forum/hotgames.php

While more sources will definitely get added as I find ways to do so, once the sample size reaches a certain place the relative positions tend to get reinforced rather than changed. The last blog network I added, for example, hardly budged the percentages despite being 600 blogs. But I'll definitely add any I can; it can only improve the system!

The Gen Con forums are an _excellent_ idea. They hadn't occurred to me! I'll definitely look into that. Social networks are much harder to extract data from - I'm currently working on Twitter hashtags, but I don't know if that will work out.

Reddit is a good idea. I don't know how possible that is, but if it can be done I'll do it!


----------



## Morrus

Hey, whatchaknow! Reddit was really easy to add! It's counting Reddit posts as of right now!

Gen Con's a bit harder. I found the RSS feeds, so it's technically easy, but is there a general RPG type forum there? I might be looking in the wrong place.


----------



## coz

Morrus, the main GenCon forums are here: community.gencon.com/forums They don't have a specific gaming related forum as far as I can tell, but I know in the lead up to the con plenty of games were discussed in the general, events and exhibitors subforums for 2013, as well as the ones dedicated to gaming groups like Kentucky Fried Gamers, I guess it might be harder to locate useful data from them unfortunately. What I meant about it being dangerous was people might see the numbers for their favorite games and then and superfluously mention their favorite a million times, like:

" Star Wars D6 is the greatest game since Star Wars D6, it uses D6s, this Star Wars D6 game, and it takes place in the Star Wars universe, all in all, much fun is to be had with the Star Wars D6 game, run out and buy some used Star Wars D6 books now."

Obviously, this facetious example is over the top, but we are gamers, someone is going to try and game the system, but again, I feel a publisher or marketer is more likely to do that than a regular gamer.

As to other sources, I see RPGGeek is one you use, the top of their "The Hotness" list on the left hand side of the page lists Torchbearer in the top slot, but it doesn't even make any of your lists (though I suppose maybe RPGGeek is the only place it's being discussed maybe?). Have you considered using show notes from popular podcasts as a source of data?

This is very cool stuff, I'm deeply interested in this and the WotC survey and whatever future similar survey you do, so much so that while I've had this account for like 10+ years here this is more or less the first post I've made in like 8 or 9 years.


----------



## darjr

Would you include the forums of game companies like Goodman games and Paizo?


----------



## darjr

On my tablet the pie chart window is to big and I can't click the x to close it. Nexus 7


----------



## Morrus

darjr said:


> Would you include the forums of game companies like Goodman games and Paizo?




I've decided not to. If I did one, I have to do them all.  So I've made a decision to stick to large independent sources only. 

It's a sample, not a census. The latter, unfortunately, is impossible. But a statistically significant sample is much more achievable.


----------



## darjr

Cool. Somewhere on g+ there is a trending web page


----------



## Morrus

coz said:


> " Star Wars D6 is the greatest game since Star Wars D6, it uses D6s, this Star Wars D6 game, and it takes place in the Star Wars universe, all in all, much fun is to be had with the Star Wars D6 game, run out and buy some used Star Wars D6 books now."
> 
> Obviously, this facetious example is over the top, but we are gamers, someone is going to try and game the system, but again, I feel a publisher or marketer is more likely to do that than a regular gamer.




If a publisher were to do that , they'd do so for much more compelling reasons than this. They'd be doing it already, and also for SEO. 



> As to other sources, I see RPGGeek is one you use, the top of their "The Hotness" list on the left hand side of the page lists Torchbearer in the top slot, but it doesn't even make any of your lists (though I suppose maybe RPGGeek is the only place it's being discussed maybe?). Have you considered using show notes from popular podcasts as a source of data?




I thought their hotness was based on average review scores? Whatever the case, Torchbearer is being counted currently as part of OSR.

Podcasts, sure - if there's a depository like the blogs of hundreds of them with an RSS feed it can be added. Does anyone collect that?


----------



## coz

Fair enough about publishers gaming things. I wasn't sure what RPGG's Hotness was based on actually, I just happened to notice that it was at the top because I backed the book and haven't seen anyone talking about it yet. How do you define OSR though? Is it enough that a game mimics the old-school style? Or does it need to have old-school rules, because Torchbearer is most certainly not a retro-clone rules-wise. Do you count Dungeon World the same way? I have no idea if there's any repository for gaming podcasts unfortunately, it just seemed a good source for another data point.


----------



## shamsael

Given the size of the sample and the secrecy of the "secret algorithm", I don't think it would be possible for anyone to intentionally significantly effect a given game's rank.


----------



## Morrus

shamsael said:


> Given the size of the sample and the secrecy of the "secret algorithm", I don't think it would be possible for anyone to intentionally significantly effect a given game's rank.




It was just a Google rankings joke. Though now I want to write one! 

I want to include pi.

( [threads] * [blogs] / ( [pi] / [podcasts] ) )


----------



## shamsael

I would look into some way of weighting sources.  Blogs are probably more valuable than forum posts.  Blogs that are gaming oriented but rarely call out specific games by name are probably more valuable than gaming blogs that regularly call out games which in turn are more valuable than mainstream blogs that occasionally call out games by name.

But still, I think at least sample size would prevent these rankings from being intentionally messed with.


----------



## dd.stevenson

Morrus said:


> I've decided not to. If I did one, I have to do them all.  So I've made a decision to stick to large independent sources only.
> 
> It's a sample, not a census. The latter, unfortunately, is impossible. But a statistically significant sample is much more achievable.



Would including all the major company forums be that difficult? Like, maybe as something that could be toggled on and off?

Because it seems like what you have now is going to skew (possibly quite heavily) your results toward out-of-print games and away from games that benefit from strong company website communities.


----------



## Morrus

dd.stevenson said:


> Would including all the major company forums be that difficult? Like, maybe as something that could be toggled on and off?




It would be next to impossible.  It'd end up simply being a chart of companies with RSS feeds on their forums.  Paizo.com has one; wizards.com doesn't. Pathfinder would shoot to the top, and D&D would almost disappear. That only works if you can do them all.  And then, you're looking at trying to achieve something closer to a census than a representative sample, which isn't realistically attainable.

The aim is to basically get a statistically significant sample. The statisticians out there can comment more on what and why that is, but it's by far the most appropriate approach.

Now, bear in mind that I'm not a statistican.  I'm just doing what I was told to do  by an actual statistician.  So my explanations may not be entirely  coherent!  But I've been told very strictly that including company websites would distort the data to beyond meaningless (unless I have the resources to do an actual complete census). 

Now, we don't have current survey data indicating the size of the market; so that's a problem.  I'm sure it exists, but I don't have access to it.  The latest public data is from 1999.  

Based on that, we have a confidence interval of within a couple of percent. (That means the 0% ones aren't "none"; they're 0% +/- x%.)



> Because it seems like what you have now is going to skew (possibly quite heavily) your results toward out-of-print games and away from games that benefit from strong company website communities.




Fortunately, the data doesn't appear to have done. The top are D&D and Pathfinder, followed by things like 13th Age, Numenera, Edge of the Empire, new Shadowrun.    New and current stuff very clearly dominates the chart.


----------



## dd.stevenson

Morrus said:


> And then, you're looking at trying to achieve something closer to a census than a representative sample, which isn't realistically attainable.
> 
> The aim is to basically get a statistically significant sample. The statisticians out there can comment more on what and why that is, but it's by far the most appropriate approach.



I took a fair amount of math in school and I'm well aware of the difference. My aim was to suggest a more representative sample, which also coincidentally would have been a lot bigger. But--of course--if it can't be done, then it can't be done.


----------



## Morrus

dd.stevenson said:


> I took a fair amount of math in school and I'm well aware of the difference. My aim was to suggest a more representative sample, which also coincidentally would have been a lot bigger. But--of course--if it can't be done, then it can't be done.




Sorry! I wasn't intending to try to teach my grandmother to suck eggs!  I somehow fell into "repeat stuff I've been told" mode and why adding and enormous data source like, say, Paizo.com would make the data less representative, not more.

When you start adding in company forums too, you begin approaching more talking about attempting a census.  Including everything.  But no, that can't be done - I can add some but not all, which would break that data completely. At least, not with the sort of resources I have available.


----------



## dd.stevenson

Morrus said:


> Sorry! I wasn't intending to try to teach my grandmother to suck eggs!  I somehow fell into "repeat stuff I've been told" mode and why adding and enormous data source like, say, Paizo.com would make the data less representative, not more.
> 
> When you start adding in company forums too, you begin approaching more talking about attempting a census.  Including everything.  But no, that can't be done - I can add some but not all, which would break that data completely. At least, not with the sort of resources I have available.



No offense taken... I'm sure you've had more than your share of criticism for this survey, and I wasn't intending to join in that.

It makes sense that if you can't add WotC then you can't add anybody. 

It also makes sense that if you did add company websites then you would need to increase the non-company site portion of the survey to census levels in order to account for that. (Though in that case I would imagine that your best course of action would be to simply present two sets of data: one that included company site data and one that excluded it. That was where I was going with the toggle suggestion.)


----------



## billd91

dd.stevenson said:


> No offense taken... I'm sure you've had more than your share of criticism for this survey, and I wasn't intending to join in that.
> 
> It makes sense that if you can't add WotC then you can't add anybody.
> 
> It also makes sense that if you did add company websites then you would need to increase the non-company site portion of the survey to census levels in order to account for that. (Though in that case I would imagine that your best course of action would be to simply present two sets of data: one that included company site data and one that excluded it. That was where I was going with the toggle suggestion.)




It's not just increasing non-company site portions, by going to company websites, you're adding a lot of bias to the sample. Naturally, a preponderance of discussion on sites like WotC and Paizo will be on their own products, making them particularly biased sources of data. Sites with a particular focus in their history, like ENWorld with its original D&D focus, already add bias to the sample despite being open to discussion of any games because the historical focus generated a sample of gamers with a systematic bias. A company website would be like injecting that sort of sample bias on steroids.


----------



## Umbran

Small nitpick that someone else may have mentioned:  It probably doesn't make much (if any) difference in the results, but you seem to be counting 13th Age in both the "D&D and Variants" and the "non-D&D" groups.


----------



## Cadence

This seems like it would be a great source of hundreds of discussion points for a first course in statistics!

I think I would actually just avoid talking about the kind of sample you have - you have a non-random convenience sample of some not-well defined larger population.  If that population is supposed to be all gamers who post on line then you are suffering from a huge source of bias due to undercoverage because your sampling frame is systematically missing all of the company web-sites. Its even farther away from all gamers.  Is it better to just say you have a census of "The most popular independent RPG sites and Blogs"?

As far as what you're measuring.  You are only getting the count of "people who post about games" if your algorithm only counts the number of distinct posters over the 90 days once each.  Further you would need to assume that most people don't have at least two boards they use regularly.      You are even farther from getting the number of "people who play each game" (which one of the charts has as a label) since a lot of us seemingly post in threads for systems that aren't related to what we are currently playing.  Going with "Most discussed RPGs" like you say in several places seems appropriately nebulous enough that it isn't misleading.

And then once you have your sample and a rough research question you get all the arguments about what data is most appropriate.  If it's threads, is a one post thread as good as a 1,000 post one?  If it's number of posts, what happens when a few 3.5ers who are on-line at the same time manage to crank out 20 pages of posts in a day and a half arguing about wizards versus fighters?   If there are lots of PF on-line games and OD&Ders like to play in person should all on-line games be excluded, or are you just going for total on-line traffic about the games?   If its threads, how does "All D&D count"?   If its post, how does a post talking about 4e as a counter example in a thread about a PF question count?   What happens if I don't like the auto-linking ENworld does when I type Pathfinder so that I use PF instead?   For posts, how does posting they wish they could give XP count versus actually gettting XP?  etc...

Hmmmm....  I'm getting more and more glad it isn't my project to run and that I just get to see the cool results.   (Which I really enjoy, thanks!).


----------



## Morrus

Umbran said:


> Small nitpick that someone else may have mentioned:  It probably doesn't make much (if any) difference in the results, but you seem to be counting 13th Age in both the "D&D and Variants" and the "non-D&D" groups.




The two lists are completely independent of each other.  Anything in one list doesn't affect the other.  Numenera and Star Wars (on the Hot Games page) are on the Sci-fi chart additionally.

Unless you mean the pie charts specifically.  All they are are the tables on the Hot Games page pasted into Excel, and me hitting the "chart" button.  On those, I slipped 13th Age out separately too because it was the largest in that group, and it gave a good visual key as to size.


----------



## Morrus

Cadence said:


> Is it better to just say you have a census of "The most popular independent RPG sites and Blogs"?




Indeed!  That's almost exactly what the Hot Games page says! "This page tracks discussion of over a quarter of a million forum members  and approaching a thousand blogs on a selection of major independent  RPG discussion forums to create an overall sample from a list including EN World, RPGnet, UK Roleplayers, RPG Geek, the RPG Bloggers network of nearly 300 blogs, the RPG Blog Alliance of nearly 600 blogs, and Reddit." Though I use the word sample, not census.  Extrapolating sales data - or anything else - from that would not be an appropriate thing to do.  

http://www.enworld.org/forum/hotgames.php



> As far as what you're measuring.  You are only getting the count of "people who post about games" if your algorithm only counts the number of distinct posters over the 90 days once each.  Further you would need to assume that most people don't have at least two boards they use regularly.      You are even farther from getting the number of "people who play each game" (which one of the charts has as a label) since a lot of us seemingly post in threads for systems that aren't related to what we are currently playing.  Going with "Most discussed RPGs" like you say in several places seems appropriately nebulous enough that it isn't misleading.




Yup!  Discussions. That's exactly what the page says!


----------



## Cadence

Morrus said:


> Indeed!  That's almost exactly what the Hot Games page says! "This page tracks discussion".




Sorry, I should have made it clear I was aiming at your post #21 ("statistically significant sample" and whether there's any need to bother with confidence intervals) and not at the actual HG page.   You can't get much more careful in a description than saying "top secret algorithm"!


----------



## Morrus

billd91 said:


> It's not just increasing non-company site portions, by going to company websites, you're adding a lot of bias to the sample. Naturally, a preponderance of discussion on sites like WotC and Paizo will be on their own products, making them particularly biased sources of data. Sites with a particular focus in their history, like ENWorld with its original D&D focus, already add bias to the sample despite being open to discussion of any games because the historical focus generated a sample of gamers with a systematic bias. A company website would be like injecting that sort of sample bias on steroids.




Absolutely. Nobody will deny that a company's website will - nearly always - be the largest source of discussion about their own products.  So the largest source of D&D discussion by far is wizards.com, the largest source of Pathfinder discussion by far is paizo.com, the largest source of Green Ronin's stuff is their website(s), and so on.  

What I would LOVE to do (but I don't believe I can) is somehow separately measure ALL of those and see if the proportions match up to the independent site data.  So if Game X is discussed 3x as much on independent sites as Game Y, is the official site/forum also 3x as busy?  That would be fascinating to find out (and if it were the case, it would indicate that including/not including_ all _of official sites would make no difference overall; but including only_ some _of them would distort the results heavily).

So in that particular case "more data" or "a larger sample" would actually be less representative, and less useful, than less data.  The statisticians in this thread no doubt have special terms for that!  Sometimes a smaller sample is better than a larger distorted sample - although our sample in this case is pretty darn large!


----------



## Morrus

Cadence said:


> Sorry, I should have made it clear I was aiming at your post #21 ("statistically significant sample" and whether there's any need to bother with confidence intervals) and not at the actual HG page.   You can't get much more careful in a description than saying "top secret algorithm"!




To secret algorothm is just a Google page rankings joke. Probably funnier to those of us who run websites!  Basicially, nobody knows Google's algorithm, and SEO experts get paid a lot of money to try to game it - all the while, Google keeps changing it to stop them.


----------



## dd.stevenson

billd91 said:


> It's not just increasing non-company site portions, by going to company websites, you're adding a lot of bias to the sample.



Emphatically no. If Morrus' goal is to answer the question "what rpgs are people talking about on the internet?" then introducing company sites into the survey cannot introduce any bias that cannot be counterbalanced by increasing the non-company site portions. Not unless you define "internet" to exclude company sites.


----------



## Mishihari Lord

Really interesting, Morrus.  Thanks for sharing.   The one point that kind of shocked me is that only 13% of the discussions counted were non-D&D.  Considering the number of games out there, I expected D&D to come in at less than 50% for all versions combined.


----------



## Morrus

Mishihari Lord said:


> Really interesting, Morrus.  Thanks for sharing.   The one point that kind of shocked me is that only 13% of the discussions counted were non-D&D.  Considering the number of games out there, I expected D&D to come in at less than 50% for all versions combined.




Pathfinder is massive; the DDN stuff I imagine will start to fall off again once the latest playtest package wears off.  So that percentage will change over the next month or so (that's just a guess though).


----------

