# Exclusive details on the DDI API (not a repeat...)



## Asmor (Jun 18, 2009)

I've figured out some exclusive details on how to programatically query the compendium. Details on my blog.

[sblock=Copied and pasted, but I'm too lazy to redo it in BB code.]So to cut a short (but uninteresting) story even shorter, I stumbled onto a way to get XML results from Compendium searches, opening the door for people to run queries against the compendium programatically. For example, I’ve just updated my Monster Maker with the ability to search the compendium for monsters and import them (this was previously available, but more complicated as you had to save the monsters to an HTML file and then import that…).

So without further ado, here’s the key:

http://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx/KeywordSearch

You can query that URL using either GET or POST If you don’t know what those are, you’re probably best off using GET, which I’ll describe.

The two parameters which seem to be necessary are “Keywords” and “tab” (I don’t think case matters, but that’s the case I’m using and it works…). Keywords is what you’re searching for, and tab is the type of results you want.

For example, to search for mind flayers, you’d look up:

http://www.wizards.com/dndinsider/c...ywordSearch?Keywords=mind flayers&tab=Monster

Note that the space was escaped to %20. I’m pretty sure you can NOT replace spaces with +. I haven’t tried dashes or underscores.

And that brings up a nice little XML document ready to be fed into whatever silly machinery you’re working on.

Also note that each entry has an associated ID. You can use that ID to find the page for the specific element. In the case of monsters, you use…

http://www.wizards.com/dndinsider/compendium/monster.aspx?id=[insert ID here]

For example, the Mind Flayer Infiltrator is ID 339, so its page is

http://www.wizards.com/dndinsider/compendium/monster.aspx?id=339

The URLs for other elements are…

http://www.wizards.com/dndinsider/compendium/race.aspx?id=
http://www.wizards.com/dndinsider/compendium/class.aspx?id=
http://www.wizards.com/dndinsider/compendium/glossary.aspx?id=
http://www.wizards.com/dndinsider/compendium/deity.aspx?id=
http://www.wizards.com/dndinsider/compendium/item.aspx?id=
http://www.wizards.com/dndinsider/compendium/monster.aspx?id=
http://www.wizards.com/dndinsider/compendium/epicdestiny.aspx?id=
http://www.wizards.com/dndinsider/compendium/paragonpath.aspx?id=
http://www.wizards.com/dndinsider/compendium/ritual.aspx?id=
http://www.wizards.com/dndinsider/compendium/feat.aspx?id=
http://www.wizards.com/dndinsider/compendium/skill.aspx?id=
http://www.wizards.com/dndinsider/compendium/power.aspx?id=

For some info on actually running queries, check out Gaming in Code, a blog run by one of the guys working on DDI. I haven’t tested to make sure that all those filters and such actually work with this, but I strongly suspect they do.[/sblock]


----------



## darjr (Jun 18, 2009)

Thanks for this.


----------



## tomBitonti (Jun 18, 2009)

Asmor said:


> Note that the space was escaped to %20. I’m pretty sure you can NOT replace spaces with +. I haven’t tried dashes or underscores.




That %20 is the escaping of a space in a URI.  See RFC 1630 for more details, or check out any of many references online (e.g. 4.1.2. Unsafe characters)

You might want to be careful before going too far down the line of figuring out the APIs.  As soon as they include access protection, or if they may be considered reverse engineering, that might subject you to the DMCA.  You might also have to deal with issues of unauthorized access if those aren't a part of a public API.

I'd have to do some research figure out what is allowed or not.  This is probably a good opportunity to invest in understanding the DMCA, as well as finding out the WotC policy for how one is allowed to access their web site.

Thx!

TomB


----------



## Asmor (Jun 18, 2009)

tomBitonti said:


> That %20 is the escaping of a space in a URI.  See RFC 1630 for more details, or check out any of many references online (e.g. 4.1.2. Unsafe characters)




I know... That would be why I escaped it thusly. 



> You might want to be careful before going too far down the line of figuring out the APIs.  As soon as they include access protection, or if they may be considered reverse engineering, that might subject you to the DMCA.  You might also have to deal with issues of unauthorized access if those aren't a part of a public API.
> 
> I'd have to do some research figure out what is allowed or not.  This is probably a good opportunity to invest in understanding the DMCA, as well as finding out the WotC policy for how one is allowed to access their web site.
> 
> ...




For a variety of reasons, I'm both not worried about any legal issues and also not worried about them taking umbrage with this use. In fact, I have good reason to believe that this is exactly what they're intending.

Of course, if they do have a problem they can email me and I'll take down anything at their request. But I seriously doubt that's going to happen.


----------



## Scribble (Jun 18, 2009)

Nice work... I've been waiting for the gaming in code part two... But you beat him to it! 

Also your monster make just got more awesomer.


----------



## Scott_Rouse (Jun 18, 2009)

Scribble said:


> Nice work... I've been waiting for the gaming in code part two.





It will go up Monday


----------



## Scribble (Jun 18, 2009)

Scott_Rouse said:


> It will go up Monday




Right on.


----------



## vagabundo (Jun 18, 2009)

Hmm it might be time to brush up on my python.


----------



## wayne62682 (Jun 18, 2009)

I thnk wotc should publish a developer API or something. I have a really great idea fir an iPhone app for D&D gamers - the only thing stopping me from seriously working on it is that I would like to be able to query for info. 

You know what I would *really* like?  A way to have someones character info from the character builder made available via web site, like the character sheet hosting that some sites provided in 3.x. If I could prompt someone for their DDI account and then grab parts of their character info to use in my app that would be awesome.


----------



## malraux (Jun 18, 2009)

wayne62682 said:


> I thnk wotc should publish a developer API or something. I have a really great idea fir an iPhone app for D&D gamers - the only thing stopping me from seriously working on it is that I would like to be able to query for info.
> 
> You know what I would *really* like?  A way to have someones character info from the character builder made available via web site, like the character sheet hosting that some sites provided in 3.x. If I could prompt someone for their DDI account and then grab parts of their character info to use in my app that would be awesome.




Something like http://iplay4e.appspot.com/characters


----------



## Asmor (Jun 18, 2009)

Scott_Rouse said:


> It will go up Monday




Yay! I've been waiting for that!


----------



## wayne62682 (Jun 18, 2009)

malraux said:


> Something like iplay4e characters




Kind of like that, yes.  My iPhone idea is at first a hit point tracker, so it will tell you when you're bloodied and whatnot... nothing major just something simple... I figure even if I can't sell it for money I can use it myself and learn iPhone programming 

If it could read/write to 4e character builder files then it could pull your powers and function like virtual power cards; you pull up your character, tap a power and it displays your bonuses and the like with it.  But that's wishful thinking on my part right now.


----------



## Cadilon (Jun 18, 2009)

This sounds neat and all, but I am no rocket surgeon.  Can you guys give me an example as to how this info from DDI would be used?


----------



## Halivar (Jun 18, 2009)

My word... the possibilities. Scott, is there any way the DDI API could be public so we can make apps for DDI subscribers?

Consider the benefits: each tool made by third-parties is value-add to a DDI subscription that required no resources on WotC's part. For me, I would kill to be able to make an encounter builder that exported colored stat-blocks to PDF. And I'd do it in a heart-beat if I thought distributing such a thing was explicitly approved by WotC.


----------



## miscreationist (Jun 18, 2009)

Considering how legal happy things have been over the last bit, I will be interested to see getting access to the data in the next Gaming in Code article because according to the terms of use when you sign up you agree not to do this.

Wizards of the Coast


> 3. Limits on Use of the Site
> You agree not to engage in any of the following: (i) use any automated means, including, without limitation, agents, robots, scripts, or spiders, to access, monitor, data scrape, copy or transfer any part of the Site (including without limitation any User data, including any Member registration information, whether individually or in the aggregate); (ii) probe, scan or test the vulnerability of the Site, or breach the security or authentication measures on the Site; (iii) reverse look-up, trace or seek to trace any information on any Member or other User of the Site, including information on any Member account other than your own, to its source, or exploit the Site in any way with a purpose of revealing any information, including but not limited to personal identification or information, other than your own information; (iv) take any action that imposes an unreasonable or disproportionately large load on the infrastructure of the Site; (v) use any device, software or routine to interfere or attempt to interfere with the proper working or authorized uses of the Site or with any other person's use of the Site; (vi) forge headers or otherwise manipulate identifiers in order to disguise the origin of any message transmittal you send on or through the Site; (vii) impersonate any other individual or entity or misrepresent your identity or your affiliation with another individual or entity; (viii) use the Site in an illegal manner or for any unlawful purpose; or (ix) violate any applicable Guidelines, including without limitation any Code of Conduct.




If this is now not the case, then I hope that these terms will be amended.


----------



## wayne62682 (Jun 18, 2009)

Halivar said:


> My word... the possibilities. Scott, is there any way the DDI API could be public so we can make apps for DDI subscribers?
> 
> Consider the benefits: each tool made by third-parties is value-add to a DDI subscription that required no resources on WotC's part. For me, I would kill to be able to make an encounter builder that exported colored stat-blocks to PDF. And I'd do it in a heart-beat if I thought distributing such a thing was explicitly approved by WotC.




Exactly.  WotC could really capitalize on D&DI by having a "Developer API" that we can use to create applications that make use of D&DI info; sort of like a D&D App Store.


----------



## Alikar (Jun 18, 2009)

miscreationist said:


> Considering how legal happy things have been over the last bit, I will be interested to see getting access to the data in the next Gaming in Code article because according to the terms of use when you sign up you agree not to do this.
> 
> If this is now not the case, then I hope that these terms will be amended.




That is for the site, not DDI.


----------



## miscreationist (Jun 18, 2009)

Alikar said:


> That is for the site, not DDI.




Exactly.  And in the member terms of service for DDI it says in the first paragraph
that "In the event of a conflict between these Member Terms and the TOU, these Member Terms shall control."  



> These Member Terms of Service (“Member Terms”) are a contract between you and Wizards of the Coast, Inc. ("Wizards"), and apply to your registration for and use of products and services (collectively the "Service") available to registered users or paying subscribers ("Members") through the Wizards website (the "Site"), and supplement the Wizards Website Terms of Use applicable to the Site ("TOU"). In the event of a conflict between these Member Terms and the TOU, these Member Terms shall control. The purchase or use of certain products and services available through the Site by Members may also be subject to certain additional terms and conditions.




There is a section under "2.User Content; Intellectual Property Ownership; Rights Granted to You":



> b. Intellectual Property Ownership; Rights Granted to You The Service is the proprietary property of Wizards or its licensors, and is protected by copyright, trademark, and other intellectual property laws. Except as otherwise provided herein, you are granted a limited, non-sublicenseable license to access and use the Service (except where specified as prohibited) for your personal, noncommercial use only; provided, that you preserve any copyright, trademark or other similar notices contained in or associated with such Service. You agree not to use the Service in an illegal manner or for any unlawful purpose. Such license is subject to these Member Terms and does not include: (a) any resale or commercial use of the Service therein; (b) modifying or otherwise making any derivative uses of the Service, or any portion thereof; or (c) any use of the Service other than for its intended purpose. Any use of the Service other than as specifically authorized herein, without the prior written permission of Wizards, is strictly prohibited and will terminate the license granted herein. This license is revocable at any time.




I am not a lawyer, but, It does not specifically allow one to automate extraction of data, so my question still stands.

My reading (I could be wrong though) indicates that you have personal use except in the case of derivative use, any use other than the intended purpose, or any resale or commercial use.  I think the derivative uses clause is the most worrisome to me.  I'd like some clarification that this is going to be fine.  I am hoping you are right and I am giving them the benefit of the doubt (especially because Scott posted here).

Scott, can this be clarified?  Is using this API going to be ok for private DDI users to use the data?


----------



## Asmor (Jun 18, 2009)

Stop getting caught up in legalese. Seriously.

As long as you're not doing it for profit, you won't get sued. I promise. Worst case scenario, they'll send you a C&D.

But just look. They've already started documenting the API themselves (which is what lead to my getting interested in it in the first place). At the end of that post on Gaming in Code, he said the next part would cover how to programatically access the data. The only issue is that it's taken to freaking long for the second part to come out.

And now The Rouse is here saying that the next part (Coincidentally? Or did I give them a kick in the butt? ) will be up Monday.

It is absolutely clear that *this is what Wizards of the Coast wants people to do*.


----------



## miscreationist (Jun 18, 2009)

Asmor said:


> Stop getting caught up in legalese. Seriously.
> 
> As long as you're not doing it for profit, you won't get sued. I promise. Worst case scenario, they'll send you a C&D.
> 
> ...




Yeah, I guess you are right.  That is what has been done thus far is to send C&Ds.  But I figured they'd go to account banning before that if you were automating the extraction of data and they didn't like it.  The only reason I had not taken an interest thus far was to avoid getting my account removed for automating access to it - something I'd not like to happen if I'd paid for an extended amount of access to DDI.

Thanks for bringing up the topic. Hopefully, this is a good step and as I said, I look forward to the upcoming article about it.


----------



## tomBitonti (Jun 18, 2009)

Asmor said:


> Stop getting caught up in legalese. Seriously.
> 
> As long as you're not doing it for profit, you won't get sued. I promise. Worst case scenario, they'll send you a C&D.
> 
> ...




IMO, this is bad advice.  Working for a _very_ major corporation, and having mandatory yearly education on this sort of stuff, generally, when in doubt, the first step is to *stop* and consult the legal team.

In this case, I don't think Mr. Rouse's posting is sufficient to present the WotC policy.  I would be careful until the online agreement and until the posted policy on the website are updated.  Mr. Rouse's posting is a positive sign, but until there is an update that is clearly from the WotC legal team, some care should be used.  (I'm actually surprised at the posting the Mr Rouse has made: That could put WotC in some danger, and is the sort of statement that usually has to be put through legal clearances.  I can't tell if that has been done.  If it has, my apologies.  In this case, absent a very clear statement that the policy has legal approval, I can't accept it as a definitive statement.  Also, this is an area where you really don't want multiple streams of statements.  That muddies the water where you want very clear policies.)

Basically, I'm looking for an official statement that appears on the WotC website that can be provided through a link.

[PostScript: Under the DMCA, a Cease and Desist is not nearly the worst case scenario.]

Thx!

TomB


----------



## Asmor (Jun 18, 2009)

miscreationist said:


> Yeah, I guess you are right.  That is what has been done thus far is to send C&Ds.  But I figured they'd go to account banning before that if you were automating the extraction of data and they didn't like it.  The only reason I had not taken an interest thus far was to avoid getting my account removed for automating access to it - something I'd not like to happen if I'd paid for an extended amount of access to DDI.
> 
> Thanks for bringing up the topic. Hopefully, this is a good step and as I said, I look forward to the upcoming article about it.




It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.

I can send you to the page, but you still need to log yourself in.

Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.


----------



## miscreationist (Jun 18, 2009)

Asmor said:


> It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.
> 
> I can send you to the page, but you still need to log yourself in.
> 
> Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.




Right.  I understand that this is totally different from screen scraping and it is clear that they are documenting the API.   I took a look at the Compendium Search service and noticed that everything accessible through its operations is data that is accessible to the public without having to log in.  If there is no access to data that requires a login then I suspect the legal questions still stand about automated access to it (and I suspect that operations like that are against the TOS).  I hoped that the next article might provide access (with authentication) to the data through the API.

Here's to hoping.


----------



## tomBitonti (Jun 18, 2009)

*Some usage that seems problematic ...*

Thinking more about the access, there are a couple of access types that seem to be problematic:

1) Anything that does queries and caches the results;
2) (1), where the cache is made available to other people.
3) *Providing a forwarding service, where a third part contacts you through an API of your design, which prompts your API to log into the for-pay DDI using your login information to complete the request.*

In a group environment, I can see (3) being very useful, where the DM and players each has network access, and where the DM has a portal to the DDI, and where the players access the DDI through the DM's portal.  That seems useful enough, and simple enough to implement, that folks are very probably already thinking about doing it.  That also seems to be exactly what WotC would *not* approve of.

Note that a web-site may have a policy that prohibits access to secondary pages without your having visited initial pages, and this is (?) actionable (?) enforceable, and has been tested through case law.


----------



## malraux (Jun 19, 2009)

Asmor said:


> It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.
> 
> I can send you to the page, but you still need to log yourself in.
> 
> Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.




Yeah, I could code up an automator script to get safari to pull everything down, but that would still be my web browser getting the info.

On a related note, have you figured out what the code is to log in.  obviously you'd want to post email and password but I still haven't figured out where that goes.


----------



## Asmor (Jun 19, 2009)

malraux said:


> Yeah, I could code up an automator script to get safari to pull everything down, but that would still be my web browser getting the info.
> 
> On a related note, have you figured out what the code is to log in.  obviously you'd want to post email and password but I still haven't figured out where that goes.




Looks to be login.aspx



> <form name="form1" method="post" action="login.aspx?page=monster&id=2351" id="form1">




Here're the relevant bits:



> <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAK5u9HLAQKyzcaDDQLyveCRDwK4+vrXBTI0l5j0hL3ej514UF0fpXH07GN4" />
> <input name="email" type="text" id="email" />
> <input name="password" type="password" id="password" />


----------



## miscreationist (Jun 19, 2009)

Asmor, from your article, http://www.wizards.com/dndinsider/c...ywordSearch?Keywords=mind+flayers&tab=Monster

Using + works fine.


----------



## Asmor (Jun 19, 2009)

Huh. That's strange. I swear I tried + and it didn't work. o_0 Thanks.


----------



## miscreationist (Jun 23, 2009)

The second part of the info on the API went up D&D Insider: Compendium API - Part 2 yesterday.

One thing to note is that in my testing, null is needed instead of NULL in the filters.  I saw the KeywordSearchWithFilters method the other day, but now that the filter strings have been disclosed, it makes it really useful.

Apparently they will be disclosing how to get to the data in a third article which addresses "security issues and concerns".  This does indeed look promising and it is clearly meant for automation, but I still think that their TOS conflicts with the use of this.  Perhaps it will get a refresh.


----------



## Asmor (Jun 23, 2009)

miscreationist said:


> Apparently they will be disclosing how to get to the data in a third article which addresses "security issues and concerns".  This does indeed look promising and it is clearly meant for automation, but I still think that their TOS conflicts with the use of this.  Perhaps it will get a refresh.




There's a fanpage policy in the works... Should have been out a while ago, frankly, though given the timeline of the vastly-more-important GSL I'm not all that surprised by the delay. My guess is that it's "in legal," since that seems to be the usual refrain.


----------



## catsclaw227 (Jun 28, 2009)

I am getting an application error when accessing the Webservice.  Typical .NET error if the custom errors aren't set up.

Is it currently down or inaccessible, or is the webservice itself not working unless you are logged in?

I have logged into the DDI and then in a different browser tab, gone directly to the webservice and used the KeywordSearch method with keywords="Human Guard" and tab="Monster".  Even trying an HTTP GET and I get the same error.


----------



## Asmor (Jun 28, 2009)

You don't need to be logged in to perform searches.

Double check and make sure the service is actually working. I've found it has a knack for noticing when I want to work on a program and it decides to stop working for a while. :/


----------



## catsclaw227 (Jun 29, 2009)

As it turns out the WSDL the webservice has tied to it, brings back incorrect URLs for the service.


```
<wsdl:service name="CompendiumSearch">
    <wsdl:port name="CompendiumSearchSoap" binding="tns:CompendiumSearchSoap">
      <soap:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchSoap12" binding="tns:CompendiumSearchSoap12">
      <soap12:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchHttpGet" binding="tns:CompendiumSearchHttpGet">
      <http:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchHttpPost" binding="tns:CompendiumSearchHttpPost">
      <http:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
  </wsdl:service>
```


----------



## Asmor (Jun 30, 2009)

Good call, catsclaw. I was able to get it working by specifying the address myself, e.g.


```
var c = new CompendiumSearch.CompendiumSearchSoapClient("CompendiumSearchSoap12", @"http://www.wizards.com/dndinsider/compendium/compendiumsearch.asmx");
```


----------



## catsclaw227 (Jun 30, 2009)

Yea, I ended up doing that too.  

Also, playing around with the API without creating a webreference to the DDI service, I also used a simple approach and just loaded the url into an XDocument.  Then I can easily traverse the monsterlist with Linq to Xml.


```
XDocument monsterXml = GetSearchResults("http://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx/KeywordSearch?Keywords=Human+Guard&tab=Monster")


    public XDocument GetSearchResults(string url)
    {
        XDocument resultsXml = new XDocument();
        try
        {
            resultsXml = XDocument.Load(url);
        }
        catch (Exception e)
        {
            throw e;
        }
        return resultsXml;
    }
```


----------



## catsclaw227 (Jun 30, 2009)

I am using the API to be able to take the monster data and convert it into a fully functional rptok file for MapTools and the 4e framework I am using (Rumble's).  

Still, it's a work in progress, and I am hesitant to write an HTML parser for the monster.aspx page because they are likely going to release the individual elements (like monster) API soon and I don't want to waste too much time writing a throw-away.


----------



## lacan2002 (Jul 12, 2009)

Hey, great thread! Any ideas on a general place to talk about DDI Api?
Seems like there are a number of interested D&D player/programmers out here, it would be nice to have a place for the hobbyist to share ideas.

It looks like they've added a needed NameOnly parameter, which I've just been setting to "false".

Thanks very much to Asmor's examples, I got a simple ruby client working.

For the interested, this requires the excellect xml-object and restclient gems, but once those are ready, getting an implicit object model that allows you to deal with the meat of the data is dead simple:


```
xml = RestClient.get('http://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx/KeywordSearch?Keywords=mind+flayers&tab=Monster&NameOnly=false')
data = XMLObject.new(xml) # That's it! Gives us a model implicitly!

# get each monster and show some data
data.Results.Monster.each do |monster|
  puts "#{monster.Name} is found in #{monster.SourceBook}"
  # do something more interesting, like put into a ul in HTML
end
```

The results for this look like this:

Mind Flayer Infiltrator is found in Monster Manual
Mind Flayer Mastermind is found in Monster Manual


----------



## Blakey (Jun 14, 2010)

I hope I don't get lynched for Thread-romancy, but is this mechanism still valid and still alive?   I've been trying it recently and all I get are 500-errors.

If not, is there a more up to date mechanism?    This whole topic is amazingly difficult to find useful, up to date and correct info about...

Cheers
Blakey


----------

