With lots of notes to be added OK! here are the notes. For anyone curious this was my first attempt with processing.js and it brought back maths and degree memories! I'll leave my notes on processing.js for later, and I'll start with the data notes!
So, people might know this, but I'll put it all down:
AO3 is an internet fanfiction archive, it's currently still in beta and doesn't yet have any APIs. Yesterday they released the ability to filter works by tags enabled again after a bit of a spruce. Previously ALL the tags had been there and it'd been a tag unusable. The new filtering displayed the top ten tags in certain categories, and one of those was Fandoms which made me want to map the crossovers between different fandoms. Tricky, but not too hard, I thought, and a great chance to try out Processing.js. As it turns out? Not quite so simple.
As mentioned, there are currently no APIs, I did a bit of a search to see if they were giving out any raw data dumps or if I could request on to have a play about with. Their support team had recently done some graphs and charts with information about the tags, I spoke to their lovely support team and they offered to see what they could do, and gave me the data they'd used for that.
I was playing about with it when I realised that it was the crossover information that I was really curious about. The team had mentioned that there were going to be issues with looking at crossovers because of both the way people tag their work, and the structure of fandom tags themselves. More notes on this later.
Knowing I'd have to compile the data myself, because I didn't want to run a crawler on AO3's servers, I realised I'd have to limit myself to a small set of data. After a quick look I decided on the top 20 TV Fandoms, sorted by work count. So, for the data itself I regexed myself a copy of the Fandom page. For each of the fandoms seletected, I found their IDs and modified the querystrings on the site so I could pull in how many works matched both tags.
http://archiveofourown.org/works?utf8=%E2%9C%93&work_search[sort_column]=revised_at&work_search[fandom_ids]=187&work_search[other_tag_names]=&work_search[query]=&work_search[language_id]=&work_search[complete]=0&commit=Sort+and+Filter&tag_id=Torchwood (Torchwood vs Smallville, if anyone's interested)
And that got me my raw data!
So this is the bit that will probably interest most people.
You'll note I mentioned the top 20, but there are only 11 fandoms represented on the chart. The original list was as below:
|Stargate - All Series||22813|
|Sherlock Holmes & Related Fandoms||21431|
|Doctor Who & Related Fandoms||16019|
|The Avengers - Ambiguous Fandom||14337|
|Avengers (Marvel) - All Media Types||13629|
|Buffy the Vampire Slayer||11161|
|Arthurian Mythology & Adaptations - All Media Types||8541|
|Teen Wolf (TV)||7387|
|Star Trek: Alternate Original Series (Movies)||6485|
The following were removed because they were not strictly TV fandoms: The Avengers - Ambiguous Fandom, Avengers (Marvel) - All Media Types, Arthurian Mythology & Adaptations - All Media Types, Star Trek: Alternate Original Series (Movies), Sherlock Holmes & Related Fandoms.
The following were removed because they were parent categories that included elements that were either represented, in my opinion and for the purposes of this chart, better by one of their subcategories on the chart, or by one that was the TV element of that fandom. For example Doctor Who & Related Fandoms included the TV movie, the radio plays and many, many more elements, where as Doctor Who included the main show itself. However, Doctor Who & Related Fandoms did cover The Sarah Jane Adventures and Torchwood. As Torchwood was already represented itself on here, I opted to go with removing Doctor Who & Related Fandoms.
Additionally you'll notice Star Trek was removed, this tag covered all of the franchise including the films and all of the TV series. As there was no tag that just covered the TV series, and no individual TV series had made it into the top 20, this was removed. As was Jossverse for a similiar reason, although the largest sub-tag, Buffy had already made it into the top 20.
Stargate - All Series was removed as it's two main sub-tags already appeared in the top 20. This left 11 fandoms to work with.
1. Chaptered fics from different fandoms - The way members use the tags at AO3 means this is in no way a perfect representation of crossovers. For example, people often posts fics from different fandoms which are all created as part of the same writing challenge as chapters of the same story, especially if they are short. These fics end up tagged with all of the fandoms that are in them and can give the illusion of a crossover.
2. AO3 is relatively young - This obviously means that the fandoms that have 'peaked' as it's been around have a higher percentage of works on the site. It's also still an invite required site with a waiting list, so there are people who are unable to add their works. This shouldn't skew it too badly as logically all fandoms would be equally affected by this, I'd assume.
2. Mistagging - Often people don't realise the intent of a tag, for example I'm pretty sure The Avengers - Ambiguous Fandom is meant for the TV series the Avengers from the 60s, but it appears to be being used by some people within the Marvel Avengers fandom.
3. 'Megafandoms' - To a small extent it's seen in this data, with the large links between Doctor Who and Torchwood, and the two Stargate shows seen. When I enquired about the data this was raised as a point and it's a very good one. I limited myself to TV shows partly to avoid some of the larger fandoms in the Movies section *coughAvengerscough* (ilu guys, but you're hard to chart <3).
4. What constitutes a 'crossover'? - Another point raised by the lovely support staff, what constitutes a crossover? For example Doctor Who and Torchwood, having Owen appear with the Doctor, would you call that a crossover? Or, because they're both within the wider Doctor Who megafandom, would that just be a fic under that? Some people would say one, some people would say the other. Looking past that as well, what about fics that use elements from one fandom in another? For example a Dead Like Me and Torchwood crossover where Ianto becomes repear, there are no Dead Like Me characters in it, but it's using all of the mythology and rules (crap, what's the good word for that?) that's built in Dead Like Me. Would that be a crossover?
If you've made it this far, hi! Some odds and ends. As I've mentioned, it was my first try with processing.js and it was fun! I'd have liked to made it more interactive, and got the text not upside down. I'd also have rather drawn the lines a bit better, I might tweak it.
I took the total amount of fics and divided it by 10, so each line on the chart represents 10 'crossover' fics. If two fandoms had less than 10 I placed one line in there.
Thoughts and discussion on every part of this welcomed! Catch me on twitter (@theaeblackthorn). There's probably lots of errors as it's 4am now and I should really be in bed, I just had thoughts and thought I'd put them down :)