In previous posts here and here, I looked at a reading list on Digital History Hacks, and examined the liked between the books and suggested new book, using Amazon's API.
I wrote my scripts with the idea of looking at some other reading lists, one such list is from the Long Now Foundation (list). Read about the foundation here, and listen to their superb seminar series here.
The majority of the code work has been done, we can take the original script (scrape1.py) and tweak it for the new list. All that's required is the new source of data (URL), an appropriate regular expression pattern to identify the ASINs and a new prefix to keen the results separate from previous tinkerings. Here are the changes...
#the page we want to scrape URL='http://www.longnow.org/shop/books/'
# the pattern we want to look for PAT="ASIN\/([0-9]+[X]*)"
# the prefix for the filename filename='ln'
This produces an new pickle of ASINs, and the file name is usefully prefixed, ln_asins.pik. Now all we have to do is alter the filename prefix in graph4.py and that will produce the following SVG, bitmap below.
I though the list would be more connected, and I was initially surprised how few connections there are. On reflection, the Long Now covers a broad range of subjects and the connection between them is the foundation.
Still we can still search through Amazon's similar product suggestion and see what recommendations we can offer. Change prefix in graph5.py and amazon offers up 280 items, we'll use any suggestion with more than a third of the connections of the more connected original. here's the results. (SVG) Well, there's plenty going on here, too much maybe. The neat and tidy cluster in the top right is by the historian and author Daniel J. Boorstin. There's another cluster where three titles, previous unconnected, become connected. So if you're interested in these...
When Good Companies Do Bad Things: Responsibility and Risk in an Age of Globalization
Built to Last: Successful Habits of Visionary Companies (Hardcover)
Guns, Germs, and Steel: The Fates of Human Societies (Paperback)
...the consider these...
The World Is Flat [Updated and Expanded]: A Brief History of the Twenty-first Century (Hardcover)
Good to Great and the Social Sectors: A Monograph to Accompany Good to Great (Paperback)
Blink: The Power of Thinking Without Thinking (Paperback)
The Tipping Point: How Little Things Can Make a Big Difference (Paperback)
The large node is actually a duplicate of another title but an alternate format. Perhaps a future version will include some dupe checking, LibraryThing's API provides for matching ISBNs and titles.
In this post we looked at Digital History Hack's proposed reading list and visualised it using using Graphviz's SVG output.
Now lets finish what was started and add in the book titles which are considered considered similar by Amazon.
From the original list, Amazon suggest nearly 780 titles, that's clearly not useful. I've chosen to limit to those which have at least a third of the number of the most connected.
The new books added as new nodes and have a double circle node shapes.
The Long Tail: Why the Future of Business Is Selling Less of More Wikinomics: How Mass Collaboration Changes Everything Everyware: The Dawning Age of Ubiquitous Computing The Victorian Internet
Heres the code graph5.py and the SVG, bitmap below.
I'm a regular reader of William J Turkel's blog, Digital History Hacks. His recent posts (here and here) about analysing his course's reading list inspired some tinkering of my own.
In William's first post on the subject, he uses the ASIN (Amazon Standard Identification Number) from each of the books in his reading list and Amazon's API to request a list of similar books. This generates a list of paired ASIN, up to 10 pairs per original title.
Original Title ASIN 1, Similar Title ASIN 1 Original Title ASIN 1, Similar Title ASIN 2 Original Title ASIN 1, Similar Title ASIN 3 Original Title ASIN 2, Similar Title ASIN 1 Original Title ASIN 2, Similar Title ASIN 2 Original Title ASIN 2, Similar Title ASIN 3
And as William shows these pairs can be used to construct a graph, which can be used to visualise the degree of connectivity of the reading list.
My code uses parts of Williams's so if it looks familiar, that's why.
First scrape the ASIN from the webpage, my version of this script is here scrape1.py.
So we have our list of paired ASIN, so let's query amazon and get some similar titles. You'll need to get your own Amazon Web Service ID which is a trivial enough process.
I have a couple of simple calls to help extract the data we want, in a module called amazon.py. The process is simple enough.
I found this article, Python MiniDom, useful when playing with the XML data returned by Amazon, and theres a more comprehensive Amazon wrapper project for python, pyAWS. OK, so now we have a list of pairs, we can start to play with Graphviz. There's a lovely Python interface for Graphviz, pydot which I think is great. Basically using pydot, graphs, nodes and edges become objects and this make it very easy to use. Let's start by graphing all the original ASINs, and show links where they are similar to each other.
Load the pickled ASINs
Load the pickled ASIN pairs
Create a graph object
Create a node object for ASIN and add to the graph
Create an edge object for each linked original ASIN
Save the graph
Here's the code, graph1.py , and here the resulting graph.
It's quite clear that many of these books are considered similar by amazon, and some stand out much more than others as hubs in a network.
We can quantify that connectedness, and use the data to alter the graphs appearance.
If we count he number of nodes which connect to or from other nodes, we can apply that to the font size, which is often done in tag clouds.
To do this is quite simple, we'll use a dictionary object to assign a value to each ASIN, and as we cycle through the pairs we'll increment the value.
Then, when we create the node, we'll set the font size.
here's the code to calculate the weights:
# for each ASIN associate a value weight={}for asin in asins: weight[asin]=0
# for each pair for pair in pairs: # only if they are one of the originals if pair[1] in asins: # increment the weight weight[pair[1]]+=1 weight[pair[0]]+=1
and here is where that value is used, and while we're at it let's make the nodes circles.
# add a node for each original titlefor asin in asins: node=pydot.Node(asin, shape='circle', fontsize=8+weight[asin]) g.add_node(node)
The larger circles make quite a difference, but for some reason my system does have the right fonts. I think it still illustrates how by calculating and using the weight data we can make the over all picture clearer. At this point the jpg files are getting large, and the quality is not that great.An alternative graphics format is Scalable Vector Graphics (SVG). Read about it at W3C and at Wikipedia.Firefox has native support for SVG and Adobe provide a viewer.
This format will provide nice fonts and smooth lines and curves, and some other useful features, which we'll get to later.
OK, let's use that weight value again, this time to add some color and view as an SVG. I have a helper module called gradi.py, which has some functions which generate colour gradients, we can use this to calculate colours for our nodes. Starting with yellow for node with no connections, and red for those with the most, everything between we'll make orangish.
Here the modified script, graph3.py and the results, click the image or here for the SVG. I recommend this Firefox Zoom and Pan extension for SVG files.
I prefer the look of the SVG files and the format provides some useful features that bitmaps can't, for instance we can have Graphviz add http links from the nodes and edges, and include tooltips too. Hovering over nodes will show the title, and clicking will open the Amazon product page.
To add this functionality we just query Amazon for the title and include that data as a node attribute.
I though the list would be more connected, and I was initially surprised how few connections there are. On reflection, the Long Now covers a broad range of subjects and the connection between them is the foundation.
Well, there's plenty going on here, too much maybe.
