Blog:

To most, the virtues of Web 2.0 are rather ephemeral; that’s always been one of its main criticisms. However, I like to think that one of the movement’s key aspects is a sense of community, an ability to create sites and applications that bring people together.

One of the most powerful ways to bring people together on the Web is to give them ownership of the sites that they frequent – make them active participants in the development and growth of the product they’re using. There’s many different approaches you can take to implementing this idea, but I’m going to talk about just one of them: setting your data free.

Data sharing isn’t a new idea. It’s a concept as old as the Web, in fact, the Web is all about sharing data. However, most of the data available on the Web is designed to be consumed by humans, through a browser. It’s encapsulated by HTML pages, trapped inside Flash files, and disguised by design. There used to be a time where, if you wanted to get any real data from a site that wasn’t your own, your only recourse would be to grab the raw HTML and scrape through that tag soup for the gem that you were looking for. Fancy writing a 15-line regular expression to get the weather off Yahoo!?

However, data providers are slowly realising that offering their data in much more flexible formats can only be an advantage to them – they get used by more applications, they get seen by more eyeballs, they get more exposure.

RSS is a great example of this. Strip all the images, colour, layout, font styles and browser dependence from a Web page, and what do you have? A lightweight, versatile data format that can be easily interpreted by pretty much any software out there. Two years ago it was barely on the radar. Now your green grocer has an RSS feed that tells you when the latest shipment of fresh guava has arrived.

RSS is good, but it’s still a bit restrictive for real application development – the content provider tells you exactly what you’re going to get, and when you’re going to get it. What if, instead of getting the last 10 blog posts from your favourite incendiary author, you could get their last 10 posts that were tagged with “free beer”; or you could get their last 10 posts that were tagged with “free beer” and were written in Melbourne?

What if you could take their last 10 moblog posts that were tagged with “free beer” and written in Melbourne, get the geolocation data of each post, plot the exact position of each free beer spot on a street map and show it in relation to your current location? That’s a good night out. And APIs can let you do it.

Application Program Interfaces are, broadly speaking, tools for building software applications. An operating system API might help you build applications with a consistent interface, but Web APIs are mostly about giving you access to data.

Even Web APIs aren’t a new idea. Google’s search API has been available via SOAP since 2002, and there’s definitely older services than that. However, the recent growth in Web API availability has been fuelled by two recent developments. The first, which I’ve already mentioned, was a philosophical change in the way that data is handled. The second was the introduction of AJAX. Again, not a new idea, or even a new technology, but sometimes it’s all about timing.

Some of the more interesting JavaScript-friendly Web APIs that are available at the moment are:

• Flickr (Photo sharing API)
• Del.icio.us (Social bookmarking API)
• Google Maps (Mapping API)
• Yahoo! Maps (Mapping API)
• Upcoming (Event planning API)
• Yahoo! Search (Search engine API)
• eBay (Online auction API)
• Amazon (E-commerce API)

The mapping APIs have received the most attention to-date, mainly through mapping “mashups” such as chicagocrime.org, or dartmaps.mackers.com. They’re also probably one of the least flexible, because you are constrained by the mapping interface that the provider gives you. When you include the Google Maps API on your page, you gain access to a number of JavaScript functions that allow you to move and zoom the map, or add your own data and visual components, but you aren’t able to manipulate any of the original data, beyond what the existing JavaScript methods allow you to do. This is perhaps just a constraint of maps themselves – how many different forms can mapping data take? A good tutorial on how to create an application with the Google Maps API is available at Hacking Maps with the Google Maps API.

Other forms of data, however, offer greater possibilities.
These web applications use AJAX to query the various non-map APIs and create new and unique ways to interact with data:

• Delicious Director (Uses the del.icio.us API)
• Yahoo! local events browser (Uses the Yahoo! search engine APIs)
• FamiliarFaces (Uses the upcoming.org API and the Yahoo! image search API)
• Mindsack integrated searching (Uses the Yahoo! web search API)

We’ll be taking a look at the how they access those APIs, so you can get right in and make your own data-sharing application.

Accessing Web APIs with AJAX

Most Web APIs are accessed via HTTP, so they’re always going to be available to server-side scripts, however, the nexus that AJAX enables between user interaction and data communication, as well as the ease with which it can be incorporated into webpages, means that the use of APIs by client-side scripts offers some of the most exciting possibilities for Web applications.

The API for Flickr has a number of methods that are specified by passing particular CGI parameters to a base script. For instance, to get a list of the most recently uploaded photos to all of Flickr, you would access their script:


http://www.flickr.com/services/rest/?method=flickr.photos.getRecent&api_key=XXXX


The “method” specifies exactly which method from the API you want to execute, and “api_key” is a way for Flickr to track who’s using what from their API. If you want to see the data that this call returns, simply sign up for your own Flickr API key, insert it into the string above, paste that string into your browser location bar, and you’ll get a nicely formatted XML file in return. Something like:

The same process applies for an XMLHttpRequest via JavaScript – you specify the URL above to your XMLHttpRequest object, and the client browser will go off and fetch it. (For a more in-depth look at creating XMLHttpRequest connections, see Remote Scripting with AJAX) Because you get XML in return, you’ll be able to use JavaScript’s in-built DOM functions to parse the data in much the same way as you do for HTML; navigating through elements, getting attributes, etc. However, as with any XMLHttpRequest, the data doesn’t necessarily have to be XML, so a Web API could send back a simple text string instead. In fact – as we’ll see later – that’s what JSON does.

XMLHttpRequest Proxies

Even though it’s theoretically possible to request this data from Flickr using XMLHttpRequest on the client-side, there’s one problem: cross site scripting. Currently, most browsers will not allow you to get data from other sites using XMLHttpRequest. Due to security restrictions that prevent cross site scripting (XSS), they will not let JavaScript access a URL that is outside of the domain that the current page is being served from. This even applies to sub-domains on the same site, for example www.example.com versus dev.example.com.

This is not an intractable problem, though. There are a couple of ways you can get around it, but they all require the server on the current domain to act as a proxy for the external data. This works because server-side scripts generally have more privileges than client-side scripts, so they are able to access any address on the Internet. With a server-side proxy, we create a script that accesses a remote server, gets the data and relays it back to the browser, making it look like the data was sent from www.example.com, when in reality it came from www.upcoming.org, or some other website which we proxy data for.

Apache Proxy

The easiest way to proxy data is to get your web server to do it for you. You can set up a proxy in Apache by making sure the mod_proxy extension is loaded and editing the httpd.conf file to include your new proxy:


ProxyPass /proxy/flickr/ http://www.flickr.com/
ProxyPassReverse /proxy/flickr/ http://www.flickr.com/


The first line forwards any requests from your server to Flickr’s, the second line handles any redirections that the Flickr server might respond with.

However, if you’re running a site off a shared server, you more than likely won’t be able to edit httpd.conf, so you can always use mod_rewrite to redirect requests by editing a .htaccess file. If I put this inside the .htaccess file for the directory /proxy/flickr/:


RewriteEngine on
RewriteRule ^(.*)$ http://www.flickr.com/$1 [P]


Anytime I make a call to a path inside http://www.example.com/proxy/flickr/, it will pass on all extra URL information to http://www.flickr.com/. So, when I reference http://www.example.com/proxy/flickr/services/rest/ it’s actually passing the request through to http://www.flickr.com/services/rest/. The [P] at the end of the RewriteRule means that the re-direction is passed through without informing the user’s browser that a change of URL has occurred.

Application Proxy

The second method to proxy data is to create a script that will do it for you – receive request URLs, create a connection, get the data, and return it to the requester. The exact code that you use to do this will differ depending upon what language you’re using, but if you are running PHP with the Curl extension installed (for creating external connections), then you could use this as your proxy script:


$remoteServer = "http://www.flickr.com";
$path = $_GET[”path”];
$proxyTarget = $remoteServer.$path;

header(”Content-Type: text/xml”);
echo $data;


This script assumes that you are passing a CGI parameter called “path” that specifies what location you want to retrieve from the remote server. Note that we don’t directly pass on this parameter to the Curl connection, as we don’t want this script to become an open proxy. Instead, we specify the exact address that we are using the proxy for – www.flickr.com – and this will limit its potential for abuse. You’d call it from a browser using this URL:


http://www.example.com/proxy.php?path=/services/rest/


If you’re sending that over an XMLHttpRequest connection, you should remember to URL endcode it using escape(), so that any special characters inside the path are passed along properly.

When we initialise the Curl connection, we tell it that we don’t want it to pass on any header information, but we do want it to pass on the content of the request. Then, once the data has been received we write the appropriate header back to the client browser and also forward on the data which we retrieved from the remote server.

The downside of proxies is that you’re double handling your data every time – a request has to come to the current domain server, which then passes it along to the actual remote data server. Then the data has to go back to the current domain server before it can be sent to the client browser. This extra traffic will always introduce some delay; the amount of which depends on the network architecture and speed of the highway at any given time, so you’ll have to ask whether that delay (however minimal it is) is acceptable to your service.

JSON

Another way to avoid XSS issues with XMLHttpRequest is to not use it all. How do you do that? You use JSON.

JSON (short for JavaScript Object Notation) is an alternative format to XML. It actually stores data as an object literal – a structure that is natively available in JavaScript. So if you receive a JSON string you can use the JavaScript evaluation function eval() and it will automatically create an object structure that corresponds to the data that you want to retrieve. Although object literals are native to JavaScript, they’re farly easy for other languages to parse. Indeed, at json.org there are links to JSON parsers for over 20 different programming languages.

Using eval() on a text string that you receive from a 3rd party can be a little scary – it opens up all sorts of security holes that could be exploited in your code, so you really have to trust the data provider if you’re blindly using JSON. There’s some parsers available that have been written in JavaScript by Douglas Crockford – the creator of JSON – that will offer some protection against malicious code. These are also available at www.json.org.

Douglas calls JSON the “fat free alternative to XML”, mainly because it’s less verbose and requires less parsing than XML, particularly if you’re using it from inside JavaScript. If we translated the XML we had from Flickr earlier into a JSON structure, it might look like this:

{
	"rsp": {
		"stat": "ok",
		"photos": {
			"page": "1",
			"pages": "10",
			"perpage": "100",
			"total": "1000",
			"photo": [
				{
					"id": "169563197",
					"owner": "98319521@N00",
					"secret": "f4002ac03f",
					"server": "67",
					"title": "turkije021",
					"ispublic": "1",
					"isfriend": "0",
					"isfamily": "0"
				},
				{
					"id": "169563188",
					"owner": "17208601@N00",
					"secret": "fbf107bffb",
					"server": "56",
					"title": "PICT0221~1",
					"ispublic": "1",
					"isfriend": "0",
					"isfamily": "0"
				}
			]
		}
	}
}

It’s entirely possible to use JSON via XMLHttpRequest – make a request to the server, receive a text string and evaluate it – but one of its real advantages is that you don’t need XMLHttpRequest in order to get it. Because JSON is a JavaScript string, if you specify a server-side script that returns JSON as the source for a