Where are you and what is around you?

Geolocation is a hot topic. Google just got the patent on geolocated advertising, mobile phones allow us to pinpoint ourselves on the planet and find things nearby and with augmented reality applications we can even find our way by filming our surrounding and finding hidden treasures by moving our mobile around. Using geolocation as a developer is quite easy, you can do a few things:

• If you are building something in a social network, you can get the geographical location from the user’s profile.
• If you work on a certain mobile platform (Android, WebOS, iPhone) you get APIs to detect the current location.
• If that is not an option you can use the W3C geo location API in browsers that support it.
• If everything else fails you can guess the visitor’s location from their IP number.

That is the where, but how to know what is around me?

Knowing the location is one thing, but what if you want to know more about the area? What about the geographical hierarchy? What part of the city/country are you in and which other geographical and administrative areas are nearby?

All of this has been available for you for quite a while. The GeoPlanet API and dataset released by Yahoo! has been out for a while but did not quite get the love from the mainstream developer crowd it deserves. The geo hackers, on the other hand already love it to bits and helped make it more accurate by providing feedback.

To make it a bit easier for you to understand what the GeoPlanet API allows you to do, I’ve put together the GeoPlanet Explorer – a tool that lets you explore all the GeoPlanet data in an interactive way:

The Geoplanet Explorer uses three tools: YQL for accessing and filtering data, YUI3 for the rich interaction and CSS layout, Yahoo Maps for display and PHP to glue all of that together. You can get the full source code on GitHub.

Simplifying matters using woeid

One big problem of geo location is ambiguity. If you have a location with latitude and longitude you can pinpoint it on the globe. You don’t know what it is though. The centre of a certain county, city and a point of interest could share the same latitude and longitude, but they are actually totally different things.

To fix this problem, Yahoo! a long time ago introduced the Where on Earth ID (or short woeid) which is a number that defines a place with its geographical location and describes what it is. For example my neighbourhood in London has the woeid of 36239 and that means I can get the following information from it when querying the GeoPlanet API:

• Stoke Newington (Suburb):
  • Country: United Kingdom
  • WOEID: 36239
  • Administrative:
    • England (Country)
    • Greater London (County)
  • Localities:
    • London (Town)
    • Stoke Newington (Suburb)
  • Postal N16 (Postal Code)
  • Location (lat/lon): 51.561199, -0.082980
  • Bounding Box: NE 51.577190, -0.058070 SW 51.546692, -0.092380

The bounding box information is very useful for displaying this information on a map as it shows you which points need to be visible in order to see the whole place.

There is just something pretty about having a number instead of a lat/lon pair which is why woeid is rapidly becoming a standard. Flickr supported it for quite a while, Dopplr uses it under the hood, the Yahoo! Weather API understands it and soon Twitter will also use it for geolocating content in an unambiguous way (right now there is only support in the Twitter Trends API).

So how could you use this for your own products?

There are a lot of ways you can use the geo tools by Yahoo! in your own solutions – by far the easiest is using YQL to access the data. This, for example allowed me to write the addmap.js solution:

Using this you can can easily take the content of a page element with a certain ID and add a map of the locations it “talks about” to the document:

<script src="http://github.com/codepo8/geotoys/raw/master/addmap.js"></script>
<script>
addmap.config.mapkey = 'YOUR_API_KEY';
addmap.analyse('content');
</script>

The API key is needed for the Google maps, not for YQL. YQL in this case under the hood uses Placemaker which is an API that finds geographical content in URLs or texts.

YQL and the geo tools offer you some other interesting options:

Geographical locations in texts/urls/RSS feeds

By using the Placemaker table in YQL you can extract geographical locations from a texts or a source on the web:

select * from geo.placemaker where documentContent =
"Hi, I am Chris, I live in London but actually I am from Germany"
and documentType="text/plain" and appid=""

This finds both “London” and “Germany” as geographical locations. You can see the result of this here as an XML file. You will find places and references. Places are what has been found and references where it was found. The start and end elements describe the character location in the whole string.

Placemaker also takes feed and web site URIs and finds the locations in them for you. For example to see the locations mentioned in my portfolio page you can do the following:

select * from geo.placemaker where documentURL =
"http://icant.co.uk" and documentType="text/html" and appid=""

Again, you can check the result as an XML file. As this is an HTML document there is no start and end but the references are reported as XPATHs.

Feeds are also supported, so if you want to get the locations in the BBC News feed you can use the following:

select * from geo.placemaker where documentURL =
"http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml"
and documentType="text/rss" and appid=""

Verify the result as an XML at your leisure – this time the XPATHs point to the RSS items.

The easiest way to see the power of Placemaker is using GeoMaker which gives you a simple interface to do turn a text or URL into a map or microformats to embed into your documents:

Location by IP

Getting the geographical location of an IP is possible with the following YQL statement:

select * from geo.places where woeid in (
  select place.woeid from flickr.places where (lat,lon) in (
    select Latitude,Longitude from ip.location where ip = "123.23.23.33"
  )
)

You can see this as an XML file here.

What we’re doing here is using the IP2location API ,get the woeid from the latitude and longitude using the Flickr findByLatLon method and run the result through the geo API to get all the information – quite some work off your hands, isn’t it? Without YQL you’d have to sign up for all these APIs and understand their inner workings to get the same information.

Getting location by IP is an old trick in the book of advertisers – you probably will have encountered ads like “meet singles in your city now” and see them changing when you went to another place – this is what they use to get this information. The accuracy of IP location is not very high though. It also can feel a bit creepy.

Location by Latitude and Longitude

Therefore it is much better to use a system that allows people to opt-in to tell their geographical location. There are services for that but we also have a W3C API built into browsers. Right now Firefox and Mobile Safari support it but it also cropped up in the latest developer build of Chrome.

Using the W3C location API is easy:

<script>
if(navigator.geolocation){
  navigator.geolocation.getCurrentPosition(function(position){
    var lat = position.coords.latitude;
    var lon = position.coords.longitude;
    alert('you are at' + lat +','+lon);
  },function(error){
    alert('Couldn\'t get your location :(');
  });
}
</script>

You test if the browser supports navigator.geolocation and if it does you use the getCurrentPosition() method to retrieve the current position. This will cause the browser to ask the visitor if they want to share their information – normally in an information bar on top of the viewport as shown in this screenshot:

The getCurrentPosition() method takes two parameters which are function to call in the success and the failure case – meaning that the visitor allowed you to get their location or not – or that there was some other error.

You can then use YQL once more to get the location from latitude and longitude – this time using the Flickr API method as the first one:

select * from geo.places where woeid in (
  select place.woeid from flickr.places where lat=51.5142271 and lon=-0.1289602
)

See the XML output here.

Putting it all together

Let’s end with an example script how you can use this. We use the W3C geolocation API to get a visitor’s location and if it isn’t available we fall back to using the IP. In any case, we use YQL to get the location information and the areas around the current location. The solution uses JavaScript except for retrieving the IP which is done in this case by PHP. You can see the full demo in action here. See the comments // for information about what is going on.

<script type="text/javascript" charset="utf-8">
<?php
  if ($_SERVER['HTTP_X_FORWARD_FOR']) {
   $ip = $_SERVER['HTTP_X_FORWARD_FOR'];
  } else {
   $ip = $_SERVER['REMOTE_ADDR'];
  }
  if(preg_match('/^[\d+\.?]+$/',$ip)){
    echo 'var IP = "' . $ip . '"';
  }
?>

// ^^
// This PHP block will get the current IP and assign it to a JavaScript
// variable - if it is the right format.

if(navigator.geolocation){
  navigator.geolocation.getCurrentPosition(
    function(position){
      getDataForLatLon(position.coords.latitude,
                       position.coords.longitude);
    },
    function(error){
      if(IP){
        getFromIP(IP);
      }
    });
} else{
  if(IP){
    getFromIP(IP);
  };
}

// ^^
// we test if the W3C location API is supported and if it is, we
// get the current lat and lon and call the appropriate method.
// if there was an error or the API is not available at all we call
// getFromIP.

function load(yql,cb){
  var src = 'http://query.yahooapis.com/v1/public/yql?q='+
            encodeURIComponent(yql) + '&format=json&callback=' + cb + '&'+
            'env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys';
  var head = document.getElementsByTagName('head')[0];
  var s = document.createElement('script');
  s.setAttribute('src',src);
  head.appendChild(s);
}

// ^^
// YQL is a web service that returns JSON. This allows us to create
// script nodes dynamically to get information from it. This function
// does that for us.

function getDataForLatLon(lat,lon){
  var yql = 'select * from geo.places where woeid in ('+
            'select place.woeid from flickr.places where lat='+
            lat + ' and  lon=' + lon + ')';
  load(yql,'foundLocation');
}
function foundLocation(o){
  if(o.query.results){
    var place = o.query.results.place
    var out = '<p>You are in ' + place.name + ' which is a '+
               place.placeTypeName.content + ' located at ('+
               place.centroid.latitude + ',' + place.centroid.longitude+
               ')</p>';
    document.getElementById('location').innerHTML = out;
    getBelongTos(place.woeid);
  }
}

// ^^
// If we can get a lat and lon from the API, we put together the right
// YQL statement. If we are successful we call the foundLocation() method
// which writes out the right HTML to the document and calls the
// getBelongTos() method

function getFromIP(ip){
  var yql = 'select * from geo.places where woeid in ('+
            'select place.woeid from flickr.places where (lat,lon) in('+
            'select Latitude,Longitude from ip.location'+
            ' where ip="'+ip+'"))';
  load(yql,'foundLocation');
}

// ^^
// If there was no luck with the W3C geo API we call this function using
// the IP we got from PHP

function getBelongTos(woeid){
  var yql = 'select * from geo.places.belongtos(0) where '+
            'member_woeid='+woeid;
  load(yql,'foundBelongs');
}
function foundBelongs(o){
  if(o.query.results){
    var out = '<p>Your location belongs to: ';
    var place = o.query.results.place;
    var info = [];
    for(var i=0;i<place.length;i++){
      var cur = place[i];
      info.push(cur.name + ' (' + cur.placeTypeName.content + ')');
    }
    var out = '<p>Your location belongs to: ' + info.join(', ') + '</p>';
    document.getElementById('location').innerHTML += out;
  }
}

// ^^
// If we found a location, then we call the belongtos method of the
// GeoPlanet API to get all the areas the current location belongs to.

</script>

In my case the result from the W3C geo location is:

You are in St. Giles’s which is a Suburb located at (51.516151,-0.125460)

Your location belongs to: WC1V 6 (Postal Code), London Borough of Camden (Local Administrative Area), 0207 London (Zone), Middlesex (Historical County), London (Town), Greater London (County), MMA London (MMA), South East England (Colloquial), DMA South East England (DMA), England (Country), England and Wales (Colloquial), Great Britain (Colloquial), United Kingdom (Country), Market United Kingdom and Ireland (Market), Europe/London (Time Zone), British Isles (Supername), Northern Europe (Supername), European Union (Supername), Western Europe (Supername), Europe (Continent), Eurasia (Supername), Earth (Supername)

Only using IP I get:

You are in Whitehall which is a Suburb located at (51.501831,-0.125760)

Your location belongs to: SW1A 2 (Postal Code), Westminster (Suburb), City of Westminster (Local Administrative Area), 0207 London (Zone), Middlesex (Historical County), London (Town), Greater London (County), MMA London (MMA), South East England (Colloquial), DMA South East England (DMA), England (Country), England and Wales (Colloquial), Great Britain (Colloquial), United Kingdom (Country), Market United Kingdom and Ireland (Market), Europe/London (Time Zone), British Isles (Supername), Northern Europe (Supername), European Union (Supername), Western Europe (Supername), Europe (Continent), Eurasia (Supername), Earth (Supername)

Quite a lot of information using a few lines of code, isn’t it? Geographical location is a very interesting topic and allows us to show much more useful content to our visitors. Using YQL and GeoPlanet it is pretty easy for you to do so – give it a go!

Our friends Jim and Nick at Doctype recently attended Future of Web Apps Miami along with their camera gear. They kindly put together a highlight reel of the day including interviews and clips from the main stage sessions. They will be releasing full length interviews over on their Facebook page shortly.

Just as a quick heads up the full length videos from the event will be made available shortly here on Think Vitamin.

Useful Links

Doctype Future of Web Apps episode: http://doctype.tv/fowa
Twitter: http://twitter.com/doctypetv
Facebook: http://facebook.com/doctype

One of the things that’s great about the Internet is that it’s open to everyone. There are a million sites, all different, and everyone is free to invent their own way of doing things. Of course, that has left us with a legacy of systems which don’t always work well together.

When you are using a web browser things mostly work because everyone adheres more or less to web standards, or at least the bits major browsers support. When you want to get data from the web to use in your app things aren’t quite as simple. Enter YQL (Yahoo Query Language).

Almost every developer knows about using SQL for getting data from databases. We wanted to do that for the Internet. YQL allows you to access all kinds of information from the internet in a very similar way to the way you would get data using SQL.

A YQL Query

A basic query in YQL is really easy:

select * from search.web where query = "javascript";

The thing we want to do is select all data (*) from Yahoo! search for web pages where the query is the term “javascript”. If you already know SQL it’s obvious, if you don’t it’s still pretty easy to read. In this case what’s happening under the hood is YQL is mapping the request to the search.web table to a pattern to call the Yahoo! BOSS (search) web service. So we are actually making a request, on your behalf, to the url:

http://boss.yahooapis.com/ysearch/web/v1/javascript?format=xml&start=0&count=10

A Sample Result

The data is returned in your choice of XML, JSON or JSONP. Like SQL most YQL data tends to be organised as rows of results.

In the case of the above query it looks like this (NB: this is truncated to show one result, view a full version in your browser):

<?xml version="1.0" encoding="UTF-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="10" yahoo:created="2010-03-02T09:22:58Z" yahoo:lang="en-US" yahoo:updated="2010-03-02T09:22:58Z" yahoo:uri="http://query.yahooapis.com/v1/yql?q=select+*+from+search.web+where+query+%3D+%22javascript%22%3B">
<diagnostics>
<publiclyCallable>true</publiclyCallable>
<url execution-time="157"><![CDATA[http://boss.yahooapis.com/ysearch/web/v1/javascript?format=xml&start=0&count=10]]></url>
<user-time>158</user-time>
<service-time>157</service-time>
<build-version>4265</build-version>
</diagnostics>
<results>
<result xmlns="http://www.inktomi.com/">
<abstract><![CDATA[<b>JavaScript</b>.com is your source for all things <b>JavaScript</b>, including tutorials, free java scripts, downloads, tools, <b>javascript</b> source code and other scripting resources.]]></abstract>
<clickurl>http://lrd.yahooapis.com/_ylc=X3oDMTQ4YzcwbGR0BF9TAzIwMjMxNTI3MDIEYXBwaWQDb0pfTWdwbklrWW5CMWhTZnFUZEd5TkouTXNxZlNMQmkEY2xpZW50A2Jvc3MEc2VydmljZQNCT1NTBHNsawN0aXRsZQRzcmNwdmlkAzc4ZHhPR0tJY3JyNGp1ZDd4R3RJcERZWXZOOFNqMHVNMlBJQUM0NkU-/SIG=10va2460d/**http%3A//www.javascript.com/</clickurl>
<date>2010/02/28</date>
<dispurl><![CDATA[www.<b>javascript.com</b>]]></dispurl>
<size>59075</size>
<title><![CDATA[<b>JavaScript</b>.com]]></title>
<url>http://www.javascript.com/</url>
</result>
</results>
</query>


Read AND Write with YQL

Looking at the example above you might agree that it’s pretty easy to use the BOSS web service already since it’s RESTful and the URL are easy to construct. Well, the same can’t really be said for the Wordpress XML-RPC API. But, with YQL we can still easily insert a new post into a Wordpress blog:

insert into wordpress.post (title, description, blogurl, username, password) values ("Test Title", "This is a test body", "http://yqltest.wordpress.com", "yqltest", "password");

A few things are interesting here. Firstly we are still using SQL syntax, but this time we are inserting data. YQL doesn’t just read data, you can write too. More than that even though BOSS uses REST and Wordpress is XML-RPC with YQL you don’t need to care. You just ask for what you want and the YQL table deals with it. Once you start using YQL to get one piece of data there isn’t any more cost to using more services.

Getting Started

So far we’ve looked at some of the things YQL can do but not how you would go about getting started, so let’s do that. The easiest way to become familiar with YQL is the YQL console. It’s a lot like the MySQL console in that you can use it to easily query available tables or to test and construct queries you want to make in your application. The console looks like this:

The first and most obvious thing is the query box. This opens with the query “show tables” which as in SQL returns a list of all the available tables. You can see that result underneath the query box. You can use the radio buttons to get the results as XML or JSON, your choice.

On the right hand side we have some navigation to get you started quicker. The list of the current tables is shown and you can open the tree to folders to show specific tables.

By default we show all the Yahoo! tables, but you can also click on a button to show all the tables created by the community. These cover a wide variety of services, almost anything you can think of.

Contribute Tables of your Own

You can also contribute new tables. They can be either really straight forward in the case of a good REST web service, or as complex as it needs to be to get the job done. A simple table might look like this one:

<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<meta>
<author>Yahoo! Inc.</author>
<documentationURL>http://craigslist.org/</documentationURL>
<sampleQuery>select * from {table} where location="sfbay" and type="sss" and query="schwinn mountain bike"</sampleQuery>
</meta>
<bindings>
<select itemPath="" produces="XML">
<urls>
<url>http://{location}.craigslist.org/search/{type}?format=rss</url>
</urls>
<inputs>
<key id="location" type="xs:string" paramType="path" required="true" />
<key id="type" type="xs:string" paramType="path" required="true" />
<key id="query" type="xs:string" paramType="query" required="true" />
</inputs>
</select>
</bindings>
</table>

You can see that it’s just XML, and only a few lines at that. The important bits are the “select” element. This represents a select statement we can make in YQL. If you wanted to insert you’d need an “insert” element too.

The select statement has a “url” which maps each “key” to the web service. In this case we have two path parameters which are inserted into the URL with { } and a query parameter which is automatically added to the end. This is the entire data required to let you select from Craigslist in YQL.

YQL is easy and extensible, and best of all there are already hundreds of tables contributed by the community to all kinds of great services like Twitter, Facebook, Etsy, bit.ly?

What are you waiting for, try the console today!

Recently I have been noticing that more and more tech companies are starting up “developer networks”. Some of these, like the Yahoo! Developer Network, consist of great tools and resources (think YQL, YUI etc) that we can all make use of in our work whilst others focus on more specific aspects of developing with a particular service or app.

Whilst tutorials and code examples are great the added bonus of a network is the community that grows around it. It’s likely that you won’t be the first person to experience a particular issue so it’s a great place to share application specific tips and frustrations.

Here are 17 networks and sites that you might not know about but are worth checking out.

Tech Company Networks

1. Yahoo! Developer Network
2. PayPal Developer Network
3. Sun Developer Network
4. Microsoft Developer Network
5. IBM Developer Works
6. Apple Developers
7. Mozilla Developer Network
8. Palm Developer Center
9. Google Code
10. Opera Developer Community

Application Specific Networks & Sites

1. WordPress Codex
2. Ning Developers
3. Freshbook Developers
4. Facebook Developers
5. Twitter API Wiki
6. Digg Developers
7. 37signals Developer Site

If you know of any more please add them in the comments, it would be great to publish a more complete list.

It’s not often Bath, the home of Carsonified, hosts full day tech events. However I was recently lucky enough to attend openMIC right here on my doorstep.

“Open Mobile Innovation Camp” is organised by Chris Book and takes place around the UK. It describes itself as “an un-conference for discussion, debate and development new mobile applications and mobile web services”. You can find out more on their Ning site.

As part of the morning sessions Bruce Lawson from Opera gave a great presentation entitled “Tips and tricks for mobile widgets”. Whilst part of the talk was directly related to mobile widgets much of it can be related to best practice web development.

What’s a Widget?

Firstly let’s be clear on what a Mobile Widget actually is. As Bruce explained we can think of Mobile Widgets as “client side apps”. They are created using standard web technologies such as HTML, JavaScript and CSS.

To create a widget you ZIP up your project files and rename it with an extension of .wgt. They differ slightly from traditional “web apps” in as much as they offer access to the local device file system and can auto-update.

Here’s a summary of Bruce’s key points, for some this may be new, for others I am sure it will serve as a good reminder.

Things to consider when creating the widget

• The physical and virtual size of the screen – How will your interface work, where will the key elements be, how will users navigate around the screen
• Readability – Make it clear and readable and get your message across
• Input mechanism – Make it easy for users to interact with your widget
• Memory, CPU and Battery limitations – Look at ways to limit their usage
• Network speed – Don’t rely on speedy networks

Work round flaky networks

• Queue XHR (XMLHttpRequest) requests – This will reduce the initial load on the network connection
• Cache data – Don’t keep polling for the same data
• Ensure try again options - What happens if you loose signal, plan for this
• Pick up where you left off - Ensure that your widget resumes where it left off after a phone call

Minimise HTTP requests

• Use CSS Sprites – Reduce the number of image requests by using one file for all your images (See http://css-tricks.com/css-sprites/)
• Consider using data URLs – These allow you to use a URL to represent an image as opposed to a file resulting in one less file request across the network as the image will effectively be present in your HTML. (See http://software.hixie.ch/utilities/cgi/data/data)

Markup

• Place JavaScript source and file references at bottom of you page – This will mean that the bulk of the page will load before your JS files, users will be able to start reading the page straightaway
• Don’t use TABLE layout – Extra overhead that isn’t necessary
• Declare image sizes in HTML – Use height and width attributes on images. This will reduce pages being redrawn once an image is downloaded, resulting in a better user experience and less work for your mobile device’s CPU.
• Consider using SVG (Scalable Vector Graphics) for illustrations or <canvas> for dynamic images - These send data as mathematical info which tells the browser how to draw it. It’s also scalable and doesn’t pixelate.

Understand accessibility

There are many similarities between understanding accessibility on the web and mobile devices. Bruce recommends reading “Relationship between Mobile Web Best Practices (MWBP) and Web Content Accessibility Guidelines (WCAG)”

• No images – What happens to your widget with images off?
• No fonts/colours - What if the device doesn’t support your preferred font or colour?
• Contrast – Ensure that it’s readable for visually impaired users

CSS for mobile devices

• Use ems not px for fonts – This is better for scalability
• Use CSS sprites – See above
• Fluid layouts – This will enable your widget to work on different resolutions
• Use media queries - These will allow you to serve different CSS files dependent on the screen size of the device
• Use SVG for background images – Declaring “width: 100%;  height: auto;” will allow your image to scale regardless
• Sniff out the device info – Like media queries this will enable you to serve a different user experience dependent on the device

Be prepared for HTML5

• Form validation is free – No further need for complicated validation JavaScript
• Client side persistent storage of key/value pairs and support for embedded SQL databases - This will reduce the need to make requests over the network
• Server sent events – The <event-source> element reduce need for polling in web apps. It will allow the receiving of push notifications from a server in the form of DOM events

Opera widgets SDK

• Documentation – Extensive documentation provided by Opera
• Test – Use the emulator to test your widget
• Opera Libraries – Lightweight libraries for animation etc
• Explore the SDK - http://dev.opera.com/sdk
• Debug your widget – Opera provide a free debugging tool called Dragonfly

Test your Widget

• Test your widget on real devices – http://perfectomobile.com/portal/cms/opera.xhtml?key=OP631R89YL2

Thoughts?

If you have developed a widget or have something to add please let us know in the comments.

View the Slides

(Updated note: the data source is an evolving project, one of the reasons we needed this app to be flexible and easily updated by a non-tech team, so this article is really focused on the technical aspects of putting this little ecosystem together.)

I use both Flash and jQuery when designing interactions for data visualization projects. A great opportunity came along to experiment with ways jQuery could be used to develop an interactive map. Before I go further, I admit that mistakes were made, but no pandas were harmed at any point and our team learned a lot, so it ended up a win-win (for us and the pandas).

The project we started out with was simple, we had a comp and a request to create a flash map that could display some static data. After diving in a little deeper, it became clear that the data source would change on a regular basis, so I recommended that we try to create a small ecosystem with various pieces of the project:

• spreadsheet (our data source)
• CMS (client was using Wordpress)
• map of the U.S. (caveat: each U.S. location needed to be able to be displayed in 1 of 3 colors: red, yellow, green for the animation, and ultimately managed as an .SVG)

jQuery as a Substitute for Flash?

We could have accomplished everything with Flash, but we wanted to learn something new and today’s browsers (shame-free plug for Chrome!) speed up DOM manipulations, so Flash isn’t the only or best choice, and there were clear benefits:

• Easy—jQuery is already packaged into Wordpress
• Styling—design can be easily restyled by almost any team member
• Integrated—everything lives integrated within the page v. living in layers
• UX first—end users never need to install a plugin
• Recession proof—we didn’t have to buy another Adobe product

Other Things we Kept in Mind

Smart architecture: All of the data is stored in a simple spreadsheet. jQuery, CSS, and simple variable placement determined by a PHP script takes the data and displays it both properly and in multiple places (click over to teachingjobsportal.com to see it in action).

Developer-free updates: The little ecosystem we created allowed us to use one data source to update both the map and each of the locations’ corresponding pages (51 pages for 50 states & DC) that are stored in Wordpress. This keeps things simple, reduces the chance for user error, saves a ton of time because a user doesn’t have to open all the individual files, and doesn’t require involving a developer.

SEO-friendly: jQuery allowed us to represent all of the data in HTML, so Google and other search engines can gobble it up and help us make it more discoverable.

Joel Sutherland: Joel Sutherland of New Media Campaigns wrote an awesome post on building an interactive map with jQuery over Flash in January 2009, and he made me believe we could do it too (albeit on a far simpler scale).

Like I Said, Mistakes Were Made

We got too fancy

At a point while we were executing, we decided to convert the data export from the CSV into XML (this also converted the .SVG and made it dynamic). While XML was easier to work with on the developer-side, it added steps and complexities to the process

Additionally, the map’s point-by-point coordinates, complex outlines (U.S. states are not nicely designed with 90 degree angles), and dynamic states required that we use software to both build the map and manage the images in an alternate library, enter ImageMagick. Simple enough, but the problem using ImageMagick to manage the library of the various U.S. state colors we also introduced “Imagick” a truly nasty PHP extension into the mix, which caused a new set of issues.

We descended into dependency hell

Installing PHP extensions can be tough, and Imagick is a huge pain in the a$$. During installation, it requires that you be prepared to manage a large set of spaghetti-like dependencies. There’s also a small bug that we ran into, which prevented compiling, so our Debian-friendly team found ourselves in Unix-land attempting to execute YUM (which usually solves the problem of “dependency hell”), and ultimately lost the battle and had to compile ImageMagick manually (both annoying and a time-suck).

We created simple, but stubborn architecture

Our little ecosystem (the one I’ve touted as mostly simple) is quiet stubborn and set in her ways. We can make updates to the data anytime, but because we didn’t build the data source as a database, we cannot easily add new fields without having to update everything.

We overlooked Hawaii & Alaska

I threw this one in for fun, but whoa! Big oversight! We accidentally forgot to include HI and Alaska and essentially redesigned the U.S. without them.

So, to Wrap it all Up

jQuery was a great substitute to Flash for us, making TeachingJobsPortal’s map. Furthermore, the execution was mostly simple, our client can control the map and its related pages’ content without having to hire a developer. (*I also didn’t have to learn Flash on the fly)

Though our animation designs are ongoing, we’ve created a new and much nicer hover state, we really like the way that the gauge works (it’s connected to the image library and as the states change color it automatically updates as well), and, as planned, the map smoothly changes colors when the data source is updated.

We’re considering ways we can eliminate using ImageMagick for the library, because we want to use the map on another project and lighten up the load on the CMS, so I’d love your thoughts on that as well as any other ideas you have on jQuery v. Flash!

I owe a lot to Joel Sutherland for sharing his team’s work, so we could learn from it, and for releasing a fantastic jQuery plug-in that I encourage you to use if you ever create zoomable interactive maps.

*Production by my brilliant partners at jjomedia.com

In case you missed it we announced Future of Web Apps Dublin 2010 earlier this week. It’s a one day conference taking place on Friday May 14th 2010 in the heart of Dublin.

We are calling this years event “Ireland’s BIG Web App Conference“. More big name speakers, new bigger venue and big value for attendees. Why not make it a long weekend in Dublin, the city practically guarantees a great time.

Interested? You can buy your ticket online on the brand new Future of Web Apps website.

Who’s Speaking?

Future of Web Apps regularly brings together leading practitioners in the field of web development and web entrepreneurship. This year is no exception, here’s a taste of our confirmed lineup. Further speakers will be announced in the coming weeks.

• Mike McDerment – CEO and Founder of Freshbooks on “Marketing Metrics and the Systems You Need to Measure Them“
• Raffi Krikorian – Twitter’s Geospatial API lead on “How to use Geolocation in Your Web App“
• Simon Wardley – Cloud computing guru working at Canonical on “Situation Normal, Everything Must Change“
• Christian Heilmann – Developer Evangelist at YAHOO on “Powerful Tools that You Need (and Probably Don’t Know About)“
• Eoghan McCabe & Des Traynor – The guys from Contrast are back with us talking about “Five Lessons We’ve Learned“
• Relly Annett-Baker – Web copywriter extraordinaire 0n how to “Write Effective Web Copy“

Full details of the speakers can be found on the Future of Web Apps website.

What can you expect?

• Amazing speakers discussing the hot web app topics
• Great networking with like minded developers and entrepreneurs
• Free refreshments throughout the day and a great lunch
• Fun times at the after party

Grab a ticket today

Tickets are selling fast but there are a handful of early bird passes left for €195 + VAT (normal price €245 + VAT). We also have student passes available for €55 + VAT. You can purchase your ticket online on our website.

Get involved

We are introducing a new option for 2010 called “Micro-Sponsorship”. It’s a great way of putting your company or web app directly in front of your target audience.

For  €100 + VAT we will email all Future of Web Apps Dublin attendees your link + 140 characters of your choice one week prior to the conference. If your interested just head on over to our registration page.

We look forward to enjoying a pint of the “black stuff” with you in May!

Twitter is certainly an interesting beast. It has been the poster child of success stories, saving lives, reporting tragedies, over turning governments and letting thousands of people know where you are going for lunch. There is no denying the impact it has on our scenius.

I wondered if there was a way to find out a bit more about how twitter was being used within the local community and if it was even possible to take a smaller stream from the river of data Twitter produces daily. I chose as my local community Iceland.

I did so for several reasons, it is small-only about 300,000 people total, it has a distinct language so it is easy to find people using the only the text search, and it is where I live so any cool information I learn about, I can actually use it.

Any of the following steps can easily be reproduced for your local area, community or group. You’ll just need to find the key to unlocking like-minded people.

RSS not API

The Twitter API has plenty of great features, but to accomplish all of this I am using the twitter search RSS results. There are plenty of libraries in your favourite language to parse RSS, so I won’t go into depth about those nuts and bolts. The key to getting quality data is to formulate very specific searches.

In Icelandic, there are several characters and words unique to the language. In your niche group there is jargon which can be used to identify the people in the know, a sort of 21st century Shibboleth.

To try and find Icelandic Twitters, an easy first pass is to search for purely Icelandic words. This link will return an RSS feed of matching tweets.

http://search.twitter.com/search.atom?q=það

<entry>
<id>tag:search.twitter.com,2005:8243287523</id>
<published>2010-01-26T17:29:23Z</published>
<link type="text/html" href="http://twitter.com/username/statuses/12345" rel="alternate"/>
<title>...</title>
<content type="html">...</content>
<updated>2010-01-26T17:29:23Z</updated>
<link type="image/png" href="http://a1.twimg.com/profile_images/1234/profile5_normal.png" rel="image"/>
<twitter:geo>
</twitter:geo>
<twitter:source>...</twitter:source>
<twitter:lang>is</twitter:lang>
<author>
<name>Display Name</name>
<uri>http://twitter.com/username</uri>
</author>
</entry>

From this, we can then extract the display names and twitter usernames and plenty of other information about the tweet.

If you try searching for some jargon within your field, you can begin to collect like-minded people.

There is also a feature in the search which allows for language detection. Using the parameter &lang=is it is possible to further filter the search. For English speakers, this might be more difficult than other languages. The system isn’t perfect either, but it can be away to limit search results for terms like “schmuck” in German to jewelry and “schmuck” in English to an insult.

Finally, you can restrict the results to a geographically close group of friends by using the near parameter &near=New+York&within=15&units=mi. You can also specific the distance and units (miles or kilometers). At the moment this is trying to using the user’s Location, but in the future it might also use their new geolocation API. People don’t always list their location, or change it out of solidarity to a cause.

These systems aren’t perfect, but when put together they allow for an easy way to thin down the millions of twitter to a much smaller more manageable size.

Create your own Groups

This is ideal for local affinity groups or video game clans. Want to see what all “EVE Online” player who are in a specific company and live in “New York City” have to say? Well we can search twitter and start to collect matching usernames, index them and search for interesting conversations. Maybe you can make new friends locally beyond the game.

The downside of using search is that you will only find new people as they tweet. Search only goes back 10 days (maybe less as twitter gets larger and larger). There will also be some false positives, but if you select some good, specific search terms, then you minimize the work pruning out mistakes.

I am searching twitter once an hour for any new posts that meet the criteria. Once an hour is enough for me, but depending on the size and frequency of your group, you can poll more or less frequently. As I loop through the RSS results and find username not already in the database, I add them so I have a nice growing list of people. I then subscribe to their RSS and save all the tweets into another table for later analysis.

Find and Subscribe

Over time, I have managed to find 3900+ Icelanders on twitter. That’s over 1% of the population. Twitter does have a limit of 2000 that people you can follow before you need to maintain a ratio of following to followers. This prevent spammers from following thousands of people and wrecking the system.

Once you have the twitter username, it isn’t necessary to follow them via twitter’s functionality, it’s enough to subscribe directly to their RSS feed if they are public. Now it seems rediculous to subscribe to thousands of RSS feeds, that’s a bandwidth hog. So I have offloaded that work to Google Reader. Google has cached versions, so all I need to do is access a single feed in Google Reader which is a composite of all the Twitter RSS feeds I have found. This makes a single RSS fire hose of your groups tweets.

Twitter Stats

Since I started mining my twitter data, some interesting statistics have emerged. I looked at the last 24 hours to see how many unique usernames had posted. Roughly 10% of the twitter population in Iceland is posting on a daily basis, but how active are they? Well, we can look at how many posts there were in the last 24 hours and divide that into active users and get about 6.5 posts per active twitter account per day.

I also looked at a longer time span of the last seven days. This allows me to get a break down by day to see the activity. As you can see, Sundays are pretty slow, then it picks up again on Monday. Presumably, as people return to work and are back in-front of the computer they get back to twittering. You can see the current stats at http://icelanders.optional.is/

These stats are valuable information if you are trying to spread your message via social networks. Do you send it out on a Sunday where there is less traffic and it is more likely to be seen by fewer eyeballs or on Monday when more people are online, but there is a deluge of other tweets?

I also broke it down further into hour-by-hour slices. As you can see, the early morning hours from midnight until around 8am are pretty quiet. Still some activity, but not the time to expect someone to reply.

Once you are archiving a sub-set of twitter, it becomes possible to search this smaller list. Maybe the types of complains within the group vary greatly than with the general public. Are discussions staying within the niche group or are links and retweets getting outside of the core group? You can even find out how many people within this group are following each other and determine the level of interconnectedness.

Recently, I went through the database and using the Twitter API, found out when each person signed-up. I could then look to see if there was a growth spike or continual steady growth. It turns out there was a spike in early 2009.

Click image to view large version

Right around the time Twitter was being touted to the masses via Oprah, the major local newspaper joined and mentioned it a few times in print. These two, along with other factors really boosted the sign-ups from a few tens of people to several hundred a month! Even now, there is a solid 100+ new Icelanders coming to Twitter each month to see what it is all about.

Building something similar for your group allows you watch for spikes in discussion or sign-ups, look for trending topics within your community, engage in the discussion and find new friends.

You have all your important data backed up, right? I mean, come on folks, it’s 2010. We may not have flying cars yet like we’re supposed to, but “having backups” is a problem that’s solved. And I’m sure you’ve solved it. I’m sure you sleep well at night knowing that if a comet hit the data center where your website lives, and everything was completely destroyed, you could get back up and running somewhere else quickly enough.

Except that the chance this is actually true for you, statistically, is very small. If we look at the statistics, then what we see is that people are storing increasing amounts of things, important things even, online because disks keep getting bigger and cheaper (though sadly not much faster, but that’s a different article).

But despite this availability of cheap storage, people for the most part still don’t back things up. This isn’t universally true of course, there are some folks who have implemented really brilliant backup systems at their companies.

Of course, it helps when you have an amazing systems guy on your team who is made out of lighting bolts and awesomeness, taking care of the nitty gritty details for you. If, however, you are one of the many many people who doesn’t have a good backup strategy in place, this article is here to help you out.

I’ve written a shell script (Ed: please remove .txt extension before use) that will serve as a good starting point, and we’ll talk through it in a little bit. For now though, let’s discuss backups in a more general sense so we can decide what our end goals are.

Types of things to back up

There are lots of different kinds of data that you may need to back up. Three of the most common are:

1. Source Code
2. Static Files (images, PDFs, etc)
3. Data in a Database

These days, there’s a good chance you have your source code files in an RCS system somewhere, such as GitHub. If you do (and you should), and your RCS system is on a different server or servers than your site, you already have your source code backed up. If your server catches on fire, you can just do a fresh checkout onto a new server. This is one of the many advantages of using an RCS system.

Static files, such as images or music or what have you, are easy to back up. A quick search will reveal dozens of different ways to ship this sort of data to some secondary place in case of emergency.

Database data, however, can be a little more tricky. You can’t just copy your binary data files safely while the database is running because if a write happens during the process, then your “backup” will be in an inconsistent state, which will probably render it useless.

Moreover, it’s not generally the best way to do a backup because restoring data from a binary copy can be tricky. What my shell script does is allow you to automate correctly backing up a standard MySQL database, or databases, with full end-to-end encryption to an offsite location.

Is my data backed Up?

This seems like an easy question to answer. In fact, it is a rather easy question to answer, but I’ve found that many people have never bothered to answer it, and it gets them in trouble. Are you doing backups of some sort currently? If so, does that mean you have backups? Actually, no, it doesn’t. If there is one golden rule of having backups that I hope you remember after reading this, it’s the following:

If your backup procedure doesn’t include instructions for doing a restore of the data, or if your restoration procedure has not been tested, then you don’t have backups.

Simply put, unless you know that you can reliably do a restoration of your data, you don’t have backups. If you haven’t tried, you don’t know. As an example, I once encountered a setup where an administrator was “backing up” several databases by using rsync to copy the binary data store to a remote machine once a day. He thought he had backups, and he was wrong.

As I noted before, if a write happens during the rsync process, those binary files aren’t guaranteed to be in a consistent state. If he’d ever tried to do a restore of those files (he hadn’t), he would have discovered this problem. So I say again, until you have tested a restore and it’s worked, and you’ve documented how to do that restore, you don’t have backups.

Another thing that I always like to stress is that you really, really want backups that are under as much personal control as possible. This typically means using a third party or offsite service distinct from your ISP to store your backups. If you ever need to restore some data, there’s a good chance it’s because something has gone wrong with your systems.

In that case, there’s also a fair chance that something has gone wrong at your ISP itself, so you’d rather not have your backups tied to your ISP. What you want is backups that are stored by some other company, preferably in an entirely different physical data center.

Basic process for setting up database backups

The steps I use to back up the databases for Virb are as follows:

• Get a dump of the database
• Compress the data
• Encrypt the data
• Send the data over an encrypted connection to an offsite service
• Automate the entire procedure with a cron job

It’s nothing fancy, and it shouldn’t be. Fancy, intricate things tend to break, and this is the last thing you want to worry about breaking.

Prerequisites

You’ll need to have a few things in place for this to all work. I’m assuming you’re using a reasonably new Debian style Linux distribution (which includes Ubuntu). You need to be able to use the mysqldump program to get a dump of the desired database data. This comes standard with the MySQL installations on every Linux distribution I know of.

I’m going to assume that you have a .my.cnf file set up in your home directory with a username and password that allow you to dump out the data you want. If you’ve never used a .my.cnf file before, it’s very easy. Create a file called .my.cnf in your home directory, and put the following in it:

[client]
user = dbusername
password = dbpassword

of course substituting the correct username and password for your systems. Then run chmod 600 $HOME/.my.cnf on the file so other people can’t read it. You can now use all the mysql* command line utilities without having to pass the -u and -p flags all the time.

Another assumption I’m making here is that you can safely run mysqldump at some point during the day, and this won’t mess anything on your site up, which practically means that you don’t have that much data. When you dump the database out, a write lock is placed on all the tables in that database (assuming you’re dumping out all the tables). So while the data is getting written out, reads can still happen, but no INSERTS, UPDATEs or DELETEs can.

If you have say a few hundred megabytes of binary data, usually located in /var/lib/mysql on Linux systems, you are probably okay just dumping the data out from whatever machine the database is on. On reasonably modern hardware, 300M of binary data takes about 15 seconds to dump out, for example. Having a write lock for 15 seconds in the middle of the night is not a big deal for most people.

If you have enough data that it really does take a long time, or your requirements are such that you can’t have a write lock happen even briefly, then the next option is to set up a mysql slave server that exists just for backups. The idea is that the slave is continuously replicating from the master.

When you do the backup on the slave, the write lock will happen there so that the data can get dumped out, and then once it’s finished the replication will catch back up to the master if it’s missed anything.

Explaining how to set up MySQL replication is beyond the scope of this article, but this “back up a slave” strategy is exactly what we do for Virb.

We’re going to be backing things up using the online back up service provided by IBackup. There are plenty of these sorts of services available, so if you want to use a different one feel free. Not much of what I’m going to describe is specific to IBackup. I have no particular affiliation to them, other than having used them for several projects and being pleased with their products and pricing. I’m going to assume you have an Economy plan with enough storage to hold your backups.

The only other assumptions I’l make are that you have stunnel4, rsync, and gpg installed. These are very standard, freely available programs. On reasonably recent Debian style systems (which include Ubuntu) you can install all of them by running this command as root:

apt-get update && apt-get install stunnel4 rsync gnupg

Setting up encryption

We are going to encrypt our database backups, and we’re going to transmit the encrypted files over a secure connection. This is technically overkill, but encryption is free and easy so we’ll do it anyway. The first thing you’ll need to do is generate GPG keys to use.

I’m not going to explain it here because this tutorial will take you all of five minutes to get through, and covers everything you’ll need. Just be sure to remember your passphrase, and put copies of your key files somewhere safe just in case.

You should now have GPG keys set up, and an account with IBackup, so create a file in your home directory called .ibackup with the following contents:

IBACKUP_USERNAME=username
IBACKUP_PASSWORD=password
GPG_EMAIL=your@email.com

The reason for this file will become apparent shortly. Just substitute the username and password for your IBackup account there, and put in the email address that was used to generate your GPG keys. Be sure to run chmod 600 $HOME/.ibackup so that nobody else can see your password.

Now we need to do a little configuration for stunnel4. We don’t need it to be running all the time, so shut it down and make sure it doesn’t start on boot with the following commands:

/etc/init.d/stunnel4 stop
update-rc.d -f stunnel4 remove

Now let’s configure stunnel4 for IBackup. First get the default configuration file out of the way.

cd /etc/stunnel
mv stunnel.conf stunnel.conf-orig

Next, with your text editor of choice, create a file called ibackup.conf in the /etc/stunnel directory with the following contents:

client = yes
[ibackup]
accept = 8455
connect = rsync4.ibackup.com:5000

These options are fairly specific to IBackup’s service, but other backup services have similar options. The accept option defines a local port for stunnel4 to use, so you don’t have to pick that particular one if, for some reason, it’s already in use.

Performing the backup

Since we have our security issues handled, we can now perform the backup. Open up that shell script (Ed: please remove .txt extension before use) I mentioned at the beginning in your text editor of choice, so we can go through it. The first 14 lines just get the names of databases we’re going to back up and identify the programs we’re going to use.

#!/bin/bash

# ibackup.com destination
IBACKUP_DEST=$1

# database names
DB_NAMES=$2

# binaries we need
MYSQLDUMP=/usr/bin/mysqldump
GZIP=/bin/gzip
GPG=/usr/bin/gpg
RSYNC=/usr/bin/rsync
STUNNEL=/usr/bin/stunnel4

If you’re not familiar with bash shell scripting, it’s not hard. Things like $1 and $2 indicate command line arguments that get passed to the script. So here, $1 is something we’re going to use in the name of the eventual backup file we’ll get, and $2 will hold the name(es) of the database(es) we will back up.

Things get more interesting from lines 23 to 39.

DOTIBACKUP="$HOME/.ibackup"

if [ ! -r $DOTIBACKUP ]; then
    echo "Need $DOTIBACKUP file for username and password!"
    exit 1
else
    PERMS=`/bin/ls -l $DOTIBACKUP | /usr/bin/awk '{print $1}'`
fi

if [ $PERMS != "-rw-------" ]; then
    echo "Permissions on $DOTIBACKUP are wrong!"
    echo "Should be -rw-------, actually is $PERMS"
    exit 1
fi

. $DOTIBACKUP
export RSYNC_PASSWORD=$IBACKUP_PASSWORD

These commands check that the .ibackup file exists and has the correct permissions. If so, they source the file, which is to say they absorb the things that file defines into the running shell script. Then it sets the environment variable RSYNC_PASSWORD to be the value of the IBACKUP_PASSWORD that you use to access the service.

Lines 41 and 42 grab the local port that stunnel4 needs in order to work.

# local stunnel4 port
STUNNEL_PORT=`/bin/grep 'accept' /etc/stunnel/ibackup.conf | awk '{print $3}'`

Next, between lines 44 and 49, we make make sure there’s a directory where we can dump out all of our data.

# place to dump everything
DUMPDIR="$HOME/.ibackup_data"

if [ ! -d $DUMPDIR ]; then
    mkdir $DUMPDIR
fi

We’re eventually going to set this up to run daily, and if we’re doing a once-per-day backup, it’s nice to keep a few days’ worth around just in case. In lines 51 through 56 we use the very convenient *NIX date program to get date stamps for the current day, the previous day, and the day before that.

# today, for tagging the dumpfile
DATE=`/bin/date +%Y-%m-%d`

# yesterday, and the day before that
ONEDAYAGO=`/bin/date -d '1 day ago' +%Y-%m-%d`
TWODAYSAGO=`/bin/date -d '2 days ago' +%Y-%m-%d`

We’re almost ready to actually dump out our data now. Lines 58 to 86 are really boring and just make sure we have a destination name and the programs we need, so I’ll gloss over those. The important part picks between lines 88 and 92, where we go into our backup directory and actually run mysqldump.

cd $DUMPDIR

echo -n "dumping $DB_NAMES to $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$MYSQLDUMP --opt --databases $DB_NAMES > $IBACKUP_DEST-$DATE.mysql
echo "done!"

Let’s take a look at that dump command to make sure we understand it. The $DB_NAMES variable was set earler in the script. It’s the second command line argument passwed to the script, and it tells mysqldump which databases we care about. The $IBACKUP_DEST variable, similarly, was the first command line argument passed to the script, and it just sets the part of the name of our dump file. Finally, the $DATE variable we just saw getting set a moment ago, and we use it as a date stamp for this dump.

Next, between lines 94 and 105, we compress and encrypt our dump file. We also check to make sure the encryption step actually worked and we exit out if not.

echo -n "gzipping $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$GZIP $DUMPDIR/$IBACKUP_DEST-$DATE.mysql
echo "done!"

echo -n "encrypting $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$GPG -e -r $GPG_EMAIL $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz
echo "done!"

if [ $? -ne 0 ]; then
    echo "encryption failed!"
    exit 7
fi

We do a little cleanup and start stunnel4 in lines 107 to 112, which are pretty self explanatory.

echo -n "removing $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz... "
/bin/rm -fv $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz
echo "done!"

echo "Starting stunnel for rsync..."
/etc/init.d/stunnel4 start

And now on line 114, we finally send our freshly made backup off to the remote servers managed by IBackup for safe keeping.

$RSYNC -r -v -z -t --delete-after --exclude $IBACKUP_DEST-$ONEDAYAGO* --exclude $IBACKUP_DEST-$TWODAYSAGO* $DUMPDIR/ $IBACKUP_USERNAME@localhost::ibackup/$IBACKUP_DEST --port $STUNNEL_PORT

This is using several different features of rsync so let’s talk about them so we know what is going on. The command line flags -r -v -z -t are what the folks at IBackup tell us to use when syncing things over to them. The --delete-after flag tells rsync to delete anything in the target directory that’s not in the local directory being sync’d. But we also have the two --exclude options there.

So what we’re really saying is “Keep the file we’re sending over, and the one from yesterday, and the day before that, but then delete everything else”. That way we always have the last three days’ worth of backups. The last point of interest is that because we tell rsync to use $STUNNEL_PORT, all the communication is encapsulated by stunnel4 and is therefore encrypted across the wire as it gets transferred. Whew!

The last few lines from 116 to 121 simply stop stunnel4 and clean up things by removing the directory we made to put the dump file into.

echo "Stopping stunnel for rsync..."
/etc/init.d/stunnel4 stop

echo -n "Cleaning up... "
/bin/rm -rfv $DUMPDIR
echo "done!"

So that should about do it. To find out, put this script somewhere and run chmod 755 db-backup.sh on it so it’s executable. Then test it to make sure it actually works. If you want the target to be called ‘mybackup’, and you have to back up the databases named ‘db1′, ‘db2′, and ‘db3′, you would invoke the script like this.

/path/to/db-backup.sh mybackup "db1 db2 db3"

If that works, then you are almost done! We just need to automate the process, which we can do trivially with a cron job. In your shell where you tested the script from, type the following:

crontab -e

This will put you into edit mode for your crontab. Assuming you want the backup to happen starting at 12:40am each day, put the following into your crontab file:

0 40 * * * /path/to/db-backup.sh mybackup "db1 db2 db3" >/dev/null 2>&1

Save the file, and then type crontab -l to list the crontab and make sure that line is in there. If it is, then there’s just one last thing to do.

Testing a restore

I said at the start of this article that if you haven’t done a restore, you don’t have backups. And seriously, I meant that. So if you’ve made it this far, then you must do one last step. The day after you set this up, you need to go to your IBackup account, download your dump file, decrypt it with the GPG key you made, and do a restore to a running MySQL instance.

Make sure you do this last step, as the consequences of missing it can be dire.

In conculsion

I covered quite a bit in this article, but if you make it through everything, you should be able to rest easier knowing that you have good, safe backups of your database. Although we used the IBackup service in these examples, very little of what I covered is specific to them.

If you wanted to use a different company, or maybe just push the backup files to a different server you have, you’ll probably just need to change some of the commands that you pass to rsync. The basic strategy though should’t have to change.

I hope you’re found this helpful, and don’t hesitate to get in touch if you have any questions.

Editors note: Interested in web apps? Join us at the Future of Web Apps Miami on February 24th to learn more about HTML5 and hear from companies such as Twitter, Facebook, Mint, Reddit and more. Buy your tickets now and get $50 off.

In this weeks 10 minute episode of Doctype Nick shows you how to create gradients without images in CSS3, and Jim takes an in depth look at local databases in HTML5.

This is the last episode we of Doctype we will be syndicating on Think Vitamin. We hope you have enjoyed them and found them valuable.