At 280 North we’ve been working on a framework called Cappuccino for creating what we like to call desktop class applications. These are not your average web sites, but instead a new breed of web application that aims to replace many of the programs that currently run on your desktop. When we launched 280 Slides, the first application built on this framework, many people were surprised to see just what was possible in the browser. Since then, Cappuccino has been steadily progressing with over 40,000 downloads and code contributions from twenty-one individuals.

The goal of Cappuccino has always been to make it as simple as possible to build high quality applications in the browser, and up until now we’ve worked on providing built-in behavior for enabling such tasks as autosaving, undo/redo, and copy/paste. But recently we’ve been focusing on an entirely new product: Atlas. Atlas is a new IDE and visual layout editor that leverages the power of Cappuccino, but drastically reduces the amount of work required for creating a truly rich application experience on the web.

WYSIWYG

With Atlas you can build and style your entire interface using drag and drop. By default we provide you with a large library of built in widgets along with the new Aristo open source theme, meaning your application will look fantastic with no additional work. However, you can also skin these applications entirely from the editor and extend the library of widgets with Atlas’ plugin architecture. And unlike a lot of existing layout editor availabe today, Atlas is not a code generator. On the contrary, Atlas lets you edit the live JavaScript objects in memory and then takes a “snapshot” of them in place. When your application runs, it ‘thaws’ these objects out, making sure that you see exactly how your application will look before you run it.

Layout that Makes Sense

Atlas provides a new take on web layout that make it easy to define precisely the behavior you want, while simultaneously doing away with existing browser inconsistencies. Instead of having to learn the variety of different ways to position and resize elements in the browser, Atlas introduces the simple concept of ‘anchors and springs’. Anchoring an interface element to one of the edges around it ensures that it will stay “stuck” there when it’s container resizes. Activating the internal ’springs’ causes it to resize with its container.

While simple to learn, these two tools enable you to create incredibly complex and fluid layouts. More importantly, they’re guaranteed to behave the same way regardless of the browser you’re running your web app in. Atlas also provides a large selection of collection views to handle common layout tasks such as split views, table views, and scroll views.

Connections Reduce Glue Code

A common problem that has plagued layout editors in the past is the need for additional glue code to “talk” to the generated interface. In Atlas, much of this work can be done with a technology called ‘connections’. Connections allow you to visually define the interactions between interface elements and other objects. For example, you can define what action a slider should take when it is dragged:

Similarly, you can ‘bind’ the contents of an array to a table, so that when it changes the table automatically updates as well, all without having to ever touch the keyboard:

Model View Controller Built-In

Cappuccino is a Model-View-Controller framework, and Atlas takes this idea to the next level by allowing you to not only create visual elements, but abstract models and controllers as well. By allowing you to interact directly with the objects in your program visually, you can focus on building unique interactions instead of learning a myriad of APIs.

Atlas has gone a step further and provided a number of pre-built controllers to existing web services that you can simply drop in to their applications. You can then simply use connections to grab information from services like RSS and Twitter. Once again, Atlas’ plugin architecture allows you to create controllers for your own web services as well, which you can then share with others.

Multi-Platform Support

Cappuccino was built from the ground up in preparation for a world of multiple platforms. It is becoming increasingly important for applications to run in several different environments: browsers, social networks, handheld devices, and much more. In general, very little application logic or backend code has to change from platform to platform, but the interface usually needs to be recreated from scratch.

With Atlas, it is drop dead simple to create a unique and custom interface for different platforms like the web browser and the iPhone. Cappuccino will then intelligently load the correct interface for your application depending on the environment it is being run on, allowing you to reduce the amount of code you need to rewrite and as well as using just one language and API for all the platforms you deploy on. We call this idea ‘Write once, layout everywhere’, and we feel it is the right way to gaurantee a high quality experience everywhere you ship your application.

View full size

View full size

Atlas will be shipping this summer and you can sign up for email updates at 280atlas.com.

[Photo credit: flickr.com/photos/atelier_tee]

It is quite interesting to see how web development turned from a niche for brochure-ware and intranets to one of the biggest software development environments in the market over just a few years.

“The web as the platform” is a hollow dream no longer ‐ you can now run and develop a web application without needing to host anything yourself and you can even get the data from other sources. With this radical shift comes a lot of change for developers, but the question is, are you keeping up?

Change is good – we need to get our act together in any case. It’s 2009 and we have the chance to start this year by tackling issues that concern everybody: the general recession, inevitable lay‐offs and lack of investment in non-profit generating parts of organizations. Now’s a good time to get real, regroup, and reassess the ways in which we approach our jobs.

We are in a crisis – have you watched the television lately? It doesn’t make cheerful viewing. However, with every crisis comes the opportunity (and the necessity) to improve and see how we can work more efficiently, producing things that are more valuable and relevant to the people we want to reach. The hype is over ‐ it’s time to get busy, working as professionals.

Back When All This Was Just Fields 

When I started web development every article, tutorial or introduction to a system started with “open your editor, read this documentation and start writing your app, (or web site, css trick or javascript trick)”. I loved it. I downloaded the article to a floppy disk at work, went home and in the evenings without the internet (remember you paid by the minute) wrote my first solutions and battled with the documentation. Later, the only thing that changed was that I didn’t work on it at home but on a laptop on the train instead.

Nowadays, things look different. First of all, for the most part it’s impossible to develop offline ‐ this is because we use hosted services, APIs and most likely you’ll write something that uses some data stored elsewhere on the web. Offline web development is becoming very rare indeed, which can be terribly frustrating if you are in a public space or hotel rooms. In other words, we build on already tested and proven concepts instead of building everything ourselves. This means we have to do much less work, but it also means that we don’t have full control. We rely on data being available to us by third parties or ‘the web’. This is pretty tough for developers as we tend to be control freaks.

Aiming Higher and Wider

The other thing that can be a hurdle for the old school developer is the fact that we tend not to build new systems any longer. Back in the day we worked on the bleeding edge: almost everything we built was the first of its kind on the web and clients came to us to build their first web presence. Users were easy to come by and people were eager to do things easier and quicker on the web. This is pretty much over.

Unless you have a ‘very’ compelling product or idea clients will already have something in place that they use and will have already spent a lot of time and money on it. Maintenance is much more common than innovation. Users are much more spoilt by choice and already spend a lot of their time using other products ‐ just building something and hoping that users will come is not enough any longer.

Most products you’ll build these days are aimed at piggy‐backing on existing user bases and use cases, rather than creating something entirely new. The web has become much more of a commodity than it used to be and a lot of people rely on it for their day‐to-day chores and social communication. For example, the amount of people that started to get in contact with me after years of silence is staggering. The reason? Facebook and how simple it is to use. These are the same people that asked me if my e-mail address would change when I moved to another city!

These social networks create huge traction, and even hardware developers jump on them to sell their newest products. Just yesterday I saw a mobile ad at the train station telling me that the new smart phone automatically gets updates from my friends on Facebook. Sounds a tad creepy to me, but I’m sure it boosted that manufacturer’s smart phone sales and I am sure many a teenager will be asking for this one for their birthday.

Social networks are taking over in all kind of environments. The same is happening to search engines.  

For example, I do see a lot of movie posters on my way to work and realized lately that the film industry has stopped bothering with domain names for movies (I guess domain squatters blackmailing them are to blame). Instead I started to see “enter xyz in Google to learn more” and one movie even used a bebo and myspace address instead of a domain. This is quite commonplace in South East Asia already.

A ‘New’ Breed of Developers 

What shall we do about it? We could sit sulking in our developer corner claiming that these things will never work but the fact is that this is where people invest and the money still flows. It is also where the users are. There is a big market in some technologically underdeveloped countries for Facebook applications and even blog systems. Using out‐of‐the‐box solutions companies can easily cover both the development and the maintenance of the content. Can you for example guess where http://s2999.com is written and maintained?  

The interesting fact for businesses is that these developers are pragmatic. They don’t cost much for a business to hire, they deliver the job without trying to lecture you and all they expect is a share of the outcome. Us on the other hand, leading and amazing developers that we are, are rumored to be tricky to work with, grumpy even. We don’t want to use things that other people made and are very happy indeed to tell any business person that they are wrong and we know all about the web. Who’d you work with?

Sure, this is nothing new – outsourcing shops have promised cut‐price development for years and most of the time the outcome was appalling (funnily enough, in a lot of cases not because the developers were bad but the management didn’t know what they wanted or failed to give precise delivery specs). With the systems in place though, development becomes much more of a ‘use’ case than a ‘build’ case and this is the biggest challenge we now face.

Good­bye White Canvas, Hello Lego Bricks

In essence, web development these days doesn’t start with a blank document but with downloading an SDK or entering your application data in some web app to get a developer key. In other words using a system that is much larger than anything you’d ever have done yourself. This concept is very hard to swallow for us battle‐hardened masters of the command line and I myself have problems getting enthusiastic about using a chunk of ‘hello world’ code and turning it into something great.

If we look at it logically, however, and leave our ego to go and play in the garden for a while it’s clear that there are many valid reasons to use SDKs and hosted development environments. It’s exactly the same thing we do as clever developers ‐ build ourselves shortcuts, hack our development tools or assemble our very own library of snippets – in order to avoid having to do the same things over and over again.

Using SDKs and hosted development environments shouldn’t mean your work will be dumbed down or that people won’t respect you as a developer. It means that they are helping to take the pain away and have taken away the hoops that they jumped through so that you can just build something interesting without worrying about the underlying architecture, security of the system or its performance.

Abstracting these away means you can easily do fixes or improvements should the need occur ‐ and boy will it occur. It is not fun to maintain a site or a server these days. The amount of people that want to hack you and inject malicious code is staggering and their skills are impressive. The only way to battle these threats is to have one system to fix ‐ not millions of little applications.

Abstraction also means that many more people can start developing applications. Geeks build geek tools, designers build designer tools, ‐ if you lower the entry barrier to developing, all groups can bring their expertise to play with the system and subsequently you can build really cool tools together. Of course, there is a lot of pointless and bad work being done too (no, I don’t want your teddy bear to travel on Facebook or catch dozens of sheep) but that’s like saying HTML is bad because people can use tables and fonts to create layouts fast instead of using CSS.

So Are We Obsolete?

Does this mean that the tinkerers, the people who spend hours fixing an IE6 bug or open the border of the screen to show sprites outside the dedicated area (yes, C64) are obsolete and should just move on? No, it doesn’t ‐ not by a long shot. If we cave in now and grumpily sit in the corner pointing at the kids with their fancy SDK toys we are indeed obsolete, but there are a lot of things that this brave new world of development needs us to do.

It needs us to analyze the systems and see where they need improvement. For example the amount of accessibility issues that can be fixed in social networks with very little effort is staggering. Most of these issues are not there because people are lazy but because they just don’t know about the consequences of their actions.

Why not help with turning these systems into a working network of systems. Open Social is a great idea but suffers from too many people shouting ‘FAIL’ at it instead of helping to fix it.

Why not work on bridging the gap between our two worlds. There is no sense in offering people a text box in an HTML page to develop in. I have used one of these and the resulting code is ‘always’ terrible. The same applies to code editors in a browser doing the indenting and colour coding for us ‐ they are flaky and when they crash they take your whole app with it (and the pictures of kittens you had in another tab). There are some Firefox extensions that allow you to edit text in your editor and then enable you to pull it back into the text area with a keyboard shortcut ‐ more of those please. Show by example how your expertise can make web applications better and how communicating with you upfront makes for better APIs and SDKs.

So What About the Future?

If I knew that… However, what I will say is that the way some web developers deal with market forces and business people is wrong. 

Our craft is becoming a commodity and people in charge don’t care about the quality of the markup, CSS or how short our JavaScript is. What matters is how fast you can get it to market, how many people it reaches and how cheaply it can be built. I think that web development has come on in leaps and bounds in the last years, not because of top‐down decisions and empowerment of the folks on the ground, but by geeks just doing the right thing without asking for permission first. We should follow standards and build solid code to make our own life easier ‐ not to impress other people. Then show the code and explain its merits to people outside of the development world to improve it for everybody.

So, (a little late) here are some of my New Year’s resolutions for this year:

1. Give things the benefit of the doubt before judging them. In other words, keep the ‘FAIL’ shouting to the funny blogs where it belongs.
2. Take time to have a look at the SDKs, hosted services and frameworks out there and write about them instead of showing yet another proof of concept.
3. Try to talk to people in charge of these systems to stop them from aggravating the developer crowd with decisions that are perfectly logical to them but don’t hold up in our world.
4. Great API developers don’t necessarily know HTML and their demos might be atrocious table and font monsters. Instead of shunning the API provide examples of how to do it better. Explaining the effects that bad API design can have goes much further than judging only by our own standards.
5. Clean up the demo code of the systems, SDKs and APIs that I have direct access to. There is nothing worse than a bad example being replicated million fold as people copy+paste instead of reading docs.
6. Write human readable documentation and don’t expect a degree in Klingon. Ask non-developers how real people use systems and what they like most about them.

What do you think? Is it time to live in the now and help build a good web 2.0,3.0,4.3212 and so on or have we lost the battle for a beautiful, valid and semantically rich web already?

Reporting is perhaps one of the biggest business deliverables for enterprise IT. Being able to say how many widgets were sold in month seven in year XX, sub-divided by customer ABC grouping, is one of the easiest ways to make management happy. But when building a web app, reporting is the last thing on many feature lists. Clear reporting is vitally important for the long term success of your web app and managing your growth. However, if you're still in the beta stage, we'll explore some ways of building simple reporting fast.

Why reporting? Because it helps you know the numbers. You need to be able to take data from web analytics, your own system stats (how many sign-ups per month, what's this month's revenue?, etc.) and be able to render that visually in some sensible way for you to be able to understand it and make the correct business decisions based upon it.

Big business style

If you're a large multi-national with large IT budgets, you're going to looking for a reporting system that can query multiple data warehouses with hundreds of users and multiple output formats. You'll also want it as near real time as possible. Packages like Crystal Reports and Jasper reports provide large scale interfaces to data sources and provide nice tools to create easy to read reports. But being as we're bootstraping upstarts, we need a solution that can pull together the data we need into one place and export it how we want.

Super simple (nearly) real time reporting

I'm not going to go into depth about database replication, load balancing your queries, redundancy or the technological challenges you face in getting your web app's data out. I am going to assume that you have a database of some sort, a web analytics package and clear objectives for what you want to measure. The most important of those is the objectives. What is it that illustrates growth and successes of your web app? This question in itself could fill thousands of articles, but for the sake of getting to some useful examples we'll look at the following:

1. Number of unique visits to your site: last 24 hours, last month, ever
2. Number of new subscribers to your rss feed: last 24 hours, last month, ever
3. Number of sign ups: last 24 hours, last month, ever
4. Number of adds of your f8/widget: last 24 hours, last month, ever
5. How much downtime: last 24 hours, last month, ever
6. How much revenue have you generated: last 24 hours, last month, year to date

If we look at the list above as our starting six key performance indicators, lets look at some ways of getting the information into a format we can use. For the web analytics based targets (1,2) we can within most analytical packages, create custom reports and get them emailed to us in a variety of formats. We're going to use XML and reformat it for our needs quickly in PHP5. Having setup our goals and metrics in Google analytics, for example, we can then set them to be emailed every morning. We then have the attachment written to a log folder. Whenever we refresh our reporting dashboard we check for the latest file in the folder and process it, then store the processed data as a small HTML fragment which we include into our dashboard. This limits the amount of processing we have to do to a minimum and keeps our dashboard quick to refresh for the more real-time metrics.

For number 3 (number of sign ups) we'll make a query of our users table, requesting the number of users created in the last 24 hours, last month, ever. As you start to get bigger, these queries will start to become very long, taking valuable time away from your production database. You can attack this problem either by using aggregates or by moving all your data to a non-production environment purely for reporting. There is a danger in this in terms of replication lag and out of date data, but facing tables locks and the problems they cause traditional LAMP based web applications, it perhaps is a risk worth managing.

For number 4 (widget adds) we can do a variety of things. If we're looking at new Facebook subscribers then we can use the CSV download from Adonomics (formerly Appaholic). For example, the Gaping Void app, is accessibile via the url: http://adonomics.com/csv.php?mode=daily&display=4703559265 We can thus make a CURL or wget call to get this data every morning:

   wget http://adonomics.com/csv.php?mode=daily&display=4703559265
 

If we're looking at a widget, say embedded into MySpace or a blog, then we could set-up a download goal in an analytics package and measure it just like we did the unique visitors or you could get a feed from a log analyzer crunching the server logs for requests for a key component of your widget.

For number 5 (downtime), we need to define what downtime is in regards to your web app. Is it not only when the server is down but also when the database is locked or when traffic causes it to go slow? Or when disk space runs out or the load is massive? However, we don't need that granularity. For our purposes we just care if it's been usable and if not how long it wasn't usable for.

The best way of doing the more in-depth testing is to monitor the system externally via a package such as Mon, Ganglia and Nagios, all of which are beyond the scope of this article. We shall assume that you care when your service or application can not be reached. Thus we need to use Synthetic monitoring, a form of monitoring that checks a particular web app action every minute or so.

For example, you could set-up a check-out every minute to check that your basket works. Or perform a site search for a particular phrase and check the response contains a link to a certain page. If we a large corporation, we'd go and buy something like Sitescope from HP However yet again we're bootstrapping a we need a super simple solution. We can again use CURL to simulate an action, check the response for what we expect and then log the data for further analysis. If you were looking to run this in production you'd have to look at using a server in a different location to do the polling, perhaps just a cheap VPS with a cron job that runs every minute. It's left as an exercise to the reader to think of what actions are triggered by a downtime event. (SMS Alert? email? Jabber?)

$test1 =  curl_init("http://www.ourapp.com/action1?search=bob“);
	try {
	    $buffer = curl_exec($test1);
		curl_close($test1);
		if (strpos($buffer, “Bob Jones”) === false) {
			throwAlert(”App Down, Server Up”);
		} else {
			// Let's log a good search - so we can track the timeline of when things go bad.
			// We could also log how long the action took to complete - an interesting
			// metric to compare against system resources over time.
			logGoodSearch(now());
		}
	catch (Exception $e) {
		   // Having caught the error do something useful with it, including both logging and alerting
		}

For number six we need to do a query in your database. If you work with a Payment Services Provider, such as Metacharge or SECPay you'll probably interface via an API or payment page where you post in the amount and other details about a transaction. You should log the transactions in a database somewhere, taking note of the time, amount and other related metadata. You can't however store any of the card details unless you are PCI-DSS compliant as enforced by your merchant account provider. If we assume you use a table called transactionsSuccessful we can query it thus:

	SELECT SUM(value) FROM transactionsSuccessful;
	# Total revenue ever from online transactions

	SELECT SUM(value) FROM transactionsSuccessful WHERE time > NOW() - 86400;
	# Total revenue in the last 24 hours

	SELECT SUM(value) FROM transactionsSuccessful WHERE time > NOW() - 2419200;
	# Total revenue in the last month

We can do that query in real-time (dependent on database size and whether you're sharding, etc.) which means we can add that to your web app reporting screen.

Displaying your data

When displaying statistical data, there is a belief that more “data porn” is better. While that may give you a false sense of knowing the real picture, what we really need to be able to understand is what actions need to taken based upon the data. Is our disk space creeping up slowly or suddenly filling up? Do we need more disks? Are people who come from blogs to our site more likely to sign-up versus general traffic? If they are, what are you going to do about it? The brilliant O'reilly book, Information Dashboard Design goes through the many pitfalls of designing dashboards for management information systems. Visual metaphors such as speed dials don't help explain or enrich your data. They just decrease the signal to noise ratio. Edward Tuft's work on design is worth studying in this field, if only for his examples of how to display information with the lowest amount of visual noise.

Conclusions

In this brief article we've looked at some examples of how to produce reporting, how to provide it in a useful form and how to look at the data over time. This is a huge field – one in its infancy. However, I hope that I have at least got you thinking about how you produce internal metrics and how you can use that business data to make better business decisions to help build and grow your web app.

Before we even get near a terminal, we’re going to want to think about the general architecture of our application and do a fair bit of planning. So put away your PowerBook, find a big whiteboard and some markers, order some pizza, and get your engineers together.

In this article, we’ll look at some general software design principles for web applications and how they apply to real world problems. We’ll also take a look at the design, planning, and management of hardware platforms for web applications and the role they play in the design and development of software. By the end of this article, we should be ready to start getting our environment together and writing some code. But before we get ahead of ourselves, let me tell you a story. . . .

Layered Software Architecture

A good web application should look like a trifle…

A well-layered trifle (photo by minky sue)

Bear with me here, because it gets worse before it gets better. It’s important to note that I mean English trifle and not Canadian – there is only one layer of each kind. This will become clear shortly. If you have no idea what trifle is, then this will still make sense – just remember it’s a dessert with layers.

At the bottom of our trifle, we have the solid layer of sponge. Everything else sits on top of the sponge. It wouldn’t be trifle without it. The sponge supports everything above it and forms the real inner core of the dessert. The sponge is big, solid, and reliable. Everything above it is transient and whimsical.

In web applications, persistent storage is the sponge. The storage might be manifested as files on disk or records in a database, but it represents our most important asset – data. Before we can access, manipulate, or display our data, it has to have a place to reside. The data we store underpins the rest of the application.

Sitting on top of the sponge is the all-important layer of jelly (Jell-O, to our North American readers). While every trifle has the same layer of sponge – an important foundation but essentially the same thing everywhere – the personality of the trifle is defined by the jelly. Users/diners only interact/eat the sponge with the jelly. The jelly is the main distinguishing feature of our trifle’s uniqueness and our sole access to the supporting sponge below. Together with the sponge, the jelly defines all that the trifle really is. Anything we add on top is about interaction and appearance.

Adding the Business Logic

In a web application, the jelly is represented by our business logic. The business logic defines what’s different and unique about our application. The way we access and manipulate data defines the behavior of our system and the rules that govern it. The only way we access our data is through our business logic. If we added nothing but a persistent store and some business logic, we would still have the functioning soul of an application – it’s just that nobody would want to eat/use it. In days of old, business logic was written in C or COBOL (seriously). These days, only the big (where performance really matters) and the entrenched (where it’s too painful to change) do it that way. It’s perfectly acceptable to write business logic in your scripting language of choice (PHP, Perl, Python, Ruby, etc.) or the corporate-friendly alternative of Java.

So we have our supporting sponge and our personality jelly (perhaps with lumps of fruit, which have no sensible analogous component, but are useful ingredients in a trifle, nonetheless). We might have a dessert and we can certainly see the shape it’s taking, but it’s not yet a trifle. What we need now is custard. Custard covers the jelly and acts as the diners’ interface to the layers beyond. The custard doesn’t underpin the system; in fact, it can be swapped out when needed. True story – I once burnt custard horribly (burnt milk is really disgusting) but didn’t realize how vile it was until I’d poured it over the jelly. Big mistake. But I was able to scrape if off and remake it, and the trifle was a success. It’s essentially swappable.

In our web application, the custard represents our page and interaction logic. The jelly of the business logic determines how data is accessed, manipulated, and stored, but doesn’t dictate which bits of data are displayed together, or the process for modifying that data. The page logic performs this duty, telling us what hoops our users will jump through to get stuff done. Our page and interaction logic is swappable, without changing what our application really does. If you build a set of APIs on top of our business logic layer, then it’s perfectly possible to have more than one layer of interaction logic on top of your business logic. The trifle analogy starts to fall down here (which was inevitable at some point), but imagine a trifle with a huge layer of sponge and jelly, on which several different areas have additional layers; the bottom layers can support multiple interactions on top of the immutable bottom foundation.

The keen observer and/or chef will notice that we don’t yet have a full dessert. There’s at least one more layer to go, and two in our analogy. On top of the custard comes the cream. You can’t really have custard without cream; the two just belong together on a trifle. A trifle with just custard would be inaccessible to the casual diner. While the hardened chef/developer would recognize a trifle without cream, it just looks like a mess to the diner/user. We need some way to convey the message of the lower layers to our diners/users.

In our web application, the part of the cream is played by markup on the web, GUI toolkits on the desktop, and XML in our APIs. The markup layer provides a way for people to access the lower layers and gives people the impression of what lies beneath. While it’s true that the cream is not a trifle by itself, or even a very important structural part of a trifle as a whole, it serves a very important purpose: conveying the concept of the trifle to the casual observer. And so our markup is the same – the way we confer the data and the interaction concepts to the user. As a slight detour from our trifle, the browser or other user agent is then represented by the mouth and taste buds of our users – a device at the other end of the equation to turn our layers into meaning for the diner/user.

The Finishing Touches

There’s once thing left to make our trifle complete, other than the crockery (marketing) and diners (users). It would already work as is, containing all the needed parts. A developer might be happy enough to stop here, but the casual user cares more about presentation than perhaps he should. So on the top of our layered masterpiece goes garnish in the form of fruit, sprinkles, or whatever else takes your fancy. The one role this garnish performs, important as it is, is to make the lower layers look nice. Looking nice helps people understand the trifle and makes them more likely to want to eat some (assuming the garnish is well presented – it can also have the opposite effect).

Atop the masterpiece of code and engineering comes our metaphorical garnish – presentation. When talking about web pages, presentation is the domain of some parts of the markup, the CSS, and the graphics (and sometimes the scripty widgets, when they don’t fall into the realm of markup). For API-based applications, this usually falls into the same realm, and for email, it doesn’t necessarily exist (unless you’re sending HTML email, in which case it resembles regular web pages).

So the point of all this? A good web application has many layers, each of which provides a discrete function. All of the layers together make up the application, but randomly combining them will get you a badly built application or a bowl of sugary foodstuffs. From the perspective of the developer/chef, the most important layers are at the bottom because they support and provide the basis for the ones above. From the perspective of the diner/user, the higher layers are the important part that set the dessert apart, with the lower layers being taken for granted, particularly the ones at the very bottom of the stack. And so it should be with web applications.

The Book

This article is an excerpt from Building Scalable Web Sites by Cal Henderson, published by O’Reilly. In the book, Cal talks about how best to tackle each of the stages of development mentioned above, and includes practical advice on the design of software architecture, how to keep your app data clean and secure, planning your hardware needs, scaling development and working with APIs.

Like this article? Digg it!

]]>

So you are building a web application… good for you! How are you doing? “Lots of people have signed up so we’re doing really well”, that sounds nice in a “you probably have no clue” kind of way. Allow me to explain.

Part of the benefit of doing business online is you can track EVERYTHING. Web services are the pinnacle of such businesses in my eyes because they are eminently trackable. But the questions you will be asking are: what to track, and where to start, and why even bother. In this article I’ll show you how to approach these questions, so you can find out (or at least have a much better idea) how your web application really is doing.

Why Track Anything?

The Eisenberg Brothers at Future Now Inc. have put together a great body of work preaching the message, not for eyeballs and traffic, but rather for conversions. To explain why conversions are so important to your business, the Eisenbergs introduced the concept of the leaky bucket. Very simply stated, if you have a leaky bucket, to keep it full you can keep adding water, or you can patch the hole. If the bucket represents your website, and water represents your sales, you can keep adding traffic to keep your sales numbers high, or you can patch up the holes in your website and increase the number of sales with your existing traffic. Constantly adding more traffic is not sustainable for many small businesses (especially start-ups with a $0 marketing budget), so the only option is to patch up the bucket.

The Conversion Funnel

A good way to start tracking all the different types of conversion for your business is to put them together in a funnel diagram like this:

This is a classic sales funnel for a web application. At each stage of the funnel, you can expect to lose a percentage of your visitors. So, let’s say 10 per cent of the people who visit your website trial your product, and let’s assume 10 per cent of those people decide to pay for your service. Therefore you are in fact losing 90 per cent of your visitors at each step of the process. This is actually not a bad conversion rate for many services and in this scenario your sales funnel would look like this:

To sum up, if you lose people at every stage – and again, it is a fairly common scenario for you to lose 90 per cent at each stage with a web service – then your bucket is leaking.

A More Sophisticated Web Service Funnel

If your bucket is leaking, then you need to identify the largest leaks and plug them fast. Breaking up your funnel into smaller pieces can help. Here is a more precise funnel, useful for services such as FreshBooks, Basecamp or DropSend:

Whoa… how did things get so complicated? They actually aren’t. Comb over that list and you will see these are merely the general steps a user must go through before they pay you for your web service. Once you see this funnel laid out on the page, identifying your weak spots becomes easier.

How to Track Your Progress

Let’s assume you commit to tracking and defining each of the six number of steps I laid out above. Here’s how we do it with FreshBooks:

To track step #1 (visitors), go get yourself a good, remotely hosted JavaScript (not server log) analytics solution. Google Analytics is a free example. We recommend IndexTools. If you can afford enterprise software (or just like data and have the money) you might want to try Omniture. Again, I would avoid server log parsing solutions like AWStats, Webalizer and Urchin 5 because they are not as accurate as you need them to be.

You may have noticed in the chart above that we don’t use our stats to track anything but the first two steps in our conversion funnel: visitors and trials. That is true. I like the accuracy of database tracking – especially when your numbers are low (ie. you are just getting started) and inaccuracies can really throw you off. Tracking does create programming and database overhead, but since you are storing very little information it does not put much strain on your database, and you can always turn the tracking off at some point down the road. Also, and I would say this is much more important, analytics do not give you good active user counts. What does give you good active user counts is tracking the number of times a user logs in.

Stats are also AWFUL at tracking paying user upgrades for a web service, because the transaction is not immediate – it can take weeks and/or months – and sometimes one user will upgrade multiple times. For these reasons I highly advocate tracking these metrics yourself using your database. You will likely want to build some custom reports for yourself as well. Allocating developing resources for internal projects like this is part of a growing application development trend known as shadow application development. It’s an investment in yourself and it will pay off.

How to Decide What to Fix First

So let’s assume you take my advice and you build and track the conversion steps. And let’s assume your bucket is leaking (if it isn’t, email me with the subject “EAT MY SHORTS”). The question is, “what do I fix first?” The answer? Start as high up the funnel as possible.

Most people need to start with visitors to trials. If 5-10 per cent of first time visitors do not trial your service, you have some work to do – fast. If you need more specific guidance, I recommend adding another step to your conversion funnel. For example, if you can’t figure out where you are losing people before they sign up to trial your service, add another step in your stats called “view sign up”. If 30 per cent of your visitors are viewing your sign-up page, but only 2 per cent are actually signing up, you’ll want to redesign your sign-up page.

Here is a chart with some hints as to where to invest your time in an effort to improve each stage of your conversion funnel:

How Much Data Do You Need to Have a Decent Sample?

Ideally, you want 100 or more records so you have a decent sample in any stage of your conversion funnel. But think about that for a second, if you need 100 paying subscribers, and you convert one per cent of all site visitors to paying users, then you need to generate 1,000 trials, and drive 10,000 visitors to your website. That can take months when you are just getting started. This is why it is so important to start focusing on the top of that funnel first (ie. visitors to trials, the trials to successful registration form completion). These metrics are the fastest to track, and if you get them set right, you will get more people into your application who will then give you feedback so you can work on your other metrics. It’s a virtuous circle.

Conclusion

You can’t improve what you can’t measure, so start measuring your conversion funnel ASAP. It’s pretty remarkable when you see the results, and if you take the time to do it, you will know how you are doing in no uncertain terms.

Shout out to those of you who run web services: I’d love to know how your conversion funnel is doing so that I can aggregate some data and share it with entrepreneurs who are trying to get started. Shoot me an email if you would like to participate. Thanks.

Like this article? Digg it!