Archive: Back-End

At 280 North we’ve been working on a framework called Cappuccino for creating what we like to call desktop class applications. These are not your average web sites, but instead a new breed of web application that aims to replace many of the programs that currently run on your desktop. When we launched 280 Slides, the first application built on this framework, many people were surprised to see just what was possible in the browser. Since then, Cappuccino has been steadily progressing with over 40,000 downloads and code contributions from twenty-one individuals.

The goal of Cappuccino has always been to make it as simple as possible to build high quality applications in the browser, and up until now we’ve worked on providing built-in behavior for enabling such tasks as autosaving, undo/redo, and copy/paste. But recently we’ve been focusing on an entirely new product: Atlas. Atlas is a new IDE and visual layout editor that leverages the power of Cappuccino, but drastically reduces the amount of work required for creating a truly rich application experience on the web.

WYSIWYG

With Atlas you can build and style your entire interface using drag and drop. By default we provide you with a large library of built in widgets along with the new Aristo open source theme, meaning your application will look fantastic with no additional work. However, you can also skin these applications entirely from the editor and extend the library of widgets with Atlas’ plugin architecture. And unlike a lot of existing layout editor availabe today, Atlas is not a code generator. On the contrary, Atlas lets you edit the live JavaScript objects in memory and then takes a “snapshot” of them in place. When your application runs, it ‘thaws’ these objects out, making sure that you see exactly how your application will look before you run it.

Layout that Makes Sense

Atlas provides a new take on web layout that make it easy to define precisely the behavior you want, while simultaneously doing away with existing browser inconsistencies. Instead of having to learn the variety of different ways to position and resize elements in the browser, Atlas introduces the simple concept of ‘anchors and springs’. Anchoring an interface element to one of the edges around it ensures that it will stay “stuck” there when it’s container resizes. Activating the internal ’springs’ causes it to resize with its container.

While simple to learn, these two tools enable you to create incredibly complex and fluid layouts. More importantly, they’re guaranteed to behave the same way regardless of the browser you’re running your web app in. Atlas also provides a large selection of collection views to handle common layout tasks such as split views, table views, and scroll views.

Connections Reduce Glue Code

A common problem that has plagued layout editors in the past is the need for additional glue code to “talk” to the generated interface. In Atlas, much of this work can be done with a technology called ‘connections’. Connections allow you to visually define the interactions between interface elements and other objects. For example, you can define what action a slider should take when it is dragged:

Similarly, you can ‘bind’ the contents of an array to a table, so that when it changes the table automatically updates as well, all without having to ever touch the keyboard:

Model View Controller Built-In

Cappuccino is a Model-View-Controller framework, and Atlas takes this idea to the next level by allowing you to not only create visual elements, but abstract models and controllers as well. By allowing you to interact directly with the objects in your program visually, you can focus on building unique interactions instead of learning a myriad of APIs.

Atlas has gone a step further and provided a number of pre-built controllers to existing web services that you can simply drop in to their applications. You can then simply use connections to grab information from services like RSS and Twitter. Once again, Atlas’ plugin architecture allows you to create controllers for your own web services as well, which you can then share with others.

Multi-Platform Support

Cappuccino was built from the ground up in preparation for a world of multiple platforms. It is becoming increasingly important for applications to run in several different environments: browsers, social networks, handheld devices, and much more. In general, very little application logic or backend code has to change from platform to platform, but the interface usually needs to be recreated from scratch.

With Atlas, it is drop dead simple to create a unique and custom interface for different platforms like the web browser and the iPhone. Cappuccino will then intelligently load the correct interface for your application depending on the environment it is being run on, allowing you to reduce the amount of code you need to rewrite and as well as using just one language and API for all the platforms you deploy on. We call this idea ‘Write once, layout everywhere’, and we feel it is the right way to gaurantee a high quality experience everywhere you ship your application.

View full size

View full size

Atlas will be shipping this summer and you can sign up for email updates at 280atlas.com.

[Photo credit: flickr.com/photos/atelier_tee]

It is quite interesting to see how web development turned from a niche for brochure-ware and intranets to one of the biggest software development environments in the market over just a few years.

“The web as the platform” is a hollow dream no longer ‐ you can now run and develop a web application without needing to host anything yourself and you can even get the data from other sources. With this radical shift comes a lot of change for developers, but the question is, are you keeping up?

Change is good – we need to get our act together in any case. It’s 2009 and we have the chance to start this year by tackling issues that concern everybody: the general recession, inevitable lay‐offs and lack of investment in non-profit generating parts of organizations. Now’s a good time to get real, regroup, and reassess the ways in which we approach our jobs.

We are in a crisis – have you watched the television lately? It doesn’t make cheerful viewing. However, with every crisis comes the opportunity (and the necessity) to improve and see how we can work more efficiently, producing things that are more valuable and relevant to the people we want to reach. The hype is over ‐ it’s time to get busy, working as professionals.

Back When All This Was Just Fields 

When I started web development every article, tutorial or introduction to a system started with “open your editor, read this documentation and start writing your app, (or web site, css trick or javascript trick)”. I loved it. I downloaded the article to a floppy disk at work, went home and in the evenings without the internet (remember you paid by the minute) wrote my first solutions and battled with the documentation. Later, the only thing that changed was that I didn’t work on it at home but on a laptop on the train instead.

Nowadays, things look different. First of all, for the most part it’s impossible to develop offline ‐ this is because we use hosted services, APIs and most likely you’ll write something that uses some data stored elsewhere on the web. Offline web development is becoming very rare indeed, which can be terribly frustrating if you are in a public space or hotel rooms. In other words, we build on already tested and proven concepts instead of building everything ourselves. This means we have to do much less work, but it also means that we don’t have full control. We rely on data being available to us by third parties or ‘the web’. This is pretty tough for developers as we tend to be control freaks.

Aiming Higher and Wider

The other thing that can be a hurdle for the old school developer is the fact that we tend not to build new systems any longer. Back in the day we worked on the bleeding edge: almost everything we built was the first of its kind on the web and clients came to us to build their first web presence. Users were easy to come by and people were eager to do things easier and quicker on the web. This is pretty much over.

Unless you have a ‘very’ compelling product or idea clients will already have something in place that they use and will have already spent a lot of time and money on it. Maintenance is much more common than innovation. Users are much more spoilt by choice and already spend a lot of their time using other products ‐ just building something and hoping that users will come is not enough any longer.

Most products you’ll build these days are aimed at piggy‐backing on existing user bases and use cases, rather than creating something entirely new. The web has become much more of a commodity than it used to be and a lot of people rely on it for their day‐to-day chores and social communication. For example, the amount of people that started to get in contact with me after years of silence is staggering. The reason? Facebook and how simple it is to use. These are the same people that asked me if my e-mail address would change when I moved to another city!

These social networks create huge traction, and even hardware developers jump on them to sell their newest products. Just yesterday I saw a mobile ad at the train station telling me that the new smart phone automatically gets updates from my friends on Facebook. Sounds a tad creepy to me, but I’m sure it boosted that manufacturer’s smart phone sales and I am sure many a teenager will be asking for this one for their birthday.

Social networks are taking over in all kind of environments. The same is happening to search engines.  

For example, I do see a lot of movie posters on my way to work and realized lately that the film industry has stopped bothering with domain names for movies (I guess domain squatters blackmailing them are to blame). Instead I started to see “enter xyz in Google to learn more” and one movie even used a bebo and myspace address instead of a domain. This is quite commonplace in South East Asia already.

A ‘New’ Breed of Developers 

What shall we do about it? We could sit sulking in our developer corner claiming that these things will never work but the fact is that this is where people invest and the money still flows. It is also where the users are. There is a big market in some technologically underdeveloped countries for Facebook applications and even blog systems. Using out‐of‐the‐box solutions companies can easily cover both the development and the maintenance of the content. Can you for example guess where http://s2999.com is written and maintained?  

The interesting fact for businesses is that these developers are pragmatic. They don’t cost much for a business to hire, they deliver the job without trying to lecture you and all they expect is a share of the outcome. Us on the other hand, leading and amazing developers that we are, are rumored to be tricky to work with, grumpy even. We don’t want to use things that other people made and are very happy indeed to tell any business person that they are wrong and we know all about the web. Who’d you work with?

Sure, this is nothing new – outsourcing shops have promised cut‐price development for years and most of the time the outcome was appalling (funnily enough, in a lot of cases not because the developers were bad but the management didn’t know what they wanted or failed to give precise delivery specs). With the systems in place though, development becomes much more of a ‘use’ case than a ‘build’ case and this is the biggest challenge we now face.

Good­bye White Canvas, Hello Lego Bricks

In essence, web development these days doesn’t start with a blank document but with downloading an SDK or entering your application data in some web app to get a developer key. In other words using a system that is much larger than anything you’d ever have done yourself. This concept is very hard to swallow for us battle‐hardened masters of the command line and I myself have problems getting enthusiastic about using a chunk of ‘hello world’ code and turning it into something great.

If we look at it logically, however, and leave our ego to go and play in the garden for a while it’s clear that there are many valid reasons to use SDKs and hosted development environments. It’s exactly the same thing we do as clever developers ‐ build ourselves shortcuts, hack our development tools or assemble our very own library of snippets – in order to avoid having to do the same things over and over again.

Using SDKs and hosted development environments shouldn’t mean your work will be dumbed down or that people won’t respect you as a developer. It means that they are helping to take the pain away and have taken away the hoops that they jumped through so that you can just build something interesting without worrying about the underlying architecture, security of the system or its performance.

Abstracting these away means you can easily do fixes or improvements should the need occur ‐ and boy will it occur. It is not fun to maintain a site or a server these days. The amount of people that want to hack you and inject malicious code is staggering and their skills are impressive. The only way to battle these threats is to have one system to fix ‐ not millions of little applications.

Abstraction also means that many more people can start developing applications. Geeks build geek tools, designers build designer tools, ‐ if you lower the entry barrier to developing, all groups can bring their expertise to play with the system and subsequently you can build really cool tools together. Of course, there is a lot of pointless and bad work being done too (no, I don’t want your teddy bear to travel on Facebook or catch dozens of sheep) but that’s like saying HTML is bad because people can use tables and fonts to create layouts fast instead of using CSS.

So Are We Obsolete?

Does this mean that the tinkerers, the people who spend hours fixing an IE6 bug or open the border of the screen to show sprites outside the dedicated area (yes, C64) are obsolete and should just move on? No, it doesn’t ‐ not by a long shot. If we cave in now and grumpily sit in the corner pointing at the kids with their fancy SDK toys we are indeed obsolete, but there are a lot of things that this brave new world of development needs us to do.

It needs us to analyze the systems and see where they need improvement. For example the amount of accessibility issues that can be fixed in social networks with very little effort is staggering. Most of these issues are not there because people are lazy but because they just don’t know about the consequences of their actions.

Why not help with turning these systems into a working network of systems. Open Social is a great idea but suffers from too many people shouting ‘FAIL’ at it instead of helping to fix it.

Why not work on bridging the gap between our two worlds. There is no sense in offering people a text box in an HTML page to develop in. I have used one of these and the resulting code is ‘always’ terrible. The same applies to code editors in a browser doing the indenting and colour coding for us ‐ they are flaky and when they crash they take your whole app with it (and the pictures of kittens you had in another tab). There are some Firefox extensions that allow you to edit text in your editor and then enable you to pull it back into the text area with a keyboard shortcut ‐ more of those please. Show by example how your expertise can make web applications better and how communicating with you upfront makes for better APIs and SDKs.

So What About the Future?

If I knew that… However, what I will say is that the way some web developers deal with market forces and business people is wrong. 

Our craft is becoming a commodity and people in charge don’t care about the quality of the markup, CSS or how short our JavaScript is. What matters is how fast you can get it to market, how many people it reaches and how cheaply it can be built. I think that web development has come on in leaps and bounds in the last years, not because of top‐down decisions and empowerment of the folks on the ground, but by geeks just doing the right thing without asking for permission first. We should follow standards and build solid code to make our own life easier ‐ not to impress other people. Then show the code and explain its merits to people outside of the development world to improve it for everybody.

So, (a little late) here are some of my New Year’s resolutions for this year:

1. Give things the benefit of the doubt before judging them. In other words, keep the ‘FAIL’ shouting to the funny blogs where it belongs.
2. Take time to have a look at the SDKs, hosted services and frameworks out there and write about them instead of showing yet another proof of concept.
3. Try to talk to people in charge of these systems to stop them from aggravating the developer crowd with decisions that are perfectly logical to them but don’t hold up in our world.
4. Great API developers don’t necessarily know HTML and their demos might be atrocious table and font monsters. Instead of shunning the API provide examples of how to do it better. Explaining the effects that bad API design can have goes much further than judging only by our own standards.
5. Clean up the demo code of the systems, SDKs and APIs that I have direct access to. There is nothing worse than a bad example being replicated million fold as people copy+paste instead of reading docs.
6. Write human readable documentation and don’t expect a degree in Klingon. Ask non-developers how real people use systems and what they like most about them.

What do you think? Is it time to live in the now and help build a good web 2.0,3.0,4.3212 and so on or have we lost the battle for a beautiful, valid and semantically rich web already?

Reporting is perhaps one of the biggest business deliverables for enterprise IT. Being able to say how many widgets were sold in month seven in year XX, sub-divided by customer ABC grouping, is one of the easiest ways to make management happy. But when building a web app, reporting is the last thing on many feature lists. Clear reporting is vitally important for the long term success of your web app and managing your growth. However, if you're still in the beta stage, we'll explore some ways of building simple reporting fast.

Why reporting? Because it helps you know the numbers. You need to be able to take data from web analytics, your own system stats (how many sign-ups per month, what's this month's revenue?, etc.) and be able to render that visually in some sensible way for you to be able to understand it and make the correct business decisions based upon it.

Big business style

If you're a large multi-national with large IT budgets, you're going to looking for a reporting system that can query multiple data warehouses with hundreds of users and multiple output formats. You'll also want it as near real time as possible. Packages like Crystal Reports and Jasper reports provide large scale interfaces to data sources and provide nice tools to create easy to read reports. But being as we're bootstraping upstarts, we need a solution that can pull together the data we need into one place and export it how we want.

Super simple (nearly) real time reporting

I'm not going to go into depth about database replication, load balancing your queries, redundancy or the technological challenges you face in getting your web app's data out. I am going to assume that you have a database of some sort, a web analytics package and clear objectives for what you want to measure. The most important of those is the objectives. What is it that illustrates growth and successes of your web app? This question in itself could fill thousands of articles, but for the sake of getting to some useful examples we'll look at the following:

1. Number of unique visits to your site: last 24 hours, last month, ever
2. Number of new subscribers to your rss feed: last 24 hours, last month, ever
3. Number of sign ups: last 24 hours, last month, ever
4. Number of adds of your f8/widget: last 24 hours, last month, ever
5. How much downtime: last 24 hours, last month, ever
6. How much revenue have you generated: last 24 hours, last month, year to date

If we look at the list above as our starting six key performance indicators, lets look at some ways of getting the information into a format we can use. For the web analytics based targets (1,2) we can within most analytical packages, create custom reports and get them emailed to us in a variety of formats. We're going to use XML and reformat it for our needs quickly in PHP5. Having setup our goals and metrics in Google analytics, for example, we can then set them to be emailed every morning. We then have the attachment written to a log folder. Whenever we refresh our reporting dashboard we check for the latest file in the folder and process it, then store the processed data as a small HTML fragment which we include into our dashboard. This limits the amount of processing we have to do to a minimum and keeps our dashboard quick to refresh for the more real-time metrics.

For number 3 (number of sign ups) we'll make a query of our users table, requesting the number of users created in the last 24 hours, last month, ever. As you start to get bigger, these queries will start to become very long, taking valuable time away from your production database. You can attack this problem either by using aggregates or by moving all your data to a non-production environment purely for reporting. There is a danger in this in terms of replication lag and out of date data, but facing tables locks and the problems they cause traditional LAMP based web applications, it perhaps is a risk worth managing.

For number 4 (widget adds) we can do a variety of things. If we're looking at new Facebook subscribers then we can use the CSV download from Adonomics (formerly Appaholic). For example, the Gaping Void app, is accessibile via the url: http://adonomics.com/csv.php?mode=daily&display=4703559265 We can thus make a CURL or wget call to get this data every morning:

   wget http://adonomics.com/csv.php?mode=daily&display=4703559265
 

If we're looking at a widget, say embedded into MySpace or a blog, then we could set-up a download goal in an analytics package and measure it just like we did the unique visitors or you could get a feed from a log analyzer crunching the server logs for requests for a key component of your widget.

For number 5 (downtime), we need to define what downtime is in regards to your web app. Is it not only when the server is down but also when the database is locked or when traffic causes it to go slow? Or when disk space runs out or the load is massive? However, we don't need that granularity. For our purposes we just care if it's been usable and if not how long it wasn't usable for.

The best way of doing the more in-depth testing is to monitor the system externally via a package such as Mon, Ganglia and Nagios, all of which are beyond the scope of this article. We shall assume that you care when your service or application can not be reached. Thus we need to use Synthetic monitoring, a form of monitoring that checks a particular web app action every minute or so.

For example, you could set-up a check-out every minute to check that your basket works. Or perform a site search for a particular phrase and check the response contains a link to a certain page. If we a large corporation, we'd go and buy something like Sitescope from HP However yet again we're bootstrapping a we need a super simple solution. We can again use CURL to simulate an action, check the response for what we expect and then log the data for further analysis. If you were looking to run this in production you'd have to look at using a server in a different location to do the polling, perhaps just a cheap VPS with a cron job that runs every minute. It's left as an exercise to the reader to think of what actions are triggered by a downtime event. (SMS Alert? email? Jabber?)

$test1 =  curl_init("http://www.ourapp.com/action1?search=bob“);
	try {
	    $buffer = curl_exec($test1);
		curl_close($test1);
		if (strpos($buffer, “Bob Jones”) === false) {
			throwAlert(”App Down, Server Up”);
		} else {
			// Let's log a good search - so we can track the timeline of when things go bad.
			// We could also log how long the action took to complete - an interesting
			// metric to compare against system resources over time.
			logGoodSearch(now());
		}
	catch (Exception $e) {
		   // Having caught the error do something useful with it, including both logging and alerting
		}

For number six we need to do a query in your database. If you work with a Payment Services Provider, such as Metacharge or SECPay you'll probably interface via an API or payment page where you post in the amount and other details about a transaction. You should log the transactions in a database somewhere, taking note of the time, amount and other related metadata. You can't however store any of the card details unless you are PCI-DSS compliant as enforced by your merchant account provider. If we assume you use a table called transactionsSuccessful we can query it thus:

	SELECT SUM(value) FROM transactionsSuccessful;
	# Total revenue ever from online transactions

	SELECT SUM(value) FROM transactionsSuccessful WHERE time > NOW() - 86400;
	# Total revenue in the last 24 hours

	SELECT SUM(value) FROM transactionsSuccessful WHERE time > NOW() - 2419200;
	# Total revenue in the last month

We can do that query in real-time (dependent on database size and whether you're sharding, etc.) which means we can add that to your web app reporting screen.

Displaying your data

When displaying statistical data, there is a belief that more “data porn” is better. While that may give you a false sense of knowing the real picture, what we really need to be able to understand is what actions need to taken based upon the data. Is our disk space creeping up slowly or suddenly filling up? Do we need more disks? Are people who come from blogs to our site more likely to sign-up versus general traffic? If they are, what are you going to do about it? The brilliant O'reilly book, Information Dashboard Design goes through the many pitfalls of designing dashboards for management information systems. Visual metaphors such as speed dials don't help explain or enrich your data. They just decrease the signal to noise ratio. Edward Tuft's work on design is worth studying in this field, if only for his examples of how to display information with the lowest amount of visual noise.

Conclusions

In this brief article we've looked at some examples of how to produce reporting, how to provide it in a useful form and how to look at the data over time. This is a huge field – one in its infancy. However, I hope that I have at least got you thinking about how you produce internal metrics and how you can use that business data to make better business decisions to help build and grow your web app.