There have been a ton of helpful and interesting tips posted to Tipster recently, so I thought we’d pull a few out for you all (and give the submitters a bit of well-deserved publicity).

Topics that we’re covering are: MySQL, CSS, JavaScript, PHP, JQuery, life hacks and server admin.

MySQL Geo Search

Here’s a MySQL statement to search for nearest objects to you in a database, ‘lat’ & ‘lng’ are fields in the ‘items’ table with the location of object and $latitude & $longitude are the users location. The distance field will be the calculated distance in Km between the user and the object:

SELECT *,(((acos(sin((".$latitude."*pi()/180)) * sin((`lat`*pi()/180))+cos((".$latitude."*pi()/180)) * cos((`lat`*pi()/180)) * cos(((".$longitude."- `lng`)*pi()/180))))*180/pi())*60*1.1515*1.609344) as distance FROM `items` ORDER BY distance;

By: Sam Machin
Vote on this tip

Transparent background images – PNG fix for IE6 – a few reminders

When using the MS filter (via The AlphaImageLoader) to fix PNG transparency for IE6 — in your CSS, remember to:

1. Set width and height for the element/s containing the transparent background image.
2. First set background to none — then apply filter for background image.
3. Apply ‘position: relative;’ to any contained links to ensure functionality

(Also, bear in mind that the background images can no longer be tiled or positioned via ‘background-position’).

By: Prisca
Vote on this tip

Address Longitude and Latitude in Google Maps

Google maps does not show the Longitude and Latitude of an address you search. To get this information, enter this piece of JavaScript into you browsers address bar and hit enter:

javascript:void(prompt('',gApplication.getMap().getCenter()));

A little window will then pop-up displaying the Longitude and Latitude for you. Bingo!

[Editor's note: If you add this to your bookmarks toolbar, it's even easier to use. Just click it whenever you're on Google Maps. Hat tip to Phil Balchin.]

By: Pete Roome
Vote on this tip

Custom Details in Code Hints – Zend Studio for Eclipse

Zend Studio is a brilliant platform for PHP development. In my view, well worth the cost. One of the best things about it now being built on Eclipse is the project-wide code hinting.

When you see the built-in code hints and highlight an option, you get loads of lovely details regarding the functions and classes that you are about to use… as you would expect from most modern IDEs. However, Zend Studio for Eclipse takes this further for your own classes and functions!

When you have written your classes and functions, start placing a multi-line comment just above the opening line, i.e. /**, hit Return and Studio will prompt you for data types and descriptions for your method variables.

This is invaluable in understanding those classes you wrote at 3:47am, 3.5 years ago and haven’t used since!

By: Simon Hamp
Vote on this tip

Use a Print Stylesheet

Print stylesheets allow you to change the style of your page when printing, without having to provide a separate “print version” of the markup.

You specify a print CSS by adding “media=’print’” as an attribute to the link tag.

<link rel="stylesheet" type="text/css" media="print" href="print.css"/>

In the print CSS you should remove unnecessary elements such as sidebars, menus, background colours, presentational images, etc so as not to waste ink and paper on parts which don’t need to be printed. You can also change the font and size to make sure the text is clear on the printed page.

You can even add some CSS to show the URLs of links on the page, which wouldn’t normally be seen when printing.

a[href]:after { content: " (" attr(href) ") "; }

By: Rich Adams
Vote on this tip

Make your links easier to read

Instead of using { text-decoration: underline; } on your links (as browsers do by default) use:

{ border-bottom: 1px solid; }.

This makes your links easier to read, as there will be more space between the text and the line underneath, so the line will no longer cross through your ‘y’s and ‘g’s.

You can also then style your underline with all the usual border stylings, much more exciting than a plain underline :)

By: Laura Kalbag
Vote on this tip

Manual filmstrips in jQuery

Really simple manual photo film-strip in jQuery. Used to swap a large image on a page with that of a thumbnail elsewhere on the page, such as different photos of a product in an e-commerce store.


$(document).ready(function(){
imageSwapper(".thumbnails a");
});



function imageSwapper(link) {
$(link).click(function(){
$('#largeimage').attr('src', this.href);
return false;
});
};


Just link the thumbnails to their larger versions and pop them in a div or paragraph of class thumbnails, and give the large image an ID of large image, and you are good to go!

By: Michael Peacock
Vote on this tip

Store user passwords as salted hashes

Just using a hash of the user password is not strong enough. If two users have the same password, they’ll have the same hash. Dictionary attacks are also still possible as the attacker can have a list of dictionary word hashes. Using a salted hash involves generating a random string of characters, which you then concatenate with the password before hashing. Two users with the same password will then have different hashes, and the stored hash will never just be a hash of a common word if the user chose a weak password. You then store the salt, and the hashed value in your database.

hash_to_store = sha1(salt + real_pass)

There are many different methods you could use too, such as concatenating the salt with a hash of the password and then hashing that, etc.

hash_to_store = sha1(sha1(real_pass) + ...

By: Rich Adams
Vote on this tip

CD/DVD stuck in Macbook

Having gotten a DVD stuck inside my Macbook the other day, i found the only way to force eject it was to hold down the trackpad button on reboot. DVD popped right out.

By: Pete Roome
Vote on this tip

Photo credit: flickr.com/photos/mance

We all know that people don’t like filling out forms online, including the folks who design and build them. Most good web designers working today know to keep them as concise and painless as possible. A minimalist sign up process on your shiny new web app is sure to produce a higher sign up rate than one that asks for a dozen pieces of irrelevant information.

Sometimes, however, you’ll want to develop a form that truly benefits from a large number of options. Let’s design a search form for a fictional used automobile site. Presenting every option in your database would probably be messy and intimidating. Just one section might look something like this:

An Alternative Presentation

There may be times when keeping all of the available options visible makes the most sense. Let’s assume for the time being, however, that you want to package the search options into a neater display.

Using this technique, we’re able to save valuable screen real-estate and the user is, hopefully, less frightened by all those checkboxes. In addition, you’ll have much more freedom to style your dropdown controls to match the look and feel of your application.

Building the Widget

This technique doesn’t require a fancy graphics treatment. You could easily use vanilla HTML elements and wire up the interaction elements with Javascript (as we’ll discuss below) and you’ll be good to go. However, if you’re wanting that little extra flair, let’s jump into Photoshop (or your graphics editor of choice).

I’ve built my widget to a fixed width with the idea that it’ll be smart enough to grow taller to accommodate longer text. Here’s how the plain widget looked:

And with the dropdown “trigger”:

As I mentioned, we want the widget to be able to expand vertically so that longer text doesn’t dangle outside it. We’ll split our image into top and bottom pieces and stretch the top one out so it’s tall enough to hold the fullest possible amount of text.

Top Piece

Bottom Piece

Be sure you don’t simply resize the image to be tall enough since it will distort your masterpiece. Simply select one horizontal row, copy it, paste it, and use the transform tool to create the “filler” region.

Styling and Wiring

Now it’s time to create the HTML, CSS, and Javascript needed to make your widget come to life. In its most basic form, there is a wrapper container:

<div class="dd_wrapper">
<div class="dd_bottom"></div>
</div>

The dd_wrapper class will contain our top background image and dd_bottom will contain the bottom one.

.dd_wrapper {
  color: #fff;
  width: 169px;
  background: url(../images/dd_top.gif) top left no-repeat;
  font-size: 11px;
  padding: 11px 30px 10px 14px;
  position: relative;
  line-height: 1.3em;
  min-height: 17px;
}

.dd_bottom {
  width: 213px;
  height: 18px;
  position: absolute;
  bottom: -1px;
  left: 0;
  background: url(../images/dd_bottom.gif) top left no-repeat;
  z-index: 0;
}

There’s nothing too fancy going on here. The width of dd_wrapper is accounting for the image width minus the left and right padding (213 – 30 – 14 = 169). The right padding is larger to allow room for the dropdown trigger graphic. The position attribute is set to relative so that we can absolutely position our other elements within in. The dd_bottom class is anchored to the bottom left of the parent container and its width is left at 213px since there is no padding for it. We specify a height of 18px to match that of the bottom image so that it doesn’t collapse and disappear due to having no inner content.

Next, let’s add our dropdown trigger and a span wrapper for our text label.

<div class="dd_wrapper">
<span class="inner">
Engine <span class="light">(6 cylinder, 8 cylinder)</span>
</span>
<a href="#" class="dd_toggle" title="Change">
<img src="../images/dd_toggle.gif" alt="Change" />
</a>
<div class="dd_bottom"></div>
</div>

The inner class will hold our text and the dd_toggle link is positioned on the right hand side.

.dd_wrapper .inner {
  position: relative;
  z-index: 100;
}

a.dd_toggle {
  position: absolute;
  top: 11px;
  right: 13px;
  z-index: 10;
}

.dd_wrapper .light {
  color: #a3bfe9;
}

The inner class is relatively positioned so that the z-index rule works properly. These are used to keep the text above the background image of the dd_bottom container.

Now it’s time to add our dropdown panel. It will be hidden by default and the dd_toggle link will call a Javascript function to handle the showing and hiding of the panel.

<div class="dd_wrapper">
  <span class="inner">
    Engine <span class="light">(6 cylinder, 8 cylinder)</span>
  </span>
  <a href="#" onclick="toggleDropdown('engine_options');return false;" class="dd_toggle" title="Change"><img src="../images/dd_toggle.gif" alt="Change"/></a>
  <div class="dd_bottom"></div>
  <div id="engine_options" class="dd_option_panel" style="display:none;">
    <p>Select which engine options you are interested in.</p>
    <ul>
    <li><input type="checkbox" id="engine_4" />
      <label for="engine_4">4 cylinder</label>
    </li>
    <li><input type="checkbox" id="engine_6" checked="checked" />
      <label for="engine_6">6 cylinder</label>
    </li>
    <li><input type="checkbox" id="engine_8" checked="checked" />
      <label for="engine_8">8 cylinder</label>
    </li>
    <li><input type="checkbox" id="engine_10" />
      <label for="engine_10">10 cylinder</label>
    </li>
    <li><input type="checkbox" id="engine_12" />
      <label for="engine_12">12 cylinder</label>
    </li>
    <li><input type="checkbox" id="engine_hybrid" />
      <label for="engine_hybrid">Hybrid</label>
    </li>
    <li><input type="checkbox" id="engine_elec" />
      <label for="engine_elec">Electric</label>
    </li>
    </ul>
    <p class="buttons">
      <input type="image" src="../images/save_choices.gif" onclick="toggleDropdown('engine_options');return false;"/>
    </p>
  </div>
</div>

The toggleDropdown() function will toggle the requested panel and it will also ensure that all other panels are hidden. This way, if one panel is open and the user clicks the trigger on another dropdown widget, only the newest panel will appear as open. I am using the Prototype Javascript library here, but this could easily be adapted to any other implementation.

function toggleDropdown(panel)
{
// toggle the requested one
Element.toggle(panel);

// then hide all of the others so that only
// one (at most) is open at a time
$$('body .dd_option_panel').each(function(node){
if (node != $(panel)) {
Element.hide(node);
}
});
}

The Requisite IE Hacks

You can always count on Internet Explorer to keep you on your toes. If you choose to align your dropdown widgets vertically like I have in my example, you’ll notice that the higher dropdown panels render behind the lower widgets. This is due to IE’s z-index implementation that is documented extensively elsewhere. The fix is to wrap each dropdown container with an additional div element whose sole purpose is to force a proper z-index stack.

Also, keep observers may have noticed my use of min-height for the dd_wrapper container. IE6, of course, doesn’t support this attribute but I’ve had success using Dustin Diaz’s Min-Height Fast Hack.

.dd_wrapper {
  ...
  min-height: 17px;
  /* IE6 min-height hack: http://www.dustindiaz.com/min-height-fast-hack/ */
  height: auto ! important;
  height: 17px;
}

<div style="position:relative;z-index:60;">
<div class="dd_wrapper">
...
</div>
</div>

<div style="position:relative;z-index:50;">
<div class="dd_wrapper">
...
</div>
</div>

Final Touches

My example includes a save button that could be used to make an AJAX call. This would be useful if your search updated the results on the fly as search options were changed. If your form was going to be submitted normally, this button could be removed.

Keep in mind that you aren’t limited to checkbox fields in your dropdown panels. You could use radio buttons or anything else. With these types of hidden panels, you will be able to include some additional helper text for your users when it makes sense, but try to keep it short and to the point.

The final thing you’d want to add is some code to handle the updating of the text label when the options are updated. Depending on your implementation, there are various ways to handle this. For example, the result of an AJAX request might determine how the label gets updated. You might want to have your toggleDropdown() function include some logic to update those DOM elements.

My final advice would be to put a little extra thought into making the text labels smart enough to show a concise view of the selections. For example, if your user selects all of the options for Fuel, “Any Fuel” is easier to read (and actually indicates an all-inclusive selection) than “Fuel (Gasoline, Diesel, Alternative)”.

Hopefully this article gives you some ideas that you can incorporate into your next project. The overall idea is quite simple, but there are endless ways that you could expand and improve upon it!

View the completed demo.

In this presentation, Matt Biddulph, CTO at Dopplr discusses how they worked with Facebook and also JavaScript (using JQuery) to create the social widgets and plug-ins that are essential to Dopplr’s make-up.

Matt also spoke about other critical stages in the development of Dopplr – including how they used the internet as a platform, and how they deal with user identities.

This session is from Future of Web Apps London 2007, hosted by Carsonified. You can listen to the entire MP3 and download Matt’s presentation at the FOWA past events page.

The next FOWA is in Miami this February 28th – more details at the usual place!

There are some things in the world of accessibility that appear, on the face of it, to be really wonderful ideas… until you scratch slightly below the service. What may seem feasible when putting together some guidelines on accessibility might not ultimately translate well to a real-world application. Hands up who can remember the last time they felt compelled to use a longdesc attribute? And what about the accesskey attribute? Oh, you have used them you say. OK, let’s back up a little and find out what went wrong here.

The Problem With Accesskeys

The accesskey attribute is a neat little feature that is supposed to enable the user to quickly access a navigation, search, or some other feature of a page quickly. For example, I could use an accesskey of ’s’ for search on a site – on all sites that I might run – and have that as the trigger. Depending on the browser or other assistive technology (AT) used, that might involve the user hitting CTRL+s to activate it, or maybe ALT+S, or perhaps some other configuration.

However, those very same commands might be the same as existing commands in the browser or AT. So which command takes precedence? There is another factor to this ‘key already assigned’ problem, and that’s one of localization. While you may think that your browser has a spare key that you could use, in many other languages this won’t be the case.

Another problem with the accesskey attribute is how do you let the user know about it? As with the longdesc attribute, the availability of the attribute is not really shouted from the rooftops (with the exception of the little-used iCab browser). Unless you make a point of informing the user of accesskeys being used on the site in a separate page, perhaps backed up by judicious use of the title attribute for the relevant links, the user may never know that you made them available.

So it seems that, while accesskeys are a great idea, they are largely hidden and also cause their own set of problems because of the potential overlap with other technology. But wait, maybe there’s life in the old dog just yet…

Let the User Decide

I have an idea, or rather I had an idea, about the way it might work. I actually used this approach a couple of years ago when I re-built accessify.com, providing a method for the user to set their own accesskeys. They are not forced upon anyone – it’s a total opt-in. But in all honesty, it was more of an experiment on a live site than anything else, and it’s not transferable to other sites unfortunately. The solution was built using PHP, but I always thought that it could be converted to a JavaScript solution. It just took me a couple of years to get around to it (and documenting it!).

The JavaScript Approach

In the JavaScript solution, the end result is exactly the same as the version used on Accessify, just the mechanics have changed. Rather than go into all the lines of code and explain everything line-by-line, I’ll summarise how it works in plain old English.

• There are a number of links on the example page (the links that we offer up as potential accesskey candidates) By default, they have no accesskey assigned, nor any title attribute:
  <h3>Navigation</h3>
<ul>
<li><a href="/" id="homelink">Home</a></li>
<li><a href="/search/" id="searchlink">Search</a></li>
…
</ul>
• There’s a warning on the page that says "you need Javascript to set your own accesskeys" – this is set to display on screen by default
  Finally, there’s a form that allows you to set the characters you want for the related sections on the page. This is hidden by default (using CSS)
  <p id="warning">Cookies are used to save these preferences, but you have either disabled JavaScript, set your browser not to accept cookies, or your browser supports neither. The site will work perfectly well without these preference options.</p>

  #warning
  {
  color:red;
  }

  function hideWarning()
{
var elwarn = document.getElementById(’warning’);
elwarn.style.display="none";
var elsetter = document.getElementById(’setter’);
elsetter.style.display="block";
}

If the user gets to the page with JavaScript disabled (or with the scripts not properly downloaded – perhaps blocked by a firewall), they will essentially get the default page that says "You can’t do this, you need JavaScript to set this stuff" and no facility to set it is displayed.

If JavaScript is available, though, it’s a slightly different scenario:

• The warning message that’s set to display by default gets suppressed (after page load, by having the CSS display property set to none)
• The form that accepts the user’s desired accesskey assignments is made visible (again, using JavaScript to change the CSS display property)

Now that we know that the user has JavaScript enabled and was able to get all the source files correctly, they can then go ahead and save their accesskeys.

Saving the Accesskey Choices

The process of saving the choices made by the user are fairly standard JavaScript/cookie methods – one long string is saved that has a number of property:value pairings (one for each link or button that can potentially be assigned an acccesskey). For example:

<form action="set-access-keys.htm" method="post" name="frmAccessKeys">
<fieldset>

<legend>Set Accesskeys for … </legend>

<div>
<label for="txthomelink">Home link</label>
<input value="" name="txthomelink" id="txthomelink" type="text" size="2" maxlength="1" />
</div>

<div>
<label for="txtsearchlink">Search Box</label>
<input value="" name="txtsearchlink" id="txtsearchlink" type="text" size="2" maxlength="1" />
</div>

...

</fieldset>
</form>

strKeyVals += "homelink=" + document.forms['frmAccessKeys'].txthomelink.value + "@@";
strKeyVals += "searchlink=" + document.forms[’frmAccessKeys’].txtsearchlink.value + "@@";

The @@ is simply there to make it easier to identify where to split the string when retrieving the data from the saved cookie later.

We also provide an option for the user to get a hint of the accesskey for each link or button with the use of a simple checkbox:

<div>
<input type="checkbox" name="chkshowhint" id="chkshowhint" />
<label for="chkshowhint" class="check">Show accesskeys in links and buttons (displayed after link in brackets)</label>
</div>

strKeyVals += "showhint=" + document.forms['frmAccessKeys'].chkshowhint.checked + "@@";

The process for capturing the information is initiated by an onclick event attached to the submit button, which passes it through to an addevent function:

addEvent(document.getElementById('cmdSubmit'), 'click', accessKeySetter, false);

(The function accessKeySetter is where the JavaScript code snippets above appear)

Remembering the Choices and Updating the Display on Page

Assuming the user has entered some values and also chosen to have the accesskeys showing in the links or buttons, once the page has reloaded following the submit, this is where the DOM really takes over.

Remember, we start with a page that has a number of links that have neither default accesskey or title attributes present, then we use JavaScript to present the option to manipulate/store accesskeys (if JavaScript is not available for whatever reason, the default page display is to warn them that they can’t do this and need to get their JS in order, or words to that effect). So, already, we’re using JavaScript in a nice progressive enhancing kinda way. The final stage is to use JavaScript and the DOM to update the page display depending on what’s found in the cookie. The steps to do this are as follows:

1. Firstly, look for the presence of the accesskey cookie
2. Assuming it’s found, take the content of that string and split it up into the individual links/buttons that it repesents (separated by the @@ characters that were shown earlier
   //get the cookie
var strAccessKeys = readCookie("accesskeys");
//split up the values
var arrAccesskeys = strAccessKeys.split("@@");
3. Then, each value found is split between the id of the link or button it represents and the value stored against each (the accesskey the user wants to assign)
   for (i=0;i<(arrAccesskeys.length-2);i++)
{
var thisAccessKey = arrAccesskeys[i].split("=");
var accessKayEl = thisAccessKey[0];
var accessKayAttrib = thisAccessKey[1];
…
4. Then it’s a case of running through the array of ids in the cookie that have values assigned and matching them up to the ids of elements on the page and then:
   1. adding the accesskey value to the element
      if (accessKayAttrib.length>0)
{
//set the attribute
document.getElementById(accessKayEl).setAttribute(’accesskey’, accessKayAttrib);
   2. and updating the content of link or value of the button text to display the accesskey (if the user has requested this), and adding a title attribute too:
      if (document.getElementById(accessKayEl).tagName=="A")
{
var strLinkPhrase = document.getElementById(accessKayEl).childNodes[0].nodeValue;
}
if ((document.getElementById(accessKayEl).tagName=="INPUT") && ((document.getElementById(accessKayEl).type=="submit")||
(document.getElementById(accessKayEl).type=="button")))
{
var strLinkPhrase = document.getElementById(accessKayEl).value;
}
document.getElementById(accessKayEl).setAttribute(’title’, strLinkPhrase + ‘, access key = ‘ + accessKayAttrib);
//display the accesskey hint, if that’s been chosen
if (blnShowHints)
{
strLinkPhrase = strLinkPhrase + " [" + accessKayAttrib + "]";
if (document.getElementById(accessKayEl).tagName=="A")
{
document.getElementById(accessKayEl).childNodes[0].nodeValue = strLinkPhrase;
}
if (document.getElementById(accessKayEl).tagName=="INPUT")
{
document.getElementById(accessKayEl).value = strLinkPhrase;
}
5. One final step, while traversing through the page, is to add the value chosen in to the form fields too, so they’re remembered, likewise the state of the checkbox.

The final result is a completely unobtrusive method for setting and remembering accesskey attributes which doesn’t enforce itself on the user – it’s completely opt-in.

[download the zipped version with all files required here.]

Possible Improvements

The biggest improvement that could happen with this script is to have it centralized somewhere, or some variant thereof, so that it could be re-used in some way. My thinking was that if this mechanism were possible to re-use across sites, then a user could potentially save some accesskey choices that would be usable for a number of different sites, but then there’s one fatal flaw in this idea – a cookie can only be read back by the same domain, thus if I were to set my choice for site whateverxyz.com, when I try to do the same for abcwhatever.com, it won’t be able to read back the same data. So I’m definitely open to suggestions as to how that might be made possible at some time in the future.

Outside Reading

This isn’t the first time someone has come up with a method for dealing with this, so here are a few other takes on how to solve the issue:

• Gez Lemon’s PHP-based accesskey setter
• Jacques Distler’s JavaScript approach

And if you want to to learn more about the background to the problem, here are a couple of recommended reads:

• Dave Shea: Keyboards and Chaos
• Derek Featherstone: Accesskey Conflicts

Forums and mailing lists are full of requests about Ajax, DOM Scripting and how to use this or that library or effect. There is also an extraordinary amount of scripts, libraries and effects being developed and showcased, and the blogs and news sites specialising in scripting hardly have time to look at the demos properly before they are on Digg.com or del.icio.us and making the rounds from these sites.
Times to celebrate for those who have hung on to their skills when the DHTML craze subsided in 2001 and JavaScript became persona non grata on your CV as a main skill.

Code that shows love to the visitor

On the whole scripts released today are a lot cleaner than those of the DHTML era – most probably because of the lack of code forking necessary with browsers these days. I am also very pleased to see that the idea of Unobtrusive JavaScript has become quite common and is regarded as a goal for aspiring developers rather than a necessary evil.
One exception might be Ajax frameworks and code generators that spit out JavaScript – which lead some ‘JavaScript evangelists’ such as Jeremy Keith to coin new terms such as HIJAX – which in essence is Ajax with progressive enhancement instead of JavaScript dependencies.
All in all, it is a great time to be in the know and enthusiastic about JavaScript. Some behaviour patterns from the old days repeat themselves though. You see a lot of code that is ‘experimental’ and ‘not fit for a live system’ getting hyped and re-used without caring about the initial concerns of the developer – sometimes even blatantly omitting these warnings when reporting about the code experiment.
You also see the old chestnut of recoding visual effects that were pretty useless in older technologies with newer technologies. An image reflected in an animated lake is a two second ‘wow’ effect and from there on a nuisance, no matter if you achieve it using Java, Flash or JavaScript.

10 PRINT ‘HELLO MAINTAINER: GOTO HELL

So we’re making life a lot easier for the visitors coming to web sites by creating scripts that are not in their way, but only help then when and if their agents support it. On the other hand, as a maintainer a lot of code flung in my direction is a pain to sift through, debug and generally find the spots where I can customise or change settings. Call me Captain Obvious to the Rescue, but isn’t it time we – knowing very well that implementing and maintaining code is annoying and causes a lot of frustration – cut future maintainers of our code the same slack?
Unobtrusive JavaScript that is easy to maintain sounds like a winner that could help us spend less time with frustrating trial and error and give us some spare time we could spend on innovating or documenting what we do.

Maintainable JavaScript? But how?

So here’s an eight step plan to show you what can be done to make your scripts easier for the maintainer. Yes, some of the steps may be very obvious to the jet-set, well travelled and seasoned JavaScript developer, but it can’t hurt to remind ourselves of their virtues.
Furthermore, I dare any one of you to look at some older code and test it against them – I am not ashamed to admit that at times I looked at code and uttered expletives about the person responsible, only to find out later on that it was in fact me a few months back. Scope and feature changes together with looming deadlines and budget cuts breed bad code; it is a fact of software development.

Step 1: Keep Your Syntax and Structure Clean and Logical

This means you indent your code, keep a line-length limit of say 80 characters and keep functions reasonably small. A rule of thumb is to keep each task in its own function rather than having one large function to do everything. As an additional benefit, this means that you can re-use these functions when you extend the script at a later stage. Don’t go the ‘IDE generated – Java’ route though and create many many many one line functions – this can become more confusing than using one massive function.

There is a holy war going on about tabs vs. spaces for indenting and curly braces on an own line or on the same of the opening command. I don’t care, as long as you keep it consistent, and also consistent with the other code, were you to extend an already existing project. As an example let’s take a function that generates ‘close’ and ‘print’ links for a popup – links that only make sense when JavaScript is available and should therefore be generated with JavaScript.

function toolLinks(){
  var tools = document.createElement('ul');
  var item = document.createElement('li');
  var itemlink = document.createElement('a');
  itemlink.setAttribute('href', '#');
  itemlink.appendChild(document.createTextNode('close'));
  itemlink.onclick = function(){window.close();}
  item.appendChild(itemlink);
  tools.appendChild(item);
  var item2 = document.createElement('li');
  var itemlink2 = document.createElement('a');
  itemlink2.setAttribute('href', '#');
  itemlink2.appendChild(document.createTextNode('print'));
  itemlink2.onclick = function(){window.print();}
  item2.appendChild(itemlink2);
  tools.appendChild(item2);
  document.body.appendChild(tools);
}

You can make this a lot more readable by indenting and separating each task out into own functions:

function toolLinks(){
  var tools = document.createElement('ul');
  var item = document.createElement('li');
  var itemlink = createLink('#', 'close', closeWindow);
  item.appendChild(itemlink);
  tools.appendChild(item);
  var item2 = document.createElement('li');
  var itemlink2 = createLink('#', 'print', printWindow);
  item2.appendChild(itemlink2);
  tools.appendChild(item2);
  document.body.appendChild(tools);
}
function printWindow(){
  window.print();
}
function closeWindow() {
  window.close();
}
function createLink(url,text,func){
  var temp = document.createElement('a');
  temp.setAttribute('href', url);
  temp.appendChild(document.createTextNode(text));
  temp.onclick = func;
  return temp;
}

Step 2: Use clever variable and function names

This is basic good coding practice, although you see it done badly a lot. Once again, quick fixes and looming deadlines are to blame. In essence it means that you use variable and function names that tell the maintainer immediately what this chunk of data is. You can use underscores or camelCase to concatenate different words – personally I prefer camelCase, as that is consistent with the methods JavaScript provides you with (eg. getElementsByTagName()). It might even be a good idea to use generic names for variables that are just computed inside a function or loop and don’t need maintaining like the ‘temp’ variable inside the createLink() function in the earlier example.
Take a leaf out of the book for good CSS design – make sure to keep your function and variable names generic and describe a task rather than a visual result. A createFatRedBoxForLeftNavigation() might become a createTopBarForSectionNavigation() in another code iteration but doesn’t have to if you had called it createSecondaryMenu() from the start.
It is also a very good to stick to English variable and function names – you never know if you have to share the code world wide or send it to another country for maintenance.

Step 3: Comment your code

Commenting can save a lot of time and grief. However, you can also overdo it. In tutorials and books you’ll find code that has a comment on each line explaining what is going on. This is because of the intended audience, and makes a lot of sense inside a tutorial but is rather superfluous in live code. Comments are needed when:

• There is a section of code that needs maintenance, eg. ‘Change ID names here’
• There is a section that might become outdated as it fixes a problem for user agents that are hip right now
• There is a reason why you programmed something in one way while there might be better ones and you want to explain that
• The code section is dependent on another script or gets parameters whose origin may not be obvious at first.

Step 4: Keep your scripts self-contained

This step ties in with unobtrusive JavaScript, so let’s not dwell on it. In essence it means that a maintainer could add your script to any page and don’t risk overwriting other code by doing so. You can keep scripts self-contained by namespacing them or wrapping them in an object via the object literal. You ensure you don’t hijack variables by keeping them in scope of your functions via the var keyword and you don’t hijack events by using addEvent() or its derivates.

Step 5: Keep maintained variables separate and test dependencies

This step is simple: Keep variables that need to be maintained at the beginning of the function and pre-set them with dummy values if necessary. That way the maintainer always knows where to change settings and doesn’t have to scan through the whole script and you won’t run into ‘not defined’ errors.

function toolLinks(){
  var tools, closeWinItem, closeWinLink, printWinItem, printWinLink;

  // link labels, please edit
  var printLinkLabel = ‘print’;
   var closeLinkLabel = ‘close’;#

   tools = document.createElement(’ul’);
   closeWinItem = document.createElement(’li’);
   closeWinLink = createLink(’#', closeLinkLabel, closeWindow);
   closeWinItem.appendChild(closeWinLink);
   tools.appendChild(closeWinItem);
   printWinItem = document.createElement(’li’);
   printWinLink = createLink(’#', printLinkLabel, printWindow);
   printWinItem.appendChild(printWinLink);
   tools.appendChild(printWinItem);
   document.body.appendChild(tools);
}

Testing for dependencies is a trait of Unobtrusive JavaScript – you test for the existence of HTML elements before you access them via DOM methods. Taking this idea further, make sure you test if there are any dependencies in your scripts and alert the maintainer if there is a problem. This is a big issue with a lot of JavaScript libraries. Scripts that rely on the Yahoo! User Interface library will throw an error when it is not available, this could be prevented by testing for it and showing an alert instead. Jquery is even worse as it seems to suppress any errors, which makes debugging a gamble.

Step 6: Separate and communicate the visuals

When it comes to Unobtrusive JavaScript there is no way around applying and removing CSS class names dynamically to and from elements. There is hardly any situation where you need to access the style collection of an object directly, lest you want to make it harder for the maintainer. Exceptions are drag and drop interfaces, but even then all you change via JavaScript is the x and y coordinates of the element. The rest of the look and feel should stay where it was meant to be: in the CSS file. This ensures that CSS designers don’t mess around with your code and you don’t need to know all the ins and outs of CSS bugs in browsers – and there are a lot more than we’d like there to be.

The most basic way to communicate the names of these classes is keeping them in variables and commenting them at the beginning of your script:

demo = {
  // CSS classes
  hideClass : 'hide', // hide elements
  currentClass : 'current', // current navigation element
  hoverClass : 'over', // hover state
  [... rest of the code ...]
}

If you want to go even further in your separation then you could create an own include file called for example cssClassNames.js and use a JSON notation in this one.

css={
  'hide' : 'hide', // hide elements
  'current' : 'current', // current navigation element
  'hover' : 'over', // hover state [... and so on for a lot of them...]
}

In your script you can reach the different class names with css.hide or css.current respectively. You could even use more natural labels like ‘hide elements’ and reach them via css ‘hide elements’, which’ll make it easier for the maintainer but harder for you as a coder.
One really easy trick is a class name to communicate with the CSS designer that you apply to the body of the document. This will allow the CSS designer to distinguish between styles that are applied to the non-JavaScript version and to the scripting enhanced version respectively.

  if(!document.getElementById || !document.createElement){return;}
  cssjs('add',document.body,'jsenabled');

CSS:

/* No JavaScript */
#nav li{
  [...settings...]
}
/* JavaScript */
body.jsenabled #nav li{
  [...settings...]
}

This is also very handy to hide a lot of elements via CSS instead of looping through all of them. In extreme cases, where you have a lot of styles and totally different settings for the scripting enabled version, you could even apply a totally different style sheet via JavaScript (either use document.write() or create a new LINK element).

Step 7: Separate textual content and code

The same tricks apply to textual content, you can either separate labels out as variables or – even better – into an own document called textContent.js in JSON format. An example that worked exceptionally well is this one:

var locales = {
  'en': {
    'homePageAnswerLink':'Answer a question now',
    'homePageQuestionLink':'Ask a question now',
    'contactHoverMessage':'Click to send this info as a message',
    'loadingMessage' : 'Loading your data...',
    'noQAmessage' : 'You have no Questions or Answers yet',
    'questionsDefault': 'You have not asked any questions yet',
    'answersDefault': 'You have not answered any questions yet.',
    'questionHeading' : 'My Questions',
    'answersHeading' : 'My Answers',
    'seeAllAnswers' : 'See all your Answers',
    'seeAllQuestions' : 'See all your Questions',
    'refresh': 'refresh'
  },
  'fr': {
  },
  'de': {
  }
};

It allowed the editorial staff to translate and maintain the labels in all the different languages without having to touch the code. I even managed to separate the IDs out for them to sort out with the HTML developers. All the JavaScript had to do was to read out the application’s locale setting and loop through the elements available:

  function doI18N(){
   var lang = yhost.PluginLocale;
   for(i in locales[lang]){
     if(document.getElementById(i)){
       document.getElementById(i).innerHTML = locales[lang][i];
     }
   }
  }

Step 8: Document your code

Last but oh so not at all least: write documentation about your script / library / effect. Good documentation takes the audience into consideration, which means that for some libraries it is good to keep it a classic API documentation with all possible properties and parameters of methods explained in ‘techspeak’, but in general I have had much more success with ‘Explain by example, then list all possibilities’ documentation. Good documentation saves the most time and money, actually a shame we hardly get the time to create it. Other programming languages tend to automatically generate documentation from code comments (PHP for example), however that does not necessarily make for better docs, just less work.

What about high traffic sites?

If you are working in a high traffic environment, or one that only allows for small code packets (mobile market) then you might frown upon the ideas explained here as they can bloat the code. The way around that problem is to maintain a code base and a live code repository, which is generated from the original code after stripping out the comments, replacing long variable names with shorter ones and performing other packing and minifying tasks. Tools to do that for you are Douglas Crockford’s jsmin and Dean Edward’s packer.

Summary

I hope that the above examples have given you an idea how to make your code more maintainable or reminded you of some ideas that lay dormant in the back of your head. If you have other ideas, or you strongly disagree with some of the practices (and you have good reason to), please drop a comment here, as with everything that is ‘best practice’, it is a work in progress and collaboration can only make it better.

You can find out more about this subject, plus get a podcast from a recent talk Christian gave on the subject at MuffinResearch

Like this article? Digg it!

I will admit straight up front – I don’t like writing long articles. There, I said it. I passed College English convincing my professors that I should be rewarded more for my creativity rather than the length of my prose. I’m the kind of guy that spent more time writing short stories than huge complicated essays. Eventually I went on to write for the school Newspaper, a.k.a. The State Hornet which gave me the freedom to keep my stories to a minimum of 500 words, or just a couple of paragraphs.

Today that legacy still stands and has transferred into my love for making widgets. No, not those widgets. I’m talking about those little goofy things you do on a Sunday afternoon from when you had that idea on Saturday night which left you frustrated on Friday after work. Yes, you’re not the only one with crazy ideas.

I’m talking about sliders, yoyos, tooltips, shakers, better looking code, faster queries, and random jubilees of DOM hoopla. It’s those small things that some of us live to create and find it rewarding even if it’s one line of code. And although these little works of ‘code art’ don’t always work on all browsers while having labels of ‘experimental’ written all over them, don’t feel ashamed of them. They come from you and you should embrace your work. You are, a Web Professional.

Segue

In this article I’m going to share some of my short stories, poems, and random sonnets of affectionate escapades I’ve had with JavaScript during the last few months, which I’ve built using the Yahoo UI utilities. Some are rather embarrassing, others useful, and yet others I feel are just downright sexy. If you see something you like, feel free to take it, tweak it, and make it your own. This is what some poetry afficionados of secret societies would call a poetry slam. So for want of a better phrase, this can be a <code slam>.

Why YUI?

The YUI utilities give me the expressive freedom to do what I want. And although comparing libraries is generally silly, the best illustration I can give is the Script.aculo.us Effects library vs the YUI Animation utility. Script.aculo.us offers you some pretty sweet looking effects like Fade, SlideUp, BlindDown, Shake, etc.. You can even add Parallel effects (way to go Thomas). On the flip side, the animation util doesn’t aim to give you these canned effects to package up with your web app, but rather it lets you make up your own effects by giving you the finely crafted tools (like a utility knife) to make what you want.

Another reason I like YUI is that it isn’t out to change the language itself, but rather it solves many of the cross-browser incompatibilities that we run into on a daily basis. Beyond that, it does quite a few other sexy things that I think you might like.

Code

Below is a compilation of fifteen things that I have either developed over the last few months, or have been inspired by a friend (who will receive full credit on the original idea). Enjoy!

DOM + The getElementBy’s

These days, when you’re working with the DOM, it’s all about getting the elements you want, with conditions applied. Thankfully the Dom util doesn’t go all out and try to do all the guesswork for you resulting in a bulky library with large sets of unwanted code. Instead it gives you a handy method for you to work with called getElementsBy.

YAHOO.util.Dom.getElementsByClassName

Let’s kick off our slam with the classic getElementsByClassName (which is actually in the utility by default). I like to think of it as our token example by the YUI folks showing off how getElementsBy can work natively within your functions. See Demo

getElementsByAttribute

This method is not in the Dom util by default, but demonstrates exactly how we can add it in if we wanted. See Demo

getElementsByExternal & getElementsByInternal

This function allows you to get outbound or inbound links returned to you as an HTMLElement collection. At that point it’s up to you what you want to do with your array of elements (eg. bind event listeners or run a method against them). Currently this is untested on IP address URIs, but I presume that won’t be an issue for most situations. See Demo

Random Effects of Love

Because everyone enjoys California cheese.

Pagination Slide Pattern

Paginating through large data sets and keeping someone’s attention can be a tedious task. Sliding results within a container is one way to keep a person entertained. See Demo

BlindUp & BlindDown (Yoyo)

These are classic effects that almost any animation lover would want in their toolkit. For the sake of making them a macro, here they are. See Demo

BlindOut

A demonstration of animating a box from any given corner. See Demo

Faux Mutation Events with CustomEvent

Most folk don’t know what mutation events are, mainly because they don’t work in IE6. These are events just like regular DOM events like ‘click’,’mouseover’, or ‘mouseout’ that you can listen for. Just to name a few there are ‘DOMNodeInserted’, ‘DOMNodeRemoved’, and ‘DOMNodeInsertedIntoDocument’. For a better reference you can see Wikipedia’s entry on DOM events and search for the Mutation events within the event table. Anyway, here’s how you can simulate some of those events using the CustomEvent class.

DOMNodeInserted

Create some elements, listen for the event, then make them draggable. See Demo

DOMNodeRemoved

Create some elements, then remove them to update an information box. See Demo

More with CustomEvents

There really is an endless amount of Custom Events that you can create. So we’ll keep this to a minimum. Here are three places that the CustomEvent can come in useful:

Extending the Drag and Drop

Currently, the Drag and Drop utility is packaged with quite a few custom events you can override such as startDrag, onDrag, or endDrag. However the problem is that you can’t subscribe to any of these, only override. Here is one trick you can use to make them subscribable. See Demo

onMenuOpen & onMenuCollapse

This is a combination of animation and custom events where we show menu items sliding into view and firing off subscribable events. See Demo

General Recipes

None of these have any rhyme or reason. But I tend to think that in some place or another, they can solve a problem.

Sweet Titles (YUI Style)

Here we have an OO version of Sweet Titles that allows us to declare multiple instances of tooltips on separate elements with some optional configuration parameters you can pass into the constructor object to get that exact look and feel you want out of a tooltip. See Demo

Faux Lightbox Effect

For the sake of doing something that’s already been done, here’s a faux light box effect that can be obtained using some core tools. See Demo

Photo Batching

YAHOO.util.Dom.batch alone may convince you of how cool the DOM Collection utility can be. This shows you how you can batch an HTML Element collection to a method. (Thanks Matt Sweeny) See Demo

Unobtrusive new windows links

For some reason or another, this always seems to create some noise regardless of its controversial nature. Here is a very simple way to do it with Event, DOM, and some of the tools we’ve defined in this very article. See Demo

Layout Switching with Grids

This is a combination of using the CSS Grids toolkit and swapping out the templates using JavaScript animation See Demo

That’s All Folks

At least for this article’s sake. There are plenty of other things you can making using the Yahoo! UI utilities and this has only scratched the surface. I hope it was as fun for you as it was for me.

Like this article? Digg it!

With our so-called "Web 2.0" applications and their rich content and interaction, we expect our applications to increasingly make use of CSS and JavaScript. To make sure these applications are nice and snappy to use, we need to optimize the size and nature of content required to render the page, making sure we’re delivering the optimum experience. In practice, this means a combination of making our content as small and fast to download as possible, while avoiding unnecessarily refetching unmodified resources.

This is complicated a little by the nature of CSS and JavaScript resources. In contrast to image assets, CSS and JavaScript source code is very likely to change many times as time goes by. When these resources change, we need our clients to download them all over again, invalidating the version in their local cache (and any versions stored in other caches along the way). In this article, we’ll look at ways we can make the whole experience as fast as possible for our users – the initial page load, subsequent page loads and ongoing resource loading as the application evolves and content changes.

I believe strongly in making things as simple as possible for developers, so we’ll also be looking at ways we can set up our systems to automatically take care of these optimization issues for us. With a little up front work, we can get the best of both worlds – an environment that makes development easy with great end-user performance – all without changing the way we work.

Monolith

The old school of thought was that we could achieve optimal performance by combining multiple CSS and JavaScript files into fewer, larger blocks. Rather than having ten 5k JavaScript files, we combine them into a single 50k file. While the total size of the code is still the same, we avoid having the overhead associated with multiple HTTP requests. Each request has a setup and teardown phase on both the client and server, incurs request and response header size overhead, and resource overhead on the server side in the form of more processes or threads (and perhaps more CPU time for on-the-fly gzipped content).

The parellization aspect is also important. By default, both Internet Explorer and Mozilla/Firefox will only download two resources from a single domain at once when using persistent connections (as suggested in the HTTP 1.1 spec, section 8.1.4). This means that while we’re waiting to download those JavaScript files, 2 at a time, we’re not loading image assets – the page our users see during the loading phase will be missing its images.

However, there are a couple of downsides to this approach. By bundling all of our resources together, we force the user to download everything up front. By chunking content into multiple files we can spread out the cost of loading across several pages, amortizing the speed hit across a session (or avoiding some of the cost completely, depending on the path the user chooses). If we make the first page slow to speed up subsequent pages, we might find that we have more users who never wait around to request a second page.

The big downside to the single file approach has not often, historically, been considered. In an environment where we will have to often change our resources, any changes to a single-file system will require the client to re-download a copy of the entire CSS or JavaScript working set. If our application has a single monolithic 100k JavaScript source file, any tiny change to our code will force all clients to suck down the 100k all over again.

A splintered approach

The alternative approach lies somewhere in the middle – we split our CSS and JavaScript resources into multiple sub-files, while at the same time keeping that number functionally low. This compromise comes at a cost – we need to be able to develop applications with our code split out into logical chunks to increase development efficiency, while delivering merged files for performance. With a few additions to our build system (the set of tools which turn your development code into production code, ready for deployment), this needn’t be a compromise we have to make.

For an application environment with distinct development and production environments, you can use a few simple techniques to keep your code manageable. In your development environment, code can be split into many logical components to make separation clear. In Smarty (A PHP templating language) we can create a simple function to manage the loading of our JavaScript:


SMARTY:
{insert_js files="foo.js,bar.js,baz.js"}

OUTPUT:
<script type="text/javascript" src="/javascript/foo.js"></script>
<script type="text/javascript" src="/javascript/bar.js"></script>
<script type="text/javascript" src="/javascript/baz.js"></script>


So far, so easy. But then we instruct our build process to merge certain files together into single resources. In our example, imagine we merged foo.js and bar.js into foobar.js, since they are nearly always loaded together. We can then record this fact in our application configuration and modify our template function to use this information.


SMARTY:
{insert_js files="foo.js,bar.js,baz.js"}

OUTPUT:
<script type="text/javascript" src="/javascript/foobar.js"></script>
<script type="text/javascript" src="/javascript/baz.js"></script>


The source code in our templates doesn’t need to change between development and production, but allows us to keep files separated while developing and merged in production. For bonus points, we can write our merging process in PHP and use the same configuration block to perform the merge process, allowing us to keep a single configuration file and avoid having to keep anything in sync. For super-bonus points, we could analyze the occurrence of scripts and style sheets together on pages we serve, to determine which files would be best to merge (files that nearly always appear together are good candidates for merging).

For CSS, a useful model to start from is that of a master and subsection relationship. A single master style sheet controls style across your entire application, while multiple sub-sheets control various distinct feature areas. In this way, most pages will load only two sheets, one of which is cached the first time any page is requested (the master sheet).

For small CSS and JavaScript resource sets, this approach may be slower for the first request than a single large resource, but if you keep the number of components low then you’ll probably find it’s actually faster, since the data size per page is much lower. The painful loading costs are spread out around different application areas, so the number of parallel loads is kept to a minimum while also keeping the resources-per-page size low.

Compression

When talk about asset compression, most people think immediately of mod_gzip. Beware, however – mod_gzip is actually evil, or at the least, a resource hogging nightmare. The idea behind it is simple – browsers request resources and send along a header to show what kind of content encodings they accept. It looks something like this:

Accept-Encoding: gzip,deflate

When a server encounters this header, it can then gzip or deflate (compress) the content it’s sending to the client, where the client will then decompress it. This burns CPU time on both the client and server, while reducing the amount of data transferred. All well and good. The way mod_gzip works, however, is to create a temporary file on disk in which to compress the source data, serve that file out, then delete it. For high volume systems, you very quickly become bound by disk IO. We can avoid this by using mod_deflate instead (Apache 2 only), which does all the compression in memory – sensible. For Apache 1 users, you can instead create a RAM disk and have mod_gzip writes its temporary files there – not quite as fast as pure in-memory compression, but not nearly as slow as writing to disk.

Even so, we can avoid the compression overhead completely by pre-compressing the relevant static resources and using mod_gzip to serve people the compressed version where appropriate. If we add this compression into our build process, it all happens transparently to us. The number of files that need compressing is typically quite low – we don’t compress images since we don’t gain much, if any, size benefit (since they’re already compressed) so we only need to compress our JavaScript and CSS (and any other uncompressed static content). Configuration options tell mod_gzip where to look for pre-compressed files.


mod_gzip_can_negotiate Yes
mod_gzip_static_suffix .gz
AddEncoding gzip .gz


Newer versions of mod_gzip (starting with version 1.3.26.1a) can pre-compress files for you automatically by adding a single extra configuration option. You’ll need to make sure that Apache has the correct permissions to create and overwrite the gzipped files for this to work.

mod_gzip_update_static Yes

However, it’s not that simple. Certain versions of Netscape 4 (specifically 4.06 to 4.08) identify themselves as being able to interpret gzipped content (they send a header saying they do), but they cannot correctly decompress it. Most other versions of Netscape 4 have issues with loading compressed JavaScript and CSS in different and exciting ways. We need to detect these agents on the server side and make sure they get served an uncompressed version. This is fairly easy to work around, but Internet Explorer (versions 4 through 6) has some more interesting issues. When loading gzipped JavaScript, Internet Explorer will sometimes incorrectly decompress the resource, or halt compression halfway through, presenting half a file to the client. If you rely on your JavaScript working, you need to avoid sending gzipped content to Internet Explorer. In the cases where Internet Explorer does receive gzipped JavaScript correctly, some older 5.x versions won’t cache the file, regardless of it’s e-tag headers.

Since gzip compression of content is so problematic, we can instead turn our attention to compressing content without changing its format. There are many JavaScript compression scripts available, most of which use a regular expression driven rule set to reduce the size of JavaScript source. There are several things which can be done to make the source smaller – removing comments, collapsing whitespace, shortening privately scoped variable names and removing optional syntax.

Unfortunately, most of these scripts either obtain a fairly low compression rate, or are destructive under certain circumstances (or both). Without understanding the full parse tree, it’s difficult for a compressor to distinguish between a comment and what looks like a comment inside a quoted string. Adding closures to the mix, it’s not easy to find which variables have a private lexical scope using regular expressions, so some variable name shortening techniques will break certain kinds of closure code.

One compressor does avoid this fate – the Dojo Compressor (there’s a ready-to-use version here) works by using Rhino (Mozilla’s JavaScript engine implemented in Java) to build a parse tree, which it then reduces before serializing it to a file. The Dojo Compressor can give pretty good savings for a low cost – a single compression at build time. By building this compression into our build process, it all happens transparently for us. We can add as much whitespace and as many comments as we like to our JavaScript in our development environment, without worrying about bloating our production code.

Compared to JavaScript, CSS is relatively simple to compress. Because of a general lack of quoted strings (typically paths and font names) we can mangle the whitespace using regular expressions. In the cases where we do have quoted strings, we can nearly always collapse a whitespace sequence into a single space (since we don’t tend to find multiple spaces or tabs in URL paths or font names). A simple Perl script should be all we need:


#!/usr/bin/perl

print $data;


We can then feed individual CSS files through the script to compress them like so:

perl compress.pl site.source.css > site.compress.css

With these simple plaintext optimizations we can reduce the amount of data sent over the wire by as much as 50% (depending upon your coding style – it might be much less), which can translate to a much faster experience for our users. But what we’d really like to do is avoid users having to even request files unless completely necessary – and that’s where an intimate knowledge of HTTP caching comes in handy.

Caching is your friend

When a user agent requests a resource from a server for the first time, it caches the response to avoid making the same request in the future. How long it stores this response for is influenced by two factors – the agent configuration and any cache control response headers from the server. All browsers have subtly different configuration options and behaviors, but most will cache a given resource for at least the length of a session, unless explicitly told otherwise.

It’s quite likely you already send out anti-caching headers for dynamic content pages to avoid the browser caching pages which constantly change. In PHP, you can achieve this with a pair of function calls:


<?php
header("Cache-Control: private");
header("Cache-Control: no-cache", false);
?>


Sounds too easy? It is – some agents will ignore this header under certain circumstances. To really convince a browser not to cache a document, you’ll need to be a little more forceful:


<?php
# 'Expires' in the past
header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");

# HTTP/1.0
header("Pragma: no-cache");
?>


This is fine for content we don’t want to be cached, but for content that doesn’t change with every request we want to encourage the browser to cache it aggressively. The “If-Modified-Since” request header allows us to get part of the way there. If a client sends an “If-Modified-Since” header with its request, Apache (or your web server of choice) can respond with status code 304 (”Not Modified”), telling the browser that its cached copy of the file is already up to date. With this mechanism, we can avoid sending the contents of a file to the browser, but we still incur the overhead of an HTTP request. Hmmm.

Similar to the if-modified-since mechanism are entity tags. Under Apache, each response for a static resource is given an “ETag” header containing a checksum generated from the file’s modified-time, size and inode number. A browser can then perform a HEAD request to check the e-tag for a resource without downloading it. E-tags suffer from the same problem as the if-modified-since mechanism – the client still needs to perform an HTTP request to determine the validity of the locally cached copy.

In addition, you need to be careful with if-modified-since and e-tags if you serve content from multiple servers. With two load-balanced web servers, a single resource could be requested from either server by a single agent – and could be requested from each at different times. This is great – it’s why we load balance. However, if the two servers generate different e-tags or modified dates for the same files, then browsers won’t be able to properly cache content. By default, e-tags are generated using the inode number of the file, which will vary from server to server. You can turn this off using a single Apache configuration option:

FileETag MTime Size

With this option, Apache will use only the modification time and file size to determine the e-tag. This, unfortunately, leads us to the other problem with e-tags, which can affect if-modified-since too (though not nearly as badly). Since the e-tag relies on the modified time of the file, we need those times to be in sync. If we’re pushing files to multiple web servers, there’s always a chance that the time at which the files are pushed are subtly different by a second or two. In this case, the e-tags generated by two servers will still be different. We could change the configuration to generate e-tags only from the file size, but this means that we’ll generate the same e-tag if we change a file’s contents without changing its size. Not ideal.

Caching is your best friend

The problem here is that we are approaching the issue from the wrong direction. These possible caching strategies all revolve around the client asking the server if its cached copy is fresh. If we could notify the client when we change a file, it would know that its own cached copy was fresh, until we told it otherwise. But the web doesn’t work that way – the client makes requests to the server.

But that’s not quite true – before fetching any JavaScript or CSS files, the client makes a request to the server for the page which will be loading those files via <script> or <link> tags. We can use the response from the server to notify the client of any changes in those resources. This is all a little cryptic, so let’s spell it out – if we change the filenames of JavaScript and CSS files when we change their contents, we can tell the client to cache every URL forever, since the content of any given URL will never change.

If we are sure that a given resource will never change, then we can send out some seriously aggressive caching headers. In PHP, we just need a couple of lines:


<?php
header("Expires: ".gmdate("D, d M Y H:i:s", time()+315360000)." GMT");
header("Cache-Control: max-age=315360000");
?>


Here we tell the browser that the content will expire in 10 years (there are 315,360,000 seconds in 10 years, more or less) and that it can keep it around for 10 years. Of course, we’re probably not serving our JavaScript and CSS via PHP – we’ll address that in a few moments.

Mistakes abound

Manually changing the filenames of resources when the contents are modified is a dangerous task. What happens if you rename the file, but not the templates pointing to it? What happens if you change some templates but not others? What happens if you change the templates but don’t rename the file? Most likely of all, what happens if you modify a resource but forget to rename it or change any references to it. In the best of these cases, users will not see the new content and be stuck with the old versions. In the worst case, no valid resource is found and your site stops working. This sounds like a dumb idea.

Luckily computers are really good at this sort of thing – dull repetitive tasks which need to be done exactly right, over and over again, when some kind of change occurs.

The first step in making this process as painless as possible is to realize that we don’t need to rename files at all. URLs we serve content from and where the content is located on disk don’t have to have anything to do with each other. Using Apache’s mod_rewrite we can create a simple rule to redirect certain URLs to certain files.


RewriteEngine on
RewriteRule ^/(.*\.)v[0-9.]+\.(css|js|gif|png|jpg)$ /$1$2 [L]


This rule matches any URL with one of the specified extensions which also contains a ‘version’ nugget. The rule then rewrites these URLs to a path without the version nugget. Some examples:


URL Path
/images/foo.v2.gif -> /images/foo.gif
/css/main.v1.27.css -> /css/main.css
/javascript/md5.v6.js -> /javascript/md5.js


With this rule in-place, we can change the URL (by changing the version number) without changing where the file lives on disk. Because the URL has changed, the browser treats it as a different resource. For bonus points, you can combine this with the script grouping function from earlier to produce a list of versioned <script> tags as needed.

At this point, you might ask why we don’t just add a query string to the end of the resource – /css/main.css?v=4. According the letter of the HTTP caching specification, user agents should never cache URLs with query strings. While Internet Explorer and Firefox ignore this, Opera and Safari don’t – to make sure all user agents can cache your resources, we need to keep query strings out of their URLs.

Now that we can change our URLs without moving the file, it would be nice to be able to have the URLs updated automatically. In a small production environment (or a development environment, for people with large production environments), we can do this really easily using a template function. This example is for Smarty, but applies equally well to other templating engines.


SMARTY:
<link href="{version src='/css/group.css'}" rel="stylesheet" type="text/css" />

OUTPUT:
<link href="/css/group.v1234567890.css" rel="stylesheet" type="text/css" />


For each linked resource, we determine the file’s location on disk, check its mtime (the date and time the file was last modified on disk) and insert that into the URL as the version number. This works great for low traffic sites (where stat operations are cheap) and for development environments, but it doesn’t scale well to high volume deployments – each call to stat requires a disk read.

The solution is fairly simple. In a large system we already have a version number for each resource, in the form of the source control revision number (you’re already using source control, right?). At the point when we go to build our site for deployment, we simply check the revision numbers of all of our resource files and write them to a static configuration file.


<?php
$GLOBALS['config']['resource_versions'] = array(

'/images/foo.gif' => '2.1',
'/css/main.css' => '1.27',
'/javascript/md5.js' => '6.1.4',
);
?>


We can then modify our templating function to use these version numbers when we’re operating in production.


<?php
function smarty_version($args){

echo preg_replace('!\.([a-z]+?)$!', ".v$version.\$1", $args['src']);
}
?>


In this way, we don’t need to rename any files, or even remember when we modify resources – the URL will be automatically changed everywhere whenever we push out a new revision – lovely. We’re almost where we want to be.

Bringing it all together

When we talked about sending very-long-period cache headers with our static resources earlier, we noted that since this content isn’t usually served through PHP, we can’t easily add the cache headers. We have a couple of obvious choices for dealing with this; inserting PHP into the process or letting Apache do the work.

Getting PHP to do our work for us is fairly simple. All we need to do is change the rewrite rule for the static files to be routed through a PHP script, then have the PHP script output headers before outputting the content of the requested resource.


Apache:
RewriteRule ^/(.*\.)v[0-9.]+\.(css|js|gif|png|jpg)$ /redir.php?path=$1$2 [L]

function go_404(){
header("HTTP/1.0 404 File not found");
exit;
}


While this works, it’s not a great solution. PHP demands more memory and execution time than if we did everything in Apache. In addition, we have to be careful to protect against exploits made possible by sending us doctored values for the path query parameter. To avoid all this headache, we can have Apache add the headers directly. The RewriteRule directive allows us to set environment variables when a rule is matched, while the Header directive lets us add headers only when a given environment variable is set. Combining these two directives, we can easily chain the rewrite rule together with the header settings.


RewriteEngine on
RewriteRule ^/(.*\.)v[0-9.]+\.(css|js|gif|png|jpg)$ /$1$2 [L,E=VERSIONED_FILE:1]

Header add "Expires" "Mon, 28 Jul 2014 23:30:00 GMT" env=VERSIONED_FILE
Header add "Cache-Control" "max-age=315360000" env=VERSIONED_FILE


Because of Apache’s order of execution, we need to add the RewriteRule line to the main configuration file (httpd.conf) and not a per-directory (.htaccess) configuration file, otherwise the Header lines get run first, before the environment variable gets set. The Header lines can either go in the main configuration file or in an .htaccess file – it makes no difference.

Skinning rabbits

By combining the above techniques, we can build a flexible development environment and a fast and performant production environment. Of course, this is far from the last word on speed. There are further techniques we could look at (separate serving of static content, multiple domain names for increased concurrency) and different ways of approaching the ones we’ve talked about (building an Apache filter to modify outgoing URLs in HTML source to add versioning information on the fly). Tell us about techniques and approaches that have worked well for you by leaving a comment.

Read more

Cal’s new book, Building Scalable Web Sites, contains more tips and tricks to help you develop and manage the next generation of web applications.

Like this article? Digg it!