Bottlenecks and Micro-Performance

Nov 25, 2015

After my last blog, I got a response regarding functional programming and performance. This is actually a common theme when people talk about functional programming versus imperative programming. Before we move into the actual performance discussion, I will openly admit, there are often times when functional programming is slower, performance-wise, than imperative programming. I have never claimed otherwise, nor will I begin doing so today.

Now, let’s move away from the very specific case of functional versus imperative programming and take a look at application performance in general. It is common to look for performance bottlenecks in applications. An application slows down and we want to dig in and uncover the performance issue. This particular situation arose at my work just a few weeks ago.

We had a form which, when big enough, slowed the entire experience to a crawl. This was bad news for us as our customers were quite unhappy with the way our application was behaving. I couldn’t blame them. The experience was miserable.

My colleague and I started digging into the offending code. I discovered a few methods which were running at O(n^2) time and he discovered a, seemingly innocuous, call to perform an external validation. When we moved our search to the validation code, it became obvious this was the problem. The entire form was being revalidated multiple times for every single element on the screen.

I fixed the O(n^2) algorithm, reducing it to an O(n) time execution, which made a visible difference, but the real win was decoupling the localized validation logic from the form logic. We estimated that for each validation call that was made, validation work was being done in the neighborhood of 60,000 times.

This demonstrates the difference between micro-performance and macro-performance. My algorithm enhancement was a macro-performance fix when looking at execution times of single lines of code, but when looking at the application as a whole, it was actually just a micro-performance tuning. The real win came when a true macro-performance fix was implemented and our total iteration count was reduced from 60,000 to about 600. That kind of performance gain can be measured in orders of magnitude and saved the experience of our customers.

Jeff Atwood talks about micro-performance benchmarking as something that only matters when a bad choice is made. If a piece of code is not optimally performant, but it is only executed once, does it matter? Jeff and I agree, it doesn’t.

Let’s take a look at two different blocks of code:

// Imperative single-loop behavior

function addEvensImperative (list) {
	var total = 0,
		listLength = list.length,
		index = 0;
	
	for(index; index < list.length; index++) {
		total += list[index] % 2 === 0 ? list[index] : 0;
	}
	
	return total;
}

// Functional, test and add abstracted, two-loop behavior

function isEven (value) {
	return value % 2 === 0;
}

function add (a, b) {
	return a + b;
}

function addEvensFunctional (list) {
	return list.filter(isEven).reduce(add, 0);
}

Clearly, addEvensImpertaive and addEvensFunctional produce the same output. If we look at the behavior with regard to constants, addEvensImperative loops over the array once so we can say it has a behavior characteristic function something like 1n + c_0. Meanwhile addEvensFunctional actually loops through the entire list twice in a pathological (worst) case. This means we can estimate the characteristic function to look something like 2n + c_1. This means for each time the functional behavior is called, the pathological behavior will be half as fast as the imperative call.

Let’s take a look at this using big-O notation. In big-O notation, the efficiency of the algorithm is reduced to the highest-power term in the approximate function. This means, all constants are discarded as well as coefficients. When we annotate our functions the imperative function performance is O(n) and the functional function performance is O(n) as well.

What this means is both of these functions have a linear growth behavior. The more values we have in our list, the longer each of these take to complete. Typically what we will see is total execution time measured in microseconds or a few milliseconds. Even large arrays of numbers can be iterated over very very quickly, so, even though the functional behavior is half as fast, the overall performance characteristic loss is negligible.

This is the pitfall of micro-optimization. Even though the perceived performance doesn’t change drastically for the user, it is fussed over because it’s easy to see and easy to measure. The problem is, there is a large blind spot that opens up when thinking about managing efficiency.

There is a phrase “what is measured is managed.” When applied to things which are properly and fully measured and everything is measureable, this is valuable. The problem is by measuring micro-optimizations, we can encounter the problem of “not seeing the forest for the trees.” This means we have accurate measurements of parts of a system, but we haven’t actually accounted for the system.

It’s common to talk about early optimization both in positive and negative light (depending on where you stand). I tend to work on the task at hand, and measure when the work is done. This means, if one function is not fully optimized, but the overall performance is as good or better than business requirements dictate, there is no need to optimize further. If, on the other hand, the system is under-performing and slow, I will look for new optimizations.

The important thing to note, however, is I prefer to optimize from the top-down. If a system is slow, it is highly unlikely that the culprit is a single function which is slower than we could be. More likely, we have a hidden optimization problem which is taking an O(n) function and looping, creating an O(n^) behavior, then looping again creating an O(n^3) behavior, and so on.

This hidden exponential growth characteristic is precisely what bit us in our slow-validating application. The reason my O(n^2) to O(n) optimization only gave us a small win is because we had another set of functions which were creating an algorithm performing at O(n^5) or so. Once we converted that behavior to an O(n) algorithm, the app sped up significantly.

In the end, the difference between a function that is performing at 2n versus n is significantly less critical than a system of functions performing at n^5 instead of 5n. Micro-performance benchmarks are important in systems like robotics, neural nets and so on. Common application behaviors like working with small, finite sets of data, typically benefit most from macro-optimizations, so fix those problems first, then measure again. In the end it is far more important to fix the bottleneck and leave the non-micro-optimized code readable and simple.

Testing Notes

Running a test on my local computer there was no measurable difference in performance between the two functions until the list of numbers got to 10,000 values, at which point the difference appears to be less than 1ms (0ms vs 1ms). At 10,000,000 the difference had grown significantly to about 850ms. Please don’t make your users process 10,000,000 numbers in their browser.

Here is my test script:

function range (start, end) {
    var output = [],
        current = start;

    for (current; current <= end; current++) {
        output.push(current);
    }

    return output;
}

module.exports = function test (endValue) {
    var rangeList = range(1, endValue),
        start,
        end;

    start = (new Date()).getTime();
    console.log(start);
    addEvensImperative(rangeList);
    end = (new Date()).getTime();
    console.log(end);
    console.log(end - start);

    start = (new Date()).getTime();
    console.log(start);
    addEvensFunctional(rangeList);
    end = (new Date()).getTime();
    console.log(end);
    console.log(end - start);
}

Output:

> runTest(100)
1448224288065
1448224288065
0
1448224288065
1448224288065
0
undefined
> runTest(1000)
1448224303600
1448224303600
0
1448224303600
1448224303600
0
undefined
> runTest(10000)
1448224309681
1448224309681
0
1448224309681
1448224309682
1
undefined
> runTest(100000)
1448224315113
1448224315114
1
1448224315114
1448224315121
7
undefined
> runTest(1000000)
1448224319977
1448224319980
3
1448224319980
1448224320063
83
undefined
> runTest(10000000)
1448224325198
1448224325222
24
1448224325223
1448224326094
871
undefined
```

This demonstrates that even a large number of arithmetic operations, my computer (an older Macbook Pro) can still push out better than 10,000,000 functional operations* a second. Arguably, in any software system processing fewer than 100,000 values at a time, the difference in performance when running each function once would be imperceptible to the end user.

* megaFLOPS so, blah blah, analogy, blah 10 megaFOPS (million functional operations per second)

    

Refactoring from Imperative to Functional

Nov 18, 2015

I was talking with a coworker one day pretty recently and I had a pretty sizeable revelation, people think I write code the way I do from the get-go. It’s actually really hard to look at the final output in a code block and understand how someone might have arrived at that solution. Even when the code is completely obvious, it might not have been obvious how the original problem was solved.

I tend to write in a very terse, functional style. I prefer declarative functions over long, tedious loops and conditionals. This style is not something you are born with and it is definitely not always the first thing to come out. I have been in situations where I KNOW I have the answer in my head, but I just can’t quite spit it out, so I take the long way around, then come back and do some cleanup.

My goal with this post is to expose what this kind of code surgery looks like on a problem which doesn’t seem to be reducible. Along the way I will introduce a variety of helper functions. These functions should not be cut and pasted. Instead they are intended for use in a library or shared module. Most (though not all) can actually be found in my JFP library, which saves the work of creating and refining these small abstractions for yourself. For the purpose of illuminating what is actually happening as I identify work, I will carry all added code forward. This is for your convenience as reference material. At the end, I will break apart the code and show which pieces are actually kept in the core source code.

Let’s start off with a function that takes an array and sums over all numbers which are either a multiple of 3 or 5. This can be done in about 15 lines including whitespace, so it’s a pretty tight little function. This is actually pretty representative of code I see in the wild.

    function sumMultiplesOf3And5 (list) {
        var total = 0,
            index = 0;
        
        if (typeof list === 'object' && Object.prototype.toString.call(list) === '[object Array]') {
            for (index; index < list.length; index++) {
                if (list[index] % 3 === 0 || list[index] % 5 === 0) {
                    total += list[index];
                }
            }
        }
        
        return total;
    }

This is pretty straightforward. We take in a list, prepare a variable for the total and one for the index, then we test that the list is actually an array. Finally, we iterate over each element, perform a modulus test and then add any matching values. It’s short, but there is a lot to digest on each line. One of the first things we can do to simplify the code is strip that top conditional out. Let’s create a helper function and sanitize our data instead of wrapping everything in a big conditional.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(list) === '[object Array]';
    }
    
    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            total = 0,
            index = 0;
        
        for (index; index < values.length; index++) {
            if (values[index] % 3 === 0 || values[index] % 5 === 0) {
                total += values[index];
            }
        }
        
        return total;
    }

That’s already a lot easier to digest. Keep in mind, since this is refactoring, some steps show an immediate impact while others will make the code worse before it gets better. In this case, we actually made a nice impact. Now we have a sanitary array we can perform safe operations on each time. In a standard imperative language, we would probably stop here and call it a day. This, however, is Javascript. We can do better!

Let’s pull the modulus check up into helper functions. It is not going to be immediately obvious why we would want to do this bit of work, since it seems to just confuse the matter, but it will make sense all in due time.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(list) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function mod (value, base) {
        return value % base;
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            total = 0,
            index = 0;
        
        for (index; index < values.length; index++) {
            if (equal(0, mod(values[index], 3)) || equal(0, mod(values[index], 5))) {
                total += values[index];
            }
        }
        
        return total;
    }

Now our checks are wrapped up in functions which will make our lives much, much easier. For now, it ends up being a little more typing. That’s alright, though. When our checks are made with functions, it exposes repeated code we might not have seen before. We are actually making a check against 0 twice. This is something we can abstract away with pure functions. Let’s add a simple compose function and create a new function we can use more freely.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(list) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function mod (value, base) {
        return value % base;
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            checkModulus = compose(equal.bind(null, 0), mod),
            total = 0,
            index = 0;
        
        for (index; index < values.length; index++) {
            if (checkModulus(values[index], 3) || checkModulus(values[index], 5)) {
                total += values[index];
            }
        }
        
        return total;
    }

Now we have a nice little function we can pass our value and modulus base into and get a boolean result. This gives us a more declarative way of performing our check. Here’s where we are going to first introduce two functions many people associate with functional programming: map and reduce. The mapping and reduction we are going to do will be so small it’s almost a direct manipulation of single values, but it gives us a way to find new pieces to pull up and out.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            checkModulus = compose(equal.bind(null, 0), mod),
            modValues = [3, 5],
            total = 0,
            index = 0;
        
        for (index; index < values.length; index++) {
            if (modValues.map(checkModulus.bind(null, values[index])).reduce(or, false)) {
                total += values[index];
            }
        }
        
        return total;
    }

Now we are actually performing a map across our modValues array and a reduce on the result. By performing this behavior inline, inside an if condition, it’s not very clear what we are doing or why. This kind of behavior is great for naming and pulling out of our core code. Really, this is a long, confusing way of asking a question: is the value a multiple of 3 or 5?

Let’s stop beating around the bush and just ask the question.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function isMultipleOf (baseValues, value) {
        var checkModulus = compose(equal.bind(null, 0), mod);
        return baseValues.map(checkModulus.bind(null, value)).reduce(or);
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            isValueAcceptable = isMultipleOf.bind(null, [3, 5]),
            total = 0,
            index = 0;
        
        for (index; index < values.length; index++) {
            if (isValueAcceptable(values[index])) {
                total += values[index];
            }
        }
        
        return total;
    }

By pulling our check up and out, we can ask if a value is acceptable. Now we are speaking in a more natural language. We want to sum multiples of 3 and 5 so we only want to work with acceptable values. Now we have a new problem, we have to wade through a loop and a conditional structure to actually see what we are looking for. Fortunately, Javascript has a built-in function for handling this kind of operation: filter. Filtering can be read as “I want to take this list of items and only keep the acceptable ones.” Let’s filter our values.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function isMultipleOf (baseValues, value) {
        var checkModulus = compose(equal.bind(null, 0), mod);
        return baseValues.map(checkModulus.bind(null, value)).reduce(or);
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            isValueAcceptable = isMultipleOf.bind(null, [3, 5]),
            multiples = values.filter(isValueAcceptable),
            total = 0,
            index = 0;
        
        for (index; index < multiples.length; index++) {
            total += multiples[index];
        }
        
        return total;
    }

Now we’re really cooking. we’ve eliminated the last conditional block around our data management. We have one ternary left for assigning the values variable, but we’ll deal with that later. For now, let’s get rid of our remaining loop. When we are performing an action like iterating over a list and summing the values, we can convert that to a reduction. Reduce is useful for lots of different purposes, but here we are going to use it at its most foundational. Let’s create an add function and reduce our list, adding all the values together.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function add (a, b) {
        return a + b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function isMultipleOf (baseValues, value) {
        var checkModulus = compose(equal.bind(null, 0), mod);
        return baseValues.map(checkModulus.bind(null, value)).reduce(or);
    }

    function sumMultiplesOf3And5 (list) {
        var values = isArray(list) ? values : [],
            isValueAcceptable = isMultipleOf.bind(null, [3, 5]),
            multiples = values.filter(isValueAcceptable);
                
        return multiples.reduce(add, 0);
    }

Reduce helped us eliminate a few bits of waste from our function. First, we removed the loop structure and just declared the objective of our function directly. Second, we eliminated the need for an accumulator variable and an index variable. This reduces the cognitive footprint of our function since we no longer need to keep track of the variables we are using in our function. Now that we have done all of this work, let’s do a little final cleanup by adding a sanitizeArray function and chaining our filter and reduce operations. This will give us our final code sample.

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function add (a, b) {
        return a + b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function sanitizeArray (value) {
        return isArray(value) ? value : [];
    }

    function isMultipleOf (baseValues, value) {
        var checkModulus = compose(equal.bind(null, 0), mod);
        return sanitizeArray(baseValues).map(checkModulus.bind(null, value)).reduce(or);
    }

    function sumMultiplesOf3And5 (list) {
        var isValueAcceptable = isMultipleOf.bind(null, [3, 5]);
        return sanitizeArray(list).filter(isValueAcceptable).reduce(add, 0);
    }

As you can see, this code is actually significantly longer than our original example. It might make functional programming actually seem like a longer path to walk than simply writing everything inline. This would definitely be true, except that much of this code can, and should, be pulled into a library and abstracted away from our core code. Let’s pull the general-use code out into its own sample block.

Library Code - Should exist in one place

    function isArray (value) {
        return typeof value === 'object' && Object.prototype.toString.call(value) === '[object Array]';
    }

    function equal (a, b) {
        return a === b;
    }

    function add (a, b) {
        return a + b;
    }

    function mod (value, base) {
        return value % base;
    }

    function or (a, b) {
        return Boolean(a) || Boolean(b);
    }

    function compose (fnA, fnB) {
        return function () {
            var args = Array.prototype.slice.call(arguments, 0);
            return fnA(fnB.apply(null, args));
        }
    }

    function sanitizeArray (value) {
        return isArray(value) ? value : [];
    }

    function isMultipleOf (baseValues, value) {
        var checkModulus = compose(equal.bind(null, 0), mod);
        return sanitizeArray(baseValues).map(checkModulus.bind(null, value)).reduce(or);
    }

All of these functions are built for reuse. We should not have to rewrite or abstract these out again. This also means, as we come across other code which uses these behaviors, we can immediately perform a drop-in replacement and get a big mess moved out of our way. Now, let’s do a side-by-side comparison of our original function and what our final product look like.

Original Function

    function sumMultiplesOf3And5 (list) {
        var total = 0,
            index = 0;
        
        if (typeof list === 'object' && Object.prototype.toString.call(list) === '[object Array]') {
            for (index; index < list.length; index++) {
                if (list[index] % 3 === 0 || list[index] % 5 === 0) {
                    total += list[index];
                }
            }
        }
        
        return total;
    }

Final Refactored Product

    function sumMultiplesOf3And5 (list) {
        var isValueAcceptable = isMultipleOf.bind(null, [3, 5]);
        return sanitizeArray(list).filter(isValueAcceptable).reduce(add, 0);
    }

    // Or if you prefer:

    function sumMultiplesOf3And5 (list) {
        return sanitizeArray(list).filter(isMultipleOf.bind(null, [3, 5])).reduce(add, 0);
    }

This is what we arrive at when we really strip out the repeated code. There’s almost nothing left!

By cleaning up our code, and preferring functional abstractions, we can strip away a tremendous amount of clutter and leave the essence of our intent. We can practically read the last function as a sentence: “To sum multiples of 3 and 5, sanitize provided list, filter for multiples of 3 and 5, then add all values together, starting with 0.” This is how we would solve the problem if we were to talk it through, so we should write our code the same way. This is how I refactor to functional code. How do you do it?

REST Verbs and HTTP

Nov 11, 2015

If your journey to programming was anything like mine, you started by looking at HTML forms to understand data transfer across HTTP. There is a lot of talk about REST and what it means on the web. I have heard people talk about the formatting of URLs and the return behavior of documents. I have even heard people discuss REST and its relation to micro-services or hypermedia API behaviors.

I am not going to get into the deep philosophical debate about what properties emerge from REST, nor will I discuss how to break services down or any other oddball topics which diverge from what someone interacting with REST should understand to begin with. Instead, let’s discuss what the REST acronym means, what REST verbs are and how we would interact with them through XMLHttpRequest (XHR).

What REST Is

REST stands for Representational State Transfer. What this means is, REST provides a standardized way to take state (user interaction created data) and transmit it across the Internet. I say “the Internet” because there is no requirement that we interact only with the standard HTTP port, 80. We can, theoretically, make an HTTP request to any port listening for and accepting HTTP requests conforming to the REST protocol.

Blah blah blah, what this really means is using the HTTP standard, if a service accepts requests passed with GET, POST, PUT or DELETE calls, it is probably RESTful. The primary reason I say it is “probably” RESTful is each of those four verbs come with a set of standards which define how the service should behave when a request is made.

Before we go further, let’s create an XHR script so we can look at how we can make browser-side requests to RESTful services. I am offering this script with no explanation, but you are welcome to use it as you see fit. I am also not promising this script is particularly robust, reliable or works at all, so use it for anything other than learning at your own risk.

Now that we have that lined up, let’s have a look at making HTTP requests. There are four common methods we can use for sending HTTP requests, GET, POST, PUT and DELETE. Although there are other request types we’re going to stick the common ones for now.

When working with REST verbs, there are certain rules we must follow. Different server technologies may or may not allow violation of the core REST rules, but when working in a RESTful system, it is best to know the conventional uses and deviate only as much as necessary to match your company or team conventions.

Before we dive into REST requests, I would recommend reviewing my post on URLs for web developers. This post assumes you are comfortable with the way URLs and their structure impact HTTP requests.

GET

GET requests are the most common and best known requests used on the web. GET requests are used all the time to view web sites, get static files and retrieve data from service endpoints. You used a GET request to view this page and all of the links on this page will initiate a new GET request to retrieve a different page.

GET requests can only use a request URL to capture data from a server. Effectively, a GET request will use all parts of the URL, though it is not allowed to use any other means to communicate information about the resource it requests. This means any body information sent to the server will likely be ignored.

Suppose we used our XHR script to make a request, let’s have a look at what the request would look like.

xhr.get('https://www.google.com/webhp', function (error, result) {
    if(!Boolean(error)) {
        console.log(result);
    }
}, { data: { q: 'chris stead programmer' } });

// Outputs a whole bunch of HTML

// final url -- https://www.google.com/webhp?q=chris%20stead%20programmer
```

We can see our request uses everything except the fragment, but it passes our data as a query string. This is the kind of behavior we should always expect with a GET request. HTML requests allow bodies to be included, however, if our data were to be included in the body, however, our GET URL would simply be https://www.google.com/webhp which would display the Google search page instead of the search results we wanted.

When a GET request is sent, the browser constructs an HTTP header which provides all of the information the server needs to resolve the request correctly.  Here's what a GET header looks like.

GET /webhp?q=chris%20stead%20programmer\n
host: www.google.com\n
\n
```

The new line character (\n) in these header examples is added to demonstrate the number of new lines that are actually part of the HTTP header. We can see the very first element in our request is the type of request, specifically GET. This is the key player in our REST methodologies.

The short version of this is, GET returns a document or resource. It always does. If something fails along the way, our browser will tell us with a different document.  Ultimately, GET always gets something, it should never write, update or delete.

DELETE

DELETE is the close cousin to the standard GET request. DELETE requests, much like GET requests, do not contain a body. In the standard REST behavior, any body in a GET or DELETE request should be ignored.  As it turns out, it is entirely likely you could make a DELETE request against your favorite website and the server would respond with the HTML content you get with a GET request.

What makes DELETE special is the way the server responds to the request. If a DELETE request is made against a service which accepts DELETE, it is likely you will receive an error.  This is due to the fact that DELETE is typically a privileged (authorization-restricted) command.

Let's construct a valid DELETE request with our XHR service with a header.  For the sake of this discussion, I am assuming the service we are calling is completely insecure and accepts credentials as clear text in the header.

```javascript
xhr.delete('http://www.insecure-service.com/", function (error, result) {
    if(!error) {
        console.log('Our record was deleted!');
    } else {
        console.log('Something bad happened.');
    }
}, {
    data: { id: 17 },
    headers: { credentials: 'chris@insecure-service.com;badpassword' }
});
```

This will send a request with extra header information.  I am not promising anything about the way credentials should be sent.  Every service will work differently.  That said, this new request is still informative. We can see credentials being provided and an id is given to tell our service what to delete.

Our callback also makes an assumption which can be considered common. DELETE requests often do not provide any data in their response body.  The response code may actually be 204 No Content. This tells us the request was successful, but there is no content to return, so the response body is empty.  Let's take a look at the request and response headers.

DELETE Request:

```default
DELETE /?id:17 HTTP/1.1
host: www.insecure-service.com
credentials: chris@insecure-service;badpassword

```

DELETE Response:

```default
HTTP/1.1 204 No Content
Date: Sat, 07 Nov 2015 20:59:30 GMT
... More header info

```

If we were to make a GET request after deleting record at id 17, we should get the following response:

```default
HTTP/1.1 404 Not Found



That record isn't here.



```

The HTML may vary, but the header should contain a 404 status code to inform the browser or client application that the record does not exist at that address.

POST

POST requests are primary request type for sending large amounts of data in a request body. Typically in a RESTful service, POST requests are used for creating a record or asset.  Our xhr service does not support all the ways information could be sent in a request body. Instead we are set up, primarily, to send JSON blobs.

JSON has become one of the most common ways to transmit information to and from web services. Most languages support JSON natively or they have a well-used, stable library for managing and manipulating JSON data.

Let's create a new POST request and create some data we can interact with. Much like DELETE, we are going to treat POST as a privileged endpoint and send our credential data.

```javascript
xhr.post('http://www.insecure-service/', function (error, response) { /* our function body */ },
{
    data: {
        foo: ['bar', 'baz', 'quux'],
        parentId: 97
    },
    headers: {
        credentials: 'chris@insecure-service.com;badpassword'
    }
});
```

Our post request will generate a rather different request structure. We will have the request header as we have seen before, but this will also pass a request body, which contains all of the information we added in the data property.  Let's take a look.

```default
POST / HTTP/1.1
host: www.insecure-service.com
credentials: chris@insecure-service.com;badpassword

{"foo":["bar","baz","quux"],"parentId":97}

```

This is actually quite similar to the response message we received when we got our 404. The header contains all of the important HTTP request information and the body contains all of the representational state, i.e. user data. Our response message may vary, but, traditionally, a representation of the record id, or some other identifying information is returned with the success response.

```default
HTTP/1.1 200 OK
asset: http://www.insecure-service/?id=103

{ id: 103 }

```

The asset URL may or may not exist in the header.  If a service provides this kind of header information is is referred to as an affordance.  Essentially, an affordance in REST and hypermedia APIs is essentially a URL to retrieve the record the service just created. This makes it convenient for the program to capture the URL and immediately request the asset from the service via GET. The JSON blob in the body contains the record ID. This service is assuming, since all data sent to create the record was assembled by the client, the only information the client doesn't currently have is the new record id.

We can use the provided ID to create a delete request if the user decides they don't want to persist that data, or we can use it to update the information when the user changes their input options.

PUT

The final REST verb we are taking a look at is PUT. PUT is the update variation on POST.  Some older, or less robust services may either not recognize PUT or they may treat PUT in the same way they handle POST requests. In either of these cases, they are not adhering to common modern expectations of REST endpoints.

It's important to understand the original specification for REST does not necessarily cover or agree with what current development practices enforce.  This means, what most places refer to as REST may not match what the originator intended, but understanding what most developers consider REST behaviors today is far more important to effective, new development than adhering to the original documentation.

PUT requests function almost identically to POST requests from the client perspective. We can actually copy our original request, make a couple small modifications and update the information stored on the server with any changes we like.  Let's take a look.

```javascript
xhr.put('http://www.insecure-service/', function (error, response) { /* our function body */ },
{
    data: {
        id: 103
        foo: ['bar', 'baz', 'quux', 'test1', 'test2', 'test3']
    },
    headers: {
        credentials: 'chris@insecure-service.com;badpassword'
    }
});
```

This xhr request would create the following HTTP request.  Please note, we did not specify a parentId.  We will take a look at how that works out at the service layer in a moment.

```default
PUT / HTTP/1.1
host: www.insecure-service.com
credentials: chris@insecure-service.com;badpassword

{id:103,"foo":["bar","baz","quux","test1","test2","test3"]}

```

Let's send a get request and see what we get back from the service.

```javascript
xhr.get("http://www.insecure-service.com/", function (error, response) {
    if (!error) {
        console.log(JSON.parse(response));
    }
}, { data: { id: 103 } });

// {
//     id: 103,
//     foo: [
//         'bar',
//         'baz',
//         'quux',
//         'test1',
//         'test2',
//         'test3'
//     ],
//     parentId: 97
// }
```

The parentId property was persisted as it was originally defined.  This behavior is common for services supporting PUT. Typically, the only properties in a record which get updated are the ones which are defined in the request. This provides security when sending data that, if the entire set of data is not known, a property isn't deleted or modified accidentally.

Conclusion

This was a rather long post for what ends up being a relatively simple concept.  The big take-away from this discussion is the interrelation between the kinds of requests that can be made and how they are interpreted and managed through HTTP and the service layer.

HTTP supports a variety of verbs which reach beyond the list of four we reviewed in this post, but these are the most common. The next time you make an asynchronous request to a server somewhere on the web, think about how that request is being processed and represented by the browser. When data comes back from the your service, take a look at the response message and see what kind of information you can uncover that you might not have known.

    

URLs for Web Developers

Nov 4, 2015

At this point it seems like everyone knows about URLs. People can rattle off google.com and myfavoritesportsteam.net, but knowing domain names is not the same as really understanding URLs. URLs, AKA Uniform Resource Locators, actually adhere to a set of standards and behaviors which apply to the way your application interacts with the browser and services which provide data and resources.

First, let’s take a look at the anatomy of a URL and break it down piece by piece regarding how each part directs the browser and informs your application. Once we have a common language to discuss the elements, we can make sense of how the pieces impact a web application.

protocol://subdomain.domain-name.top-level-domain:port-number/path?query-string#fragment

Protocol

Commonly, the protocol is http or https, however it doesn’t stop there. Browsers can support a variety of protocols such as ftp, mailto and even gopher. If you opt for an alternate protocol, be aware there is no promise that all browsers will support it. Mailto, for instance, used to function for sending emails since mail servers were wide open, but browsers typically attempt to open a mail client on your computer instead, now. Some browsers may have dropped mailto support altogether. In short, your mileage may vary.

The protocol can be accessed directly via Javascript. The window object contains a location property. Location contains several properties including protocol. Knowing the protocol can be useful, especially when it is not clear whether your site will be communicating via standard http or through a secure channel. The protocol is typically returned with a trailing colon, so you may want to strip the colon. Let’s take a look at two different lines of code for interacting with the protocol.

var fullProtocol = window.location.protocol,
    trimmedProtocol = window.location.protocol.replace(':', '');

console.log(fullProtocol)); // http:
console.log(trimmedProtocol); // http

Domain

Although the domain is broken into three parts in our diagram, domains are hard to actually split programmatically. Domain names, including the subdomain, domain name and top-level domain may be as simple as foo.bar.com, domains can also look like foo.bar.baz.quux.co.uk. Clearly co is not the domain name since co.uk is the top-level domain for companies in the United Kingdom. This means that quux is the domain name and foo.bar.baz is the subdomain.

Unfortunately, this means we can’t parse a fully-qualified domain without some sort of prior knowledge. However, if we make certain assumptions, we can safely work backwards through a fully-qualified domain, specifically, if we assume that all domains we are interacting with are .com, .net, .org, .gov or .edu top level domains, we can then parse out the pieces pretty easily.

function parseHost (hostname) {
    var domainTokens = hostname.split('.');

    return {
        topLevelDomain: domainTokens.pop(),
        domainName: domainTokens.pop(),
        subdomain: domainTokens.join('.')
    };
}

var parsedDomain = parseHost('foo.bar.baz.quux.com');

console.log(parsedDomain.subdomain); // foo.bar.baz
console.log(parsedDomain.domainName); // quux
console.log(parsedDomain.topLevelDomain); // com

This knowledge actually lines us up for beginning to understand an HTTP header as it is compiled by the browser for requests to a server. Let’s take a look at what a basic HTTP header would look like using a url like http://www.chrisstead.net/.

GET / HTTP/1.1
host: www.chrisstead.net

That header information is a fully-formed acceptable HTTP request. If you were to issue this header through a telnet client you would get the front page of this site. This makes the relation between a URL and the request that is issued fairly transparent.

Port

The port is precisely as it sounds: a number representing the port to communicate through. HTTP typically goes through port 80, however other ports can and have been used for HTTP communication like 8080 and 8000. Express (a Node web service framework) defaults to port 3000. We can specify a different port like the following.

http://www.chrisstead.net:8080

This call will, of course, result in a refused connection. This website uses the default web port and does not respond on 8080 or any other non-standard ports. If, however, this site were to accept requests on port 8080, you could open a connection with a telnet connection in the following way.

telnet www.chrisstead.net 80
```

This is effectively what the browser does when connecting to a website. Our URL model is actually getting pretty rich and we haven't introduced any of the other variables.  If we wanted to capture the port that was requested programmatically when your browser script loads up, we can do that with the port property on location.

```javascript
console.log(window.location.port); // typically returns 80
```

Path

The path in a URL is nearly as familiar as the .com in a big company URL. This path can represent a variety of different resources. It can be a path to a file resource like an HTML file or an image, or it could represent a path to a service resource with a specific data query associated. A good example of a non-physical resource is the page displaying this blog post.

```javascript
http://www.chrisstead.net/archives/970/urls-for-web-developers/
```

We need to be able to break this down, so let's take a look at how we can capture the path from the location. We can access the path from the location object in a similar fashion to the port. Window.location has a pathName property which will give you just the path without any other data attached to it.

```javascript
// The path for this page looks like this:
console.log(window.location.pathName); // /archives/970/urls-for-web-developers/
```

When a request is sent through the browser with a path, our request header changes a little to manage the enhanced request data. Our original header actually included a path, but it simply pointed at the root directory.  Our new header will reflect our correct path.

```default
GET /archives/970/urls-for-web-developers/ HTTP/1.1
host: www.chrisstead.net

```

Query String

The query string is one of the last and most interesting parts of the URL. This is traditionally where information can be stored and passed along to the server.  This ties in with REST, which we will discuss in another post. Let's take a look at a representation of user state in a query string and how to retrieve it with Javascript.

```javascript
// example query string: ?foo=bar&baz=quux

console.log(window.location.search); // ?foo=bar&bar=quux

function setPair (finalObj, keyValueStr) {
    var queryTokens = keyValueStr.split('='),
        key = queryTokens.shift(),
        value = queryTokens.join('=');

    finalObj[key] = value;

    return finalObj;
}

function parseQueryString (queryString) {
    return queryString.slice(1).split('&').reduce(setPair, {});
}

var queryData = parseQueryString('?foo=bar&baz=quux');

console.log(queryData.foo); // bar
console.log(queryData.baz); // quux
```

Unfortunately, Javascript doesn't provide any special facility for parsing or managing query strings, so we have to handle it ourselves. Fortunately parsing out the information is fairly simple.  Of course, if we want to change the data, we need to reconstruct the original string with updated values.  Let's add a new key/value pair and change one of our values, then we'll update and refresh the URL.

```javascript
function addPair (queryData, queryArray, key) {
    queryArray.push(key + '=' + queryData[key]);
    return queryArray;
}

function buildQueryString (queryData) {
    return '?' + Object.keys(queryData)
                       .reduce(addPair.bind(null, queryData), [])
                       .join('&');
}

queryData.foo = 'bar0';
queryData['test'] = 'quack';

window.location.search = buildQueryString(queryData);
console.log(window.location.search); // ?foo=bar0&baz=quux&test=quack
```

Most importantly, when we update the search string, the page is refreshed and a request is sent to the server for any data updates that might need to occur. This means, for standard page views, form data can be sent via query string to a service if an AJAX call is inappropriate.

Much like we did when we did with our previous request updates, let's have a look at the update to our request header including a query string. You'll note the change is merely an appended string on the request path.  Let's have a look.

```javascript
GET /archives/970/urls-for-web-developers/?foo=bar HTTP/1.1
host: www.chrisstead.net

```

Fragments

The final part of a url is the fragment, otherwise known as the hash.  The hash data is the only part of the URL which does not affect the request sent back to the server. Any fragment information that is included in the URL is only used for the browser to handle the document returned from the server. Originally the fragment information was used to move to the focus of the browser to a specific part of the document.

Now the fragment information has been claimed by single page applications (SPAa) to maintain state and route information.  This new use stems from the very same fact that the fragment is in place for browser use alone and does not refresh the page, or send any new information to the server.

The first application I can recall using a fragment was Twitter with a #!/ to denote the SPA route.  It seems they were borrowing against the Unix script shebang used to identify the executable to use when interpreting the script. Useless trivia. It won't be on the test.

Fragments can contain anything including #, and will be made available to any client-side script running in the window. Let's have a look at a URL containing a hash.

```default
http://www.chrisstead.net/#/foo/bar/baz
```

This setup is precisely what we would need to handle routing for a SPA. Without digging into the routing logic, let's access the fragment and log it.

```javascript
console.log(window.location.hash); // /foo/bar/baz
```

It might have been the long way around, but this is how URLs and web development tie together. Hopefully this filled in any information gaps you might have wondered about. It's really important for someone working in web development to understand how requests and URLs are related and what each does for the other.

    

3 Rules to Improve Your App State Management

Oct 28, 2015

As SPAs become more common for web application development, the concept of state is moving almost completely out of the realm of servers and services and into the browser, and rightly so. Ideally services should only maintain state as persistent storage like databases, document stores and file systems. Application state like where the user is and what they are interacting with should be maintained as close to the user as we can keep it.

Systems like Angular have built in state machines as well as third-party solutions, but these solutions are often not enough, on their own, to truly manage all the state your application needs. By relying on just a finite-state machine and persistent storage, your application takes on a chatty characteristic and leaves a lot of data living in limbo before it is actually ready for consumption by the services which persist the data from one application session to the next.

The goal of this post is to fill in some of those gaps and smooth out the application development process. Three main ideas are going to be presented: keep state where you can get it, make state mutation obvious, keep state out of meddling hands. Each of these principles will guide our direction and create something that will serve us well as we work on integrating data transit into our systems both through the client-side application as well as moving data back and forth between our application and our data persistence system.

Keep State Where You Can Get It

The first main idea is, we actually do want our state to be accessible to our application. A common reaction I have seen from many developers I’ve worked with is to simply create a global variable and push data right into an object on the window scope. We will see later, this is a violation, not only of general best practice, but also of one of our three rules.

Nevertheless, we need some place where we can access our data, so let’s create a very simple state cache factory, which will give us the most fundamental piece of the puzzle. We will create an object, if one doesn’t already exist, and hand it out to components which request it. This kind of design keeps state off the global scope, while still making it readily accessible when we need it.

** Note: Our state factory makes use of the either function from my sanitary data post.

var stateFactory = (function () {
	
	var stateCache = {
		data: false
	};
	
	function build () {
		stateCache.data = either({}, stateCache.data, 'object');
		return stateCache.data;
	}
	
	return {
		build: build
	};
	
})();

var myStatePointer = stateFactory.build();
myState['foo'] = 'bar';

var secondStatePointer = stateFactory.build();
console.log(secondStatePointer.foo); // bar

So far so good. We can easily reach our state, but it is kept out of the global scope. This makes our state a little more stable, but it also exposes a big problem. We are hand-editing an object. This means, each time we update state, we are actually modifying all state for everyone. What’s worse, it’s really, really difficult to identify where the state is getting modified. We can’t search the document for any particular behavior because anyone could write their function to work in any way they like and potentially blow away all state for any part of the app. Yikes!

Make State Mutation Obvious

Simply handing around an object pointer is anything but obvious. We are all quite familiar with working with object literals, and most SPAs end up littered with them, so our state object will quickly become part of the crowd. Losing track of your state object is a dangerous game that will lead to some very difficult to identify bugs. Ideally your state should have a well-defined interface to interact with so it is easy to identify when state properties are added or existing properties are accessed and modified.

The post about creating a new generic type introduced structs as an object concept. Structs are a good step in the right direction. Each property has its own accessor and mutator, so we get a lot of what we need for free. Let’s update our state factory build method to return a struct and lock down our state interface.

(function () {
	
	/* ... State factory code here ... */
	
	function build () {
		stateCache.data = either(new Struct(), stateCache.data, 'object');
		return stateCache.data;
	}
	
	/* ... Remaining state factory code here ... */
	
})();

var statePointer = stateFactory.build();

statePointer.addProperty('foo');
statePointer.set.foo('bar');

var secondStatePointer = stateFactory.build();

console.log(statePointer.get.foo()); // bar

The interface is a little more verbose than directly manipulating an object literal, but the clarity we get when interacting with the state outweighs the pain of an extra couple key strokes. We can now easily find every instance of state access by looking for get.keyName() and every instance of state modification by looking for set.keyName().

With an interface distinct to the struct, our state interactions become obvious. This kind of clarity makes programming much easier since our local data can be anything we like, while we keep our state data clean and obvious. There is still one issue: we still have a lot of access to do terrible things to the data stored in our state. The struct definition comes with a merge function so we could easily do the following:

var statePointer = stateFactory.build();

statePointer.merge({
    keyName1: someNewObject,
    keyName2: someOtherObject,
    keyName3: someLastObject
});

Now all of the data for three keys has been blown away which could quickly send our app into a tailspin. There is no way to be certain our objects will continue to line up with the state our app expects in other places. This kind of uncertainty makes for a scary programming experience which could bring even the most senior programmer to tears.

Keep State Out of Meddling Hands

There is a saying that goes a little like this, “if one person gets a cold everyone gets a cold.” What this really means is, when everyone is working closely together, if someone introduces something bad into the system it becomes everyone’s problem. There is no place this becomes more obvious than when dealing with state.

When dealing with state access, handing off a pointer becomes a liability. What we really need to do is wrap up our struct with a state-specific interface. Structs as created in the related post, are a mutable data type. This means anyone can go in and change them any way they want. What we really want to do is temper that mutability and define how our program actually interacts with our struct.

The best way to do this is to wrap our struct up in a state object, and expose methods which tightly control how our data flow works. This new construct uses the clone function from JFP v2.4.0 to allow copied data out through a getter, but stops data propagation back up the object tree. This forces programmers working with our state object to explicitly set the new state once they have modified the object copy.

function State (initialState) {
	var sanitizedInitialState = j.either({}, initialState, 'object'),
		stateStruct = new Struct(),
		stateKeys = Object.keys(sanitizedInitialState);
		
	Struct.prototype.constructor.apply(stateStruct, stateKeys);
	stateStruct.merge(sanitizedInitialState);
	
	this.get = {};
	this.set = {};

	this.addState = this.addState.bind(this, stateStruct);
	stateKeys.foreach(this.attachStructMethods.bind(this, stateStruct));
}

State.prototype = {
	
	accessorBase: function (struct, key) {
		return j.clone(struct.get[key]());
	},
	
	attachStructMethods: function (struct, key) {
		this.get[key] = this.accessorBase.bind(this, struct, key);
		this.set[key] = struct.set[key];
	},
	
	addState: function (struct, key, value) {
		struct.addProperty(key);
		struct.set[key](maybe(value));
		
		this.attachStructMethods(struct, key);
	}
	
};

Our new state object also introduces the concept of an initial state. This allows us to bootstrap our application into a particular state by making use of the struct merge behavior. We will want our stateFactory to account for this and only allow state bootstrapping once, throwing an error if someone tries to fully re-bootstrap the app in the middle of the user’s experience. Let’s take a look at our final build behavior.

(function () {
	
	/* ... State factory code here ... */
	
	function build (initialState) {
		if(typeof initialState !== 'undefined' && stateCache.data instanceof State) {
			throw new Error('Cannot bootstrap existing state object.');
		}
		
		stateCache.data = either(new State(initialState), stateCache.data, 'object');
		return stateCache.data;
	}
	
	/* ... Remaining state factory code here ... */
	
})();

This final iteration gives us the remaining stability we need to really handle state well. Now when we access state, we get a copy of the data, which may be all we need. If we need to actually update the data, we have exposed methods to merge new data back into the state system. Finally, we can make use of the merge functionality in our structs, which allows us to bootstrap our state with little effort, which takes the user to exactly the place they wanted to be when we pick up from a non-trivial state. Let’s take a look at what our new interactions would look like.

var statePointer = stateFactory.build({ 'foo': ['bar'] }),
    fooState = statePointer.get.foo();

console.log(fooState[0]); // bar

var secondStatePointer = stateFactory.build(),
    secondFooState = statePointer.get.foo();

console.log(secondFooState[0]); // bar

fooState[0] = 'baz';
statePointer.set.foo(fooState);

console.log(fooState[0]); // baz
console.log(secondFooState[0]); // bar

secondFooState = secondStatePointer.get.foo();
console.log(secondFooState[0]); // baz

stateFactory.build({ 'foo': ['bar', 'baz', 'quux'] }); //Throws error

By taking time to build out a stable state system, we shield our application from sudden, unexpected changes in our state. This means each of our modules can act on state data independently, and then feed the updates back into the system. Ideally, we would take care to merge state updates into our local data before we merge our changes back into the core state, but that’s another post for a different day.

As you build your client-side applications, don’t only think about state that your routing system provides, but also consider the true, robust state of your application. Create state data your application can access, make it obvious when that data changes and keep the original state data out of meddling hands. With just a little bit of preparation, your applications will be stable and fun to work on.

  • Web Designers Rejoice: There is Still Room

    I’m taking a brief detour and talking about something other than user tolerance and action on your site. I read a couple of articles, which you’ve probably seen yourself, and felt a deep need to say something. Smashing Magazine published Does The Future Of The Internet Have Room For Web Designers? and the rebuttal, I Want To Be A Web Designer When I Grow Up, but something was missing.

  • Anticipating User Action

    Congrats, you’ve made it to the third part of my math-type exploration of anticipated user behavior on the web. Just a refresher, the last couple of posts were about user tolerance and anticipating falloff/satisficing These posts may have been a little dense and really math-heavy, but it’s been worth it, right?

  • Anticipating User Falloff

    As we discussed last week, users have a predictable tolerance for wait times through waiting for page loading and information seeking behaviors. The value you get when you calculate expected user tolerance can be useful by itself, but it would be better if you could actually predict the rough numbers of users who will fall off early and late in the wait/seek process.

  • User Frustration Tolerance on the Web

    I have been working for quite a while to devise a method for assessing web sites and the ability to provide two things. First, I want to assess the ability for a user to perform an action they want to perform. Second I want to assess the ability for the user to complete a business goal while completing their own goals.

  • Google Geocoding with CakePHP

    Google has some pretty neat toys for developers and CakePHP is a pretty friendly framework to quickly build applications on which is well supported. That said, when I went looking for a Google geocoding component, I was a little surprised to discover that nobody had created one to do the hand-shakey business between a CakePHP application and Google.

  • Small Inconveniences Matter

    Last night I was working on integrating oAuth consumers into Noisophile. This is the first time I had done something like this so I was reading all of the material I could to get the best idea for what I was about to do. I came across a blog post about oAuth and one particular way of managing the information passed back from Twitter and the like.

  • Know Thy Customer

    I’ve been tasked with an interesting problem: encourage the Creative department to migrate away from their current project tracking tool and into Jira. For those of you unfamiliar with Jira, it is a bug tracking tool with a bunch of toys and goodies built in to help keep track of everything from hours to subversion check-in number. From a developer’s point of view, there are more neat things than you could shake a stick at. From an outsider’s perspective, it is a big, complicated and confusing system with more secrets and challenges than one could ever imagine.

  • When SEO Goes Bad

    My last post was about finding a healthy balance between client- and server-side technology. My friend sent me a link to an article about SEO and Google’s “reasonable surfer” patent. Though the information regarding Google’s methods for identifying and appropriately assessing useful links on a site was interesting, I am quite concerned about what the SEO crowd was encouraging because of this new revelation.

  • Balance is Everything

    Earlier this year I discussed progressive enhancement, and proposed that a web site should perform the core functions without any frills. Last night I had a discussion with a friend, regarding this very same topic. It came to light that it wasn’t clear where the boundaries should be drawn. Interaction needs to be a blend of server- and client-side technologies.

  • Coding Transparency: Development from Design Comps

    Since I am an engineer first and a designer second in my job, more often than not the designs you see came from someone else’s comp. Being that I am a designer second, it means that I know just enough about design to be dangerous but not enough to be really effective over the long run.

  • Usabilibloat or Websites Gone Wild

    It’s always great when you have the opportunity to built a site from the ground up. You have opportunities to design things right the first time, and set standards in place for future users, designers and developers alike. These are the good times.

  • Thinking in Pieces: Modularity and Problem Solving

    I am big on modularity. There are lots of problems on the web to fix and modularity applies to many of them. A couple of posts ago I talked about content and that it is all built on or made of objects. The benefits from working with objectified content is the ease of updating and the breadth and depth of content that can be added to the site.

  • Almost Pretty: URL Rewriting and Guessability

    Through all of the usability, navigation, design, various user-related laws and a healthy handful of information and hierarchical tricks and skills, something that continues to elude designers and developers is pretty URLs. Mind you, SEO experts would balk at the idea that companies don’t think about using pretty URLs in order to drive search engine placement. There is something else to consider in the meanwhile:

  • Content: It's All About Objects

    When I wrote my first post about object-oriented content, I was thinking in a rather small scope. I said to myself, “I need content I can place where I need it, but I can edit once and update everything at the same time.” The answer seemed painfully clear: I need objects.

  • It's a Fidelity Thing: Stakeholders and Wireframes

    This morning I read a post about wireframes and when they are appropriate. Though I agree, audience is important, it is equally important to hand the correct items to the audience at the right times. This doesn’t mean you shouldn’t create wireframes.

  • Developing for Delivery: Separating UI from Business

    With the advent of Ruby on Rails (RoR or Rails) as well as many of the PHP frameworks available, MVC has become a regular buzzword. Everyone claims they work in an MVC fashion though, much like Agile development, it comes in various flavors and strengths.

  • I Didn't Expect THAT to Happen

    How many times have you been on a website and said those very words? You click on a menu item, expecting to have content appear in much the same way everything else did. Then, BANG you get fifteen new browser windows and a host of chirping, talking and other disastrous actions.

  • Degrading Behavior: Graceful Integration

    There has been a lot of talk about graceful degradation. In the end it can become a lot of lip service. Often people talk a good talk, but when the site hits the web, let’s just say it isn’t too pretty.

  • Website Overhaul 12-Step Program

    Suppose you’ve been tasked with overhauling your company website. This has been the source of dread and panic for creative and engineering teams the world over.

  • Pretend that they're Users

    Working closely with the Creative team, as I do, I have the unique opportunity to consider user experience through the life of the project. More than many engineers, I work directly with the user. Developing wireframes, considering information architecture and user experience development all fall within my purview.

  • User Experience Means Everyone

    I’ve been working on a project for an internal client, which includes linking out to various medical search utilities. One of the sites we are using as a search provider offers pharmacy searches. The site was built on ASP.Net technology, or so I would assume as all the file extensions are ‘aspx.’ I bring this provider up because I was shocked and appalled by their disregard for the users that would be searching.

  • Predictive User Self-Selection

    Some sites, like this one, have a reasonably focused audience. It can become problematic, however, for corporate sites to sort out their users, and lead them to the path of enlightenment. In the worst situations, it may be a little like throwing stones into the dark, hoping to hit a matchstick. In the best, users will wander in and tell you precisely who they are.

  • Mapping the Course: XML Sitemaps

    I just read a short, relatively old blog post by David Naylor regarding why he believes XML sitemaps are bad. People involved with SEO probably know and recognize the name. I know I did. I have to disagree with his premise, but agree with his argument.

  • The Browser Clipping Point

    Today, at the time of this writing, Google posted a blog stating they were dropping support for old browsers. They stated:

  • Creativity Kills

    People are creative. It’s a fact of the state of humanity. People want to make things. It’s built into the human condition. But there is a difference between haphazard creation and focused, goal-oriented development.

  • Reactionary Navigation: The Sins of the Broad and Shallow

    When given a task of making search terms and frequetly visited pages more accessible to users, the uninitiated fire and fall back. They leave in their wake, broad, shallow sites with menus and navigtion which look more like weeds than an organized system. Ultimately , these navigation schemes fail to do the one thing they were intended for, enhance findability.

  • OOC: Object Oriented Content

    Most content on the web is managed at the page level. Though I cannot say that all systems behave in one specific way, I do know that each system I’ve used behaves precisely like this. Content management systems assume that every new piece of content which is created is going to, ultimately, have a page that is dedicated to that piece of content. Ultimately all content is going to live autonomously on a page. Content, much like web pages, is not an island.

  • Party in the Front, Business in the Back

    Nothing like a nod to the reverse mullet to start a post out right. As I started making notes on a post about findability, something occurred to me. Though it should seem obvious, truly separating presentation from business logic is key in ensuring usability and ease of maintenance. Several benefits can be gained with the separation of business and presentation logic including wiring for a strong site architecture, solid, clear HTML with minimal outside code interfering and the ability to integrate a smart, smooth user experience without concern of breaking the business logic that drives it.

  • The Selection Correction

    User self selection is a mess. Let’s get that out in the open first and foremost. As soon as you ask the user questions about themselves directly, your plan has failed. User self selection, at best, is a mess of splash pages and strange buttons. The web has become a smarter place where designers and developers should be able to glean the information they need about the user without asking the user directly.

  • Ah, Simplicity

    Every time I wander the web I seem to find it more complicated than the last time I left it.  Considering this happens on a daily basis, the complexity appears to be growing monotonically.  It has been shown again and again that the attention span of people on the web is extremely short.  A good example of this is a post on Reputation Defender about the click-through rate on their search results.

  • It's Called SEO and You Should Try Some

    It’s been a while since I last posted, but this bears note. Search engine optimization, commonly called SEO, is all about getting search engines to notice you and people to come to your site. The important thing about good SEO is that it will do more than simply get eyes on your site, but it will get the RIGHT eyes on your site. People typically misunderstand the value of optimizing their site or they think that it will radically alter the layout, message or other core elements they hold dear.

  • Information and the state of the web

    I only post here occasionally and it has crossed my mind that I might almost be wise to just create a separate blog on my web server.  I have these thoughts and then I realize that I don’t have time to muck with that when I have good blog content to post, or perhaps it is simply laziness.  Either way, I only post when something strikes me.

  • Browser Wars

    It’s been a while since I have posted. I know. For those of you that are checking out this blog for the first time, welcome. For those of you who have read my posts before, welcome back. We’re not here to talk about the regularity (or lack thereof) that I post with. What we are here to talk about is supporting or not supporting browsers. So first, what inspired me to write this? Well… this:

  • Web Scripting and you

    If there is one thing that I feel can be best learned from programming for the internet it’s modularity.  Programmers preach modularity through encapsulation and design models but ultimately sometimes it’s really easy to just throw in a hacky fix and be done with the whole mess.  Welcome to the “I need this fix last week” school of code updating.  Honestly, that kind of thing happens to the best of us.

  • Occam's Razor

    I have a particular project that I work on every so often. It’s actually kind of a meta-project as I have to maintain a web-based project queue and management system, so it is a project for the sake of projects. Spiffy eh? Anyway, I haven’t had this thing break in a while which either means that I did such a nice, robust job of coding the darn thing that it is unbreakable (sure it is) or more likely, nobody has pushed this thing to the breaking point. Given enough time and enough monkeys. All of that aside, every so often, my boss comes up with new things that she would like the system to do, and I have to build them in. Fortunately, I built it in such a way that most everything just kind of “plugs in” not so much that I have an API and whatnot, but rather, I can simply build out a module and then just run an include and use it. Neat, isn’t it?

  • Inflexible XML data structures

    Happy new year! Going into the start of the new year, I have a project that has carried over from the moment I started my current job. I am working on the information architecture and interaction design of a web-based insurance tool. Something that I have run into recently is a document structure that was developed using XML containers. This, in and of itself, is not an issue. XML is a wonderful tool for dividing information up in a useful way. The problem lies in how the system is implemented. This, my friends, is where I ran into trouble with a particular detail in this project. Call it the proverbial bump in the road.

  • Accessibility and graceful degradation

    Something that I have learnt over time is how to make your site accessible for people that don’t have your perfect 20/20 vision, are working from a limited environment or just generally have old browsing capabilities. Believe it or not, people that visit my web sites still use old computers with old copies of Windows. Personally, I have made the Linux switch everywhere I can. That being said, I spend a certain amount of time surfing the web using Lynx. This is not due to the fact that I don’t have a GUI in Linux. I do. And I use firefox for my usual needs, but Lynx has a certain special place in my heart. It is in a class of browser that sees the web in much the same way that a screen reader does. For example, all of those really neat iframes that you use for dynamic content? Yeah, those come up as “iframe.” Totally unreadable. Totally unreachable. Iframe is an example of web technology that is web-inaccessible. Translate this as bad news.

  • Less is less, more is more. You do the math.

    By this I don’t mean that you should fill every pixel on the screen with text, information and blinking, distracting graphics. What I really mean is that you should give yourself more time to accomplish what you are looking to do on the web. Sure, your reaction to this is going to be “duh, of course you should spend time thinking about what you are going to do online. All good jobs take time.” I say, oh young one, are you actually spending time where it needs to be spent? I suspect you aren’t.

  • Note to self, scope is important.

    Being that this was an issue just last evening, I thought I would share something that I have encountered when writing Javascript scripts.  First of all, let me state that Javascript syntax is extremely forgiving.  You can do all kinds of  unorthodox declarations of variables as well as use variables in all kinds of strange ways.  You can take a variable, store a string in it, then a number, then an object and then back again.  Weakly typed would be the gaming phrase.  The one thing that I would like to note, as it was my big issue last evening, is scope of your variables.  So long as you are careful about defining the scope of any given variable then you are ok, if not, you could have a problem just like I did.  So, let’s start with scope and how it works.

  • Subscribe

    -->