Story Driven Design

Articles about building interactive, content-rich websites usually focus on the software structure and demonstrate the use of advanced JavaScript combined with a modern framework such as Angular, React or Vue. Useful as they often are, though, this is a very techno-centric way of looking at things and is really putting the cart before the horse.

Websites don't in general start as technical projects but as stories - descriptions, sometimes with pictures, sometimes not - of what the site should do. Stories are written by domain experts - the customers whose real-world needs are to be met by a new website, and it is with these stories we should start.

To explain this we need an example. The one I will use is called Here On The Map and you can see it working at https://hereonthemap.com. It's a demo site built to meet a somewhat artificial need, chosen to bring in some specific technical features. In design terms it's simple; the important aspect is the interactivity.

The 'story' of Here On The Map

Here On The Map (HOTM) has quite a simple user story. It displays a live map on which there are colored pins. The user can zoom and pan to reveal any part of the Earth's surface. When the user clicks a pin a panel of information appears about the location of that pin. Users can register with the site; they are then able to add their own pins and write about them using a rich text editor. Images can be accessed by URL or uploaded to the system. Here's what it looks like when viewing:

and while editing:

That's a not-untypical story of part of the website's functionality at least. From there we find a technical team and go to the implementation. But before we jump in with the coding, here are some general issues.

General implementation issues

To start with, the platform here is WordPress. Why? Because its the simplest way to get basic functionality in place, on top of which we can build our website. The theme chosen here is called No Header; it's a very basic theme that provides little more than a sidebar, leaving two-thirds of the screen for our own content. It also provides some basic responsiveness for mobile compatibility. Our application knows nothing about it and doesn't interact with it in any way.

This website is likely to be front-end driven. It's not impossible to deliver something along these lines with traditional server-side code, but good responsiveness to user actions would be hard to achieve. Furthermore, the visual structure is pretty basic so it's quite feasible - maybe even preferable - to implement it entirely in code. There's a rather fuzzy line between a web page and a browser application; the former has more emphasis on content and the latter on function. Here we're rather leaning towards a browser application.

One thing that stands out when looking at the story is the way the site is made up of functional blocks. Quite big ones, in fact. In particular:

• A Google Map
• A user registration/login module
• A Rich Text editor component
• Picture upload and storage

The way these blocks interact with each other is the key to getting the site to work well, but it's hard to predict all of the details without first building the site. Once it's running, the nature of some of these interactions will become clearer. For example, when you click a pin you see the corresponding article. You also get a link to the author and a list of tags. When any of these are clicked, the map updates to show only the pins matching what was selected. A line of text appears under the map to tell you what filter you currently have in place.

A second interaction: When you zoom the map the current zoom level is shown at the end of a line of text below the map.

A third interaction: When you log out by clicking the link in the line under the map, the viewer or editor panels disappear, just in case you were viewing something that shouldn't be seen by someone who is not logged in. Also, if you were viewing a pin you created yourself it would be green. When you log out it goes yellow.

There are also various rules, such as what state information gets persisted as the application runs. We preserve the current latitude, longitude and zoom, also the name of the pin we're viewing. In the File Manager we hold onto the directory path so the user can return there next time. And so on.

These interactions and rules together form stories and they belong to the domain expert. They should be preserved in a form (s)he can understand and where they can easily be accessed to make changes, not buried in arcane code. Think long-term and think of the site maintainer, who will often be someone other than the original site builder.

Stories and Components

From the above we can see a structure begin to emerge, one of functional components joined together by stories. If we can find a way to manage these two things independently the website will be much easier to work with than if we jumble everything up together. With interactive websites, maintenance is a serious issue. If the stories are not easy to find in the code, a future maintainer will have trouble understanding what should be happening and is likely to break things.

Some components are just large third-party JavaScript libraries. Here we have two in particular; Google Maps and CKEditor, the component that provides our rich text editor. Each of these is freely available from a CDN or can be installed along with the rest of the website. The key thing is they are both black boxes. We have no interest in how they are implemented, only in the interfaces they present to the world.

The rest of the components might be provided as standard libraries, WordPress plugins, one-off code specifically written for this project or some combination of code and library. Whichever is the case, we should design our application to use components in a clean and consistent manner.

That just leaves the stories. For reasons I've already given, these should remain visible in a form that's as close to the original as it's possible to get, and a good precedent for this exists in the form of SQL. Database applications may do all sorts of things with data, whether transforming it for display or using it to control layout, but the actual process of extracting it from a database is done in a way that is readily understood by database domain experts as well as by programmers. Can we express our stories in a similar way?

Levels of coding

General-purpose computer languages are for programmers. The rest of us speak and write in English or another human language, and we do it entirely using words. Human languages look nothing like JavaScript code; there are no curly braces, fat arrows or triple-equals, yet we humans process them pretty well and manage to organize whole societies by using them. Another clear difference between human and computer languages is that one you can speak; the other you can't. Not without sounding daft, that is. This may sound like a trivial point but most human communication is verbal, not written, and in most languages the way we write closely mirrors the way we speak.

When we come across a complex entity, rather than spell it out longwindedly every time we give it a name such as a Defibrillator. Or, to return to the context of HOTM, a Map or an RTF Editor. We don't concern ourselves with how it works, only with what it does, that is, the interfaces it presents to us. This works in the computer world too. You can draw a map with just 3 items of information, such as

"draw a map in the box, with latitude 45.234, longitude 5.82 and zoom 5.5"

The one and only requirement for a viable computer language is that it must be unambiguous, so the above qualifies as a valid computer language statement while remaining perfectly readable English.

Of course, any compiler able to process the above and execute it will itself be written in a conventional programming language, but that part doesn't have to be visible to the person implementing the stories. All (s)he needs is a syntax that's rich enough to express all the features required by the stories while providing access to the APIs of the major components.

By layering our application in this way we can maintain high readability, which gives us some confidence that future modifications can be made quickly and safely by whoever is available to do the work.

How to do it

Here On The Map was built to demonstrate the principles outlined above. It's written entirely in high-level script, using a syntax derived from English and a vocabulary that encompasses all the concepts and activities required. The compiler and runtime package is called EasyCoder and it's available from the WordPress plugin library. To illustrate the general flavor, here's a simple fragment from a user story:

"When given the id of a particular pin record in the database, read the record, set up the map and display the article for that pin"

and here's the corresponding section of script:

rest get Record from `_/ec_markers/id/` cat RequestedID or
begin
    print `Failed to load the requested pin.  Error: ` cat the error
    stop
end
if property `id` of Record is not 0
begin
    set the latitude of Map to property `latitude` of Record
    set the longitude of Map to property `longitude` of Record
    set the zoom of Map to property `zoom` of Record
    update Map
    fork to ShowStory
end

You should be able to see that there is no computer code to learn, as everything is expressed in English words. Admittedly, the syntax is a bit clunky, but it's still understandable. Names of things are camel-cased with an initial capital letter, while language commands remain in all lower case.

The first command requests a record from the database by calling a REST server that's part of the language plugin. This is one of the minor functional components that come with the system and which are largely hidden from the application programmer. In fact, though conceptually simple this is a fairly complex operation, the details of which are best kept well hidden. In the event of an error a report is logged but nothing else happens and the screen remains unchanged.

Assuming a valid record came back it will have a non-zero id, so we can extract the 3 map parameters and update the map. This time we're using the Google Maps module, wrapped in simple commands that let us access its API without worrying about the details.

And finally we go off to show the article for the pin. The 'fork' command merely ensures that the story is shown in parallel while processing continues in this thread. (No, we haven't discovered a hitherto-unknown threading feature in JavaScript; threads in this context are provided by cooperative multitasking inside the language engine.)

The HOTM website documents all the scripts used for the application. The total amount of code is a lot less than the equivalent JavaScript would be, with or without a React framework to help it. Of course, driving all this is a pile of JavaScript; a compiler/runtime engine of about 50kB and further components that bring the total close to 200kB if they are all used. Plus the scripts themselves, loaded from GitHub. The site loads in under 3 seconds; over half of this is just getting the initial WordPress page from the server, after which things fly along. Scripts compile in a few tens of milliseconds and compilation usually happens in parallel with other things like image downloads.

A website can be built very quickly using EasyCoder, partly owing to the way each command does a lot of work but also because so much of the usual infrastructure needed by websites is already provided. The bulk of HOTM took less than 2 weeks from the initial idea to a fully working website. Errors are rare because the internal functions are used over and over again, leaving bugs with few places to hide. However, it has to be acknowledged that this is a fairly small website using readily available technologies, so what if you need to build some really complex functionality?

Mixing technologies

People have a tendency to favor particular ways of working, and programmers are no exception. The current hot subject is JavaScript frameworks, with React leading the pack, and its adherents often suggest it should be used for everything. I disagree. There's nothing wrong with React but I wouldn't use it to write any code that is likely to be modified - or maybe even seen - by my customer. All that complex stuff belongs in the lower levels of coding.

The reason is my wish to keep the stories visible. JavaScript code of almost any kind fails to reveal user stories in a form that can be easily understood by anyone other than the original programmer. Without understanding, many more problems soon follow. I've demonstrated above that functionality can be split into one level for the stories and another for the infrastructure, and it's in the latter area that React belongs.

A key feature of React is that it's a component technology; another is its 2-way mapping between the component and the model. This works well in the environment I use, where major units of functionality are built as components and joined together by the random glue of stories. There are, after all, two different kinds of programming going on here. The one that deals with components looks at contained functionality and seeks to optimize it, while the other, that deals with stories and frequently-changing relationships between components, seeks to retain clarity at almost any price. These two aims are directly contradictory when applied to the same entity but not if the entities themselves are separated.

The EasyCoder high-level scripting language used by HOTM is built out of individual plugin modules that each perform a bounded range of duties, either for a specific domain or for a particular technology. The core of it just handles the random glue that keeps the user stories visible. Writing new plugins is simple; you code them to handle the syntax chosen to deal with functionality exposed by a component, which itself will often be delivered as minimized JavaScript from a CDN. How these components are engineered is entirely down to their author, and if (s)he is most comfortable using React then that's what it will be.

Remember at all times that what we're doing is creating a top-level syntax that's readily understandable by English speakers who are also domain experts. The complexity behind this syntax is not for them; it's for the programmer, who is then free to pick the best tools for the job. You are free to construct a language that has just one word; "doit", everything else being hidden inside that one command. That's how websites are in effect built today, but I don't advocate doing it that way. We can do better.

To a certain extent, adopting this strategy tends to focus the mind on building better reusable components. When you build a component alongside the code that uses it there's a tendency for ad-hoc interfaces to be added to solve short-term needs, making it progressively more difficult for the component to be used outside of the context for which it was originally built. The discipline of having to build the component in such a way that it can be accessed solely through published interfaces forces a reappraisal of just what the component is offering to the world, and in the process makes it a better component. And better components lead to better, more reliable and less costly websites.