code development NMandelbrot programming

Thoughts on Node.js

Focus on the flow
Focus on the flow

In my post about developing in the cloud, I started promising a few nuggets about the project I was working on, and following my diversion onto talks and security, I’m ready to start discussing it again. The project itself is fairly straightforward. It was an excuse for me to try a realistic project in Node.js using cloud development tools, to see what was possible, and to decide if I wanted to use Node.js for anything more substantial.

Partly, I wanted to immerse myself in a completely callback-driven, “non-blocking” model, so that I can see how it affected the way I thought about the software I was writing, and also to see if I could see the benefits. I’ll expand on some of these thoughts as I talk more about it, but I wanted to start with my initial impressions.

What Node.JS does well

Firstly, from looking at a test example based on graphing data from this great read about green energy, it was clear to me that the promises of Node.js lie very much in the I/O performance side, rather than anything else. If you’re trying to do anything more complex than throwing as many bytes at the response buffer as possible, Node.js is probably the wrong tool. It’s not built for complex business logic. It is good if your server is a thin layer between a JS front-end and something specialised on the backend (or a series of specialist pipes on the backend – the web services support is top notch, as you’d expect from the language that gave us AJAX and JSON).

One of the reasons I started to enjoy using Node.js was the ability to run the same code on client and server, which meant that I could write tests for my JavaScript logic that would run as unit tests in the server context, and then ship the exact same JavaScript to the client, knowing that the internal logic was correct, even if the performance wouldn’t be. This was greatly helped by the fact that I could use Require.JS to bridge the gap between the client and Node’s extensive NPM package repository, which isn’t as easy to search as the apt-get and NuGet repositories I’m more used to, but is fairly comprehensive, although it does suffer from the apt-get problem of having a lot of packages for doing the same thing, and it can be hard to choose which one to use. There are definitely popular options that bubble to the surface, but I get the feeling I’d need to try a few of them to really start to feel which was the right one to use, especially as a few of them have APIs that feel quite alien to the core libraries, and feel like unthinking ports from other languages, or at least non-callback philosophies.

In the end, I got a proof of concept up over a weekend, which is about as long as I’d normally spend on The Mandelbrot Set, which is a nice quick result, and I got multiple users up on the site, which is more of a testament to Cloud 9 as it is to Node, but the resultant code had fewer features and messier code than the alternatives I’d written in other languages. It certainly felt more as if I was fighting against the flow than in previous incarnations, despite the refactoring tools and examples I had available, and it was a lot harder to keep the program flow in my head, since I had to ensure I understood the context that the callback was operating in : which paths could lead to the current code being executed and what I could rely on being set. I tried to trim the code back as much as possible, but I was still debugging a lot more than I was used to in other incarnations, despite more testing.

What Node.JS does to make you grind your teeth

At this point, I’ve written two proofs-of-concept in Node.JS, and whilst I think the 2 projects I tried weren’t the optimal projects for Node.JS, I’m getting a feel for what is can and cannot do. I can see places where I would use it if I was doing streaming or similar low-latency, high-throughput tasks that were just about routing bits, and I have watched it updating several clients at once, but it’s very easy to write blocking code that will slow the server, and has an instant impact on all clients, making them all hang until the blocking operation is completed. That type of error is not unique to Node.JS, but I found the chain of callbacks increasingly more difficult to reason about, making that type of error more likely. It feels like writing GOTOs.

At this point, I can’t see myself using Node.JS for anything other than a very thin layer to some business logic, and at that point, it seems odd to use a thin layer of web services just to call other services in another language. That’s what I’d do to start migrating out a legacy app, but I wouldn’t start a design that way. If I wanted to build a web service backend, there’s no benefit in Node.JS that I couldn’t get in WebAPI. However, I’m wondering whether my next project should be to re-write the backend in Go, to see if that’s any easier.


Node.JS is fun to play with as a proof of concept of callback-driven code, but I’ve seen the basic idea of a stripped back, high performance web server delivered far more elegantly in Erlang, Go, and other places. Node.JS throws out threads, and doesn’t offer enough in return to justify the switch for me. It’s fast, but not the fastest, and my productivity is definitely lower in Node.JS than it was in my first program in C#, or Java, or Python, or even my first work with client-side JavaScript. It’s an interesting experiment but I’ve got better things to do with my time than reason about which callback failed to trigger. For the right project, I’d give it another run, but the right projects for Node.JS are very specialised.

code development programming

DDD Scotland 2011 Voting and Teasers

DDD Scotland voting With voting now open for Developer Developer Developer Scotland 2011, I thought it was time to give everyone a teaser of the 3 talks I submitted to inspire you to vote for me πŸ˜‰ – but be sure to vote for some of the other sessions too, there’s a great line-up and some tough competition for speakers places.

HTML5 for mobile

HTML5 Canvas : Card back
A simple HTML5 game in 1 hour. Covering real-world usage of canvas, CSS3, ECMAScript 5, and a few other goodies. I’m not going to say too much, except that the image here is rendered via canvas (which neatly, in Firefox, offers a “Save As…” png option in the context menu). This canvas works in Firefox, Chrome, Android and iPhone, and possibly others.

The talk will be given using Firefox, but participants will be able to follow along on their Androids and iPhones (well, it’s one way to get people off Twitter πŸ˜‰ – @craignicol btw, if you hadn’t guessed)

If angle brackets or JavaScript scare you, this is not the talk you are looking for.

Genetic Algorithms

Lights Out
Want computers to solve problems themselves? Maybe find a solution to light out like the game on the left?

I’ll run through the theory and a bit of practice, using the game above as an example of how to encode a problem to be solved by a Genetic Algorithm.

There may be some C++ (yeah, I am old skool πŸ˜‰ )



IronPython has been around for a little while as an official .Net language. Whilst many .Net developers are fleeing to Ruby on Rails, wouldn’t it be nice to program in a dynamic language using the libraries you already know? Wouldn’t you like to write the same program with much less code. And wouldn’t you like to see how a language where everything is dynamic really works?

code development programming

DDD9 Apologies

After all my excitement at being accepted in my last post, it is with a heavy heart that I have to apologize that I will not be speaking at DDD9 this year. Many thanks again to all of those who voted for me, and I am very sorry I will not be there to speak.

For those of you who did vote for my talk, please have a look at the video of my talk from BCS Aberdeen and please ask here or on Twitter if you have any questions and I will do my best to answer them in a future blog post.

Many thanks for your support and hope to see you at another conference soon.

code development programming

Developer Developer Developer 9 : HTML 5

I’ve been selected for DDD9 to give an updated version of my HTML 5 talk. Many thanks to everyone who voted for me. I hope it goes smoother than Dublin (I won’t be using the pink netbook for one thing).Β  Unfortunately the registration is now full, but if you’ve already signed up, I’ll see you there.

Now. What am I going to submit for ddd Scotland in May?

development programming

HTML 5 in Aberdeen

I recently gave my html5 talk in Aberdeen for the local BCS branch. There were lots of good discussions, and most of them are on the accompanying video at which also includes a great list of HTML 5 and CSS 3 resources.

Many thanks to everyone for making me feel welcome and for your fascinating questions and insights. I hope to return with another talk.

beta code development lifehacks programming

Html5 (.1) : the language of the cloud?

I’m back in Scotland now, so I’ve posted up my updated presentation for Techmeetup and DDD Dublin. This presentation should work in Firefox 4 (and 3.5 up with html5 turned on in about:config) and will mostly work in Chrome, although you will need a microformat reader such as Tails Export for Firefox. If anyone can test this on Safari or can recommend a Chrome extension, please comment below and I’ll add it to my follow up post when the talk hits Channel 9.

For those of you in Dublin who want to see the video that refused to play during the session, here it is :

As before, I would like to acknowledge the following that I used in preparing my presentation:

I would also like to thank the organisers of Techmeetup and of DDD Dublin for all the hard work, particularly for DDD where the Irish audience were unfortunately thin. I hope all the attendees will spread the word for next year.

A few things I learnt from these two sessions :

  1. The smaller room and audience in Techmeetup were far more conducive to audience participation. The room in Dublin was a little intimidating, and a lot of the audience were strangers, which doesn’t happen much at Glasgow events these days.
  2. If you want to connect a widescreen netbook to a 4:3 projector, don’t expect it to know what to do. Set it up manually or spend 15 minutes with 2 screens that cannot sync and needing a reboot.
  3. Make sure your backup is compatible with at least one other machine (hint: Windows or Mac are more popular than Linux)
  4. If all else fails, make sure you have a video of the presentation to hand.
code development programming

HTML5 and the Beta Browsers

I’ve dug out my HTML5 talk again, as I’ve been invited to present at TechMeetup in Glasgow on Wednesday. As I now use FireFox 4 Beta as my main browser (App tabs and Panorama are too good to give up), I’ve been trying to get my presentation, written in HTML5, working on it, and on the Internet Explorer 9 beta. I also tried the it on the latest release versions of Chrome and Opera. Oddly, Opera is the only browser not to display anything, given that the S5 presentation framework I use is ultimately based on an Opera demo. This may be something I have broken in the template. Opera is stuck playing the loading screen.

The presentation (that I will upload after the talk) now validates according to the W3C validator. Chrome, Firefox and IE all display the content, and all hide content when moving to the next slide, but Chrome and FireFox format the content to fit on the screen whilst IE simply lists the content. The CSS3 demos do not work in IE, suggesting that CSS may be part of the problem, although the CSS support in IE9 is light years ahead of previous versions. Certainly it appears that the layout model on IE is different enough from that used by Chrome and Firefox to merit having to redesign any site for IE. Since the w3c does not publish a rendering validator, I will not comment on which one is most standards compliant. I will merely say that even in the brave new world of HTML5 and CSS3, where rendering is more strictly defined, web development will still need to be done to the lowest common denominator.

Of the new HTML5 features in my demo, FireFox 4 has some changes to svg handling which interferes with the inline svg in my presentation, and has tidied up the UI for features such as geolocation from the previous version. Other parts of my presentation are held back by the lack of FireFox 4 support for some of the extensions I use. In particular, certain aspects of FireBug and Web Developer I use for live editing either do not work at all or are unavailable for local pages (I’m not sure which). IE follows Chrome and FireFox in not offering support for the new form input types, in partcular date and datetime. Given the problems datepickers are giving me in my day job, this is a big disappointment to me. I notice that IE, like Chrome does not animate PNG files such as the ones here, but unlike Chrome, also has the wrong background colour. IE is also the only one of the 4 browsers not to support the Mozilla <video> demos – although youtube html5 does appear to work, so this could be a codec issue. I don’t currently have access to Safari to see how well that does – I suspect it too will fail.

IE9 does have some good debugging support through its F12 Developer Tools, and, whilst the add-ons catch up, IE9 is easier to use to debug web pages than FireFox 4. Browsing through remy’s html5 demos, it looks like content-editable is more persistent than in should be in IE, because editing that page seems to persist the lifetime of the tab. Still, that’s why it’s a beta. I’m also disappointed to see that the Offline Application Manifest is not supported, as that would seem to be the perfect partner to the pin-to-taskbar functionality. Still, I’m happy to see support for canvas and inline svg, particularly given FireFox’s problems, although I hoped for a higher score on the HTML5 test (where it gets less than 100 compared to 159 for Opera and over 200 for Firefox and Chrome) and the ACID3 test (where 95 beats FireFox on 91 but falls behind Chrome and Opera on 100).

Now that I’ve finally got my hands on IE9, I’m less excited by it than I am by Chrome and FireFox. It looks as if it will still need plugins (Silverlight?) for rich internet applications, meaning that developing a full cross-platform web application means either limiting functionality for certain browsers (which will need to be done for IE6 for a while yet anyway), or writing a plugin for IE to bring it up to the level of the other desktop and smartphone browsers.

Given the pace of releases, I’m surprised how little new I can add to my presentation. Whilst every browser is moving forward, the baseline standard still appears to be where it was, there’s no cross-browser video format, no cross-browser forms support (input types or validation), no cross-browser Web Application support, no cross-browser geolocation support and no cross-browser CSS transforms. In fact, the canvas and structural elements seem to the only features in my survey with cross-browser support, and given the poor showing of canvas in the results, it’s going to be a long road to adoption, unless you’re in the mobile space.

code programming

HTML5 : The language of the cloud

I had a great time at DDD Scotland, so many thanks again to the organisers and everyone who came. As promised, here are the slides for my talk : HTML5 : The language of the cloud? in HTML5 format. The talk was given in Firefox 3.6 with the MiniMap (geoDiscovery) and Tails Export (microformat) extensions. If you have to make any tweaks to get it working in your favourite browser, let me know and I’ll update the file.

For a bit of fun, I’ve put together a short survey on HTML5 to gauge responses to some of the discussions I’ve had since the talk. I will publish the results in an upcoming post.

Finally, for ease of access, here’s a list of the important links from the talk for those of you who missed them:

And here’s Google’s demo using the dev version of Chrome :

code google programming

The Cloud Promise

Having submitted a talk on html5, and calling it “The Language of Cloud Computing” (please vote for it if you are interested), I thought I should take this opportunity to discuss how I see the possibilities of the cloud. I do web development in my day job and we use some of the technology that is now discussed as cloud technologies, but this is my perspective as an end user. I will put a warning here, that there is a bit of blue-sky thinking in this post.

My original germ of this post came after reading this post by Gary Short: Cloud Computing – It’s that New Old Thing – Gary’s Blog and as I’ve been writing this post, I think that he’s the angel sitting on one shoulder telling you to be careful, to watch out for snake oil and to mind the gap, which makes me the devil on the other shoulder telling you to go on, jump in, the water’s lovely, and what could possibly go wrong? With that in mind, I recommend you read his post and keep it in mind.

There are two distinct shifts I have seen that are classed as “Cloud Computing”. On the server side, machines are becoming virtualised which allows for greater flexibility and a more efficient use of resources (go green rangers). I use VMs all the time, and I think they’re great. Of course, mine aren’t hosted across the world, like Amazon Elastic Compute Cloud (Amazon EC2) or Windows Azure. On the client side, it’s about the trend to save your data “somewhere else” so you don’t need local storage and you can access your data from everywhere. That seems to be the underlying vision of Google Chrome OS and the Apple iPad and it has a certain appeal when you don’t have to worry about backups, running low on disk space, and other maintenance tasks. Why not give it to someone else to worry about? And if that’s not enough, put your photos on Flickr!, edit them on Picnik, and then share them on Facebook, giving a lot more options than you could have on your own, and allowing others to manipulate, improve, and remix your data.

But what if you pick the wrong service? Picnik currently doesn’t work with Windows Live Photos so you miss out on that service unless you upload the photos from your computer. And what happens when you’ve been using Yahoo 360 for blogging and it disappears from the web?

What we realy need is some proper data portability. You don’t want “Facebook compatible” or “Works with Flickr”. Google Reader is great because it works with RSS, and there’s a lot of RSS out there to work with. We want something as simple as “Yes, it’s a JPG, I can work with that”, just like I can on my computer. And more than that, we need to be able to move our data from one cloud provider to another.

My ideal model for how “The Cloud” should be working, for me, would be that every cloud provider would support standard data transfer protocols, whether it’s ftp, imap for email, or something new, that supports open authentication to transfer directly between providers as well as to my local storage. The key is making it easy to synchronise my data. Yes, some of it will be world-accessable, with all the privacy issues that presents, and yes, some providers will be better at some things than others, but making it easy to transfer my data between providers means that I can move around to get the best of all worlds, instead of choosing this place for greater storage instead of that one that lets me share, or the other one with better editing. There’s loads of people who make it easy to get your data in, they support multiple clients, have a nice web uploader, annd so on, but it’s often a lot harder to get your data out. The Data Liberation Front (the Data Liberation Front) is a nice step in the right direction from Google, and I think is a good, basic philosophy for Cloud providers to follow, but that should be the bare minimum I should expect from a provider. If they don’t follow that model, they are not “The Cloud”, they are just silos full of quicksand, and the more you struggle to get you data out, the more it seems to be lost forever.

So, will html5 help with any of this? AJAX has become a reason for providers to become more open, to encourage traffic back to their own sites, and adverts, but with html5, we get, for example:

  • microdata, to help define areas of pages that other sites can understand, such as calendar events and contact details;
  • structural elements, like nav, to help parsers find the interesting data;
  • Cross-document messaging to allow pages from one domain to communicate with pages on another domain, with a security model to prevent XSS attacks; and
  • protocol handler registration to allow a website to declare that it can handle fax: or mailto: links, or jpg or apk files so I can grab a link or file from one site and sent it directly to another site that I trust.

I don’t know how many cloud providers will start to use these features, but until they do, the cloud is stunted.

Blogged with the Flock Browser