Software Development as a Christmas Story

Oh Tanenbaum

We decorated the house for Christmas. A smaller project than most of the software I’ve worked on, but a useful reflection.

The Cost of context switching

Sometimes, in the middle of one task, you need to do another. Either because you were interrupted, or because you weren’t prepared. Consider the tree, a tall tree, one you need a ladder to decorate. But you left the in the attic.

You don’t just have the cost of picking up the star, you need to get down the ladder, fold it up, carry it upstairs, open the attic, unfold the ladder, climb to the attic, find the star, then pick it up and unwind all the other steps just to get back where you stated before you can continue the task of putting the Star on the tree.

It’s just a minute to get the star…

Yak shaving

Sometimes it’s more than just one thing. Sometimes to get to where you need to go, there’s a cascade of other tasks.

You want to put the tree up, but it’s the first time you’ve had a real tree, so you need a new base. It’s bigger than last year’s tree, so your lights don’t fit. So you need to go to the shops. But you need to fill up your car. And the tyre pressure warning light comes on so you need to top them up. And you need to check the tread depth whilst you’re there, and so on.

Programming in a nutshell

Performance at scale

Our tree stood up fine when it was delivered. But as it scaled out and the branches widened, it pushed against the wall, making it unstable in the condensed space. It fell over.

Luckily no lights or baubles were using it at the time, but it’s an interesting challenge holding up a heavy tree in one hand, trying to adjust it’s position with the other hand, as I avoided the puddles of water in the floor as my wife mopped them up. If you’ve ever worked support, this may sound familiar.

Turns out that it was harder to stabilise than I anticipated.

The branches were unevenly partitioned, providing more load on one side, so I had to stabilise it against the wall. And the tree was almost a foot taller than expected, which turned it out to be 2 foot taller than the base was rated for.

We upgraded the base to handle the load. It’s bigger than we need.

Technical debt

I have some old pre-LED tree lights and they slowly fail so each year I replace the unlit ones from the packet of spares but I haven’t been replacing the spares. Eventually, they will run out, and the spares will no longer be available. I’ll have to throw them all out and start again.

The new led ones don’t let me replace them individually. But they last longer and are cheaper to run.

Which debt is easier to live with?

With a big tree, those old lights aren’t long enough. So, do I buy another set for the rest of the tree that doesn’t match them, or throw out the existing ones and buy a longer set? The latter looks better, but throws out the sunk cost along with the lights.

You know computers, can you look at this for me?

No, I can’t fix your tree. I can navigate a garden centre. I know enough physics to keep a tree upright, eventually, and safely put the angel on, but I’ve no Idea how to grow one, or what that weird stand with 17 gears you bought does, or how to assemble your plastic one. And I’ve no idea what that bug in the tree is.

Merry Christmas. Catch you all in the New Year.

Sense and Respond by Jeff Gothelf and Josh Seiden

Sense and Respond: How Successful Organizations Listen to Customers and Create New Products Continuously by Jeff Gothelf

My rating: 5 of 5 stars

Having read Lean UX, I was expecting a good book on how organisations could deliver projects. This book is about much more than that. The authors are coming from a point of frustration where agile delivery and Lean UX projects have failed to reach their potential because the organisations surrounding them were not supportive, so this book is about how to reshape an entire organisation to meet the challenged of the modern world, with lots of positive and negative examples, which is refreshing to see in this type of book. If you’ve ever had a strategy meeting, or ever found your project success limited by your organisation, go read this book.

Easily the best book about dysfunctional organisations and organisational change I’ve read since Peopleware.

View all my reviews

Buy on Amazon UK :

Reducing waste by automation

Reducing waste is one of the key concerns of agile development, and is a defining character for Kanban, where blockages are ruthlessly identified and resolved. Timeboxing provides a low-cost means of identifying waste and a framework for tackling it, but doesn’t provide many solutions on its own.

For tasks that continually cause blockages, such as the release process, or testing, automation is a good way to eliminate the repetition, and the chance of human error. Indeed, the engineers in the audience will immediately recognise the key drivers for Continuous Deployment and Continuous Integration in the examples I’ve given.

And yet, whilst we appreciate the gains it can make, how many teams set aside specific time in their schedules for “automation” as an activity alongside “refactoring”, “upgrading dependencies” and “deleting dead code (including tests)”.

Check your backlog, and if automation, in some form, isn’t there, ask yourself why those repetitive manual steps still exist in your workflow and why they have priority over the other work you have to do.

What engineers want

How to keep engineers interested, and understand the people the company is so keen to tell us are the core of the business :

• engineers love problems, keep the work interesting;
• engineers are creative. Give them space to do so;
• engineers need autonomy to do the above;
• we do the work we do because no-one else has solved the problem before. That means we fail sometimes. We need the freedom to fail and recover, because sometimes new things don’t work;
• engineers love creating value. If you want to demotivate them, give them meaningless work that no-one will see. We want projects that people will use.

Thinking in Code

What is coding? The practice of breaking down big problems into very small problems. Turning everything into logic : recipes and decisions. Taking the fuzziness and ambiguity of the world and human language and turning it into something definitive and concrete, something that can be reasoned about.

And then you start to notice things about human behaviour and the processes they generate. With the exception of a few professions, human processes are hard to reason about. They’re designed with ambiguity built in because it’s not human nature to dive deep into every problem.

Have a principle, not a process “we need to be more secure than that dating site”, “users need to be able to manage their privacy settings easily”. Every process you bake into the code will support the principles, if you have the right conversations. But how does it deal with the outliers, the user with 1 name, the one organisation with 1 million staff, the one day the stock market price changes by 60% rather than 0.6%?

For Software to be repeatable and reliable, we have to remove flexibility. How do you delegate to humans?

I’ll come back to that in a later post.

DRM and plastic knifes

I tried to do something I wasn't allowed to do but they refused to tell me what I'm not allowed to do.

I thought Digital Rights Management was dying when iTunes no longer required it. I was wrong. It’s sneaking into HTML, it’s in coffee machines, it’s being discussed for JPEG, it’s led to the introduction of boot lockers to prevent users from modifying the operating system on their machine.

So what? Surely copyright holders have the right to protect their content, no matter that the entire music industry could be bought out with Google or Apple’s spare change. Surely they need the money to support the poor struggling up and coming artists, who they can’t afford to pay from streaming revenues. Surely we should just let Hollywood have their way?

And anyway, doesn’t restricting operating systems and the software we can install protect us from the bad guys? Apple will keep us safe in their protected iOS and App Store, right?

When you restrict tools via drm, only the professionals can use the full tools, but where do the next professionals come from? We’re suffering from a shortage of people who want to learn to develop software, and we’re putting extra barriers in place, so that we need specialist devices for teaching development in the form of the Raspberry Pi and others, because the main machines we use are locked down and restricted.

Knives are dangerous and they can kill, so Scotland has restrictions on their sale and carrying them, but we still teach children to cook, sharp knives and tools for sharpening them are still widely available, and you don’t need to be a licenced chef to use a real knife.

If we can trust society enough that we don’t need plastic knives outside preschool, fast food vendors and cheap airlines, why can’t we trust them with access to a fully functional computer?

Why can’t the IT industry deliver large, faultless projects quickly as in other industries?

Glasgow Tower

The title and inspiration of this post is an old question on StackOverflow : Why can’t the IT industry deliver large, faultless projects quickly as in other industries? – Programmers

There is a continuing question of why IT consistently fails to deliver large projects, when other industries such as construction, civil engineering, and aircraft companies consistently deliver on time and to budget, and never have any problems in their first few years. Just ask anyone in Edinburgh about the trams.

However, there are a few things that make software projects more likely to fail, as I see it, throughout the process, and the successful methodologies recognise and address these problems directly.

The first key difference I see is best demonstrated looking at architecture vs IT. I’ve seen a few design competitions for key projects, and the bidding always involves paper or 3D-rendered models of the final structure, with lots of trees, and several people milling about, looking happy. It’s been very rare for me to see that in a software bid, and that’s probably a good thing. Aside from some rough sketches of UIs, what really matters is the relationship between the developers and the customer, because software changes dramatically according to use, especially after first use when the users start to see what’s possible rather than just talking about it.

The buildings we see are not version 1. Before the models in the bidding stage, there may be sketches, and after the models may come prototypes, scale models, physics simulations, walkthrough renderings, and many other versions iterating towards the final design that actually involves tonnes of steel and concrete driven into big holes in the ground.

Software is version 1, or maybe version 2. The design is executable, and malleable. Code can be used to simulate itself via test frameworks. Software is the best model for software, after all simulations such as paper prototypes are doomed to succeed, because they won’t have real world data with apostrophes in names, they won’t have anyone living in Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch, and all network interactions will be instantaneous.

Every model and sketch built before a building is a level of abstraction that considers a subset of the details of the finished product. In software, everything is done at the same level of abstraction, the production code, the unit tests, the integration tests, the behaviour driven tests, the factory testing, are all done on the same business logic, and often in the same language, so the design is the product, and if the design is wrong, the product is wrong, and often the only way to test the design is to deliver the product. Users are not going to care about curly braces and angle brackets. They care that hitting that button does the right magic to send that email. If the design is wrong, then the magic is wrong, and the user is disappointed. So we iterate, we gather feedback, and we improve, step by step, polishing the magic until the experience shines.

And that’s what other industries do, whether we admit it to ourselves or not. Walls in buildings are knocked down, offices are reconfigured, and the Boeing and Airbus planes are improved in iterations. Carriers are offered new seat styles, and get offered stacking accommodation, flight navigation systems are removed and upgraded, and so on. Improvements are made around an expensively tested and maintained core, which improves at a slower pace, because the gap between design and implementation is large, and the feedback cycle is very long, although it’s getting better, at least for architects.

Is software uniquely complex? Are software projects too large? No. But the nature of software puts us in a much tighter feedback cycle between design and code. That’s what the agile manifesto cuts to at its core. We want to test our designs, but the best way to do that is to implement them and get them in front of users, and then refine them. Upfront design doesn’t work because users understand products, not requirements.

Software can deliver large, faultless projects, but it’s much easier to deliver many smaller, faultless iterations, and take more risks whilst you’re doing so, because losing 1 weeks’ work is a lot less painful that losing a year’s work.