code development programming

Software Development as a Christmas Story

Gingerbread Christmas Tree
Oh Tanenbaum

We decorated the house for Christmas. A smaller project than most of the software I’ve worked on, but a useful reflection.

The Cost of context switching

Sometimes, in the middle of one task, you need to do another. Either because you were interrupted, or because you weren’t prepared. Consider the tree, a tall tree, one you need a ladder to decorate. But you left the ūüĆü in the attic.

You don’t just have the cost of picking up the star, you need to get down the ladder, fold it up, carry it upstairs, open the attic, unfold the ladder, climb to the attic, find the star, then pick it up and unwind all the other steps just to get back where you stated before you can continue the task of putting the Star on the tree.

It’s just a minute to get the star…

Yak shaving

Sometimes it’s more than just one thing. Sometimes to get to where you need to go, there’s a cascade of other tasks.

You want to put the tree up, but it’s the first time you’ve had a real tree, so you need a new base. It’s bigger than last year’s tree, so your lights don’t fit. So you need to go to the shops. But you need to fill up your car. And the tyre pressure warning light comes on so you need to top them up. And you need to check the tread depth whilst you’re there, and so on.

Programming in a nutshell

Performance at scale

Our tree stood up fine when it was delivered. But as it scaled out and the branches widened, it pushed against the wall, making it unstable in the condensed space. It fell over.

Luckily no lights or baubles were using it at the time, but it’s an interesting challenge holding up a heavy tree in one hand, trying to adjust it’s position with the other hand, as I avoided the puddles of water in the floor as my wife mopped them up. If you’ve ever worked support, this may sound familiar.

Turns out that it was harder to stabilise than I anticipated.

The branches were unevenly partitioned, providing more load on one side, so I had to stabilise it against the wall. And the tree was almost a foot taller than expected, which turned it out to be 2 foot taller than the base was rated for.

We upgraded the base to handle the load. It’s bigger than we need.

Technical debt

I have some old pre-LED tree lights and they slowly fail so each year I replace the unlit ones from the packet of spares but I haven’t been replacing the spares. Eventually, they will run out, and the spares will no longer be available. I’ll have to throw them all out and start again.

The new led ones don’t let me replace them individually. But they last longer and are cheaper to run.

Which debt is easier to live with?

With a big tree, those old lights aren’t long enough. So, do I buy another set for the rest of the tree that doesn’t match them, or throw out the existing ones and buy a longer set? The latter looks better, but throws out the sunk cost along with the lights.

You know computers, can you look at this for me?

No, I can’t fix your tree. I can navigate a garden centre. I know enough physics to keep a tree upright, eventually, and safely put the angel on, but I’ve no Idea how to grow one, or what that weird stand with 17 gears you bought does, or how to assemble your plastic one. And I’ve no idea what that bug in the tree is.

Merry Christmas. Catch you all in the New Year.


Reducing waste by making your goals visible

Metrics are good, but they’re not enough. If you want metrics to drive change, they need to be visible. Not just to managers, but to the team. The right metrics, visible to the team, drive improvement. Visible only to management, drive control. Visible to no-one, drive nothing.

Good metrics need to be communicated clearly and persistently. There needs to be a continual focus on whatever the priorities are. If it’s out of sight, it will be out of mind. If it’s digital, rescue it from individual screens that are easily covered up.

It’s why kanban boards can be so effective when used well. It’s easy to see when the team is taking on too much work, or where there’s a queue waiting for tasks to be pulled. The board radiates information to the team, about how much work is in progress, and who is working on what. It becomes the centre of communication about tasks and removing waste. Make it physical, if you like, and the team is co-located. Use Post-its. Let dog ears and grubbiness indicate age. Do your backlog pruning via environmental effects – older & more used notes are more likely to fail behind the radiator and get lost.

Put (work-related) personal tasks on there too. Keep yourself honest. It’s good for the team to know you’re reviewing CVs, or preparing for that strategy meeting, or learning Haskell the hard way.

But keep it clear. Use a Kanban if WIP or queueing are your key metrics. Use Scrum if velocity and deadlines are key. Use a dashboard if that makes your priorities and progress clearer.

And make sure your key priorities (4 or less please) are visible at a glance for the team, and easy for anyone in the team to explain to a passerby. They don’t need detail, but pointing to a big pile of waiting tasks to justify recruiting a new tester is a clear message.

development leadership

Measuring the wrong thing

Process improvement requires measurements. How do you know what to improve if you can’t see where things aren’t working, and how do you know you’ve made the right change without seeing those numbers going in the right direction?

But measuring doesn’t mean you’re measuring the right thing, and measuring the wrong thing can lead to the wrong outcomes, and can be very harmful (see Liz Keogh’s talk on perverse incentives¬†)

The key to the success of any metric is that it is owned by the team so they have complete control over the changes needed to affect it, so they feel ownership, and that improving the metric has a useful business outcome.

Metrics that I’ve found useful in the past include:

Number of bugs introduced by a release.

This is a tricky one to work with because it can easily be a perverse incentive, and the feedback cycle can be slow, especially with the usual 6-12 month release cadence. However, on one waterfall project I took over there was a strong negative perception of the quality of the code, and big count was a useful proxy as the customer had access to JIRA so the big list was already visible. Reducing this number was a combined effort where testers and developers had to approve requirements, developers and testers invested in automated testing at all levels, and testers joined the developer daily stand-up in order to catch bugs before they were released “have you checked that page in Welsh?“, “We found problems on that page last time because IE was too slow“.

Number of releases per month.

On an agile project we noticed that releases were getting held up because testing was slow and produced a lot of rework, which were then tested against more features that had been pulled from the backlog, increasing testing time. Each release also took half a day, so we had to schedule them carefully.

So we set a goal of focusing on releases for a month and measuring how many we did. There were other measures within that, such as monitoring work in progress on the testers, time taken to do a release, and cycle time from start of development to live release, but they all drove the big number, visible to the whole company, of more quality releases.

Questions answered per day.

This can be a very useful metric for research projects, especially when following a fail fast approach, when you want to prove something can’t be done before you invest in doing it. In order to do that, you need lots of questions with quick answers, to accelerate learning. Any answer, positive or negative, is progress. It means we have learned something.

“I cheerily assured him that we had learned something.
For we had learned for a certainty that the thing couldn’t be done
that way, and that we would have to try some other way.” – Thomas Edison

Age of Pull Requests

Successful agile projects rely on peer review, either via pairing, or a formal review process such as git PRs (and I won’t discuss that in this post). However, when we started working with these on one project, we discovered that we were building up a large debt of Work In Progress because the team wasn’t incentivised to review each other’s code, so one of the developers set up a nag bot for any PR older than 4 days. It lasted 2 weeks before it was no longer needed.

What about you?

What metrics have you used to incentivise the team? What problem were you trying to solve, and did the metric work?

data security

Privacy is not your only currency 

If you’re not paying, you’re the product.

But you’re not. In security, we talk about 2-factor authentication, where 2 factor is 2 out of 3 : who you are, what do you know, and what do you have. Who you are is the product, a subset of a target market for advertising, or a data point in a data collection scoop. The former requires giving up privacy, the latter less so.

Advertising is about segmenting audiences and focusing campaigns, so views and clicks both matter, to feed into demographics and success measures. Ad blocking is a double whammy – no ads displayed, and no data on you. Websites tend to argue that the former deprives them of revenue, many users argue that the latter deprives them of privacy.

What you have is money, and who you are is part of a demographic than can be monetised in order to advertise to you to get your money.

But what else do you have? If you’re on the web you have a CPU that can be used to compute something, whether it’s looking for aliens or looking for cancerous cells. If you’re happy to give up your CPU time.

Who else are you? You might be an influencer. You might be a data point in a freemium model that makes the premium model more valuable (hello, LinkedIn).

What do you know? If you’re a human you know how to read a CAPTCHA (maybe), you could know multiple languages. Maybe you know everything about porpoises and you can tell Wikipedia.

Your worth to a website isn’t always about the money you give them, or the money they can make from selling your data. It’s the way we’ve been trained to think, but there’s so much else we can do for value.

development leadership

Bad change : the hokey cokey requirements

It goes on and on and on

When I said I don’t trust change, there were a number of situations I was specifically thinking about, changes that actively cost time and money and decrease business value. One example of this is what I call the hokey cokey requirements.

You can identify these where the business value of the requirement is unclear. Where the product owner is making decisions on behalf of a user without verifying them, or not considering the impact on other requirements, for example, how changing a date for one screen will cause a dashboard somewhere else to report additional failures.

If you’re lucky, you’ll identify the requirement before you start implementation, but more likely, it’ll be in until you release to test or even live, and you’ll immediately get a change request to take it out again.

If you’re really unlucky, you’ll then end up in a tug of war between 2 teams.

How can you identify them in advance?

For some systems, you’ll have a well defined dependency graph between requirements, or at least enough prep time to generate the dependencies from the code prior to accepting the requirement.

For others, you’ll need a big user group to cover the edge cases, and accept that big user groups are not a comfortable fit for agile development.

How do you deal with them when you find them?

Make sure the right users are involved, at design and test. And make sure there’s no hierarchy to invoke. What works best for the CEO might break the workflow for the administration team. On the projects I tend to work on, keeping the administration teams happy are the key to a successful delivery and simpler support.

Sometimes the smallest possible change is best. Don’t build the full solution, build just enough for all sets of users to see the implications.

Make sure you understand the impact, so you can explain it clearly.

But most of all, if you see it happening, try and build a ripcord into the code to pull it out again, like a feature toggle, because despite best efforts, you may still hear “I know that’s what I signed off, but I didn’t expect it to do that!”

How have you dealt with hokey cokey requirements?

code data programming ux

#dunddd Analyse This : The dangers of big data

Thanks to everyone who came to my DunDDD talk. Lots of interesting questions, although I’m not a lawyer so couldn’t answer them all.

If you want the slides, with references in the notes, you’ll find them here. All the images are creative commons, and you can use the sides yourself under CC by Attribution. Link to slides : Dunddd Analyse This – The Dangers Of Big Data (Google Drive)

If you missed the talk, the arguments I made and the references, apart from the privacy sections, are in this

Link to previous post

If you want the references for the Personal Data and anonymisation parts, have a look at these :

AOL searches are not private

IBM privacy-preserving data mining