Somebody else’s problem

Sometimes dividing tasks creates inefficiency. When one person both cooks and cleans, they tend to create less washing up than if the tasks are divided, because washing becomes “somebody else’s problem”. However, when one person washes and the other dries, the load is equivalent. This setup can be prone to stoppages when the drainer is empty and the washer is trying to get that stubborn burnt bit off the pan, at which point any good ToC practitioner will ask the drier-upper to help with the washing to maintain throughput.

When teams are divided by layer instead of features, the interface between them becomes a fracture because each team has context missing from the other. Data has the wrong shape, validation requires fields that can’t be captured. If you build both sides, the transition is smooth.

When developers test, they can focus on smaller, independent chunks, which means fewer tests with greater coverage. Writing integration tests for complex calculations is painful and slow. Testing those same calculations as an xUnit Theory is cleaner and faster.

Specialists are great, but where is specialisation causing fractures and more work?

This is raw

This is raw chicken : 🐤

If you eat it like that, you may get hurt immediately, by its beak, or its claws. It may grab your money and run off with it.

If you want to eat it, better to kill it first. 💀

If you eat it like that, you may get hurt or die, in a few hours, or days. Washing it won’t help.

Cook it. Cook it well. If there’s any sign of pink, cook it some more. 🔥

It might still kill you, but at least you’re a lot safer than when you started.

---

This is raw data : 🐤

If you display it, users will get hurt immediately, whether by cross-site scripting, cookie sniffing, crypt-currency mining, or something else. If you’re lucky, it will be something your user’s see immediately and leave your site never to return. Otherwise they may get infected.

If you want to use it, better to validate it first. 💀

If you save it like that, your users are still vulnerable. It might appear on the front end in a different form. It might be a string of unicode characters that crashes your phone. It might be a link to somewhere they can’t trust.

Encapsulate it. Sandbox it. Never trust it, in or out . HTML encode, whitelist the output as well as the input.

And if you need to avoid spam, or incitement, or solicitation, maybe you need editors. Computers can’t fix all the social problems. 🔥

NMandelbrot : running arbitrary code on client

As part of my grand plan for map-reduce in JavaScript and zero-install distributed computing, I had to think about how to gain user trust in a security context where we don’t trust the server. I couldn’t come up with a good answer.

Since then, we’ve seen stories of malicious JavaScript installed to mine cryptocurrencies , we know that JavaScript can be exploited to read kernel memory, including passwords, on the client, and I suspect we’ll see a lot more restrictions on what JavaScript is allowed to do – although as the Spectre exploit is fundamentally an array read, it’s going to be a complex fix at multiple levels.

I had ideas about how to sandbox the client JavaScript (I was looking at Python’s virtualenv and Docker containers to isolate code, as well as locking them into service workers which already have a vastly more limited API), but that relies on the browser and OS maintaining separation, and if VMs can’t maintain separation with their levels of isolation, it’s not an easy job for browser developers, or anyone running on their platform.

I also wondered if the clients should be written in a functional language that transpiled to JavaScript, to have language level enforcement of immutability and safety checks. And of course, because a functional style and API provides a simpler context to reason about map-reduce, by avoiding any implicit shared context.

Do you allow someone else’s JavaScript on your site, whether a library, or a tracking script, or random ads from Russia, Korea, botnets and script kiddies? How do you keep your customers safe? And how do you isolate processes you trust from processes that deal with the outside world and users? JavaScript will be more secure in the future, and the research is fascinating (JavaScript Zero: real JavaScript, and zero side-channel attacks) but can you afford to wait?

Meltdown and Spectre shouldn’t change any of this. But now is a good time to think about it. Make 2018 the year you become paranoid about users, 3rd parties and other threats. The year is still young, but the exploits are piling up.

 

De-pluralisation: strategic blinkers

The process of de-pluralisation takes all your existing problems and combines them into one simple manageable problem: “it’s JavaScript”, “it’s Windows”, “it’s the CDO” with a simple manageable solution “kill it”. Like all simple solutions it’s almost always wrong.

“People aren’t complying with our data quality. Users keep putting in wrong email addresses.”

“The new computer system will fix that”

“Users complain that there’s too many steps to sign off expenses”

“Each step will be faster in the new computer system”

“Staff have low job satisfaction”

“New computer system!”

“Our users aren’t interested in that new feature”

“New computer system!”

“We’ve been hacked with a social engineering attack”

“NEW COMPUTER SYSTEM!”

Changing one thing is rarely the answer. There are many pain points and problems we all face day to day. It’s tempting to try and fix them all together, but unless you understand why the problem exists and what the parameters are to fix that problem, you can’t fix it.

Sometimes New Computer System (TM) will fix multiple problems, if they’ve been defined and the system has been designed to do so. But don’t expect it to fix everything because it’s new. That’s how to make people disenchanted.

If you think one system will rule them all, you may as well throw your team in the fire now before you waste time and money on a misguided transformation.

The importance of confidence

Have you ever been given a project to deliver and thought you can’t do it? That you don’t know enough, that the deadlines are too tight?

Ever felt like panicking?

Have you been on that team, and somehow still delivered, with absolute pride in what you’ve delivered, bar a few teething issues?

As a leader, to inspire your team, you need to exude confidence. Confidence in the outcomes, and in the team. You don’t deny problems or paper them over. Hiding is a sign of weakness. But the team needs confidence to deliver and they get that from you.

If you’re not confident, don’t fake it. Revise the outcomes to a scope you’re confident in. Narrow your horizon. Get answers to your doubts, and support to conquer them. When times are tough, it’s your job to set the right expectations that the team can believe in, and deliver on. It’s your job to find a clear path through the chaos. Or it’s your job to help them pick up the pieces and move on.

These are the tests of your leadership. If the team can weather the crisis, if it can keep it’s collective head whilst all about are losing theirs, then you have earned the right to be a leader.

Be confident. But be authentic.

Sense and Respond by Jeff Gothelf and Josh Seiden

Sense and Respond: How Successful Organizations Listen to Customers and Create New Products Continuously by Jeff Gothelf

My rating: 5 of 5 stars

Having read Lean UX, I was expecting a good book on how organisations could deliver projects. This book is about much more than that. The authors are coming from a point of frustration where agile delivery and Lean UX projects have failed to reach their potential because the organisations surrounding them were not supportive, so this book is about how to reshape an entire organisation to meet the challenged of the modern world, with lots of positive and negative examples, which is refreshing to see in this type of book. If you’ve ever had a strategy meeting, or ever found your project success limited by your organisation, go read this book.

Easily the best book about dysfunctional organisations and organisational change I’ve read since Peopleware.

View all my reviews

Buy on Amazon UK :

You own your dependencies

I mentioned as part of my Your API Sucks series that I don’t want your API to be the weak point in my application. But it runs deeper than that.

Every dependency you add to your project is a codebase whose maintenance schedule you need to know, whose security vulnerabilities you and your customers are exposed to, whose existence you depend on – whether it’s a long established company or a guy who wrote 12 lines of Javascript that everyone uses. Know how you’re going to keep using it when the existing support isn’t there.

Because it’s not just one dependency, it’s dependencies all the way down.

Of course, modern software can’t be built without collaboration, without using dependencies written by others, but where you have a choice, always choose the dependency that works closest to how you’d do it yourself, just in case you have to.

Usable APIs follow-up

Following the Usable APIs guided conversation at CodeCraftConf, I wanted to capture some of the thoughts that came out.

Starting an API (as a user or a developer)

• Does the API documentation include examples of usage (i.e. have they thought about the client)
• How mature is the API?
• How well maintained is it?
• How long does it take to get to the first success (e.g. 200 OK – assuming success doesn’t mean error).
• What’s the versioning policy?
• What’s the contract?
• What’s the shape of the data?

Changing and retiring APIs

• Never, ever, ever, change the endpoint.
• Give as much notice as possible of changes (and never negative notice).
• Provide migration guides to clients, or automation scripts, such as the Python 2to3 migration scripts and guide.
• Be proactive – there was an example given of a web API (I can’t remember which one, sorry) that changed their endpoint, and created a bot that searched Github for usages of the old URL, and submitted pull requests for the new usage.
• Deprecate, then kill, once usage falls below a threshold.

Foolproof APIs

• Provide helpful errors, and treat unhelpful errors as a bug.
• Guide users into the pit of success.
• Follow existing conventions and standards.
• Be consistent.
• Document all the options and overloads.
• Postel’s Law : “Be liberal in what you accept, and conservative in what you send”.
  • But it’s very, very easy to get it wrong. Just look at IE6 and the broken web.

Isolation and proxies

• Log everything to detect unreliability.
• Make sure proxies are kept up to date with the underlying API, and fail gracefully when the API changes.
• Expect failure.
• Data is key – don’t give up more than you have to.
• Make sure, as a server writer you understand the client, and the network.

Developers are Users Too : Why the User Experience of Your API sucks #yourapisucks

Many thanks to those of you who came along to my talk on why your API sucks. There were some great discussions during and after, and I hope I’ll be seeing slightly fewer reasons to tear my hair out in the near future.

A few things that people mentioned that I want to discuss again, as they’re not on the slides.

APIs still need a view model

There was a question about the API view of your data vs the database view. A good API still follows MVC principles. Just because a View is written in JSON instead of HTML doesn’t make it a model. Isolate it, because otherwise a database change is an API change. You can use automappers to save you having to write convertors, but always create a view for your API so you can handle versions and create consistency.

There was also a big discussion on how deep the hierarchy should be, particularly relating to data obesity. And it’s a different tradeoff if you’re optimising for poor latency, where larger packets with redundant informatio make more sense, or timeliness where small packets that can be built and parsed quickly are preferred. It all depends on your users. It may be the case, as the Trello API does, that returning the deep hierarchy is a request option so that the user can decide.

User Experience doesn’t always mean asking the user

Developers will ask for lots of things that aren’t useful. Sometimes you need to discover what the user needs are. Maybe it’s a test-first design. Maybe it’s a loosely typed API to help you discover what users want to do (think Google search vs Yahoo’s hierarchy – Google is more loosely types so was more likely to return odd results, but Google had lots of data on what people were searching for that it could learn from).

Link suggestions from the audience

Thanks for these suggestions. I’ll have to try them out myself, so I’m putting them here without warranty.

Kong – to consolidate your APIs, and create the polished surface that doesn’t expose the dirty, inconsistent history behind it.

RAML – to design a typed RESTful API up front so you know what it will look like.

Heroku’s HTTP API design guide

Slides from Google Docs

Slides from Slideshare

Your API sucks : testing

Finally, you’re ready to launch, you’ve integrated the API, you’ve decoded the documentation, you have passing integration tests, and FAT passed against their test server, with their test data. You go live. And everything breaks.

Strawman testing

Sometimes the test data or the API out of sync with live, and you find you’ve been testing the wrong thing. Maybe the test server is a version ahead of the live API (if they’ve ignored the advice not to version HTTP APIs).

More frustrating is when the data is plain wrong. When the returned record has { Firstname = “Lincoln”, Lastname = “Abraham” }, or an invalid postcode (did you know, The final two letters do not use the letters CIKMOV) which means the test site may require a postcode that your validator will reject.

Tests doomed to succeed

Sometimes the test data is correct, but incomplete. Imagine a postcode lookup where test addresses are all SW… which means you can’t test for:

• Flat x/y addresses
• BFPO addresses
• Irish or other EU addresses

Or imagine a payment processor test stub that always returns success – so you can’t test for declined cards, or 3DS cards, or check that you only accept debit cards.

If your API supports it, put it in your test, and make it available to users for their tests. If it’s not tested, it’s not complete.