2022 reflections

2022 seems to have been a strange year for a lot of people. There’s a lot of bloggers I follow whose output dropped a lot this year, myself included. Some of that I’m sure is a seeming loss of community, with changes to Twitter and Facebook, and I’m sure Google’s AMP as well, there’s been less drive-through traffic and less engagement.

I also think online discourse in many places is following the lines we see in politics where subtlety and nuance are increasingly punished and every platform is pushing shorter form content. We’re not giving ourselves time to digest and reflect.

And we should.

The pandemic is still here, but we’re adjusting, working from home is a natural state for many of us in tech, although that’s not an arrangement that plays to everyone’s strengths, so let’s make space for different companies with different cultures. There’s new ways of working to explore (hello the UK 4 day week experiment), people have moved jobs to take advantage of the change and create more family time.

But we can’t escape the world outside tech, and many of us are burning mental cycles on disease, on the massive weather events from climate change, on war, on the continued assaults by the far right, and watching inflation tickling upwards. It’s not an environment that leads us to our best work. It’s not an environment that helps us be in the moment.

Through 2016-2021 the world stared into the abyss of the rise of the far right, and the dismantling of certainties, before we were all thrown into lockdown. We were hoping for a turning point this year, but our leaders were lackluster in improvements, pulled us further to the right or were just plain incompetent. Instead of hope to counter the dispair, we got indifference at best Rather than turning away from the abyss, we collectively chose to build a car park next to it.

The greatest minds of our generation are building pipelines for ads for things we don’t need and can’t afford, whilst the AI engineers are building complex transformations that churn out uncanny valley versions of code, of mansplaining and of other people’s art. But of course the AI is built on a corpus of our own creations, and I don’t think we like the reflection looking back at us.

Ethics in technology isn’t just about accurately reflecting the world as it is, or how the law pretends it is (or seeks to adjust what is), STEM at its most important shows us the world as it could be. An airplane isn’t just a human pretending to be a bird. A car isn’t just a steel horse.

Yes, these advances in AI are cool parlor tricks, and they will lead to great things, but just like drum machines didn’t replace drummers, we need to get past the wave of novelty to see what’s really behind the wizard’s mask.

AI is dangerous. Look at how machine learning projected racial predictions on zip codes based on historical arrest data. Look at how many corrections Tesla’s “Self-Driving Mode” requires. Look how easily ChatGPT can be manipulated to return answers it’s been programmed not to. But, with the right oversight AI encompasses some very useful tools.

Let’s get out of the car park and look away from the abyss. What does the world AI can’t predict look like? After years of despair, what does a world of hope look like? What does the world you want for your children, grandchildren, nieces and nephews look like?

Land on your own moon. What’s your 10 year plan to change your world?

Advertisement

NMandelbrot : running arbitrary code on client

As part of my grand plan for map-reduce in JavaScript and zero-install distributed computing, I had to think about how to gain user trust in a security context where we don’t trust the server. I couldn’t come up with a good answer.

Since then, we’ve seen stories of malicious JavaScript installed to mine cryptocurrencies , we know that JavaScript can be exploited to read kernel memory, including passwords, on the client, and I suspect we’ll see a lot more restrictions on what JavaScript is allowed to do – although as the Spectre exploit is fundamentally an array read, it’s going to be a complex fix at multiple levels.

I had ideas about how to sandbox the client JavaScript (I was looking at Python’s virtualenv and Docker containers to isolate code, as well as locking them into service workers which already have a vastly more limited API), but that relies on the browser and OS maintaining separation, and if VMs can’t maintain separation with their levels of isolation, it’s not an easy job for browser developers, or anyone running on their platform.

I also wondered if the clients should be written in a functional language that transpiled to JavaScript, to have language level enforcement of immutability and safety checks. And of course, because a functional style and API provides a simpler context to reason about map-reduce, by avoiding any implicit shared context.

Do you allow someone else’s JavaScript on your site, whether a library, or a tracking script, or random ads from Russia, Korea, botnets and script kiddies? How do you keep your customers safe? And how do you isolate processes you trust from processes that deal with the outside world and users? JavaScript will be more secure in the future, and the research is fascinating (JavaScript Zero: real JavaScript, and zero side-channel attacks) but can you afford to wait?

Meltdown and Spectre shouldn’t change any of this. But now is a good time to think about it. Make 2018 the year you become paranoid about users, 3rd parties and other threats. The year is still young, but the exploits are piling up.

 

Why can’t the IT industry deliver large, faultless projects quickly as in other industries?

Glasgow Tower

The title and inspiration of this post is an old question on StackOverflow : Why can’t the IT industry deliver large, faultless projects quickly as in other industries? – Programmers

There is a continuing question of why IT consistently fails to deliver large projects, when other industries such as construction, civil engineering, and aircraft companies consistently deliver on time and to budget, and never have any problems in their first few years. Just ask anyone in Edinburgh about the trams.

However, there are a few things that make software projects more likely to fail, as I see it, throughout the process, and the successful methodologies recognise and address these problems directly.

The first key difference I see is best demonstrated looking at architecture vs IT. I’ve seen a few design competitions for key projects, and the bidding always involves paper or 3D-rendered models of the final structure, with lots of trees, and several people milling about, looking happy. It’s been very rare for me to see that in a software bid, and that’s probably a good thing. Aside from some rough sketches of UIs, what really matters is the relationship between the developers and the customer, because software changes dramatically according to use, especially after first use when the users start to see what’s possible rather than just talking about it.

The buildings we see are not version 1. Before the models in the bidding stage, there may be sketches, and after the models may come prototypes, scale models, physics simulations, walkthrough renderings, and many other versions iterating towards the final design that actually involves tonnes of steel and concrete driven into big holes in the ground.

Software is version 1, or maybe version 2. The design is executable, and malleable. Code can be used to simulate itself via test frameworks. Software is the best model for software, after all simulations such as paper prototypes are doomed to succeed, because they won’t have real world data with apostrophes in names, they won’t have anyone living in Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch, and all network interactions will be instantaneous.

Every model and sketch built before a building is a level of abstraction that considers a subset of the details of the finished product. In software, everything is done at the same level of abstraction, the production code, the unit tests, the integration tests, the behaviour driven tests, the factory testing, are all done on the same business logic, and often in the same language, so the design is the product, and if the design is wrong, the product is wrong, and often the only way to test the design is to deliver the product. Users are not going to care about curly braces and angle brackets. They care that hitting that button does the right magic to send that email. If the design is wrong, then the magic is wrong, and the user is disappointed. So we iterate, we gather feedback, and we improve, step by step, polishing the magic until the experience shines.

And that’s what other industries do, whether we admit it to ourselves or not. Walls in buildings are knocked down, offices are reconfigured, and the Boeing and Airbus planes are improved in iterations. Carriers are offered new seat styles, and get offered stacking accommodation, flight navigation systems are removed and upgraded, and so on. Improvements are made around an expensively tested and maintained core, which improves at a slower pace, because the gap between design and implementation is large, and the feedback cycle is very long, although it’s getting better, at least for architects.

Is software uniquely complex? Are software projects too large? No. But the nature of software puts us in a much tighter feedback cycle between design and code. That’s what the agile manifesto cuts to at its core. We want to test our designs, but the best way to do that is to implement them and get them in front of users, and then refine them. Upfront design doesn’t work because users understand products, not requirements.

Software can deliver large, faultless projects, but it’s much easier to deliver many smaller, faultless iterations, and take more risks whilst you’re doing so, because losing 1 weeks’ work is a lot less painful that losing a year’s work.

Google Code Migration : Genetic Algorithm Templates

With the closure of Google Code, I’ve moved some projects to github. All personal projects so far, but related to talks our blog posts from the past, so may still be of interest.

The first project I want to highlight again is written in C++ and implements genetic algorithms using mainly C++ templates, just to see how powerful they were. It taught me a lot about generic code, and how a poor type system can interfere with the clarity of your code. It also prepared me for one of my first talks, about Genetic Algorithms at a Beauty of Code techmeetup.

I’d like to look at a Python port, to see if my expectations of using dynamic typing would answer the concerns I have about code clarity. For now though, it’s available for reference. It’s not production tested, and there are parts that are embarrassing, but it might be interesting if you want to know what genetic algorithms might look like.

I don’t trust change

Everything changes

I don’t trust change. I know change is what we do, it’s why people need new software to do things they couldn’t do before, to sweep away the cobwebs and start everything anew. To change. For the better.

The better what? Faster, more efficient, more user friendly. “Just better”. “An improvement”. “The new shiny”. “Make it cool”.

Great, can you write me a requirement for that, or some acceptance criteria?

“We’ll know it when we see it.”

But how will we know?

“It doesn’t matter. We’ll know.”

And when you come back and say it doesn’t have enough “zing” or “pizazz” or isn’t “cool” or “new” enough?

“Well then, it shows you didn’t listen to us.”

A change for the better?

If you want to change, you should know why. What will the users notice? What pain will it take away? What will it allow you to do that you couldn’t do before?

Those are questions I can pick away at, questions with an answer where I can measure something tangible. I can change the time it takes to do something, I can change a system to do something new, and I can show you exactly what the change looks like.

A change without a diff is a break.

If you don’t know why you’re making a change, the chances are, you’re making the wrong change, and you won’t know when the change is done. Make a small improvement, or a big one, but know why. If you can’t articulate the difference between the present reality and the desired future, all I can tell you is that the change will break. It might be beneficial, it might not, but there will be no way to trace from now to the future, and no way to know if anything has improved.

The best change may be no change

We love writing software, coming up with new ideas, new approaches. We want the new shiny. But that doesn’t mean it will do us, or our customers, any good. Angular.js makes it much easier to write web apps a certain way, but is that the type of web app that most suits your customer, and do they really need a web app at all? Is a web page easier for them?

And maybe what needs to change isn’t the software. Maybe there is a process that people have forgotten that will fix the problems, maybe they’re solving the wrong problem.

Software isn’t always the answer, and neither is change.

When to embrace change.

Change when things cause you pain. Subversion makes branching and merging painful. If that’s a problem for you, try git or mercurial. Manipulating the DOM from a callback is painful. If that’s a problem for you, try Angular or another framework. If they’re not causing you pain, chances are something else is, go change that first.

Change when there’s a functionality gap. When you’re chasing a new regulations, new APIs, new competitors, or new customers. And make sure that functionality does something to help close that gap. You might have a few ideas, so pretotype and prototype them first, make the change as small as possible, and build on it, because a small change is easier to throw away, physically and psychologically.

Embrace change.

But don’t trust it, until you know where you want the change to take you, and you have a way to check you’re on the right path.

DOOR Oasis Office Reporting

I’ve been hinting in my status in various places (twitter, facebook, etc..) that’s I’ve been playing about with an idea to do some database/web service reporting using the ODF format, powered by a bit of XSLT. The grand idea is that ODF editors are easier to use and to get hold of than the existing reporting frameworks, ODF is reasonably easy to edit by hand if things go horribly wrong (or you need to change a server name), and, as an XML format, I can use my data to change anything in the output, content, styling, could even generate different letters for US and EU customers by selecting paper size based on country. It’s still very much in the requirements capture stage at the moment, but if it sounds like something you’d be interested in, hit the link below and have a nose around. There’s a discussion group too, which I might have to tweak if things get popular, so pop on over and give me your thoughts.

door-reports – Google Code

Blogged with the Flock Browser