There is a continuing question of why IT consistently fails to deliver large projects, when other industries such as construction, civil engineering, and aircraft companies consistently deliver on time and to budget, and never have any problems in their first few years. Just ask anyone in Edinburgh about the trams.
However, there are a few things that make software projects more likely to fail, as I see it, throughout the process, and the successful methodologies recognise and address these problems directly.
The first key difference I see is best demonstrated looking at architecture vs IT. I’ve seen a few design competitions for key projects, and the bidding always involves paper or 3D-rendered models of the final structure, with lots of trees, and several people milling about, looking happy. It’s been very rare for me to see that in a software bid, and that’s probably a good thing. Aside from some rough sketches of UIs, what really matters is the relationship between the developers and the customer, because software changes dramatically according to use, especially after first use when the users start to see what’s possible rather than just talking about it.
The buildings we see are not version 1. Before the models in the bidding stage, there may be sketches, and after the models may come prototypes, scale models, physics simulations, walkthrough renderings, and many other versions iterating towards the final design that actually involves tonnes of steel and concrete driven into big holes in the ground.
Software is version 1, or maybe version 2. The design is executable, and malleable. Code can be used to simulate itself via test frameworks. Software is the best model for software, after all simulations such as paper prototypes are doomed to succeed, because they won’t have real world data with apostrophes in names, they won’t have anyone living in Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch, and all network interactions will be instantaneous.
Every model and sketch built before a building is a level of abstraction that considers a subset of the details of the finished product. In software, everything is done at the same level of abstraction, the production code, the unit tests, the integration tests, the behaviour driven tests, the factory testing, are all done on the same business logic, and often in the same language, so the design is the product, and if the design is wrong, the product is wrong, and often the only way to test the design is to deliver the product. Users are not going to care about curly braces and angle brackets. They care that hitting that button does the right magic to send that email. If the design is wrong, then the magic is wrong, and the user is disappointed. So we iterate, we gather feedback, and we improve, step by step, polishing the magic until the experience shines.
And that’s what other industries do, whether we admit it to ourselves or not. Walls in buildings are knocked down, offices are reconfigured, and the Boeing and Airbus planes are improved in iterations. Carriers are offered new seat styles, and get offered stacking accommodation, flight navigation systems are removed and upgraded, and so on. Improvements are made around an expensively tested and maintained core, which improves at a slower pace, because the gap between design and implementation is large, and the feedback cycle is very long, although it’s getting better, at least for architects.
Is software uniquely complex? Are software projects too large? No. But the nature of software puts us in a much tighter feedback cycle between design and code. That’s what the agile manifesto cuts to at its core. We want to test our designs, but the best way to do that is to implement them and get them in front of users, and then refine them. Upfront design doesn’t work because users understand products, not requirements.
Software can deliver large, faultless projects, but it’s much easier to deliver many smaller, faultless iterations, and take more risks whilst you’re doing so, because losing 1 weeks’ work is a lot less painful that losing a year’s work.