We May Look Back Fondly on Our 68% Project Failure Rate

The report [PDF] referenced here -- focusing on failure originating in the business requirements realm and offering a "68% project 'failure'" headline -- inspired two thoughts.

First, it remains a head-scratcher why nearly every project, despite talk of risk analysis and mitigation, expects to be in the 32% success area. Even if a company has the best resources (and generally it will not), many causes of project failure originate in organizational factors -- friction, process, politics -- and externalities (e.g., no budget this quarter for the tools in the original plan).

Since these issues are rarely known -- and, I would argue, actively denied in the sense that whistle-blowers and problem-spotter are systematically excluded from planning (at best) or forced down or out -- the average "expected value" of the d100 role is way less than a 68.

In startups, I've heard the excuse that the whole company is a "long shot" -- as though that justifies taking on disproportionate risk in each project implementation.

In enterprises, it seems as though management is so used to failure (in a timeline or budgetary sense), that they have simply redefined success to mean anything that they can pull off in their tenure -- and if that means a system that kinda works, but no one likes, shipped in 200% time and 400% budget, well, that's just the "reality" (which, to be sure, it is once all other possible paths have been discarded).

This redefinition of success also has the side effect of removing accountability and pretty much assuring a nice bonus and making strict accountability impossible.

Another thought inspired by the report is how the gap between enterprise development and small / startup development is widening. On the one hand, large businesses could benefit from the high-productivity tools and agile approaches popular with startups; for a variety of reasons ranging from policies to personnel, they are not exploiting the latest wave of technology, and it's costing them.

On the other hand, what they need regardless of tech is solid analysis and estimation capabilities. Analysis and estimation are possible, but hard, and the agile camp has moved mountains to try and reframe the problem so that they can advocate punting on all of the hard parts of these disciplines. That works great for the hourly agile consultants, but inside large businesses and large projects, it just doesn't cut it. A business needs to be able to attempt to estimate and plan months or even years of a project (hence the prevalence of "fake" agile in companies that purport to use it).

The fact that the business does a horrible job with the estimation today does not mean that the organization (which, generally, is run by business people not developers) won't keep planning and targeting in the large.

The result of these two pieces (different tech, different process) is that enterprise and small development are moving farther and farther apart, which is a damaging long-term trend. Ideally, these two groups should be learning from each other. They should spend more time moving in the same world.

The enterprise learns that it probably doesn't need JavaEE and Oracle and a big planning process for myriad internal utility apps that could be done with Rails at a fraction of the cost and effort. The small company learns that relational integrity, transactions, estimation, and operations management are sometimes both necessary and profitable.

Even more importantly, individual employees in the industry can cross-pollinate ideas as they move between these environments over their careers, sorting fact from fiction and getting better at determining what will work.

The farther apart these groups are, the less appealing it will be for "startup guys" to work in an enterprise or a startup to hire on an "enterprise guy."

This trend -- since it keeps useful knowledge hidden -- can only help a 68% failure rate go up.

Software Discipline Tribalism

Unfortunately, people seem rarely able to stop at a reasoned preference -- e.g., "I like X, since X may offer better outcomes than Y" ... and too often end up, at least whenever group persuasion is involved, somewhere more dramatic, personal, extreme, and narrow -- e.g., "I'm the X kind of person who rebels against Y, since Y can offer worse outcomes."

This is as true in cultures of software development as anywhere else.

While many people have been guilty of this unproductive shift in attitudes, it seems many of the Agile development, dynamic languages, small-tools/small-process/small-companies crowd has long since fallen prey to it.

To be sure, there was much temptation to rebel.

At the start of the decade, proprietary Unix was strong; many processes came with expensive consultants, training, books, and tools ... and overwhelmed the projects they were meant to guide; many tools were expensive and proprietary. Web services were coming onto the scene and large players, with licenses and consulting hours to sell, created specs that were unwieldy for the small, agile, and less-deep-pocketed.

When the post-dot-com nuclear winter set in, small companies had no money to pay for any of that, and we got LAMP, Agile, TDD, REST, etc. Opposition to OO, which had been strong in many quarters, suddenly faded as OO was no longer identified (rightly or not) with certain problematic processes. Ironically, many new OO language fans had been ignoring the lightweight, free (speech and beer) processes that some OO advocates had been producing for years.

These have all proven to be useful tools and techniques and have created whole companies and enormous value ... but somewhere along the line, instead of being in favor of these tools and techniques because in some cases they produced better outcomes, either the leaders or the converts started thinking they were the rebels against anything enterprise, strongly typed, thoroughly analyzed, designed and well tooled.

This shift in attitudes does not help the industry ... nor even the clever consultants who lead the charge, deprecating last year's trend for a new one which they just happen to have written a book about.

We desperately need a broader perspective that integrates all of these pieces. There are things manually-written tests just won't do -- tools like Pex can help immensely, even if (or because... )they are from a big company.

Analysis and design are not bad words, while Agile can get dangerously close to simply surrendering to the pounding waves of change (and laughing at goals all the way to the bank) rather than building against the tide, and trying manage to a real outcome on a real budget.

Static languages can get hideously verbose for cases with functor-like behavior (Java and C# [pre-3.0], I'm looking at you). At the same time, go talk to some ActionScript developers -- who have had dynamic and functional for years -- and you'll see an amazing appreciation for the optional strict typing and interfaces in AS3.

REST is great, but in playing at dynamic, it turns out to be rather like C -- it's as dynamic as the strings you pipe into the compiler, and no more. Absent proper metadata, it cannot reflect and self-bind, so it sacrifices features that dynamic language developers love in their day-to-day coding.

Ironically, most of the critical elements of this "movement" -- along with open source -- are being subsumed into the big enterprise software companies at a prodigious pace. Sun owns MySQL and halfway owns JRuby; Java servers may serve more Rails apps than Mongrel/Ebb/Thin/etc. soon, Microsoft is all over TDD, IronRuby, IronPython...

I suppose the sort of tribalizing we see here is at least partly inevitable in any field. But it would serve the entire industry if that "part" could be made as small as reasonably possible. As a young industry with a poor track record and few rules, we ought to be more interested in better software outcomes than in being rebellious.

Want More Money? Repair, Remediate, Refactor

In the time it takes you to read this sentence, about $6.02x10²³ dollars will have been wasted in lost productivity due to old, rickety, hard-to-use, non-performant software systems.

Well not so many dollars, but the loss is real -- whether it's the energy and cost-of-operation for more servers, when fewer could do the same job; whether it's hours spent waiting for CSRs to tab through tens of screens in legacy line-of-business apps to find the one field they still use; or worse, when a police dispatcher makes a mistake because software makes it hard to click the right thing fast and dangerously easy to click the wrong one.

Strategies for making "bad old systems" better are not the most popular topic for the tech elite. This makes sense for a bunch of reasons. The mythologies we live by are generally startup oriented. But, more importantly, real roll-up-your sleeves work on old systems doesn't scale. The hard problems are business-specific and not ones that licensable products can solve ... so anyone on a mission to save the world from these systems ends up with a services approach -- i.e., a process approach -- that is bottlenecked by human quality-and-scaling problems.

That's a bummer, because there is a lot that can be done with old systems once a really good team starts thinking about them. Forget the big rewrites and the magic bullets. Forget the awful HTML front ends that barely hide COBOL.

In between are some great possibilities. A lot of legacy architectures lend themselves to SOA without even knowing it. Inexpensive hardware and storage means, in some cases, legacy client bits can be moved back to the datacenter and integrated into a middle tier. Other times, the clients can be done away with altogether if someone has the chutzpah to figure out the "old" protocols and implement them with modern tech.

While I mostly work with startups, I've been involved with at least three projects in the last five years that included some re-engineering of legacy systems. One of these systems was almost 40 years old and used emulation of custom hardware (and a custom physical network) just to communicate. Two of the three were barely documented, or not documented at all. And at least one was a total clusterfudge, in the sense that it hadn't aged and "crufted" naturally, but was the result of crackpot work that was known to be broken from the relatively recent start.

In all of these cases, it was possible to improve the architecture, performance, and functionality of the system without the mega-rewrite. And since the existing systems had tons of data and transactions running, the new software needed to run smoothly side-by-side with existing software, allowing a limited and careful transition.

Here's the kicker: in two out of the three cases, although the "new" software was a vast improvement, it was never allowed to take on a larger role in the enterprise outside of a niche for which it had been developed. A bunch of startup hotshots writing cutting-edge stuff was simply not in the Enterprise IT Legacy Technology Lifecycle Planning Regime and so, like the dismissal of Heron's steam engine in the first century, the new technology was regarded as a mere novelty.

There are some reasons for conservatism and stodginess around legacy systems. But there is also a ton of value waiting to be liberated as soon as some of that stodginess can be shed.