We May Look Back Fondly on Our 68% Project Failure Rate

The report [PDF] referenced here -- focusing on failure originating in the business requirements realm and offering a "68% project 'failure'" headline -- inspired two thoughts.

First, it remains a head-scratcher why nearly every project, despite talk of risk analysis and mitigation, expects to be in the 32% success area. Even if a company has the best resources (and generally it will not), many causes of project failure originate in organizational factors -- friction, process, politics -- and externalities (e.g., no budget this quarter for the tools in the original plan).

Since these issues are rarely known -- and, I would argue, actively denied in the sense that whistle-blowers and problem-spotter are systematically excluded from planning (at best) or forced down or out -- the average "expected value" of the d100 role is way less than a 68.

In startups, I've heard the excuse that the whole company is a "long shot" -- as though that justifies taking on disproportionate risk in each project implementation.

In enterprises, it seems as though management is so used to failure (in a timeline or budgetary sense), that they have simply redefined success to mean anything that they can pull off in their tenure -- and if that means a system that kinda works, but no one likes, shipped in 200% time and 400% budget, well, that's just the "reality" (which, to be sure, it is once all other possible paths have been discarded).

This redefinition of success also has the side effect of removing accountability and pretty much assuring a nice bonus and making strict accountability impossible.

Another thought inspired by the report is how the gap between enterprise development and small / startup development is widening. On the one hand, large businesses could benefit from the high-productivity tools and agile approaches popular with startups; for a variety of reasons ranging from policies to personnel, they are not exploiting the latest wave of technology, and it's costing them.

On the other hand, what they need regardless of tech is solid analysis and estimation capabilities. Analysis and estimation are possible, but hard, and the agile camp has moved mountains to try and reframe the problem so that they can advocate punting on all of the hard parts of these disciplines. That works great for the hourly agile consultants, but inside large businesses and large projects, it just doesn't cut it. A business needs to be able to attempt to estimate and plan months or even years of a project (hence the prevalence of "fake" agile in companies that purport to use it).

The fact that the business does a horrible job with the estimation today does not mean that the organization (which, generally, is run by business people not developers) won't keep planning and targeting in the large.

The result of these two pieces (different tech, different process) is that enterprise and small development are moving farther and farther apart, which is a damaging long-term trend. Ideally, these two groups should be learning from each other. They should spend more time moving in the same world.

The enterprise learns that it probably doesn't need JavaEE and Oracle and a big planning process for myriad internal utility apps that could be done with Rails at a fraction of the cost and effort. The small company learns that relational integrity, transactions, estimation, and operations management are sometimes both necessary and profitable.

Even more importantly, individual employees in the industry can cross-pollinate ideas as they move between these environments over their careers, sorting fact from fiction and getting better at determining what will work.

The farther apart these groups are, the less appealing it will be for "startup guys" to work in an enterprise or a startup to hire on an "enterprise guy."

This trend -- since it keeps useful knowledge hidden -- can only help a 68% failure rate go up.

One Reason Software Development is Such a Malleable Process

This post is the fifth (the others are here, here, here, and here) -- and last planned one -- summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

At the beginning of Chapter 23, Steve writes:

Philip Metzger observed decades ago that technical staff were fairly good at estimation but were poor at defending their estimates [ ... ] One issue in estimate negotiations arises from the personalities of the people going the negotiating. Technical staff tend to be introverts. [ ... ] Software negotiations typically occur between technical staff and executives or between technical staff and marketers. Gerald Weinberg points out that marketers and executives are often at least ten years older and more highly placed in the organization than technical staff. Plus, negotiation is part of their job descriptions. [ ...] In other words, estimate negotiations tend to be between introverted technical staff and seasoned professional negotiators.

This is a valuable observation in estimation scenarios. It also reminded me of some hand-waving I did about 6 months ago in reply to a comment on this post. I wrote that the reason some industrial process control type procedures fail when applied to producing software is that "There are too many social vulnerabilities -- the software process itself is inherently 'soft' in a group dynamics sense, so it gets reshaped by the org's internal disagreements."

I stand by that comment, but it seemed a bit vague. I think Mr. McConnell's summary of the personalities that push and pull around software commitments is quite helpful. His description clarifies one of the software group's social vulnerabilities. After finding themselves agreeing to a problematic "estimate," engineering groups may be able to make up for their poor performance at the negotiating table by trying to "reroute power from the warp core" ... but we know that statistically, over the long term, that approach fails, leaving chaos in its wake.

What to do? A proverb says that a man representing himself has a fool for a lawyer. I.e., right or wrong, he is likely no match for a trained, experienced legal professional on the other side.

So let's get the software team some representation. Not a lawyer, but an executive advocate, someone with the personality and negotiation skill-set to go to the mat with other parties.

If this person has a good grasp of the technical issues, all the better; if not, he or she can take it offline and come back to the engineering team for a briefing.

Of course, if the engineering group somehow gets the impression that the representative has sold them out, making unauthorized or unreasonable commitments, then we are back at square one.

In Software Estimation, Fewer Inputs Can Trump More Inputs. Here's Why:

This post is the fourth (the others are here, here, and here) summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

There are estimation models that take a large number of inputs, and others that rely on only a few inputs. Some of the input data are more objective (e.g., the number of screens in a spec) while other data are more subjective (e.g., a human-assigned "complexity" rating for each of several reports).

One thing that Steve explains is that, when many subjective inputs are present, models with fewer inputs often generate better estimates than models with "lots of knobs to turn."

The reason for this is that human estimators are burdened with cognitive biases that are ridiculously hard to break free of. The more knobs the estimator has to turn, the more opportunities there are for cognitive biases to influence to result, and the worse the overall performance.

It should be reiterated that this same problem does not apply to systems that take many objective measures (such as statistics or budget data from historical projects) as inputs.

LOC Counts to Compare Individual Productivity? No Way.

"Measuring programming progress by lines of code is like measuring aircraft building progress by weight." - Bill Gates

The other day, someone on my project team proposed ranking developers by their lines-of-code committed in Subversion. I really hope the suggestion was meant to be tongue-in-cheek. But lest anyone take LOC measures too seriously, it's worth pointing out that LOC is a bad, bad metric of productivity, and only gets worse when one tries to apply it across different application areas, developer roles, etc.

Here are a few reasons why LOC is not a good measure of development output:

• LOC metrics, sooner or later, intentionally or unintentionally, encourage people to game the system with code lines, check-ins, etc. This is bad enough to be a sufficient reason not to measure per-developer LOC, but this is actually the least bad of the problems.
  
• LOC cannot say anything about the quality of the code. It cannot distinguish between and an overly complex and bad solution to a simple problem, and a long, complex, and necessary solution to a hard problem. And yet it "rewards" poorly-thought-out, copy-paste code, which is not a desirable trait in a metric.
  
• In software development, we desire nicely factored, elegant solutions to a problem -- in a perfect world, we want the least LOC solution to a problem that still meets requirements such as readability. So a metric that evaluates the opposite -- maximum LOC for each task -- is counterproductive. And since high LOC certainly doesn't necessarily mean bad code, there isn't even a negative correlation to use from the measurement.
  
• In general, spending time figuring out the right way to do something, as opposed to hacking and hacking and hacking, lowers your LOC per unit time. And if you do succeed in finding a nice compact solution, then it lowers your gross LOC overall.
  
• Even in the same application and programming environment, some tasks lend themselves to much higher LOC counts than others, because of the level of the APIs available. For example, in a Java application with some fancy graphics and a relational persistence store, Java2D API UI code probably requires more statements than persistence code leveraging EJB 3 (based on Hibernate) based on the nature of the API. Persistence code using straight JDBC and SQL strings will require more lines of code than EJB 3 code, although it’s most likely the “wrong” choice for an application for all sorts of well-known reasons.
  
• In the same application and environment, not every LOC is equal in terms of business value: there is core code, high-value edge-case code, low-value edge-case code. To imagine that every line of every feature is the same disregards the business reality of software.
  
• You may have read that the average developer on Windows Vista at Microsoft averaged very few lines of code per day (from 50 down to about 5 depending on who you read). Is Microsoft full of lazy clueless coders? Is that why the schedule slipped? I doubt it. There were management issues, but Microsoft also worked extremely hard to get certain aspects of Vista to be secure. Do security and reliability come with added lines of code? Unlikely – in fact, industry data suggest the opposite: more code = more errors, more vulnerabilities.

But no one on our team would ever write bad code, right? So we don’t need to worry about those issues.

Not so fast… Developers, even writing “good” code, generally make the same number of errors on average per line (a constant for an individual developer). So if I write twice as many lines of code, I create twice as many bugs. Will I find them soon? Will I fix them properly? Will they be very expensive sometime down the line? Who pays for this? Is it worth it? Complex questions. Never an easy “yes.” Or, as Jeff Atwood puts it, “The Best Code is No Code At All.”

And beautiful, elegant, delightful code is expensive to write (because it requires thought and testing). The profitability of my firm depends on delivering software and controlling costs at the same time. We don’t fly first class and we don’t use $6000 development rigs, even if we might offer some benefit to the client or customer as a result. And we don’t write arbitrary volumes of arbitrarily sophisticated code if we can help it.

Ok, so why, then, is the LOC metric even around? If it’s such a bad idea, it would be gone by now!

Here’s why: while LOC is a poor measure of developer output, it’s easy to use, and it’s a (primitive but functional) measure of the overall complexity and cost of a system. When all the code in a large system is averaged together, one can establish a number of lines per feature, a number of bugs per line, a number of lines per developer-dollar, and a cost to maintain each line for a long, long time into the future.

These metrics can be valuable when applied to estimating similar efforts in the same organization under similar constraints. So they’re worth collecting in aggregate for that reason.

But for comparing individual productivity? I don’t think so.

Estimates, Targets, and Commitments: Mix 'Em Up and You're on the Way to Pain

This post is the third (the first two are here and here) summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

When doing a little searching to see who had written about this point before, I found a fabulous post that covered exactly what I wanted to cover. I also found that McConnell has put the first bit of his discussion about this topic online as a free excerpt. He comes back to these topics later in the book, but the key definitions are here [~ 1MB PDF].

Since these good folks have done the heavy lifting on my post for me, I'll proceed right to the controversial part:

One would think, given the clarity and common sense of these definitions, and the nonjudgmental approach which McConnell takes toward difficult business situations, that software companies would embrace this clarity and let the light shine in on their estimates, targets, and commitments.

But, in my experience, the opposite is true. Groups will actively obfuscate the distinctions here, or deny that one or another of the terms is distinct or relevant. I believe that the root causes of this practice are

1. Diminished tolerance for ambiguity common in group settings and
2. Fear of the potential reaction if it is conceded that, e.g., a target value and estimate value may be far apart

Covering up these distinctions does not affect reality, but only attempts to manage perceptions. And not everyone's perceptions will be successfully managed. Such behavior is rarely helpful and often harmful.

Software Estimation: All Estimates Are Range Estimates

This post is the second (the first is here) summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

All estimates are range estimates, whether we realize it or not. If we say that a project will finish on July 14, or that, by the end of the year, 56 function points will be complete, we likely do not mean we are 99% certain that these numbers are dead on.

We are implying or suggesting some kind of range (e.g., "sometime in July"). And without making the bounds of the range explicit, nor specifying a level of confidence that the value being estimated lies inside that range, we are probably doing more harm than good with our "estimates."

McConnell points out that in many areas of life we systematically do a bad job at constructing range estimates wide enough to include the value we're trying to estimate. He includes examples -- and an exercise for the reader -- showing that even when given specific instructions to generate an arbitrarily wide range that will include a target value, we still fail to make it wide enough.

We have a cognitive bias that causes us to mistake a wide estimate range for an unacceptably vague answer even when it is not. No wonder, then, that in real-world business scenarios, where pressures exist to create estimates that are both hyper-precise and artificially small, the estimation process comes apart right out of the gate.

How to think about ranges and estimates? Here are a few points to get started:

1. Many values can be estimated in a software development project. Typical values to estimate include total effort or resources to implement a set of functionality, or quantity of functionality that can be implemented with fixed constraints.
2. A point estimate is really just a very narrow range estimate -- and almost certainly an inaccurate range estimate. 
3. How wide should a range estimate be? Wide enough that one can have a specific confidence level that the range includes the actual number being estimated.
4. If the confidence level is fixed, then the width of the range necessarily depends on how much is known to inform the estimate (as well as on the estimation techniques being used, etc.) For example, if you know the specs for a project, it is in the best case possible to achieve a narrower range estimate at given confidence level than if you don't yet know the specs at all.
5. The amount if information known (feature details, effort to implement each feature, unexpected dependencies, etc.) increases over time and over the course of the project.
6. Therefore, updated range estimates can narrow over the course of the project. Initial ranges, if they are accurate, will necessarily be quite wide. McConnell refers to these narrowing intervals over time as the "Cone of Uncertainty" based on the cone-like shape that the graph (range vs. time) makes.
7. Despite the convergence of the "cone" graph lines, the cone concept reflects a best-case scenario of mapping limited information to limited accuracy in estimation. If other estimation best practices are not followed, it is possible to do much worse than the ranges reflected in the cone.

For the Love of God, Count Something

I've been reading Steve McConnell's Software Estimation: Demystifying the Black Art.

It's a mostly-fabulous book, and I highly recommend it. I have no doubt this book will take its place as a classic alongside his Code Complete and Rapid Development titles.

This post is the first of a few summarizing and commenting on key estimation principles that McConnell covers in the book.

As my title suggests, step zero: count something ... count anything. Count features, functions point, user stories, use cases, web pages, GUI widgets... count what you can. It'll take some work to get from that count to a useful heuristic for estimation, but it's way better than the all-too-common practice of estimating by licking a finger and holding it up in the air. Hmmm ... 7 knots, from the north-west.

Steve spends many chapters exploring what to count, when, and how; who should do the counting; and what to do with the totals. But the premise is that "take a SWAG and double it" is not only unreliable, but it's indefensible. How can you explain an overrun or ask for more resources if there's no process to justify your numbers? There's no way anyone in your organization can plug your "gut" into a spreadsheet in a useful way, so you're perpetually at a disadvantage in negotiating to ensure positive outcomes.

Another vulnerability in the gut-feel approach is that -- even if the manager or programmer who makes the gut estimate is right on the money -- she can only be right on relative to the scope she is imagining to be on the table.

In many organizations, scope is not made sufficiently explicit to guarantee a proper understanding up front. Thus the "counting" part of the estimation process actually brings new specifications to light regarding the project being estimated. A couple of iterations of this process, and a lot more is known by all parties. More knowledge generally leads to less risk and more predictable outcomes.