Engineering Departments Should Take Their Medicine Too

Without getting into gloomy predictions, it's clear a lot of companies will be feeling some pain soon, if they aren't already feeling it.

Software development groups at tech companies should take some medicine too -- cutting costs, becoming more competitive, improving morale, and attracting good talent at the same time.

How is that possible? Here's one way:

Unless your company is brand new, and you've got geniuses agile-ly coding your hot frog-sticker-collectors' social network, your project has a bunch of history, cruft, and mistakes in it.

It's nothing to get upset about, that's just the way of the world when developing software over the course of years in an organizational setting.

Yet many companies don't make any effort -- or actively resist any effort -- to identify those legacy problems and mitigate them. Perhaps it's fear of blame (why didn't you see this before? why didn't you say something? aren't you guys supposed to be experts?) or fear of appearing backward-looking rather than forward-looking (we'll make do with all that stuff; now can we shrink wrap and ship your latest proof-of-concept and call it next quarter's product?)

Suppose that, instead, your development group listed all the crufty bits that bug them. Stuff that maybe made sense at the time, but just isn't right any more -- wrong code, wrong architecture, wrong network protocol, wrong database, wrong format, whatever. Suppose the team got to rank these in order of annoyance factor, and impediment to productivity. Then, picking the top handful, they got to decide how to use the latest and greatest (and in many cases less expensive) technology to refactor and fix those modules.

A shortsighted manager might complain that 'if it ain't broke, don't fix it' -- it's a waste of resources.

But we know better than that.

In many projects a significant percentage of resources (up to two-thirds in extreme cases, based on my research and experience) can be spent wrestling with these "legacy cruft" issues. So, from a simple economics standpoint, it's definitely "broke" if you're spending $1 million per year on a dev group that could theoretically deliver the same functionality on $333,000, or 3x as much for the same $1 million.

A project that removes these old bits becomes more competitive. Why? The competition, particularly if it's a newer company or on a newer platform, isn't running with these parking brakes holding them back. Why should your team? If you can release the brake, you can deny your competitors something they definitely view as an advantage against you.

Moreover, these moves can boost morale and make your company more attractive to prospective employees in several different ways.

Getting rid of old morale-busting code makes everyone feel good.

Using a newer technology -- not as a novelty but because it's solving a real problem better than existing code -- is appealing to developers who want to learn new skills.

Doing a refactor of critical code without breaking existing code, wrecking release schedules, or introducing excessive ops downtime is a challenging and rewarding skill, kind of like working on the engine while plane is flying -- and top developers relish this kind of "black-diamond" assignment.

Finally, it tells everyone, inside the company and out, that this isn't Initech, where an engineer will have to work on 15-year-old tech for the next 15 years.

One Reason Software Development is Such a Malleable Process

This post is the fifth (the others are here, here, here, and here) -- and last planned one -- summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

At the beginning of Chapter 23, Steve writes:

Philip Metzger observed decades ago that technical staff were fairly good at estimation but were poor at defending their estimates [ ... ] One issue in estimate negotiations arises from the personalities of the people going the negotiating. Technical staff tend to be introverts. [ ... ] Software negotiations typically occur between technical staff and executives or between technical staff and marketers. Gerald Weinberg points out that marketers and executives are often at least ten years older and more highly placed in the organization than technical staff. Plus, negotiation is part of their job descriptions. [ ...] In other words, estimate negotiations tend to be between introverted technical staff and seasoned professional negotiators.

This is a valuable observation in estimation scenarios. It also reminded me of some hand-waving I did about 6 months ago in reply to a comment on this post. I wrote that the reason some industrial process control type procedures fail when applied to producing software is that "There are too many social vulnerabilities -- the software process itself is inherently 'soft' in a group dynamics sense, so it gets reshaped by the org's internal disagreements."

I stand by that comment, but it seemed a bit vague. I think Mr. McConnell's summary of the personalities that push and pull around software commitments is quite helpful. His description clarifies one of the software group's social vulnerabilities. After finding themselves agreeing to a problematic "estimate," engineering groups may be able to make up for their poor performance at the negotiating table by trying to "reroute power from the warp core" ... but we know that statistically, over the long term, that approach fails, leaving chaos in its wake.

What to do? A proverb says that a man representing himself has a fool for a lawyer. I.e., right or wrong, he is likely no match for a trained, experienced legal professional on the other side.

So let's get the software team some representation. Not a lawyer, but an executive advocate, someone with the personality and negotiation skill-set to go to the mat with other parties.

If this person has a good grasp of the technical issues, all the better; if not, he or she can take it offline and come back to the engineering team for a briefing.

Of course, if the engineering group somehow gets the impression that the representative has sold them out, making unauthorized or unreasonable commitments, then we are back at square one.

The Hardest Thing About Having Plan B is Realizing You Might Be Wrong About Plan A

The second law of thermodynamics describes why it's easier to break a glass than put it back together; easier to write the wrong code than the right code; and easier to metabolize nutrients than to create them. Wilfred Bion's group dynamics theories provide a bit of a social parallel -- some of the reasoning why groups and organizations persistently do the "incorrect" (by their own definitions of "correct"!) thing. By default, groups tend to devolve from their own goals in more or less spectacular ways.

Our social interactions don't rigidly follow laws of physics, we can occasionally climb out of the potential well of typical group dynamics. Doing so improves our odds of success on technology projects.

One of my favorite success stories is Intel's "Yamhill Project." Yamhill is why Intel is still in business. But it's also something few management egos could have lived with, and the Intel exec team's willingness to deal is why Intel is still a going concern.

Once upon a time, Intel wanted to clean up all the messiness that had evolved over the history of the x86 architecture. Make a clean break and launch Itanium, a new 64-bit architecture and instruction set. The new chips would not be backwards-compatible, so the gamble was that if Intel said that the future is Itanium, then no matter how painful the transition for customers and partners, they would have to go along.

Given the egos, and the "all-in" nature of chip development, the typical move would be to wave the Itanium flag, charge into battle, and either win the war or die trying. Lots of famous companies have died (or become irrelevant) fighting these battles.

But Intel's team did something much better. A few folks early on said that the market might not tolerate the change and ... if an alternative emerged from a competitor ... the result for Intel would be catastrophic. Officially, Craig Barrett backed Itanium / IA-64 all the way. But in secret he gathered together some of the brightest engineers in the company and sent them off to work in a remote location on Plan B, a non-Itanium 64-bit chip that would be backward compatible.

Barrett and his team possessed an all-too-rare willingness to come to grips with the fact that the big IA-64 play might possibly go wrong. Instead of ostracizing whoever made the suggestion (more typical organizational behavior), they integrated the information and took a decision about how to work with that possibility.

As it happened, AMD came to market with a backwards-compatible 64-bit implementation. The market loved it, and, outside of a few server niches, it might have been game over for Intel right then. If they hadn't had Yamhill.