Is It Too Early or Too Late for an Open RIA Design/Dev Toolchain?

I was playing with the Raphael JavaScript graphics library (a sort of script-based, cross-browser, implementation of SVG) and started thinking how helpful this library would be in creating a browser-based (as opposed to plug-in based) RIA.

That lasted for about 15 seconds before I remembered that creating large, non-trivial RIAs generally involves designers, and most designers don't like creating vector art by coding a set of "path" statements, or animations as a collection of key-value pairs and millisecond-based transition times.

That's why tools like Microsoft Expression, and Adobe Illustrator, Catalyst, and Flash exist.

And why Adobe and Microsoft are investing so heavily in the designer-developer workflow: the ability of designers to turn graphics and animations into app skins and interaction which are immediately available to coders.

In order for an open RIA solution to be competitive and realistic -- whether it's open in the pure-browser sense, using JS via dojo.gfx, or Rapael, etc., or whether it's via an open plug-in (Java/JavaFX seems like the closest, though it's not 100% open yet and may never be) -- this full toolchain needs to exist.

We need to be able to export vector art from mainstream design programs such that they can be incorporated as assets into the RIA. It doesn't matter if this is via SVG, XAML, AI/EPS, or something else entirely. What does matter is that the import/export is robust enough that designers -- whose jobs, after all, include making stuff look just right -- are confident that what they design is what end-users will see. The Microsoft and Adobe tools can do this. To date most OSS attempts cannot.

Next up, we need a truly usable, designer-friendly authoring tool for animations and interactions. It is often argued that some standard tools (*cough* Illustrator *cough*) are not paragons of usability themselves. No matter -- it's hard enough to get converts.

Happily, there seems to be emerging some consensus among the big vendors about how these tools should work (both on-screen and in terms of in intermediate data formats). That blueprint lowers the risk and challenge for an open source contender.

The biggest obstacle remaining is a classic open-source triangle-of-trouble:

1. The toolchain/workflow will not be viable until it is quite solid, since the commercial alternatives (Flash, mainly) are so entrenched.
2. It's hard to get enough contributor man-hours against such a huge project without an active user base.
3. Since the user base is not developers, the bootstrapping for #2 that makes many OSS projects work (devs are tolerant -- even excited -- about getting up on an 0.1 release) is unlikely.

The Second Law of Thermo-Specifications

A couple of days ago, Adam Milligan posted an eponymous "law" on the Pivotal Labs blog.

It includes a corollary stating, "The full definition of correct behavior of code exists in the tests for that code."

Now, there seemed to be something fundamentally off about this proposition ... and I wanted to figure out what that was.

Surely Mr. Milligan doesn't mean this in the trivial, tautological sense (define the spec to be no more than what test cases happen to exist at a point in time) ... even though Agile dogma often borders on that view, pretending it's some kind of paradoxical path to enlightenment like a Zen koan.

The comments to his post start to touch on "spec" vs. "test," and whether 100% test coverage is practical, desirable, or conclusive.

Of course even 100% code coverage, with a missing code path and a matching missing test and behavior spec equals ... test success, and wrong functionality.

In such a case, though, the problem has been externalized from the Agile context and put onto a faceless "business person" who "doesn't get" Agile because he or she actually wants to plan ahead and describe some certain specs that persist over time.

This is a neat trick. In physics, all sorts of magical things can happen if you look only inside of one context (or frame of reference) without looking at what's happening outside, or at what's happening to the frame itself. In finance, there can certainly be a free lunch ... if you can make its cost into an externality and remove it from your model.

Once you make the tests and code-based behavior spec a part of the application you're building -- and clearly, once it's a real cost center as well as a critical deliverable part of the project, you have done so -- then you are in a sense simply externalizing that troublesome human interaction (specification, functional analysis, planning). Yes, the tests match the code under test. But seen at a different level of abstraction, it's just another flavor of interface-driven development, or debugging.

Most engineers would love to realize the dream of self-describing, self-verifying code, whether that description be some kind of formal model, or a textual DSL (as with Microsoft's Oslo), or a set of tests and code-based behavior specs. And, indeed, these systems all improve the transparency of the code, propagating requirements from the outside in, and revealing when they are not met.

But even with the most "bought-in" business stakeholders, it is impossible to escape the the outermost specification layer, the one with the humans in it.

We May Look Back Fondly on Our 68% Project Failure Rate

The report [PDF] referenced here -- focusing on failure originating in the business requirements realm and offering a "68% project 'failure'" headline -- inspired two thoughts.

First, it remains a head-scratcher why nearly every project, despite talk of risk analysis and mitigation, expects to be in the 32% success area. Even if a company has the best resources (and generally it will not), many causes of project failure originate in organizational factors -- friction, process, politics -- and externalities (e.g., no budget this quarter for the tools in the original plan).

Since these issues are rarely known -- and, I would argue, actively denied in the sense that whistle-blowers and problem-spotter are systematically excluded from planning (at best) or forced down or out -- the average "expected value" of the d100 role is way less than a 68.

In startups, I've heard the excuse that the whole company is a "long shot" -- as though that justifies taking on disproportionate risk in each project implementation.

In enterprises, it seems as though management is so used to failure (in a timeline or budgetary sense), that they have simply redefined success to mean anything that they can pull off in their tenure -- and if that means a system that kinda works, but no one likes, shipped in 200% time and 400% budget, well, that's just the "reality" (which, to be sure, it is once all other possible paths have been discarded).

This redefinition of success also has the side effect of removing accountability and pretty much assuring a nice bonus and making strict accountability impossible.

Another thought inspired by the report is how the gap between enterprise development and small / startup development is widening. On the one hand, large businesses could benefit from the high-productivity tools and agile approaches popular with startups; for a variety of reasons ranging from policies to personnel, they are not exploiting the latest wave of technology, and it's costing them.

On the other hand, what they need regardless of tech is solid analysis and estimation capabilities. Analysis and estimation are possible, but hard, and the agile camp has moved mountains to try and reframe the problem so that they can advocate punting on all of the hard parts of these disciplines. That works great for the hourly agile consultants, but inside large businesses and large projects, it just doesn't cut it. A business needs to be able to attempt to estimate and plan months or even years of a project (hence the prevalence of "fake" agile in companies that purport to use it).

The fact that the business does a horrible job with the estimation today does not mean that the organization (which, generally, is run by business people not developers) won't keep planning and targeting in the large.

The result of these two pieces (different tech, different process) is that enterprise and small development are moving farther and farther apart, which is a damaging long-term trend. Ideally, these two groups should be learning from each other. They should spend more time moving in the same world.

The enterprise learns that it probably doesn't need JavaEE and Oracle and a big planning process for myriad internal utility apps that could be done with Rails at a fraction of the cost and effort. The small company learns that relational integrity, transactions, estimation, and operations management are sometimes both necessary and profitable.

Even more importantly, individual employees in the industry can cross-pollinate ideas as they move between these environments over their careers, sorting fact from fiction and getting better at determining what will work.

The farther apart these groups are, the less appealing it will be for "startup guys" to work in an enterprise or a startup to hire on an "enterprise guy."

This trend -- since it keeps useful knowledge hidden -- can only help a 68% failure rate go up.

Software Discipline Tribalism

Unfortunately, people seem rarely able to stop at a reasoned preference -- e.g., "I like X, since X may offer better outcomes than Y" ... and too often end up, at least whenever group persuasion is involved, somewhere more dramatic, personal, extreme, and narrow -- e.g., "I'm the X kind of person who rebels against Y, since Y can offer worse outcomes."

This is as true in cultures of software development as anywhere else.

While many people have been guilty of this unproductive shift in attitudes, it seems many of the Agile development, dynamic languages, small-tools/small-process/small-companies crowd has long since fallen prey to it.

To be sure, there was much temptation to rebel.

At the start of the decade, proprietary Unix was strong; many processes came with expensive consultants, training, books, and tools ... and overwhelmed the projects they were meant to guide; many tools were expensive and proprietary. Web services were coming onto the scene and large players, with licenses and consulting hours to sell, created specs that were unwieldy for the small, agile, and less-deep-pocketed.

When the post-dot-com nuclear winter set in, small companies had no money to pay for any of that, and we got LAMP, Agile, TDD, REST, etc. Opposition to OO, which had been strong in many quarters, suddenly faded as OO was no longer identified (rightly or not) with certain problematic processes. Ironically, many new OO language fans had been ignoring the lightweight, free (speech and beer) processes that some OO advocates had been producing for years.

These have all proven to be useful tools and techniques and have created whole companies and enormous value ... but somewhere along the line, instead of being in favor of these tools and techniques because in some cases they produced better outcomes, either the leaders or the converts started thinking they were the rebels against anything enterprise, strongly typed, thoroughly analyzed, designed and well tooled.

This shift in attitudes does not help the industry ... nor even the clever consultants who lead the charge, deprecating last year's trend for a new one which they just happen to have written a book about.

We desperately need a broader perspective that integrates all of these pieces. There are things manually-written tests just won't do -- tools like Pex can help immensely, even if (or because... )they are from a big company.

Analysis and design are not bad words, while Agile can get dangerously close to simply surrendering to the pounding waves of change (and laughing at goals all the way to the bank) rather than building against the tide, and trying manage to a real outcome on a real budget.

Static languages can get hideously verbose for cases with functor-like behavior (Java and C# [pre-3.0], I'm looking at you). At the same time, go talk to some ActionScript developers -- who have had dynamic and functional for years -- and you'll see an amazing appreciation for the optional strict typing and interfaces in AS3.

REST is great, but in playing at dynamic, it turns out to be rather like C -- it's as dynamic as the strings you pipe into the compiler, and no more. Absent proper metadata, it cannot reflect and self-bind, so it sacrifices features that dynamic language developers love in their day-to-day coding.

Ironically, most of the critical elements of this "movement" -- along with open source -- are being subsumed into the big enterprise software companies at a prodigious pace. Sun owns MySQL and halfway owns JRuby; Java servers may serve more Rails apps than Mongrel/Ebb/Thin/etc. soon, Microsoft is all over TDD, IronRuby, IronPython...

I suppose the sort of tribalizing we see here is at least partly inevitable in any field. But it would serve the entire industry if that "part" could be made as small as reasonably possible. As a young industry with a poor track record and few rules, we ought to be more interested in better software outcomes than in being rebellious.

Microsoft Pex Moves the Needle Bigtime on Software Testing and Correctness

Over two years ago, I wrote about how neither the assurances of static compiler technology nor the ardent enthusiasm and discipline of TDD (and its offshoots) represent major headway against the difficulty and complexity of large software projects.

At the time, this issue came up in the context of static languages versus dynamic languages. There still exists a political issue, although today it is more transparently about different organizations and their view of the computing business. I will revisit the successor debate in my next post.

For now, however, I want to talk about a tool. In my post of two years ago, I suggested that significantly better analysis tools would be needed in order to make real progress, regardless of your opinion languages.

So I've been excited to see the latest tools from Microsoft Research fast-tracking their way into product releases -- tools which can really move the ball downfield as far as software quality, testing, productivity, and economy.

The most significant of these is called Pex, short for Program Explorer. Pex is a tool that analyses code and automatically creates test suites with high code coverage. By high coverage, it attempts to cover every branch in the code and -- since throwing exceptions or failing assertions or contracts count as branches -- it will automatically attempt to determine all of the conditions which can generate these occurrences.

Let me say that again: Pex will attempt to construct a set of (real, code, editable) unit tests that cover every intentional logic flow in your methods, as well as any exceptions, assertion failures, or contract failures, even ones you did not code in yourself (for example, any runtime-style error like Null Pointer or Division by Zero) or which seem like "impossible-to-break trivial sanity checks" (e.g. x = 1; AssertNotEqual(x, 0))

Moreover, Pex does not randomly generate input values, cover all input ranges, or search just for generic edge cases (e.g., MAX_INT). Instead, it takes a complex theoretical approach called "abstract interpretation" coupled with a SMT (satisfiability modulo theories) constraint solver to explore the space of code routes as it runs the code and derives new, significant inputs.

In addition, so far as I can understand from the materials I've seen, Pex's runtime-based (via IL) analysis means that it should work equally well on dynamic languages as on static ones.

To get an idea of how this might work, have a look at this article showing the use of Pex to analyze a method (in the .Net base class library) which takes an arbitrary stream as input.

For those of you who are inherently skeptical of anything Microsoft -- or anything that sounds like a "big tool" or "big process" from a "big company" -- I'll have more for you in my next article. But for now keep in mind that if Microsoft can show that the approach can work and is productive, user friendly, and fun (it is), then certainly we will see similar open source tools. After all, it appears the same exact approach could work for any environment with a bytecode-based runtime.

Last, I do recognize that even if this tool works beyond our wildest expectations, it still has significant limitations including

1. reaching its full potential requires clarity of business requirements in the code, which in turn require human decision making and input and
2. for reasons of implementation and sheer complexity this tool operates at the module level, so you can't point it at one end of your giant SOA infrastructure, go home for the weekend, and expect it to produce a report of all the failure branches all over your company.

That said, here are a couple more great links:

• "Contract Checking and Automated Test Generation with Pex" from PDC 2008
• "Pex: Automated White Box Testing for .NET" on MS DevLabs

On BASIC and JavaScript and 25 Years of Coding

I realized I've been putting programs together on a regular basis for 25 years now. I distinctly remember some of the earliest programs I worked on, around October 1983, and the fourth-grade teacher who let me get away with a lot more access to my school's computers than was the norm. When I somehow convinced may parents later that year to get me a machine ... things got even more out of control.

I worked with a bunch of the classic 80s "home computers": Tandy Color Computers (1, 2, and 3) and Apple II (and II+, IIe, IIc, IIgs, ...), and some not-so-classics like a CP/M machine that ran via an add-in board (with its own Z80) inside an Apple.

The languages and tools were primitive, performance was always a concern, most serious programs required at least some assembly-language component for speed and hardware access and, even if they didn't, compatibility across computer types was nearly impossible.

A lot like programming in JavaScript nowadays (I guess replace assembly language and hardware with native browser plug-in [e.g., gears] and OS APIs).

I could flog the 80s-BASIC / JavaScript analogy to death, but (1) if you read this blog, you likely can fill in the blanks yourself and (2) my goal isn't to bash JavaScript, which would be a side-effect of the drawn-out analogy.

What I find interesting is the question of why these things seem similar, and I have a hypothesis.

I have noticed that many members of my peer group, who started programming at a young age on these early microcomputers, have an affinity for tools, structured languages, and to a lesser extent models and processes. I wonder whether this affinity is not some kind of reaction against the difficulties of programming these early microcomputers in what can only be called a forced-agile approach, where debugging and testing took an inordinate proportion of the time, "releases" were frequent, and where the only evidence of working software was ... working software.

I will be the first to admit I am quite conscious that my experiences in the years before the C/DOS/Windows/C++/Mac era make me appreciative of and (depending upon your perspective) perhaps overly-tolerant of -- process, tools, models, definitions, infrastructure, etc. as a kind of reaction.

Let's stretch the hypothesis a little further: Gen-Y, who missed this era in computing (I would say it really ended between 1987 and 1989) will have typically had their first experience with coding in a structured, well documented, "standardized" ecosystem -- whether that was C on DOS, or Pascal on a Mac, or for those still younger perhaps C++ on Windows or even Java on Linux.

Depending on their age today, and the age at which they first took up coding, this generation had compilation, linking, structured programming, OS APIs, perhaps even OO and/or some process from the beginning. For them, it is possible to imagine, the overhead of the structure and process was something to rebel against, or at least a headache worth questioning.

Hence their enthusiasm for interpreted languages and agile approaches, and the general disdain for processes with formal artifacts, or extensive before-the-fact specs.

That's the hypothesis.

A simple study could validate at least the correlation -- by gathering data on age, age when someone started coding, the language/machine/environment they were using, perceived benefits or disadvantages in those years, and their interests today. And even the bare correlation data would be fascinating.

Considering that these approaches are often "best fits" for rather different kinds of development projects, knowing what sort of prejudices ("everything looks like a nail") individuals bring to the debate might add a lot of context to decisions about how to make software projects more successful, and how to compose the right teams and resources to execute them.

hquery: Separate data from HTML ... without templates!

Out of a bunch of cool presentation at least week's San Francisco Ruby meetup, my favorite was Choon Keat's demo of a working implementation of his hquey project -- a lib that lets you use Ruby, CSS, and jQuery patterns to bind data to views ... in the DOM, on the server side.

He had previously blogged about the motivation -- the core idea is that 10-15 years into the web application era, we are largely using template languages for integrating data into HTML. We've come a long way toward avoiding procedural code in our templates, enforcing MVC, etc. But aside from collectively agreeing to avoid a set of 'worst practices,' we're still inserting data into HTML in a manner reminiscent of 1998.

For well designed pages, with CSS classes and/or IDs, it should be possible to specify the data binding using CSS on the server, without any special binding tokens or markup. hquery does just that. So a designer can create full mockups with dummy data, and hquery can swap in the live data

The current demos and syntax might be touch verbose, but this is just an intial proof-of-concept and Ruby lends itself to easy re-arrangement an API if you need to.

This sort of radical division of dynamic data from HTML, while still using standards and not introducing yet-another-meta-templating-scheme reminds me a little of the old idea of using XML+XSLT to create pages. We all know how popular that ended up.

The hquery approach seems about fifty times more accessible to the present community of developers ... so the question is: how do we make this more popular and build community interest?

Engineering Departments Should Take Their Medicine Too

Without getting into gloomy predictions, it's clear a lot of companies will be feeling some pain soon, if they aren't already feeling it.

Software development groups at tech companies should take some medicine too -- cutting costs, becoming more competitive, improving morale, and attracting good talent at the same time.

How is that possible? Here's one way:

Unless your company is brand new, and you've got geniuses agile-ly coding your hot frog-sticker-collectors' social network, your project has a bunch of history, cruft, and mistakes in it.

It's nothing to get upset about, that's just the way of the world when developing software over the course of years in an organizational setting.

Yet many companies don't make any effort -- or actively resist any effort -- to identify those legacy problems and mitigate them. Perhaps it's fear of blame (why didn't you see this before? why didn't you say something? aren't you guys supposed to be experts?) or fear of appearing backward-looking rather than forward-looking (we'll make do with all that stuff; now can we shrink wrap and ship your latest proof-of-concept and call it next quarter's product?)

Suppose that, instead, your development group listed all the crufty bits that bug them. Stuff that maybe made sense at the time, but just isn't right any more -- wrong code, wrong architecture, wrong network protocol, wrong database, wrong format, whatever. Suppose the team got to rank these in order of annoyance factor, and impediment to productivity. Then, picking the top handful, they got to decide how to use the latest and greatest (and in many cases less expensive) technology to refactor and fix those modules.

A shortsighted manager might complain that 'if it ain't broke, don't fix it' -- it's a waste of resources.

But we know better than that.

In many projects a significant percentage of resources (up to two-thirds in extreme cases, based on my research and experience) can be spent wrestling with these "legacy cruft" issues. So, from a simple economics standpoint, it's definitely "broke" if you're spending $1 million per year on a dev group that could theoretically deliver the same functionality on $333,000, or 3x as much for the same $1 million.

A project that removes these old bits becomes more competitive. Why? The competition, particularly if it's a newer company or on a newer platform, isn't running with these parking brakes holding them back. Why should your team? If you can release the brake, you can deny your competitors something they definitely view as an advantage against you.

Moreover, these moves can boost morale and make your company more attractive to prospective employees in several different ways.

Getting rid of old morale-busting code makes everyone feel good.

Using a newer technology -- not as a novelty but because it's solving a real problem better than existing code -- is appealing to developers who want to learn new skills.

Doing a refactor of critical code without breaking existing code, wrecking release schedules, or introducing excessive ops downtime is a challenging and rewarding skill, kind of like working on the engine while plane is flying -- and top developers relish this kind of "black-diamond" assignment.

Finally, it tells everyone, inside the company and out, that this isn't Initech, where an engineer will have to work on 15-year-old tech for the next 15 years.

Presentations from SF Flash Hackers August '08

A couple of weeks ago I gave two mini-presentations to the SF Flash Hackers group, on topics I've talked about here before ... but I figured I'd post the slides to slideshare.

Here are slides about porting a large Windows app (Mindjet MindManager 7.2 with Connect) to run in the browser via Flash (developerd with Flex):

Porting a large Windows app to Flex

View SlideShare presentation or Upload your own.

 

And here are slides on generating ActionScript 3 code from UML class diagrams using my VASGen tool -- which is really a contribution building on two existing tools, the Violet UML modeler, and the Metaas ActionScript 3 meta-library (in Java):

As3 Code Gen from Uml

View SlideShare presentation or Upload your own.

Refactoring: Bittersweet if it Could Easily Have Been Avoided

I've been doing some refactoring on a decent sized app lately. Since there are lots of other people doing stuff and the app is close to a supported 1.0 release, it's like working on the engine of a car while it's cruising down the freeway. Not just tightening the cap on the coolant reservoir, more like converting the engine to run on hemp seed oil and corn cobs instead of gasoline.

On one hand, it's fun: mundane tasks that might feel academic become a lot more interesting when the task is part of the Rubik's cube of after-the-fact architectural change.

On the other hand, a lot of the refactor is an attempt to deal with unnecessary complexity in less complex way. The underlying complexity -- not mediocre code or an initial pass at red-green development -- is really the enemy.

And here's the rub: this app has some network protocol requirements that are (1) outrageously baroque; (2) opaque and undocumented; and (3) completely unnecessary for the app to function for its users.

As a result, it's already eaten over 120 man-months (and a staggering amount of money, let's just say over 8x what you're probably thinking the labor cost was).

At least 2/3 of that time and money (maybe 75%) went to nothing the user will ever see or care about, and nothing that offers any inherent business value or competitive advantage for the owner. That 2/3 is all expended in dealing with the bizarro protocol requirements ... and the issues and complexity it spawns: the design compromises, the testing, the bug analysis and remediation, the refactoring. I'm not even counting the actual delays and feature cuts in this cost.

Where did I get the 2/3 number? Well, the project was initially scoped without this network protocol issue. The estimates were written down. And, in the initial iteration -- about 30 man-months of work over 5 calendar months -- we tracked resource usage and found that core functionality and QA was reliably 1/3 and closely matched the estimates, while accommodating these "other" issues was 2/3. Week in, week out. No surprises. Things haven't changed since.

It's ironic because this pattern is something that often comes up when developing against a true legacy system -- the formats, protocols, and systems in place simply cannot be changed right away for the convenience (or productivity) of a new team. And yet this initiative was green-field, not legacy. It could have used any protocol in the world. There was 0 (zero) install base that needed compatibility with this particular system.

I almost wish I didn't know this information ... It's the one thing that puts a real damper on the refactoring fun: knowing that with even a tiny bit of management planning and the willingness to deal with team issues, it would be unnecessary.

When Corner Cases Attack

Notwithstanding the credit given to MTV's Real World, I think reality TV took off with that Fox special that featured a deer knocking a hunter on his rear. That guy just never saw it coming.

Same thing seems to happen too often when we start making architectural adjustments around a corner case in a software application.

The real problem is how to tell when an exception to the initial model -- the corner case or edge case -- is truly a strange bird, versus when it is a big piece of the model that was missed the last time through.

Sounds like something that should be easy to resolve with proper analysis, whether of the thick-stack-of-paper variety, OO variety, or an agile story-based style.

But it's not.

I've recently been doing some work on a project where this mistake was made. At some point in the past, this team looked at an odd duck, thought hard, and realized it was a big piece of their world. Big enough that the underlying infrastructure -- database, network services, etc. -- should be designed with sufficient generality that this corner case would become a mainstream case.

I understand how the analysis went, and why it made sense.

Only problem is, with 20/20 hindsight we can now see it was wrong. Bringing this edge case into the tent made the system an order of magnitude more complex, and it should have been left as an odd hack in a dozen places ... but as development happened over time, on several different module teams, the problem of the added complexity wasn't clear until it was too late.

To be a little more specific, I'll offer an analogy: you're writing a driving game or simulation. A big part of the fun is the spots where the players catch air, jumping their vehicles over obstacles to get ahead. But how to model the behavior of the cars as they fly through the air? A quick hack? or ... integrating elements of flight simulation or 3-D physics simulation?

Well, this team went the equivalent of the flight simulator route, and their hard-enough-to-code 2-D driving game has become a brutal 3-D simulation. All for a few jumps that in retrospect should have been hacked.

My point is not to complain about a historical decision that probably didn't seem bad given the info available at the time. It's to ask about how to avoid these problems in the future.

If you grant my assertion that, in some cases, no straightforward analysis reveals the problem at the moment the "how do we handle this case?" question is asked, then it seems like we need a set of milestones for a posteriori analysis.

We make a design call when we need to make a call, and then we need some specific questions -- and times or situations to ask them -- that will tell us if we're on the right track.

Specifically: What is the best and earliest way to ascertain if we're

1. monkeyhacking in an edge case that really demands a proper refactor
   OR
2. doing a refactor that way overengineers a solution that really just needs a monkeyhack

What do you think?

Higher Energy Prices == More Value in Software, Bandwidth

I got to see Arthur Rosenfeld speak at UC Berkeley's Physics Dept. graduation last week, when my younger brother finished up there as an undergrad. Dr. Rosenfeld's lab and LBNL worked in both research and policy, producing technical advancements in energy efficiency focusing on physical buildings.

His talk got me thinking that not only can information processing produce efficiencies ... but actually that higher energy costs may also be a boon to all manner of information processing and software applications, because it increases the general (relative) value both of processing and of bandwidth (and associated infrastructure).

The gains apply not only to applications directly focused on saving fuel (e.g. logistics systems), but also all of the productivity apps that allow more (or the same) industrial capability with less effort, time, and raw materials. This productivity argument is a standard ROI type argument about information systems. It just takes on a new urgency as forecasts for the energy component in these industrial processes show a bigger number.

Also worth re-examining are the applications which are meant not to make moving physical stuff more efficient -- but to obviate the need completely. The U.S. Mail's logistics capability may make Netflix much more efficient than driving an SUV to Blockbuster ... but BitTorrent makes Netflix look like the stone age. It's no surprise that the easiest thing to move as data is a data-like-object.

The other category is meat-like-objects. Telecommuting and satellite work sites are vastly more usable and useful now than they were even 6 years ago when companies (famously Sun) ditched office space after the dot-com crash.

While issues remain with 100%-offsite work, I believe the era of traveling to the office five days a week for 'info-worker' jobs is pretty much over. Already, I see many companies allowing or encouraging people to work somewhere else part of the week, coming in only 2 or 3 days. That movement can become broader based as the support software and bandwidth improves. In the verticals, the telemedicine model will become more prevalent.

We can also create improvements in industries where the physical world is the point, like grocery stores -- and no, I'm not thinking about stuff like WebVan even if Safeway.com does pretty well, and amazon.com can ship you a year's worth of pasta. I'm thinking that sharing and augmenting data from a store (and its upstream supply chain) can reduce the number of trips people take to the store ... you won't go when they don't have what you're looking for; you'll look at the produce or the microbrews remotely and decide whether to bother; you'll spend less time in the store, which means a smaller (both in business hours and space) site can service the same volume of sales.

But I'm getting too specific: my point is not to envision kooky Internet schemes for avoiding bruised apples. It's that all of the software we have (operating systems, applications), hardware (GPUs, quad-core procs), and bandwidth becomes measurably more valuable as they becomes a viable substitute good for energy. And where we always measured productivity by looking at time-replacement, that time becomes doubly valuable when we are also paying more to heat, cool, light, or motorize the environment where people need to spend that time.

No Uber-Soft Launches and No Stealth Mode

Uber-Soft Launches and Stealth Mode are two common practices that are usually big red flags of impending trouble.

To be clear, an Uber-Soft Launch is not a classic, small-scale launch where you release a decent version (maybe beta) of your product, but don't blast every PR trumpet you can find until you get a first round of feedback and some perf stats. There's nothing wrong with that; it borders on a best practice.

On the contrary, an Uber-Soft Launch is when the CEO or entrepreneur starts hedging about whether this launch is really the product or really the big launch he's been working toward and talking up. "We're going to just try this out and see what happens ... "

That statement is legit if it's intended as faux-modest understatement from a guy (or gal) who's clearly going for broke to make the product succeed. But if the firm or leader is really this wishy-washy and diffident about the launch, forget it. It's game over.

But then, in that case it doesn't matter because what the entrepreneur is really saying is that he doesn't expect to succeed so he's simply covering himself so he doesn't look silly after the failure. And that CYA attitude is one of the key things that indicates impending failure. It's a symptom of lack of conviction, a fear of failure that drives systematic bad decision-making.

Stealth Mode is a little less black-and-white, as there are a few cases where it may pay off. A few. Meaning not many. Luckily, Web 2.0 seems to involve far less "stealth mode" than Web 1.0 so it's less of a problem.

When might stealth mode be useful?

(1) A company has a specific physical or algorithmic invention (no, not Amazon one-click), and intends to patent it and to defend the patent vigorously (= has the massive cash to do so). Company wants to make sure it's documented and filed before anyone else files. If this is you, then you'd better be working toward the patent filing as fast as you can, no excuses. And get a good lawyer. If you're not ready and planning to defend, then stealth mode doesn't matter. Someone else can implement your technique if they want, and even patent it. You'll win or lose on execution and customer acquisition.

(2) A company's value is going to be based on a "network effect" play rather than a "hard-to-duplicate" play, and already has big the PR for the launch you lined up. In this case, since you know you're not "hard to duplicate," you don't want to spill the beans until you can fire off the giant PR cannons, at which point, you'll either grab a big enough chunk of network effect to sustain you, or else you'll drift. An example of this is Ning, which was in stealth mode for a long time. Marc Andreessen's celebrity and connections were the big PR blast, timed to match Ning's actual launch. But what if you're not Marc Andreessen or Kevin Rose and you're not going to make any headlines with your launch? Then you're not going to get a big "pop" when you come out of stealth mode, so really you're just:

Afraid of someone stealing your idea

But that's not a good reason for stealth. There are very few new ideas. If an entrepreneur thinks he or she has one, it almost certainly indicates insufficient research to find the people with the same or very similar idea before (and now), and consequently ignorance of why they failed (or might fail).

If this is you, get over yourself. Make it part of your "leadership agenda" to systematically find the previous incarnations of your idea -- or related ones. Analyze the heck out of them. Do better. Or be more popular. (Pick one or both).

But here's the twist, and if you're introspective then you saw this coming: you're not really afraid of someone stealing your idea, you're really

Afraid of someone not liking your idea

and you don't want to deal with that. You think that by waiting until you have perfect execution you will stun disbelievers with the beautiful product. That's just a delay/procrastination tactic. The underlying fear (of the product falling flat) will just make you want to delay and delay, making the product more and more "mature," so as to defeat nay-sayers.

Doesn't work. There will be nay-sayers. Embrace them. Love them. If you have money, make them into a focus group and pay them! Separate the whining pessimists from the ones with specific advice. Don't waste your time on the former, but realize that the latter are creating value in your company for free.

They are doing what your best product managers and designers should be doing -- finding and clearing roadblocks to adoption. Listen to them and verify what you think they're saying, by checking with other real people. Then you have a real bug list to work on, not getting the that drop-shadow AJAX doodad the right shade of pink.

As an entrepreneur, you're already drinking enough Kool-Aid by necessity; if you hide from naysayers, it just leads to a Kool-Aid overdose death-spiral.

I've been involved with companies that have committed both of these sins (and many more) so of course my perspective is warped by that. But don't take my word for it. Find the companies and entrepreneurs that you look up and want to learn from. Read their stories or go talk to them (being careful to filter out the Spiel and the 20/20 hindsight). Whatever you do, don't hide out convincing yourself you've invented cold fusion.

MozyHome Backup Fails to Backup Designated Files

I have been testing out Mozy's MozyHome remote backup product, and have found that it sometimes ignores new or changed files in the backup set marked to be backed up. Sometimes it "discovers" these files (or changes) days or weeks later and backs them up; other times, if I modify a file "nearby" (relative to the way my backup set is constructed), it will suddenly discover all the other changed files and back them up. Still other times, no poking or prodding seems to make it back up these files.

This is a serious problem. After all, the raison d'être of the product is backup. Imagine that a home user or a business installs this solution, marks files to be backed up, observes that the backup process is running successfully and then -- after a data disaster -- learns that, well, some of the files are backed up and some simply are not.

If Mozy were a fly-by-night outfit, one might say that better due diligence is required in choosing a backup provider. But with its recent acquisition by storage giant EMC and its global contract with GE, Mozy appears to be a solid company.

The software, though not perfect, basically works. The backup and restore are straightforward. You can encrypt locally with your own key so that no one but you could ever decrypt your data in the case of a breach (although interestingly, the file names are not encrypted, so don't count on hiding the existence of my_illegal_off_balance_sheet_transactions.xls, or gifts_to_my_mistresses.doc).

But we're talking about the number one, sine qua non, only -- really -- important use case for backup. It has to back files up. Or at least, if it doesn't, it needs to tell you what failed, when, and why.

When I first discovered this problem, I realized that publicizing it could have a negative impact on Mozy's business. So instead of blogging, I contacted them directly to learn more. Unfortunately, after a few back-and-forths, including my running their diagnostic tools and sending them the reports and explaining that I was not going to give their support personnel full remote access to my box without more information, they have gone radio silent.

As a software engineer, I am fully aware that this problem could be a strange corner case. Perhaps it is so narrow that it never affects anyone besides me. But I doubt it. And, in any case, this problem is severe enough that it warrants a little investigation to determine its breadth.

Why do I doubt it is an unusual corner-case failure? Simply because my configuration of the service is so "typical." I'm running XP SP2, NTFS on a well-maintained, modern machine, on a secure home network behind NAT, with no strange services or applications of any kind running (e.g., the kind that might hook and hack kernel file system operations, or leverage alternate data streams). The affected files are not under unusually named file paths, or have any funky attributes set on them.

The only things I am doing that are not defaults are using my own encryption key, and adding a few files to my backup set that aren't in the "My Documents" tree. My conclusion is that it is likely that whatever glitch is causing the software to miss files on my file system, is also missing files on other people's file systems. And they have no idea.

Perhaps this problem doesn't affect other users -- but without trying to verify the bug, how can Mozy know? In my last email to them, I specifically asked if their QA team had even attempted to replicate the bug. Had they tried and failed to reproduce it? Fair enough, maybe I could offer some help. But if they haven't tried, it makes you wonder what bug report could possibly be a higher priority? Maybe if their app runs off the rails and reformats your drive, that's a higher priority. But barring active destruction of your data, or a major security bug that could compromise you to a third party, I can't think of anything.

The ultimate problem here is not even an engineering problem. Yes, there's a bug in the software, but there are bugs in almost all software. Rather, it's a process problem. How does the QA process work? How does customer support work? What steps do you take if someone reports the Really Big Bug? Is the right thing to assume they're a crackpot? Can you afford to do that and not even look into it? (Hint: No.)

</end of regular post> <free:bonus>

Since I'm really not out to get these guys, but ideally want to help, I'm gonna offer a free first step: it's really easy to tell after the fact if this bug is manifesting, since the set of files actually backed up simply doesn't match the local description of the backup sets. So an easy diagnostic is to write a list of these two sets of files and diff them. If there are deltas, you've got a problem.

So... you push out an update to the client that creates a list of the files in the active backup sets and sends it over to QA as the last step in the online backup process. Then QA just has to generate a matching file list (the match is on the account id [email address] and either the date/time or the id number of the backup) from the Mozy meta-data store and compare.

Agile Clients Present Challenges for Fixed-Price Consulting Firms

As more companies adopt agile development techniques, their increased desire for flexibility is causing pain for fixed-time, fixed-price consulting partners who have not yet figured out how to make the new dynamic work.

Traditionally, there are two models for technology consulting: "Time and Materials" (T&M), where the consultant bills per unit time and is only minimally responsible for the overall timeline, and "Fixed-time, Fixed-price," where the consultant is responsible for proposing a combination of scope, budget, and timeline and then delivering it.

The problem is that the initial time and price estimation models are based on a particular deliverable target. And while it's traditionally been a consulting challenge to keep change under control during an engagement, now clients are being completely upfront and saying, "We are developing alongside you and we intend to make substantial changes to the target every month, as the project develops."

With agile methods, this is not a shocker. And the client is justified in believing that if their organization can deliver new features and make spec adjustments every few weeks, surely high-priced consultants should be able to keep up.

The good consultants can keep up. The problem is figuring out a way for them to get paid for it; otherwise they won't be able to do these kinds of projects for long. Without a model that specifically addresses the possibility that expensive goal changes may arise, the consultants are forced into (a) being less agile than the client, (b) eating the cost of the changes, or (c) asking for more money or time. None of these options are good ones for the consulting relationship.

A first step toward a solution is to change agile co-development from an externality to a core part of the service delivery model. An iteration length should be defined as part of the project work statement. So far so good. The tricky part is allowing the consultants to make time or price changes at the checkpoints between iterations. After all, wasn't this whole project sold as fixed-time, fixed-cost? Sounds like having one's cake and eating it too. The reality is that both parties need to allow these changes (the client needs to change the specs; the consultant needs to change the cost) in order for the process to work.

The problem on the client's end is that development methods like agile, even if they tie in to product development processes, are often more "micro" in the budgeting scheme than the department budget or consulting budget. That is, the client's finances are not as agile as their development.

On the consultant's end, if the engagement is renegotiated -- even at a micro level -- between iterations, the overhead of managing and selling the project goes through the roof. And of course it's always possible that agreement can't be easily reached to match a spec change with a cost change. In that case, the whole engagement can fall though adding massive risk to delivery and reputation all the way around.

While not a complete solution, a number of elements would be helpful

• Guidelines that throttle the rate of change between iterations via specified metrics, so that both sides have some ability to predict cost
• Escalation chains for approval based on the amount of money or time involved, so that small and large changes alike don't necessarily need to go to a VP Finance or the legal department every time
• Emphasizing these agile mechanisms in the sales process, so that an agile consulting firm (which might potentially cost a little more) remains competitive with others (who might offer a lower price but not be able to speak to how they will deal with these issues).

At this time, technology consulting opportunities abound and make good sense for clients and service providers alike. But to succeed in the long term, fixed-cost consultants need to evolve their models a few steps to keep up.

Double Black Diamond Software Projects

I've spent a good part of my career working on particularly challenging development gigs that I have come to call "Double-Black-Diamond Software Projects"

What exactly is a double-black-diamond project?

The name comes from ski trail markings, where a single black diamond indicates (at least in the U.S.) an "advanced" trail, while two black diamonds indicate "expert."

In reality, the difference between single- and double- black trails is that a strong skier can typically waltz onto a single diamond slope and have confidence that it may be interesting or challenging, but it will have a predictable outcome.

The double black trails, on the other hand, can feature hidden obstacles, cliffs, terrain features that vary with the snow conditions, and even areas which may be practically unskiable depending on conditions. (E.g., the cliffs in the background of this image are the double-black "Palisades" at Sugar Bowl.)

In software development, a double-black-diamond project is one where the outcome is in question from the beginning, not because of resource issues (e.g., too few developers or too little time), but because some fundamental unknowns are involved. Something new is being created, and it is sufficiently unique as to make it unclear whether it is even possible to succeed, or exactly what "success" will look like.

These unknowns might involve integration to an unknown external system, performance of problematic features like voice recognition, custom hardware, etc. If these challenges seem feasible enough, or success seems valuable enough, to make the software project worth a try (ultimately an investor decision), but the sheer implementation risk (as opposed to business risk) is clear from the start, you've got a double-black-diamond slope in front of you.

I will be writing a number of posts on these double-black-diamond software projects, and I plan to cover

• typical elements ("signposts") indicating a double-black project
• why, as a developer, you might ever want to get involved in such a project when there are lots of other opportunities
• gearing up: the skills, knowledge, and/or people to bring with you
• the art of keeping project sponsors properly informed (the subtle part is the definition of properly)
• how to handle risk and predict outcomes (as much as possible)
• maneuvers and techniques to gain leverage and minimize the odds of failures or surprises
• managing the definition of "project success" (and why it's essential to do so, even as a coder)

Project Manager as Diplomat: Herald or Negotiator?

If software project managers are diplomats -- and I think most project management roles see them this way -- then there are at least two distinct flavors: heralds (low level diplomats) and negotiators (high level).

The difference is sometimes imposed by the project/work situation, but more frequently appears to be self-imposed by the project manager.

The herald project manager sees his or her role as a courier bringing messages back and forth between various parties. The parties may be friendly or hostile, and the communications may be straightforward, complex, or threatening. But if the herald gets the info to the recipient in a timely way, his job is done and he expects to pass unmolested.

The negotiator project manager, on the other hand, sees his role as keeping everyone on the same page about decisions and outcomes. If (when) parties are not in agreement, the negotiator tries to bring about sufficient discussion and confrontation that something can be agreed upon.

The project manager rarely has direct authority over multiple parties in the project (e.g., engineering, product management, and marketing). He can, perhaps, control one closely aligned party through resource allocation. In general, though, I've seen more of a carrot-and-stick approach:

The PM has opportunities (e.g., a recurring meeting) where he holds the floor and escalates all of the known issues, so that everyone is painfully aware of the potential problems. He then reminds everyone of all the negative consequences coming down the pike for the project, if key decisions and compromises are not made. He also points out how well things will turn out for everyone if the project can reach completion on time, on budget, and with everyone more or less happy with what was done.

With enough persistence, it is usually possible to keep things on track. How? The negotiator's secret weapon in the business arena is his willingness to make people consider negative possibilities that they are not likely to raise on their own, and to make it seem matter-of-fact. Business groups (at least in the U.S.) are extremely uncomfortable thinking about and planning for negative outcomes. So the project manager gets to make a whole room full of normally confident people very uncomfortable. In order -- literally -- to alleviate this discomfort, they start to talk and become a bit more flexible.

It is clear that the negotiator PM is doing the heavy lifting, while the herald PM is a glorified secretary. The herald may be useful in a large project where so much information is flying around that it's worth having full time staff to keep it straight. He's holding off entropy but not solving big problems. The negotiator feels like his success depends on getting people to work together for success, which is a bigger challenge and produces bigger rewards.

When you are choosing, hiring, or staffing project managers for software projects, keep this dichotomy in mind. Unless your world is all rainbows and unicorns, you probably need a negotiator, and getting this sort of PM will pay off handsomely. You can supplement him, if the project is really big, with a herald PM to "do the paperwork."

But if you enter a challenging project with nothing but a herald PM, you are increasing your project risk considerably.

One Reason Software Development is Such a Malleable Process

This post is the fifth (the others are here, here, here, and here) -- and last planned one -- summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

At the beginning of Chapter 23, Steve writes:

Philip Metzger observed decades ago that technical staff were fairly good at estimation but were poor at defending their estimates [ ... ] One issue in estimate negotiations arises from the personalities of the people going the negotiating. Technical staff tend to be introverts. [ ... ] Software negotiations typically occur between technical staff and executives or between technical staff and marketers. Gerald Weinberg points out that marketers and executives are often at least ten years older and more highly placed in the organization than technical staff. Plus, negotiation is part of their job descriptions. [ ...] In other words, estimate negotiations tend to be between introverted technical staff and seasoned professional negotiators.

This is a valuable observation in estimation scenarios. It also reminded me of some hand-waving I did about 6 months ago in reply to a comment on this post. I wrote that the reason some industrial process control type procedures fail when applied to producing software is that "There are too many social vulnerabilities -- the software process itself is inherently 'soft' in a group dynamics sense, so it gets reshaped by the org's internal disagreements."

I stand by that comment, but it seemed a bit vague. I think Mr. McConnell's summary of the personalities that push and pull around software commitments is quite helpful. His description clarifies one of the software group's social vulnerabilities. After finding themselves agreeing to a problematic "estimate," engineering groups may be able to make up for their poor performance at the negotiating table by trying to "reroute power from the warp core" ... but we know that statistically, over the long term, that approach fails, leaving chaos in its wake.

What to do? A proverb says that a man representing himself has a fool for a lawyer. I.e., right or wrong, he is likely no match for a trained, experienced legal professional on the other side.

So let's get the software team some representation. Not a lawyer, but an executive advocate, someone with the personality and negotiation skill-set to go to the mat with other parties.

If this person has a good grasp of the technical issues, all the better; if not, he or she can take it offline and come back to the engineering team for a briefing.

Of course, if the engineering group somehow gets the impression that the representative has sold them out, making unauthorized or unreasonable commitments, then we are back at square one.

In Software Estimation, Fewer Inputs Can Trump More Inputs. Here's Why:

This post is the fourth (the others are here, here, and here) summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

There are estimation models that take a large number of inputs, and others that rely on only a few inputs. Some of the input data are more objective (e.g., the number of screens in a spec) while other data are more subjective (e.g., a human-assigned "complexity" rating for each of several reports).

One thing that Steve explains is that, when many subjective inputs are present, models with fewer inputs often generate better estimates than models with "lots of knobs to turn."

The reason for this is that human estimators are burdened with cognitive biases that are ridiculously hard to break free of. The more knobs the estimator has to turn, the more opportunities there are for cognitive biases to influence to result, and the worse the overall performance.

It should be reiterated that this same problem does not apply to systems that take many objective measures (such as statistics or budget data from historical projects) as inputs.

LOC Counts to Compare Individual Productivity? No Way.

"Measuring programming progress by lines of code is like measuring aircraft building progress by weight." - Bill Gates

The other day, someone on my project team proposed ranking developers by their lines-of-code committed in Subversion. I really hope the suggestion was meant to be tongue-in-cheek. But lest anyone take LOC measures too seriously, it's worth pointing out that LOC is a bad, bad metric of productivity, and only gets worse when one tries to apply it across different application areas, developer roles, etc.

Here are a few reasons why LOC is not a good measure of development output:

• LOC metrics, sooner or later, intentionally or unintentionally, encourage people to game the system with code lines, check-ins, etc. This is bad enough to be a sufficient reason not to measure per-developer LOC, but this is actually the least bad of the problems.
  
• LOC cannot say anything about the quality of the code. It cannot distinguish between and an overly complex and bad solution to a simple problem, and a long, complex, and necessary solution to a hard problem. And yet it "rewards" poorly-thought-out, copy-paste code, which is not a desirable trait in a metric.
  
• In software development, we desire nicely factored, elegant solutions to a problem -- in a perfect world, we want the least LOC solution to a problem that still meets requirements such as readability. So a metric that evaluates the opposite -- maximum LOC for each task -- is counterproductive. And since high LOC certainly doesn't necessarily mean bad code, there isn't even a negative correlation to use from the measurement.
  
• In general, spending time figuring out the right way to do something, as opposed to hacking and hacking and hacking, lowers your LOC per unit time. And if you do succeed in finding a nice compact solution, then it lowers your gross LOC overall.
  
• Even in the same application and programming environment, some tasks lend themselves to much higher LOC counts than others, because of the level of the APIs available. For example, in a Java application with some fancy graphics and a relational persistence store, Java2D API UI code probably requires more statements than persistence code leveraging EJB 3 (based on Hibernate) based on the nature of the API. Persistence code using straight JDBC and SQL strings will require more lines of code than EJB 3 code, although it’s most likely the “wrong” choice for an application for all sorts of well-known reasons.
  
• In the same application and environment, not every LOC is equal in terms of business value: there is core code, high-value edge-case code, low-value edge-case code. To imagine that every line of every feature is the same disregards the business reality of software.
  
• You may have read that the average developer on Windows Vista at Microsoft averaged very few lines of code per day (from 50 down to about 5 depending on who you read). Is Microsoft full of lazy clueless coders? Is that why the schedule slipped? I doubt it. There were management issues, but Microsoft also worked extremely hard to get certain aspects of Vista to be secure. Do security and reliability come with added lines of code? Unlikely – in fact, industry data suggest the opposite: more code = more errors, more vulnerabilities.

But no one on our team would ever write bad code, right? So we don’t need to worry about those issues.

Not so fast… Developers, even writing “good” code, generally make the same number of errors on average per line (a constant for an individual developer). So if I write twice as many lines of code, I create twice as many bugs. Will I find them soon? Will I fix them properly? Will they be very expensive sometime down the line? Who pays for this? Is it worth it? Complex questions. Never an easy “yes.” Or, as Jeff Atwood puts it, “The Best Code is No Code At All.”

And beautiful, elegant, delightful code is expensive to write (because it requires thought and testing). The profitability of my firm depends on delivering software and controlling costs at the same time. We don’t fly first class and we don’t use $6000 development rigs, even if we might offer some benefit to the client or customer as a result. And we don’t write arbitrary volumes of arbitrarily sophisticated code if we can help it.

Ok, so why, then, is the LOC metric even around? If it’s such a bad idea, it would be gone by now!

Here’s why: while LOC is a poor measure of developer output, it’s easy to use, and it’s a (primitive but functional) measure of the overall complexity and cost of a system. When all the code in a large system is averaged together, one can establish a number of lines per feature, a number of bugs per line, a number of lines per developer-dollar, and a cost to maintain each line for a long, long time into the future.

These metrics can be valuable when applied to estimating similar efforts in the same organization under similar constraints. So they’re worth collecting in aggregate for that reason.

But for comparing individual productivity? I don’t think so.