Microsoft Pex Moves the Needle Bigtime on Software Testing and Correctness

Over two years ago, I wrote about how neither the assurances of static compiler technology nor the ardent enthusiasm and discipline of TDD (and its offshoots) represent major headway against the difficulty and complexity of large software projects.

At the time, this issue came up in the context of static languages versus dynamic languages. There still exists a political issue, although today it is more transparently about different organizations and their view of the computing business. I will revisit the successor debate in my next post.

For now, however, I want to talk about a tool. In my post of two years ago, I suggested that significantly better analysis tools would be needed in order to make real progress, regardless of your opinion languages.

So I've been excited to see the latest tools from Microsoft Research fast-tracking their way into product releases -- tools which can really move the ball downfield as far as software quality, testing, productivity, and economy.

The most significant of these is called Pex, short for Program Explorer. Pex is a tool that analyses code and automatically creates test suites with high code coverage. By high coverage, it attempts to cover every branch in the code and -- since throwing exceptions or failing assertions or contracts count as branches -- it will automatically attempt to determine all of the conditions which can generate these occurrences.

Let me say that again: Pex will attempt to construct a set of (real, code, editable) unit tests that cover every intentional logic flow in your methods, as well as any exceptions, assertion failures, or contract failures, even ones you did not code in yourself (for example, any runtime-style error like Null Pointer or Division by Zero) or which seem like "impossible-to-break trivial sanity checks" (e.g. x = 1; AssertNotEqual(x, 0))

Moreover, Pex does not randomly generate input values, cover all input ranges, or search just for generic edge cases (e.g., MAX_INT). Instead, it takes a complex theoretical approach called "abstract interpretation" coupled with a SMT (satisfiability modulo theories) constraint solver to explore the space of code routes as it runs the code and derives new, significant inputs.

In addition, so far as I can understand from the materials I've seen, Pex's runtime-based (via IL) analysis means that it should work equally well on dynamic languages as on static ones.

To get an idea of how this might work, have a look at this article showing the use of Pex to analyze a method (in the .Net base class library) which takes an arbitrary stream as input.

For those of you who are inherently skeptical of anything Microsoft -- or anything that sounds like a "big tool" or "big process" from a "big company" -- I'll have more for you in my next article. But for now keep in mind that if Microsoft can show that the approach can work and is productive, user friendly, and fun (it is), then certainly we will see similar open source tools. After all, it appears the same exact approach could work for any environment with a bytecode-based runtime.

Last, I do recognize that even if this tool works beyond our wildest expectations, it still has significant limitations including

1. reaching its full potential requires clarity of business requirements in the code, which in turn require human decision making and input and
2. for reasons of implementation and sheer complexity this tool operates at the module level, so you can't point it at one end of your giant SOA infrastructure, go home for the weekend, and expect it to produce a report of all the failure branches all over your company.

That said, here are a couple more great links:

• "Contract Checking and Automated Test Generation with Pex" from PDC 2008
• "Pex: Automated White Box Testing for .NET" on MS DevLabs

What Does Pownce Have to Do With Bogus Internet Movie Rentals?

Maybe my point about Pownce got lost in a larger, broader, and more academic argument. That's fine; I'm glad to see Ted Dziuba (yes, uncov) pretty much made the same point about Pownce. He has a larger audience than I do. My larger point, of course, is that we're not always the hiking-in-redwoods, biking-to-work, inventing-computers-in-the-garage, tossing-whole-industry-paradigms-out-the-window-with-our-wicked-code folks that our narrative says we are. We're as ridiculous, self-absorbed, and self-referential as the folks we take swipes at on our blogs.

What does this have to do with Internet movie rentals? Well, Internet movies -- and VOD in general -- is one of those always-coming-never-here technologies. Even if does arrive, it'll be 10-15+ years into the (U.S.) broadband-penetration era, meaning it will kind of dribble out the exhaust pipe of innovation as an incremental use case long after its revolutionary potential has disappeared. But Netflix (with LG) and Apple are back to the baccarat table for another round of tech's most pleasant losing gamble.

While Apple and Netflix have made little gains, big ol' Safeway is now selling a wad of recent hit films at 2-for-$6. On DVD. With a non-revocable license to watch in perpetuity, carry around, re-encode (if you're "careful" how you do it), and pass on to one's heirs for the next 1000 years. Or you can just rent the films for $0.99 - $1.49. The studios aren't going to hassle Safeway about "$3=watch forever" pricing (and heck maybe Safeway is paying $2 for those DVDs), because it's a retail supermarket. They understand it and it makes sense in their world. The same reason Joe Teenager can sign up with a record club and, if he works his deals optimally, pay a better per-track price than Apple does (maybe 30% of retail).

The house will have the advantage as long as tech keeps thinking like a player (and believing its own mythos) and not like the casino (the other that it tells stories about). Who's the "house" ? ... well, studios, producers, broadcasters, cable networks, telcos, CE makers (guess what, they don't think of 'tech innovation' the way software folks do), even government ... and that's just a start.

I made this point before. Now I'm gonna take it a step further, and talk about one way to move forward.

If you're an innovator in one of these areas, and you need to get a foot in the door, take on another perspective in addition to your own. Try and find a place to live in the other industry's model of the world (their value chain/web/network). I don't mean philosophically, I mean literally. Think of it as hacking their worldview, if it makes you feel better.

Here are some examples:

• When ecast needed legal clearance to get running with a digital library of songs in 1999, "MP3" and "Internet download" were conversation killers with record labels. ecast needed to be a jukebox, don't worry about the gears and levers inside, nothing to see here, move along. There were some hiccups in licensing until they shifted to the latter approach. Record labels know what a jukebox is and how they make money. Problem solved.
• Mediabolic ended up taking a different course, but at one point when the firm was working on  VOD with a set-top box, they looked into subleasing excess digital broadcast spectrum time for moving their data. What does that mean? It meant they would have been in some real sense a broadcaster using TV spectrum to reach customers. Which is a whole different conversation with content owners from the "renegade startup conversation."
• When Skip Interaction focused on travel data management, and mobile transactions, I advocated becoming a travel agency. While Skip would not likely have made much (any?) money on bookings, it changes the conversation with airlines, travel agencies, and companies whose employees travel a lot. Skip would have appeared as a "known" entity in the industry. All of a sudden, instead of requiring custom arrangements to do anything, Skip would have had a huge pile of boilerplate data and access to work from.

These examples are just from my personal experience -- I'm certain that this strategy is not at all uncommon. But we don't hear about it a lot, because it doesn't fit the plotline we want.

Post-structuralism for Dollars in Silicon Valley

On the eve of one of our grand theatrical spectacles, featuring one of our industry's undisputed showmen as emcee, let's pause for a minute and imagine that we've got our mental map of the tech world all wrong. We've been drawing lines not just in the wrong places, but where the model isn't about lines at all. As a result, we're misunderstanding ourselves and a lot of other stuff. And, since the economy doesn't care about our degree of self-awareness, it means we all -- entrepreneurs, investors, engineers, designers, marketers, journalists -- are leaving big money on the table.

I'm hoping that last sentence might catch someone's attention, especially that of the VCs who have a lot to gain or to lose based on their understandings or misunderstandings.

Where do we draw lines by mistake? We live in a structuralist mythology that we make up because it's easy and it's a shared set of beliefs that we can use to argue (in circles sometimes):

• The "new" versus the "old" is of course the big daddy duality
• How about Client / server, Startup / enterprise
• Geek / management, Engineering / marketing
• East Coast / West Coast, Stanford / Berkeley
• Open source / proprietary
• Content / code
• NorCal / SoCal, Hollywood / Silicon Valley
• Entertaining / Functional

These just scratch the surface -- you can probably think of fifty pairs of your own now, along with examples of how they are convenient, self-reinforcing, and yet ultimately invalid.

They are a cliche, but they own the conversation. I want to talk about the last few pairs: think of the narratives we make and live inside, as regards Northern California vs. Southern California, Hollywood vs. Silicon Valley, and Entertaining vs. Functional.

Now look at Steve Jobs, an entertainer who could get his fans to poke out their own eyes and get in line for an iPatch. Apple is theater, the Apple store a set, a Disney experience for people who feel superior making snarky comments about Disney. Do you really think computer geniuses work at the Genius Bar? "Space may be the final frontier / But it's made in a Hollywood basement" -- and in this case the fans have already reserved tickets to the sequel.

But, wait! I'm not bashing Apple or Steve -- that would be buying into the very dualism I'm pointing up here...

Now consider also: we have another award show, the Crunchies, coming up. Before you laugh, remember this event isn't intended as a joke. And donating money to charity doesn't make you serious, just generous.

We have our gossip writers, our A-lists and even our stunts that get a third of the Valley looking self-consciously at the floor, embarrassed at having to wonder whether Carly Fiorina or Leah Culver has set feminism back further, a third watching our great-standup-with-the-dirty-mouth pretending they all get the joke, and another third just saying the heck with it and jumping on.

Ya know, maybe we're not so far away from our SoCal cousins as we think. And just to keep it clear: I'm not saying there's a middle position, a compromise, where we're all gonna end up. I'm saying that the duality itself is a fiction. So there is no middle. Until we get this, we're going to continue to have a heck of a time getting some things right. Or making money with them!

Like what? Like things that aren't quite code and aren't quite content. We never got SVG right, because it didn't fit into the categories we thought made up the world.

Is a Flash media code or content? When my mom gets an e-card, she thinks it's content, not an app.

We don't have a cross-platform vector graphics standard in 2008 (!) ... because practical vector graphics in the real world means some logic as well as geometry.

Why can music acts sell (via the carriers) all manner of wallpapers and ringtones for cold hard cash to technically unsophisticated folks, and we still can't get the average person to install or run a mobile app on their phone for free? We're playing that game wrong and we can't see it.

We have every manner of "media center" -- hardware, software, open, closed, expensive, free. Even Microsoft has been in on the act for five years now. And I still don't know any regular folks at all who use any of them.

We have tons of great ideas on the shelves because we can't understand 'the other' well enough to make the deals we need on content licensing. (They don't understand us either.)

Three quarters of "Web 2.0" is not about any kind of functionality at all; it's not even about interaction design. It's about the glossiest thinnest veneer of user experience. It's about buttons and realistic smoke ... the kind of thing that some folks refer to as "production values." Moreover, three quarters of it is free and ad-supported. Not unlike most radio and television. Unlike radio and television, its reach is minimal. Ask your aunt in Duluth what del.icio.us is. Before we spend another session making fun of the studios for not realizing where their business really is, we might think about what an adjustment in ad rates will do to our own.

One last time for the folks in the cheap seats: it's not about right or wrong, steak or sizzle. We're just not where we like to think we are. We're somewhere else. To get a better idea where we are, we need to do some demolition on our usual Monday morning narratives. It's uncomfortable. The good news is, there's a ton of opportunity. And eventually it feels good to realize we've outgrown the notion that the world is flat.

Now go to Macworld and enjoy the, uh, show.

Who Considers These Folks the Elite Again?

I just read David Megginson's "Programming languages of distinction" in the context of Jeff Atwood's great post on The Shiny. The Scala cartoon cracked me up. But in real life, what truly "elite" programmers are going to move from Megginson's "Step 1" (too many riff-raff using their current language) to "Step 2" (look for a new, little-known language with fewer annoyances)?

At the end of the day, all of these trends/fads/fashions fall somewhere between two well known poles:

• Assembly language, as specified for typical mass-produced hardware (x86, 680x0, ARM, etc.)
• Lisp, and in particular Scheme

If you start with Assembly as your deck of cards (stacks and registers and jumps), and start shuffling and doing slight-of-hand tricks, you can get C, C++, Java, et al. If you start with a Scheme deck (lambda expressions), you can shuffle to get Haskell and Erlang.

If you throw all the cards on the floor and mix-and-match from both decks, you get your Ruby and your Scala and your "JavaScript 2/ES4" (well, maybe).

The reasons for religiously sticking to one pole or the other are not compelling for coding at the "application level." That's a fairly trivial statement. Once we specify the sort of typical end-user or business application that makes up the bulk of development, we evade the issue of whether a primitive imperative language (C/Assembler, close to the metal as far as specifying instructions) or a functional one (Haskell/Erlang/etc., potentially easier to parallelize on multi-core hardware) will perform better.

In the application world, we have hybrids like C#/Java/Ruby/ECMAScript. Sure, Scala. At that point, all sorts of other considerations come into play, from design to cost to maintainability to security, which have more bearing than whether a curly brace is a bad thing or /: is evil as an operator.

And guess what? With ANTLR and some open source code you can go make your own half-a-cup of Assembler-half-a-cup-of-Scheme language quite easily. You can use Kanji for the keywords, Roman numerals, a keyboard with a special key that generates Unicode chars above 0xFFFF for declarations, and maybe change the text direction to indicate which way functions are evaluated. What a niche! You'll be the leading authority! You can write a book. You can entice mediocre engineers and terrorize bad ones. You can entice mediocre engineering managers and terrorize bad ones. Or you can work on software that solves real problems for real people.