HD or Blu? Format Wars are Over, Winner is x264

Holiday sale $99 refurb HDDVD players couldn't win the war for the HD camp, and the PS3 still shows no sign of ending it in BluRay's favor.

Meantime, the de facto standard online for HD disc content, legit or not, is x264.

After DivX/XviD/MPEG-4 replaced MPEG-2 as a generally practical format for video content from DVDs, it took some time before gadget makers released decent players that could load up a DivX stream off of any CD-R or DVD-R and play it (reliably).

My money says we'll have a much shorter wait this time. In fact, in the end-of-year spirit of goofy tech prognostication, I'll wager that by Christmas of next year

1. The "official" format war between BluRay and HD-DVD will still not be meaningfully resolved
   
2. But 200 bucks will get you a DVD player that can load up an x264 stream and play it back, upsampled and/or at HD resolution
3. That same player (or maybe the $300 version) will either have a USB port so you can plug in your 1TB external enclosure and play HD movies that you've, um, obtained, without using a disk at all, or else it'll have a network port.
4. Game over.
   

Loving RoR with NetBeans 6

I've switched to NetBeans 6 for Ruby coding, and I'll never go back to those bogus text editors.

Some time ago, I wrote about IDEs for Ruby. I'm not sure I did a great job pointing out out that for Ruby (like JavaScript and other dynamic languages) some of the basic features we'd like from an IDE (code completion, refactoring) are non-trivial problems. The Sapphire in Steel blog has some great articles on working those problems.

The milestones for NetBeans were promising; the other free contender, the Eclipse bits formerly known as RadRails, are now part of the "Aptana IDE." Aptana is a reasonable tool for JavaScript, but last time I checked, Aptana was in "1.0" and the Ruby support still crashed and failed in interesting ways. The fine print points out that Ruby/Aptana/RadRails isn't 1.0. Bummer.

I'm very excited about this NetBeans release. For a detailed discussion of features, check out Roman Strobl's review (part 1 and part 2). Once the app starts and gets warmed up (which admittedly takes longer than booting the OS on my Windows server box), it does a pretty good job of the hard stuff, like refactoring. And it's rock-solid on the rest, like interactive debugging.

Unfortunately, Rails 2.0 became official at about the same time NetBeans 6 did. And Rails 2 changes a few things, like filenames/extensions on erb templates, that NetBeans doesn't know about. But it's easy to work around those things, and I'm sure an update will be forthcoming.

Amazon SimpleDB isn't Astoria... it Could Be, but Does it Need to Be?

A while back I wrote about Microsoft's Astoria REST-based relational data store in the cloud (or in your data center, if you want it there).

With Amazon's SimpleDB, we're a step closer to making this vision a reality. Now we're almost on track for competition (and sooner-than-later commoditization) of the new world where you don't even need MySQL to store your stuff.

Why almost? Because SimpleDB is not a full RDBMS, but is looks more like a flavor of triple store. Now, a typical (i.e., SQL-style) RDBMS can be built on top of a triple-store fairly easily. So we could will see a SQL processor, JDBC drivers, and the like, from the community pretty soon.

Another way to look at that "top layer" is to take a REST API like those used by Astoria or ActiveResource [PDF link] and simply implement that. Not as expressive as hardcore SQL, but easier, and probably enough for many applications.

What I don't see -- in the long run anyway -- is applications developing against thin wrappers specific to the Amazon triple store service itself. There's nothing fundamentally flawed in doing so ... it's just that, for a variety of reasons, data storage has evolved very slowly. The relational model is going on 40 years old, but still reigns supreme in terms of popularity, even if it has conceptual or technical flaws when put to work in today's applications.

Given the brilliant data storage alternatives that have fallen flat time and again, I doubt Amazon SimpleDB will change the way people talk about storing structured data. So SimpleDB doesn't need to be SQL but it will probably need to at least be RESTful.

How Bay Area Subprime Mortgages Relate to High-Tech Startups

Earlier this week, Tom Campbell, Dean of Berkeley's Haas School of Business, gave a long interview about "the mortgage crisis" on KCBS' in-depth segment [MP3]. At one point, the conversation turns to the general paucity of housing in the Bay Area (relative to demand, anyway), and the notion that landlords may be beneficiaries as folks suddenly discover that (1) they can't afford $975,000 for that 3-bedroom house and (2) it isn't worth $975,000 anymore either.

But aren't these b-school types the ones who are always telling us to "think outside the box"? The best Tom can come up with is a kind of housing trust or co-op, that lets you own, say, 25% of an expensive house, while you still get to live in it (an investor group owns the rest).

That's a really cute way of pumping up housing prices and speculation even further, by letting investors who would never do the real-estate transaction themselves pool their investments and then spread them across a bunch of properties (doesn't this sound a little like the mortgage problem we just saw?), while leaning on loan guarantees to cover their downside.

The cost of housing and difficult commutes are a big brake on the tech industry; they consistently rank at or near the top of Silicon Valley business leaders' concerns about the growth of their companies and the success of the region.

In the positions I've held over the last 4 years or so, I've been responsible for hiring engineers. Not just punch-the-clock engineers, but wicked smart, willing-to-build-it-from-scratch-but-wise-enough-not-to startup-minded engineers. But it is brutally hard. Even with robust salaries, it is difficult to get applicants in the door, let alone hired. And, no, it's not due to a shortage of U.S. geeks. Housing costs and impractical commutes seem to be the primary limit on the realistic pool of applicants.

Over time, if we don't do something about it, we'll only have the old-guard geeks (who bought homes a long time ago or live in rent-controlled places in SF or Berkeley) and immediate college grads (who are up for an adventure and happy to have roommates).

The problem is, when your only tool is sprawl, and you run out of farmland to sprawl on, you either throw in the towel (Silicon Valley) or you sprawl farther away (American Canyon, Fairfield, Tracy, even Monterey). When your only tool for traffic congestion is building more lanes, that's what you try to do, even though latent demand means that more freeway creates more traffic in the long run, not less.

If I'm so clever, what am I suggesting? We need to use a different tool, namely smart growth.

Among other things, we need significantly higher density, infill development, and more mixed-use development (residential, commercial, and light-industrial uses in the same or nearby buildings). This isn't a speculative proposal: where it has happened locally, it has been popular. Look at Santana Row, San Mateo, downtown San Rafael, even South of Market/Mission Bay SF (although arguably the planning there didn't go nearly far enough, leaving too few housing units to make any dent in affordability).

Luckily, the Bay Area does not share America's poorly grounded prejudice against living in towns. So communities like the ones I propose, on the peninsula and in the East Bay (Hayward, anyone?) would likely be embraced. And infill opportunities abound: Every time you see a big-box store in the Bay Area, whether you love them or hate them, imagine a few stories of residences on top and additional businesses lining the enormous "blank" sides of the store at street level.

Will this generate so much housing that no one will be tempted to take on debt at crazy terms they can't afford? Of course not. But if it can help keep that housing-cost-to-salary ratio from growing quite so fast for the folks we want (and need!) to work with, while bringing the ancillary benefits of denser communities, it seems like a no-brainer.

No Uber-Soft Launches and No Stealth Mode

Uber-Soft Launches and Stealth Mode are two common practices that are usually big red flags of impending trouble.

To be clear, an Uber-Soft Launch is not a classic, small-scale launch where you release a decent version (maybe beta) of your product, but don't blast every PR trumpet you can find until you get a first round of feedback and some perf stats. There's nothing wrong with that; it borders on a best practice.

On the contrary, an Uber-Soft Launch is when the CEO or entrepreneur starts hedging about whether this launch is really the product or really the big launch he's been working toward and talking up. "We're going to just try this out and see what happens ... "

That statement is legit if it's intended as faux-modest understatement from a guy (or gal) who's clearly going for broke to make the product succeed. But if the firm or leader is really this wishy-washy and diffident about the launch, forget it. It's game over.

But then, in that case it doesn't matter because what the entrepreneur is really saying is that he doesn't expect to succeed so he's simply covering himself so he doesn't look silly after the failure. And that CYA attitude is one of the key things that indicates impending failure. It's a symptom of lack of conviction, a fear of failure that drives systematic bad decision-making.

Stealth Mode is a little less black-and-white, as there are a few cases where it may pay off. A few. Meaning not many. Luckily, Web 2.0 seems to involve far less "stealth mode" than Web 1.0 so it's less of a problem.

When might stealth mode be useful?

(1) A company has a specific physical or algorithmic invention (no, not Amazon one-click), and intends to patent it and to defend the patent vigorously (= has the massive cash to do so). Company wants to make sure it's documented and filed before anyone else files. If this is you, then you'd better be working toward the patent filing as fast as you can, no excuses. And get a good lawyer. If you're not ready and planning to defend, then stealth mode doesn't matter. Someone else can implement your technique if they want, and even patent it. You'll win or lose on execution and customer acquisition.

(2) A company's value is going to be based on a "network effect" play rather than a "hard-to-duplicate" play, and already has big the PR for the launch you lined up. In this case, since you know you're not "hard to duplicate," you don't want to spill the beans until you can fire off the giant PR cannons, at which point, you'll either grab a big enough chunk of network effect to sustain you, or else you'll drift. An example of this is Ning, which was in stealth mode for a long time. Marc Andreessen's celebrity and connections were the big PR blast, timed to match Ning's actual launch. But what if you're not Marc Andreessen or Kevin Rose and you're not going to make any headlines with your launch? Then you're not going to get a big "pop" when you come out of stealth mode, so really you're just:

Afraid of someone stealing your idea

But that's not a good reason for stealth. There are very few new ideas. If an entrepreneur thinks he or she has one, it almost certainly indicates insufficient research to find the people with the same or very similar idea before (and now), and consequently ignorance of why they failed (or might fail).

If this is you, get over yourself. Make it part of your "leadership agenda" to systematically find the previous incarnations of your idea -- or related ones. Analyze the heck out of them. Do better. Or be more popular. (Pick one or both).

But here's the twist, and if you're introspective then you saw this coming: you're not really afraid of someone stealing your idea, you're really

Afraid of someone not liking your idea

and you don't want to deal with that. You think that by waiting until you have perfect execution you will stun disbelievers with the beautiful product. That's just a delay/procrastination tactic. The underlying fear (of the product falling flat) will just make you want to delay and delay, making the product more and more "mature," so as to defeat nay-sayers.

Doesn't work. There will be nay-sayers. Embrace them. Love them. If you have money, make them into a focus group and pay them! Separate the whining pessimists from the ones with specific advice. Don't waste your time on the former, but realize that the latter are creating value in your company for free.

They are doing what your best product managers and designers should be doing -- finding and clearing roadblocks to adoption. Listen to them and verify what you think they're saying, by checking with other real people. Then you have a real bug list to work on, not getting the that drop-shadow AJAX doodad the right shade of pink.

As an entrepreneur, you're already drinking enough Kool-Aid by necessity; if you hide from naysayers, it just leads to a Kool-Aid overdose death-spiral.

I've been involved with companies that have committed both of these sins (and many more) so of course my perspective is warped by that. But don't take my word for it. Find the companies and entrepreneurs that you look up and want to learn from. Read their stories or go talk to them (being careful to filter out the Spiel and the 20/20 hindsight). Whatever you do, don't hide out convincing yourself you've invented cold fusion.

MozyHome Backup Fails to Backup Designated Files

I have been testing out Mozy's MozyHome remote backup product, and have found that it sometimes ignores new or changed files in the backup set marked to be backed up. Sometimes it "discovers" these files (or changes) days or weeks later and backs them up; other times, if I modify a file "nearby" (relative to the way my backup set is constructed), it will suddenly discover all the other changed files and back them up. Still other times, no poking or prodding seems to make it back up these files.

This is a serious problem. After all, the raison d'être of the product is backup. Imagine that a home user or a business installs this solution, marks files to be backed up, observes that the backup process is running successfully and then -- after a data disaster -- learns that, well, some of the files are backed up and some simply are not.

If Mozy were a fly-by-night outfit, one might say that better due diligence is required in choosing a backup provider. But with its recent acquisition by storage giant EMC and its global contract with GE, Mozy appears to be a solid company.

The software, though not perfect, basically works. The backup and restore are straightforward. You can encrypt locally with your own key so that no one but you could ever decrypt your data in the case of a breach (although interestingly, the file names are not encrypted, so don't count on hiding the existence of my_illegal_off_balance_sheet_transactions.xls, or gifts_to_my_mistresses.doc).

But we're talking about the number one, sine qua non, only -- really -- important use case for backup. It has to back files up. Or at least, if it doesn't, it needs to tell you what failed, when, and why.

When I first discovered this problem, I realized that publicizing it could have a negative impact on Mozy's business. So instead of blogging, I contacted them directly to learn more. Unfortunately, after a few back-and-forths, including my running their diagnostic tools and sending them the reports and explaining that I was not going to give their support personnel full remote access to my box without more information, they have gone radio silent.

As a software engineer, I am fully aware that this problem could be a strange corner case. Perhaps it is so narrow that it never affects anyone besides me. But I doubt it. And, in any case, this problem is severe enough that it warrants a little investigation to determine its breadth.

Why do I doubt it is an unusual corner-case failure? Simply because my configuration of the service is so "typical." I'm running XP SP2, NTFS on a well-maintained, modern machine, on a secure home network behind NAT, with no strange services or applications of any kind running (e.g., the kind that might hook and hack kernel file system operations, or leverage alternate data streams). The affected files are not under unusually named file paths, or have any funky attributes set on them.

The only things I am doing that are not defaults are using my own encryption key, and adding a few files to my backup set that aren't in the "My Documents" tree. My conclusion is that it is likely that whatever glitch is causing the software to miss files on my file system, is also missing files on other people's file systems. And they have no idea.

Perhaps this problem doesn't affect other users -- but without trying to verify the bug, how can Mozy know? In my last email to them, I specifically asked if their QA team had even attempted to replicate the bug. Had they tried and failed to reproduce it? Fair enough, maybe I could offer some help. But if they haven't tried, it makes you wonder what bug report could possibly be a higher priority? Maybe if their app runs off the rails and reformats your drive, that's a higher priority. But barring active destruction of your data, or a major security bug that could compromise you to a third party, I can't think of anything.

The ultimate problem here is not even an engineering problem. Yes, there's a bug in the software, but there are bugs in almost all software. Rather, it's a process problem. How does the QA process work? How does customer support work? What steps do you take if someone reports the Really Big Bug? Is the right thing to assume they're a crackpot? Can you afford to do that and not even look into it? (Hint: No.)

</end of regular post> <free:bonus>

Since I'm really not out to get these guys, but ideally want to help, I'm gonna offer a free first step: it's really easy to tell after the fact if this bug is manifesting, since the set of files actually backed up simply doesn't match the local description of the backup sets. So an easy diagnostic is to write a list of these two sets of files and diff them. If there are deltas, you've got a problem.

So... you push out an update to the client that creates a list of the files in the active backup sets and sends it over to QA as the last step in the online backup process. Then QA just has to generate a matching file list (the match is on the account id [email address] and either the date/time or the id number of the backup) from the Mozy meta-data store and compare.

Free ActionScript Code Generation: VASGen 0.2 Release

I finally got a new rev up of VASGen, my Violet UML ActionScript 3 Generator.

It is built against Alexandre de Pellegrin's enhanced version of Violet UML, which is designed to run either standalone, as an Eclipse plug-in, or via JNLP ("Java Web Start"). This version of Violet also has a much glossier look and feel, which make it particularly pleasant to use (thanks, Alexandre!)

     

The main functional enhancement to VASGen is in the area of package support. This version supports placing classes and interfaces (and nested packages) inside of packages, using the Violet modeling tool. When code is generated, the proper folders and qualified type names will be used.

If you don't want to package all of your types, or you are just sketching around inside what will end up one package, you can specify a default package by just dropping a package icon anywhere in the diagram and marking the name of it with a '+' (e.g., "+foobar").

I've tossed in a couple of bug fixes too. Details and the download are here.

If you uncover problems, I would love to hear about them and hopefully fix them.

Hazardous Attitudes: Disregarding VC Due Diligence

The FAA identifies five "hazardous attitudes" that have proven so dangerous to pilot decision making over the years that they are explained in the FAA "Pilot's Handbook of Aeronautical Knowledge" and included by reference in exams and regulations. These attitudes are Anti-authority ("Don't tell me."), Impulsivity ("Do it quickly."), Invulnerability ("It won't happen to me."), Macho ("I can do it."), and Resignation ("What's the use?").

Working with startups, I've seen entrepreneurs exhibit all of those attitudes when trying to convince others (and themselves!) that they needn't worry about VC due diligence.

I would advise those entrepreneurs -- and any wannabe entrepreneurs -- to read Rick Segal's fabulous post on due diligence. In addition to giving advice, Rick explodes a commonly held notion that VC due diligence is just a formality and that, if you get to the d.d. stage with a VC, you're all set, you can just wait for the wire transfer to hit your bank account.

Due diligence is real -- Rick suggests that one or more of three deals he's currently looking at will not close because of problems at the due diligence stage. Read that sentence until you believe it. Then, before you tell yourself that it doesn't matter because you're going to bootstrap, you'll never need a VC or a corporate investor, go back and read the five hazardous attitudes again. If you think success means believing in Plan A so much that you don't bother preparing a Plan B, the odds are you'll be laughing about this failure some time down the line over a beer.

Now that we've got that out of the way...

A couple of Rick's points that deserve emphasis:

• Financial Forecasts. Of course they'll be rosy. What's important are the assumptions. Where did you get your data? How hard did you try to get good data? Is the logic that ties the data together sound?
• Business Thesis and Assumptions. "... [W]hat do I have to believe?  What do you believe?  And, of course, what are the assumptions behind those beliefs. ... You have to have the same story, metrics, thesis, etc, from day one." If you change your execution plan a few times, that's to be expected. If your fundamental beliefs about the space are changing faster than you can execute, you have little chance.

At some point between the kitchen table stage of your startup, and the time when you walk into a VC meeting, the following things will become relevant. Plan accordingly:

• "Understandings" or "Gentlemens' Agreements" with investors or employees. Unusual terms can be changed or dealt with at VC time, but I wish I had a buck for every time a CEO told me these issues would just melt away because everyone would be so pleased at the prospect of making a big deal or closing a round.
• Questionable expenditures. This can either be a few big items or systematic spending that adds up. Don't do it, and if you do, don't try to hide it or hand-wave. I've seen a VC identify a six-figure sum missing from the books, and still make the investment. The firm identified the issue and decided they would arrange the deal so as to handle it and get it under control. It wasn't a showstopper for the company, but it was the end of the line for the guy who tried to cover it up.
• Team issues. Although Rick says that not every VC would talk to all the employees in a small company (he would), it's a good bet that your core exec team -- and probably everyone in your first 8 people -- will get a good grilling regardless of the VC. The team has to be a team, and you won't be able to fake it. This doesn't mean everyone needs to agree or be buddies, but they need to function properly as a group. If you don't have a team, the VC will figure this out, and your funding is unlikely.
• Product. Ok, this should be obvious. You can spin the marketing claims ("a better way to manage your contacts" could be anything) but you can't fake the technical claims ("syncs up to 5,000 contacts to any device" is either true or it's not).
• Customers. Although some VCs seem to be trying to get all the risk out of their portfolios by investing only in companies with a solid customer base, that is their problem, not your excuse to pretend you have customers that are fictional, occasional, or just plain unlikely.

If you still think this is all hypothetical, or won't affect you, then take another look at the five hazardous attitudes and ask yourself: are you really planning for success? just closing your eyes and hoping? or fooling around and enjoying the ride?

Overclocking: I So Ought To Have Known Better

Ok, I'm not a big pc modder guy, but I was trying to squeeze some extra life for gaming out of a machine that isn't the latest. Using techniques I had successfully executed before, I was in the process of kicking the 3.2 GHz CPU up to 3.65, when I toasted my HDD. The thing was, I started skipping some steps I had done the previous times, and that turned out to be my downfall.

I know, the high-clock-speed chips are so 2005, because a slower CPU can have 7 cores, and execute 19 instructions per clock cycle and have so much on-die cache that you just mirror the entire system RAM into cache on startup and then pull out the DIMMs and use them to level that kitchen table with the short leg.

The fun part is seeing which component will fail from the overclocking. It's not the CPU of course. At one time it was the video card; fixed that. This time it looks like either the SATA bridge or the drive electronics themselves. Ah well. It'll totally be worth it once I get that pixel shader 3.0 action going. (I told you it was an older machine, besides, don't get me started on the awful experience I had trying Vista/DX10/PS4).

500 Channels, er, Feeds and Nothing to Read

The blogosphere often writes about itself, and the discussion of group blogs vs. individual ones is no exception. Nevertheless, I have seen an increase in quantity and decrease in quality (per post anyway) from a number of top blogs that I read. Maybe coincidentally, these blogs are team affairs now.

It kind of snuck up on me.

Slowly I started finding more and more items each day from certain feeds, and skimming over more and more useless posts. As the post-per-day count increased, it seemed that more filler was appearing. The filler was less interesting, lower quality, pseudo-news (i.e., the obligatory regurgitation of tech semi-news that was already published elsewhere, plus two sentences of uninspired commentary).

Just to keep up with the flow, and to stop wasting my time, I started dialing feeds that had been on full-text mode down to headlines only, to make it easier to skip the garbage. And now I've started tossing a couple of these so-called A-list feeds in the trash altogether; days were going by without an impactful or truly newsworthy item.

These are commercial blogs. Like the commercial TV and mass media from which they originally tried to differentiate themselves, the publishers have elected to fill lots of airtime with bad shows, and then sell the commercial breaks.

The publishers have every right to do that. But if I wanted fake news and comment, I would watch CBS or read cbs.com (not sure if that's even their URL, that's how irrelevant they've made themselves).

But I'm not going to bash these guys (and gals) -- after, they are digging their own graves, at least as far as being in the vanguard media and not the legacy media.

Instead I want to mention and thank a few great bloggers who have elected not to do this even though their audience size is such that they might have. I look forward to reading their work every time an item shows up in my aggregator:

• Marc Andreessen
• Jeff Atwood
• Don Box
• Don Dodge
• Rick Segal
• Scott Guthrie
• Tim Lee

Thanks, guys!

Lessons from Porting a MS-Office-Scale App to Flex 3/Flash 9

Earlier this year, I led a consulting team which ported a large Windows client app (similar to a member of the Microsoft Office family; think something like Visio but with different functionality) to run in the Flash 9 runtime, using Flex 2 and Flex 3 Beta, in under 6 months.

I was invited to present a (very!) quick talk on some of our "lessons learned," as part of O'Reilly Ignite at Adobe MAX in October. Alas, schedule issues made it impossible to present at MAX. So here, in Ignite style (15 seconds per slide), are the key lessons from the project:

1. The learning curve for Flex/ActionScript 3 is very shallow for experienced software engineers, so a strong team can jump right in and be productive even without any Flex/Flash experience. Only two of our five developers had done any Flex or ActionScript work before at all. That said, Flex is still a “leaky abstraction” – some aspects (e.g., deferring an operation to a subsequent frame) of Flash Player still bubble up – so it’s useful to have at least one engineer who is familiar with coding for Flash.
   
2. There is no need to think about a Flex app as a Flash movie on steroids – simply think of it as a VM-based app (like a C# or Java app) from day one. Bringing standard OO pattern experience is very helpful. UML/Code generation tools for AS3 may be helpful too depending on your dev process.
   
3. Premature optimization is still the root of all evil. Most optimizations are a non-issue with the performance levels of Flash 9, and we found that a small number of relevant optimizations could be easily done toward the end of the project without significant pain in refactoring. Network operations, which were outside of our scope, were the only cause of performance issues. Nor did we compromise in terms of space: We had tens of thousands of lines of code, hundreds of features, and most of our display via custom code (the app is partly a diagramming app) … and it compiled to a .swf in the 670 KB range.
   
4. ECMAScript 4, JavaScript 2, ActionScript 3: whatever you want to call it, throwing static typing into the mix (without requiring it everywhere) rocks! Our unscientific feeling is that we enjoyed a 2-4x productivity improvement over a similar language (e.g., JavaScript) because of static analysis capabilities and compile-time error checking.
   
5. E4X on a large scale is a beautiful and terrible thing.
   
• The beautiful part is: It did not lead us to make a mess of our application with snippets of XML processing scattered everywhere (something we worried about at first), and the performance is amazing. We were doing a ton of XML for integration to existing systems so we worried at first. We measured it, and benchmarks of XML processing speed with E4X were far and away more than adequate.
  
• The terrible part is: the E4X API has some quirks and inconsistencies that are not documented, and in a number of cases we needed to cook up some ripe hacks to work around them. These issues can and should be cleaned up. E4X also misses a couple of (admittedly rarely used) aspects of the XML schema spec around attribute namespace scoping, that just happened to cause us some difficulty.
  
6. The Flex framework is in serious need of a native AS3 rich text, full [X]HTML editing widget. Workarounds currently include AJAX in the browser, or AS2-based components that can’t run immediately in an AS3 app's VM space.
   
7. For building some applications, the async-callback approach can create substantially more chaos than it eliminates. To address this, Flash/Flex needs threads, or at least continuations. In fact, this is the first time in my career I’ve really seen a problem (handling certain callbacks) where continuations would be an ideal solution and not just a clever one.
   
8. It is easy to mix open source Flash tools and commercial Adobe tools – my team did, and it worked well all the way around.

Since Flex/Flash are likely to stay key contenders for any RIA implementation project, I hope these learnings are helpful when embarking on that sort of work!

Beware the Hail Mary Meta-Hyper-Framework

A while back I spoke with the CEO of a midsize enterprise software company. He invited me to work with a small team building a framework for customers to create applications that would run with his company's app.

I spent some time with the team, looked at the framework, and declined. As diplomatically as possible, I explained what I saw as the key problems with the initiative:

• The term "architecture astronauts" barely touched these guys. I think they actually used the term 6GL in conversation. They were architecture hyperspace/time-travelers.
• At the same time, I'd call them "infrastructure spelunkers" as well. The team trusted nothing, and was reinventing everything. They were rebuilding so far down the stack I started wondering where they built their own hardware. Then I realized that was silly -- after all, they would mine their own copper before they'd use someone else's wires.
• The initiative had been running for several years with no concrete delivery. There was no deadline for a future delivery either. Given that the predominant tools and platforms were evolving underneath this team faster than the team was moving, I suggested I did not expect they would ever release.
• No real world app -- either inside the company or among its customers -- was using it yet. That is, there was no dogfooding or validation of any kind that the framework was learnable or made the relevant problems easier to solve than existing approaches.

The CEO appreciated my frankness, and said that from his point of view, it was a big long-shot bet:

• For him, the 15+ man-years that it cost to work on this framework were a non-issue. With plenty of revenue and people, if this initiative went nowhere, it would not be a significant loss to the company.
• The framework was not critical-path for anything else, so he didn't anticipate any major downside.
• Potential upside was, he felt, huge. If this framework somehow succeeded, he could grow his company much faster than he could any other way he could envision.

Well, I thought, I wouldn't make the same choice -- I would probably look to gain the same leverage using a radically simpler framework or mechanism -- but then I wasn't looking to take on an executive role, so it wasn't really my call.

DHH, the man behind the Rails web app framework, wrote about frameworks when he explained "Why there's no Rails Inc":

"... Rails is not my job. I don't want it to be my job. The best frameworks are in my opinion extracted, not envisioned. And the best way to extract is first to actually do. ... That's really hard if your full-time job is just the extraction part since you now have to come up with contrived examples or merely live off the short bursts of consulting. For some that might work, but I find that all my best ideas and APIs come from working on a real project for a sustained period of time."

Nicely put.

"Wrong Date" Emails == SPAM

On a regular basis, I get emails from vaguely reputable entities (airlines?!) sent with the wrong date. For example, on Wednesday I'll get a message "sent" on Monday. Because these messages show up out of order in a mail reader, and are marked "unread" among mostly "read" messages, they catch my attention. And sometimes I read them.

It's time to call this out for what it is: an annoying spamming technique designed to try and get my attention. And, like sending all your email flagged "High Importance," it gets old fast and makes you look bad.

There are plenty of time servers on the net. Plenty of bulk mailing mechanisms that don't get several days behind in getting the mail out. And plenty of docs on how to leverage UTC and offsets in your environment of choice.

So I don't believe anymore that a "misconfigured server" or "backed up bulk remailer" is to blame.

If I were an ISP, I'd think about creating spam filter rules that tweak an entity's rating every time bulk mail comes through with a date that's way off. Do this enough, and your messages go to my users' filtered spam folder.

Android? Give Me a Call When (If?) Phones Go On Sale

In this era of cluetrains, agile, getting real, perpetual beta, and all the rest, I would have thought that FUDly tactics like announcing big game-changing vaporware just to assert your position couldn't be taken seriously.

Not that Google's Android phone software is vapor (you can download it today) -- rather, the proposition of a viable phone ecosystem (devices, carriers, software -- in short, something you can actually use) is vapor.

It pains me to write this. After all, I proposed an platform-oriented MVNO (which Google may become with its possible spectrum bid), and I also stated that, to change the game, Google should support J2SE (and more), which Android does. And Google's challenging the carriers, who are the big problem in wireless software.

But I gotta call it like I see it, and announcing a big platform initiative when the first real handsets are at least a year away is a movie I've seen before.

There's an ALP announcement like this every six months; Palm announced the mythical Cobalt (with SDK!) over and over; Motorola made a big deal about its Linux phones. To be fair, Moto does ship a number of phones running Linux under the hood. But they're not really open, and neither developers nor consumers have taken much note. In early 2006 we were told MIDP 3 devices would be shipping by, er, now. Oh, and Sun's magic wand is going to bring us JavaFX mobile devices.

It's old-school (bad!) tech marketing at its worst. At best, these announcements need a bogo-coefficient to convert, say, "Just 12 months away!" to "Just 36 months away!" ... but really it's best not to believe in anything you can't buy at your local Sprint store.

Producing the SDK for public consumption now is not necessary to provide development lead time (how long did it take developers to build apps for the iPhone?). Plus freezing APIs now just makes it that much harder to alter the them when unforeseen issues crop up closer to mass production.

Although Apple is famous for taking a haughty approach toward their developers and customers, there is something to be said for putting the hardware and carrier together first, releasing the phone second, and then when you see what people really want to do with it, release an official SDK to help them.

I wish Google the best of luck with 700MHz, the carriers, the handset makers, the FCC, and the public. I'm just not sure this is off to an auspicious start.

Agile Clients Present Challenges for Fixed-Price Consulting Firms

As more companies adopt agile development techniques, their increased desire for flexibility is causing pain for fixed-time, fixed-price consulting partners who have not yet figured out how to make the new dynamic work.

Traditionally, there are two models for technology consulting: "Time and Materials" (T&M), where the consultant bills per unit time and is only minimally responsible for the overall timeline, and "Fixed-time, Fixed-price," where the consultant is responsible for proposing a combination of scope, budget, and timeline and then delivering it.

The problem is that the initial time and price estimation models are based on a particular deliverable target. And while it's traditionally been a consulting challenge to keep change under control during an engagement, now clients are being completely upfront and saying, "We are developing alongside you and we intend to make substantial changes to the target every month, as the project develops."

With agile methods, this is not a shocker. And the client is justified in believing that if their organization can deliver new features and make spec adjustments every few weeks, surely high-priced consultants should be able to keep up.

The good consultants can keep up. The problem is figuring out a way for them to get paid for it; otherwise they won't be able to do these kinds of projects for long. Without a model that specifically addresses the possibility that expensive goal changes may arise, the consultants are forced into (a) being less agile than the client, (b) eating the cost of the changes, or (c) asking for more money or time. None of these options are good ones for the consulting relationship.

A first step toward a solution is to change agile co-development from an externality to a core part of the service delivery model. An iteration length should be defined as part of the project work statement. So far so good. The tricky part is allowing the consultants to make time or price changes at the checkpoints between iterations. After all, wasn't this whole project sold as fixed-time, fixed-cost? Sounds like having one's cake and eating it too. The reality is that both parties need to allow these changes (the client needs to change the specs; the consultant needs to change the cost) in order for the process to work.

The problem on the client's end is that development methods like agile, even if they tie in to product development processes, are often more "micro" in the budgeting scheme than the department budget or consulting budget. That is, the client's finances are not as agile as their development.

On the consultant's end, if the engagement is renegotiated -- even at a micro level -- between iterations, the overhead of managing and selling the project goes through the roof. And of course it's always possible that agreement can't be easily reached to match a spec change with a cost change. In that case, the whole engagement can fall though adding massive risk to delivery and reputation all the way around.

While not a complete solution, a number of elements would be helpful

• Guidelines that throttle the rate of change between iterations via specified metrics, so that both sides have some ability to predict cost
• Escalation chains for approval based on the amount of money or time involved, so that small and large changes alike don't necessarily need to go to a VP Finance or the legal department every time
• Emphasizing these agile mechanisms in the sales process, so that an agile consulting firm (which might potentially cost a little more) remains competitive with others (who might offer a lower price but not be able to speak to how they will deal with these issues).

At this time, technology consulting opportunities abound and make good sense for clients and service providers alike. But to succeed in the long term, fixed-cost consultants need to evolve their models a few steps to keep up.

Red Pill Shoe #2

So XSpace is actually, well, everyone in the social networking space except for Facebook.

And the Red Pill app itself is still a hypothetical, but it's getting closer and it's clear where that data would be helpful now.

Scoble's suggestion that his friends would all have to move before he'd look outside the "walled garden" of Facebook, and Don Dodge's suggestion that Facebook users won't just leave are both missing the point.

Users don't need to make up their minds now, they only have to decide that at some point in the future they might want to spend more time somewhere else. That's where Red Pill and its data come in.

Meantime, Bebo's press release headline ("[...] REVEALS PLANS TO LAUNCH FACEBOOK-COMPATIBLE DEVELOPERS PLATFORM") appears to overreach the actual article content -- namely, that Bebo will "make it easy for Facebook developers to port their applications," not necessarily host Facebook apps directly.

But there seems to be no clear reason why an OpenSocial proxy system could not actually host F8 applications. Except for maybe some terms-of-service issue. All those "fb" abbreviations and tags may have to get ROT13'ed or the like.

Want a Real Cheap, Real Usable Windows Laptop? Don't Stop at XP, Go All the Way With Windows Server

As we move from back-to-school shopping shenanigans into holiday shopping shenanigans, a number of retailers (Wal-Mart, BestBuy, CompUSA) are selling dirt-cheap underpowered Windows Vista laptops for $300-$400 from vendors including Toshiba, Acer, and Gateway.

A few articles have been published on "upgrading" these machines from Vista Home Basic, which is more or less unusable on them, to XP Home or Pro. But why stop there? If you want to get down to business on a dirt-cheap laptops, and Linux isn't your thing, take a look at Windows Server.

I've used Windows Server 2003 on extremely old boxes (Pentium 3, 500 MHz, 384MB RAM) for testing and development purposes, and these machines are snappy. Server has a lot less eye candy and overhead than XP (I often catch it idling with only 120 MB of RAM in use), which makes it ideal for these underpowered machines.

How do you do the "upgrade"? Well, the the biggest challenge is the same as with XP: the laptop vendors provide Vista drivers only, and you'll need a Win2000-XP era driver for Server 2003. So before dropping the cash, do some research and make sure you can find XP drivers for the video card (on-board or not), Ethernet, and WiFi devices at the least. Sound card support can be hard to find too.

Once you've got the drivers, it's a straightforward install. Throw in another GB of memory (these laptops are sold with 512MB) for $30 or so, then go into the system control panel, boost the virtual memory setting and tell Windows to prioritize console apps rather than services (Server does the opposite by default, as one would expect). You probably want to uninstall the high-security mode of IE, if you're planning to use IE at all. Do this under Add/Remove Programs, and then Add/Remove Windows Components.

There are a small number of applications that can differentiate the server from the desktop Windows OSes and will complain (Grisoft's free AVG anit-virus product, for example, as well as some Windows Live apps). For the most part, though, all of your applications will run fine.

Now ... where to find a Server 2003 license? (The OS requires activation like XP.) At $999 for "Standard Edition" and $399 for the cheaper "Web Edition," this OS was never meant to be used with a $300 laptop. Although it would be nice if Microsoft offered a single-user non-server price, that's about as likely as Apple selling you MacOS for your Acer laptop. Good bets are companies buying new licenses of Server 2003 R2 or Server 2008 to replace Server 2003. You'd be surprised what unused or retired licenses could be kicking around your own company.

Another approach is to download the Server 2008 Beta and try that. 2008 is based on the same kernel as Vista, and so should use Vista drivers rather than XP drivers. However, I haven't used Server 2008, and I don't have any immediate experience that would inform performance, installation on random hardware, etc.

While this is not a recipe for building a cheap laptop for your less-than-computer-savvy relatives, if you're already a hacker looking for an extra portable machine to do Windows work on, it's almost certainly the most bang-for-the-buck possible.

Facebook Conundrum: Will it Tolerate a Red Pill Application?

Suppose I build a skeleton social networking app, call it "XSpace" for convenience. Maybe it has a couple of clever, unusual features, but there's nothing much there, and no users.

Now I build a Facebook application called "Red Pill."

If you add the Red Pill app, and give it permission, it will export data from your profile into storage on XSpace. None of this data is published or shared or anything else that would violate the current Facebook TOS. In fact, at this stage, it's a lot like FriendCSV, a current friend data exporter app -- it's actually a strict subset of FriendCSV, since the data is not actually downloaded as a file, just stored in a private spot in XSpace. (The FB Terms of Service prohibit storing user data for more than 24 hours, but this is less of an issue than it seems: we'll talk about this later)

Now I add a feature to Red Pill called "Take Me"

Take Me brings you into the XSpace app, with all of the social graph structure already in place, from everyone who has installed Red Pill, even if they've never "taken" it.

If a lot of people run Red Pill, then the core value of Facebook -- the network itself and to a much lesser extent the profile data -- is replicated into XSpace.

What happens next?

In the Facebook-success scenario, XSpace becomes an interesting alternative world to Facebook, linked by Red Pill "tunnels," and mirroring some parts of the social graph. Maybe a nice symbiosis evolves or XSpace is subsumed into a pure traditional Facebook app.

In the Facebook-failure scenario, something kicks off an exodus from the 'book. Whether it's a change in functionality, rules, service level, or just "cool factor," a mass of people pop the Red Pill and just start logging into XSpace instead. They keep all their original friend and profile data, so it's a smooth transition. Maybe XSpace also implements the Facebook API (it's a small, public API after all) so that Facebook apps can run in XSpace as well...

It is only a matter of time before the "Red Pill" and "XSpace" show up. That's the risk you take building a platform with a API -- it's actually to be expected, even desired.

But in the world of network-effect applications, there's a twist: these apps do not derive their value from being "the best implementation of the platform" (typically the way a platform implementer tries to assert value). Instead, the value is page views driven by the social graph itself and by the apps on the platform. Since both of these areas can be trivially replicated in XSpace, there's not much left. That is, there is no core value-add (aka Sustainable Competitive Advantage) left to Facebook as such.

Could Facebook cut off (or dial down) the API? Sure, but at the risk of slower growth and possibly aggravating users and developers who might like to play in someone else's open sandbox. And if they were too late in doing so, the move could actually spark the emigration to XSpace.

Could Facebook assert intellectual property rights over the data? Maybe, but facts cannot be "owned" as intellectual property. So the fact of my having a friend relationship to someone, or having met them in a class, or being married, are all things that no one owns. There is some user-generated content that the user has "signed away" rights to, but we're not talking about that. I.e., we are not proposing scraping out and re-using any content outside the core profile facts (age, name, etc.) and social graph.

Could Facebook complain about XSpace storing user data more than 24 hours? Maybe, but it depends on how the boundaries of the systems are defined, and in any case the better solution is for Red Pill to pull updated user data every 24 hours rather than archive the old data. If the app is ever cut off from retrieving this data, then it will keep the last snapshot, and then who cares, 'cause it's "game on."

As long as Facebook's valuation can be reinterpreted as an artifact of a Microsoft marketing expenditure (i.e., not a true investment, as some have suggested), this doesn't matter to anyone except maybe Microsoft.

But if Facebook is looking at taking really big money in, or eyeing an IPO, they'll be forced to think about cutting some of the Matrix exits. What will they do? What can they do?

Gratuitous Post: Gmail + IMAP = Sweeeeeeet

Ok, this is neither news nor particularly clever-monkey opinion writing.

But Gmail has been opening up full IMAP access on accounts, and it is truly a beautiful thing. Web mail is fine when it's all you've got, but whether it's stripped down (Gmail, old Yahoo! interface, etc.) or supa-deluxe JavaScript (new Yahoo!, Hotmail, er, I mean Windows Live) it can't beat the smart client.

And email clients are the original smart client: rich client with offline capabilities + access to network resources.

I'll still use the Gmail web interface sometimes, so it's not like Google won't get a chance to present me with plenty of ads, but having access from Outlook and especially Outlook Mobile on my Windows smartphone (sorry, guys, it just works better than the J2ME Gmail client) is absolutely killer.

I didn't think the free-online-email game was going to change a whole lot at this point, but this is a game-changing move by Google.

Microsoft Will See Your Web Services and Your Horizontal Database Scaling, and They'll Raise

Don Box has come out in defense of Microsoft's support of REST technologies a number of times. And this year we've started to see what else is behind the curtain, with project Astoria. Still being designed and built, but with bits available and integrated into VS2008 today, Astoria includes both local (your own machine/datacenter) and cloud services for data storage that support HiREST, relational modeling, and support for various data formats (JSON, POX, Web3S [the last a play on, and jab at, S3?], ATOM).

If you haven't seen these services, you need to check them out. In addition to being technically interesting (e.g., since queries can be expressed via REST-style URLs, your network appliances and front-end web servers can actually participate in execution or caching strategies!), these services are likely to be a big part of the web service landscape.

Whether you love Microsoft or not, it is fairly clear that the original ASP.NET SOAP implementation (and client generation) were years ahead of anyone else in terms of no-nonsense ease of use, compatibility, and extensibility.

These SOAP components were made available to developers in 2000 or earlier. They brought things like "Type [WebMethod], ok now you have an XML-RPC SOAP service. You're done, go home, have a beer," while Java was still trying to figure out how which alphabet to jam into the end of their JAX* wildcards, and inventing APIs where you just start off with a nice ServiceFactoryBuilderConfiguratorFinderInstantiatorFactory and go from there.

Why rehash this history? Because in its final form, the Microsoft solution is going to be influential. They may be late to the party here, but don't discount them.

Enterprises need service description layers for REST. Someone is going to give it to them in a way that they can use it. And an Amazon-S3-scale relational data service in the cloud (no, Freebase and the like don't count) could be really interesting to everyone who doesn't have enterprise, need-my-data-physically-in-house requirements. With ActiveResource a core part of Rails 2.0, I could see building read-heavy apps using caching and Astoria, and no local database at all! My clustering problems (and expense) have all just become someone else's problem!

There's something else to see here too: read the Astoria dev team blog and the comments. You're watching Microsoft designing and implementing a big API in real time with interaction from the community. Don't look for an open-source experience -- you can't check out the code and send patches. But there is a lively discussion going on between the development team and outsiders to try and come up with the best solution that fits the constraints.

Clowns on Parade: Giving Administaff Your Keys Isn't Much Better Than Leaving the Door Open

Chains and their "weakest links" are used all the time in metaphor. But I realized this metaphor was wrong after seeing an odd "chain of locks" securing a no-vehicle gate last week near the GGNRA in Marin.



The only practical reason I could imagine for using this chain of locks is that a large number of people all need to be able to open the gate (e.g., park staff, firefighters, police). Instead of having one lock and sharing copies of the key, someone decided to give each party a lock and key. By chaining them together, any opened lock allows the gate to be opened.

I'm still not sure why they would choose this approach (if any reader is familiar with this construct, please tell me!)

With personal information, we may not share a "master key" with many people, but we offer a lot of locks and keys to a lot of different parties. Any one of them can leave us wide open. Like last week, when Administaff -- a huge co-employment organization that my employer uses -- announced ... (drumroll) ... a laptop was stolen with personal info, including SSNs, for everyone on every payroll they processed in 2006 (approximately 159,000 people total).

With friends like this ... you know the rest.

There's a wonderful FAQ on the theft, where Administaff explains that it's not the organization's fault: "the information was not saved in an encrypted location, which is a clear violation of our company’s policies." In other words, they're blaming the employee for violating the company policy.

I don't buy it.

Yes, I believe there's a company policy somewhere that says not to copy the entire human resources database onto your laptop in plain text.

But I don't believe Administaff made reasonable efforts to see that this policy would be carried out.

I suspect there were at least three distinct failures:

Failure #1: The employee whose laptop was stolen was tasked with an activity for which the easiest workflow involved loading the entire database onto his or her laptop. How do I know this? Most workers do not take the hardest route to doing their job. They take the easiest one they can.

In this case, someone took the easiest route even though it meant violating a policy (that he most likely never took note of anyway). When Administaff management allows the easiest workflow to be one with this much security exposure, they share the blame. If they don't know what workflows are being used for Social Security data, then they are failing at a bigger level, namely not auditing sensitive processes in their own identity-theft-prone line of business.

Failure #2: At best, the server system which "owns" the stolen data allowed this employee to produce a report containing critical data for a very large number of records. (At worst, this data is not stored in any controlled application at all, but rather in something like Access, FoxPro, or Excel. While I know this is a real possibility, it's such a revolting idea that I will ignore it for now.) Assuming this application has a user/role model, why would this user have such a reporting privilege?

Even if the application is designed to support some "work offline" workflow, so a that network connection is not required to access each record, this can be accomplished without any mass download of records. A modest number of records could be downloaded and cached for an offline work session, and synched back later. The record cache would, of course, be secured with a passphrase and/or other elements.

My point here is that there's no way the employee accessed and then copied/saved each of 160,000 records, one at a time. The application had to help, making it easy to do some operation on "all" or on a large set of records (birthdate in a specific year, last name starting with a certain letter, etc.) Awful idea. Administaff is leaving the door wide open, no surprise that the employee stumbles on through.

Failure #3: How long was this data on the laptop before the laptop was stolen? At one large financial institution, any computer connected to the network -- whether on site or via VPN, virtual machine or real -- was subject to regular scanning from the mother ship. The security group would check all of these machines not only for vulnerabilities (viruses, vulnerable services), but also for content. Were they after pr0n? Not so much. They wanted to find out if any disproportionate amount of their data ended up on any of your machines.

If, say, they found a file that looked like a bunch of credit-card numbers, you'd have some explaining to do. While this approach would not stop a clever data thief (who would employ steganography or removable drives), it would do a great job at stopping any accidental hoarding of customer data. In fact, it would do a great job at stopping this pervasive stolen-laptop-stolen-data problem.

Apparently Administaff really cares about this stuff. Surely enough to spend the half-hour thinking about it that I did when I wrote this post. Just not enough to actually do anything.

Double Black Diamond Software Projects

I've spent a good part of my career working on particularly challenging development gigs that I have come to call "Double-Black-Diamond Software Projects"

What exactly is a double-black-diamond project?

The name comes from ski trail markings, where a single black diamond indicates (at least in the U.S.) an "advanced" trail, while two black diamonds indicate "expert."

In reality, the difference between single- and double- black trails is that a strong skier can typically waltz onto a single diamond slope and have confidence that it may be interesting or challenging, but it will have a predictable outcome.

The double black trails, on the other hand, can feature hidden obstacles, cliffs, terrain features that vary with the snow conditions, and even areas which may be practically unskiable depending on conditions. (E.g., the cliffs in the background of this image are the double-black "Palisades" at Sugar Bowl.)

In software development, a double-black-diamond project is one where the outcome is in question from the beginning, not because of resource issues (e.g., too few developers or too little time), but because some fundamental unknowns are involved. Something new is being created, and it is sufficiently unique as to make it unclear whether it is even possible to succeed, or exactly what "success" will look like.

These unknowns might involve integration to an unknown external system, performance of problematic features like voice recognition, custom hardware, etc. If these challenges seem feasible enough, or success seems valuable enough, to make the software project worth a try (ultimately an investor decision), but the sheer implementation risk (as opposed to business risk) is clear from the start, you've got a double-black-diamond slope in front of you.

I will be writing a number of posts on these double-black-diamond software projects, and I plan to cover

• typical elements ("signposts") indicating a double-black project
• why, as a developer, you might ever want to get involved in such a project when there are lots of other opportunities
• gearing up: the skills, knowledge, and/or people to bring with you
• the art of keeping project sponsors properly informed (the subtle part is the definition of properly)
• how to handle risk and predict outcomes (as much as possible)
• maneuvers and techniques to gain leverage and minimize the odds of failures or surprises
• managing the definition of "project success" (and why it's essential to do so, even as a coder)

Mozy Keeps Your Data Safe ... Once You Get It Working

A while back I wrote about Carbonite, a consumer PC online backup solution. I thought the user experience was fantastic, but I didn't love the idea that my data could be decrypted with just my (low-entropy) site password.

More recently, I decided to give Mozy a try. Mozy is another leading online backup app, and they offer 2 GB of personal backup for free. Interestingly, Mozy seemed to me to have the opposite qualities (both positive and negative) as Carbonite in my trial.

The security is hard-core if you choose: Mozy generates a 448-bit encryption key from any chunk of text or file that you give it. Naturally that means your source should have some decent entropy in it, and you'd better have a copy of either the key source material or the generated key file if you ever want your data back. But the folks who really want to keep their own key will know this already.

Mozy does encryption (and presumably decryption in a restore) locally, and ships the encrypted files off to storage. So your data is pretty darned safe from anyone inside or outside of Mozy.

The "average user" experience, though, had a number of annoying snafus.

The GUI on the client tool that manages your backups and filesets is neither pretty nor intuitive. I'm tempted to compare it to some of the more mediocre Gtk front ends to Linux command-line tools. Perhaps that's too harsh, but it does have a number of similar quirks, like multiple widgets that control the same setting without being clear about it, checkboxes becoming checked or unchecked "on their own", etc.

More troubling was that the client appears non-robust in the face of network outages. When the network dropped during my initial config, the app crashed. Upon restart, it did not appear to have saved state: I had to redefine my backup sets. I then started a backup, and the app crashed again when the network momentarily dropped. This time when I restarted it did have my backup sets, but the history window showed no trace of my failed backup.

Then, during my next backup attempt, after getting several hundred megabytes onto the net, my machine rebooted itself (courtesy of a Microsoft "patch Tuesday"). When I looked at Mozy, its history again showed nothing. I would have liked some information about the failed backup, and a way to "resume." Instead, I had to start the whole backup from byte 0.

This latter time, I achieved success. But in an era of WiFi access (which can be flaky), I would expect not only robustness in the face of network connectivity issues, but also a really solid resume mechanism. After all, how many machines will succeed with the initial multi-gig upload in one go?

To finish on a positive note, I should point out first that for paranoid^H^H^H^H^H^H^H^H security conscious people, it's great to have a tool that handles strong crypto inline and gives me control over the only key. Also, a quick search suggests Mozy is ready to bring out the heavy guns to address customer problems. If they keep that up, they'll be able to overcome almost anything else.

Thanks for the Wakeup Call! Jeff Atwood Explains Why I Shouldn't Write a (Traditional) Book

Jeff Atwood's Coding Horror is one of my favorite blogs.

He recently wrote about his new ASP.net book, and instead of telling everyone to go buy a copy, he spent his words explaining why one should not write a technical book.

To summarize, he pointed out

1. The format and process was painful to work with
2. "Writing a book doesn't pay"
3. One shouldn't associate any extraordinary credibility with publication. Jeff's a little more blunt: "Anyone can write a book. ... The bar to publishing a book is nonexistent..."
4. "Very few books succeed ... In the physical world of published atoms, blockbusters rule" or, in case you forgot the pre-web-2.0 world, the long tail isn't supported by the economics.

These arguments hit home with me because I had been preparing a book "proposal proposal," and chatting with some publishers (I say "proposal proposal" because a proper book proposal has a certain form and content which I have not fully developed; what I have is a more informal proposal; i.e., a proposal for a full proposal).

At a certain level, I already knew I am living in the wrong decade for the dead-tree book. But I also imagined that, since my topic is a little bit more process- and project- oriented, rather than "how to write an app with the new foobar 3.0 API," my book might have more long-term relevance or staying power.

But Jeff's post hit me like the knocking at the door in Macbeth. He's right. At the end of the day, the publisher and bookstores are not going to do anything impactful to promote my book; it will be hard to find on a shelf with 1500 tech books that change every 3 months, and I wouldn't expect it to sell remarkably well.

My main goal in writing is to share some knowledge and experience with folks who are interested and who might benefit. And writing and publishing online, I have a lot of confidence that my audience will find my content.

This belief comes from looking at the web analytics for my blog: through the magic of Google, I get solid traffic for meaningful keywords. For example, my post on the .net implementation of J2ME hidden (?) in Y!Go, but which you can use to port your own MIDlets to Windows Mobile, is result #8 (today) if you Google "j2me .net implementation" ... and it drives a bunch of traffic.

That's good enough for me.

Now I'm realizing that there are no advantages to the legacy tech book publishing process for me (I'm sure that for established, famous authors who write for a living, it's a different story). But there are a lot of advantages to avoiding that publishing process. Beyond the simple advantages of being accessible via Google, etc., I will surely save an enormous amount of time interacting with the machinery of the publishing industry.

Time saved is time I can spend writing and refining my core content. And if it's not perfect, that's ok, because there's no offset printing and physical distribution. I can fix typos or code errors instantly. I can revise and re-post if I realize I've said it wrong. I can offer old versions if anyone cares to see them.

Meantime, if anyone wants a printed and bound paper copy, there's always lulu.

Thanks, Jeff!

Project Manager as Diplomat: Herald or Negotiator?

If software project managers are diplomats -- and I think most project management roles see them this way -- then there are at least two distinct flavors: heralds (low level diplomats) and negotiators (high level).

The difference is sometimes imposed by the project/work situation, but more frequently appears to be self-imposed by the project manager.

The herald project manager sees his or her role as a courier bringing messages back and forth between various parties. The parties may be friendly or hostile, and the communications may be straightforward, complex, or threatening. But if the herald gets the info to the recipient in a timely way, his job is done and he expects to pass unmolested.

The negotiator project manager, on the other hand, sees his role as keeping everyone on the same page about decisions and outcomes. If (when) parties are not in agreement, the negotiator tries to bring about sufficient discussion and confrontation that something can be agreed upon.

The project manager rarely has direct authority over multiple parties in the project (e.g., engineering, product management, and marketing). He can, perhaps, control one closely aligned party through resource allocation. In general, though, I've seen more of a carrot-and-stick approach:

The PM has opportunities (e.g., a recurring meeting) where he holds the floor and escalates all of the known issues, so that everyone is painfully aware of the potential problems. He then reminds everyone of all the negative consequences coming down the pike for the project, if key decisions and compromises are not made. He also points out how well things will turn out for everyone if the project can reach completion on time, on budget, and with everyone more or less happy with what was done.

With enough persistence, it is usually possible to keep things on track. How? The negotiator's secret weapon in the business arena is his willingness to make people consider negative possibilities that they are not likely to raise on their own, and to make it seem matter-of-fact. Business groups (at least in the U.S.) are extremely uncomfortable thinking about and planning for negative outcomes. So the project manager gets to make a whole room full of normally confident people very uncomfortable. In order -- literally -- to alleviate this discomfort, they start to talk and become a bit more flexible.

It is clear that the negotiator PM is doing the heavy lifting, while the herald PM is a glorified secretary. The herald may be useful in a large project where so much information is flying around that it's worth having full time staff to keep it straight. He's holding off entropy but not solving big problems. The negotiator feels like his success depends on getting people to work together for success, which is a bigger challenge and produces bigger rewards.

When you are choosing, hiring, or staffing project managers for software projects, keep this dichotomy in mind. Unless your world is all rainbows and unicorns, you probably need a negotiator, and getting this sort of PM will pay off handsomely. You can supplement him, if the project is really big, with a herald PM to "do the paperwork."

But if you enter a challenging project with nothing but a herald PM, you are increasing your project risk considerably.

Writing Javascript on a Phone, on a Phone

I'm on vacation, so I'm writing some less, um, in-depth posts. Hopefully. Like this one:

I've gotten bored waiting in lines, on airplanes, etc., and I thought it would be a fun way to waste time if I could program my phone.

I wanted to be able to do it without a PC, just with the phone itself. And I didn't want to do it over the network (i.e., edit some code, send it off to a web page or network app to compile, run, etc., and return results, even thought that could be a fun way to tinker when you're on a good connection).

I thought there must be some kind of compiler or interpreter for my Blackberry or, now, Blackjack. And yes, I know, if I had a real Linux phone, then I could just download any tools I wanted. But Linux smartphones with QWERTY keyboards are hard to come by.

Enter Javascript.

I used to hate Javascript. A whole lot. Then I watched Douglas Crockford's fabulous lectures on Javascript, and it turned my whole attitude around. I realized I had never understood what Javascript was about in the first place. And here was a really smart guy who knew all of the problems, and didn't say "get over it," but talked about finessing your way to the true nature of Javascript. Which, incidentally, are lambda expressions. Crockford asserts that Javascript is more or less the only "lambda language" to organically receive mainstream adoption in the development industry.

Now I only hate the browser hosting environment for Javascript. And with enough plug-ins, even that isn't so bad.

Since Pocket IE has Javascript support, and I can edit text files and save them on my phone, I can build up a library of the functions that do what I want, and this library becomes the web page that I run and hack in with PIE. This is where having a real smartphone, not a crippled one whose manufacturer tries to deny me access to the local filesystem, is helpful.

Goofy? Kinda. But it keeps me out of trouble.

One Reason Software Development is Such a Malleable Process

This post is the fifth (the others are here, here, here, and here) -- and last planned one -- summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

At the beginning of Chapter 23, Steve writes:

Philip Metzger observed decades ago that technical staff were fairly good at estimation but were poor at defending their estimates [ ... ] One issue in estimate negotiations arises from the personalities of the people going the negotiating. Technical staff tend to be introverts. [ ... ] Software negotiations typically occur between technical staff and executives or between technical staff and marketers. Gerald Weinberg points out that marketers and executives are often at least ten years older and more highly placed in the organization than technical staff. Plus, negotiation is part of their job descriptions. [ ...] In other words, estimate negotiations tend to be between introverted technical staff and seasoned professional negotiators.

This is a valuable observation in estimation scenarios. It also reminded me of some hand-waving I did about 6 months ago in reply to a comment on this post. I wrote that the reason some industrial process control type procedures fail when applied to producing software is that "There are too many social vulnerabilities -- the software process itself is inherently 'soft' in a group dynamics sense, so it gets reshaped by the org's internal disagreements."

I stand by that comment, but it seemed a bit vague. I think Mr. McConnell's summary of the personalities that push and pull around software commitments is quite helpful. His description clarifies one of the software group's social vulnerabilities. After finding themselves agreeing to a problematic "estimate," engineering groups may be able to make up for their poor performance at the negotiating table by trying to "reroute power from the warp core" ... but we know that statistically, over the long term, that approach fails, leaving chaos in its wake.

What to do? A proverb says that a man representing himself has a fool for a lawyer. I.e., right or wrong, he is likely no match for a trained, experienced legal professional on the other side.

So let's get the software team some representation. Not a lawyer, but an executive advocate, someone with the personality and negotiation skill-set to go to the mat with other parties.

If this person has a good grasp of the technical issues, all the better; if not, he or she can take it offline and come back to the engineering team for a briefing.

Of course, if the engineering group somehow gets the impression that the representative has sold them out, making unauthorized or unreasonable commitments, then we are back at square one.

In Software Estimation, Fewer Inputs Can Trump More Inputs. Here's Why:

This post is the fourth (the others are here, here, and here) summarizing and commenting on elements from Steve McConnell's excellent Software Estimation: Demystifying the Black Art.

There are estimation models that take a large number of inputs, and others that rely on only a few inputs. Some of the input data are more objective (e.g., the number of screens in a spec) while other data are more subjective (e.g., a human-assigned "complexity" rating for each of several reports).

One thing that Steve explains is that, when many subjective inputs are present, models with fewer inputs often generate better estimates than models with "lots of knobs to turn."

The reason for this is that human estimators are burdened with cognitive biases that are ridiculously hard to break free of. The more knobs the estimator has to turn, the more opportunities there are for cognitive biases to influence to result, and the worse the overall performance.

It should be reiterated that this same problem does not apply to systems that take many objective measures (such as statistics or budget data from historical projects) as inputs.

Lifecasting will be Prevalent, and has a Down-To-Earth Future

Lifecasting is one of those things that is interesting on a theoretical, academic, political, and artistic level. Yet, right now, it's pretty boring in "real life."

But there is a big bourgeois, commercial opportunity coming for lifecasting, and with that will come low prices, ubiquity, and all sorts of new content also relevant to the avant-garde. In the same way that cellphone cameras now catch politicians off guard and police beating protesters, the surveillance (sousveillance?) society will take a quantum leap forward.

Walgreen's sells a low-end USB web cam for $14.99 today, and for well under $100 one can get a higher-end unit. I predict that in 2-5 years, I'll be able to drop $40 in Walgreen's and get a wearable cam/mic with an 8-ounce belt-clip battery that will plug in to my cell phone ... and I'm in the game with justin.tv.

This is not live-without-a-net futurism here, either. I'm making a modest argument by extrapolation on the hardware side. The original justin.tv rig was a hassle to put together with today's technology. The biggest challenges involved upstream mobile bandwidth, battery power, and data compression.

With EVDO Rev B, HSUPA, and WiMAX, bandwidth looks to be less of a problem than giving customers a reason to buy it. As MPEG-2 fades away in favor of MPEG-4 flavors like H.264 and 3GP, cheap hardware compression is already becoming less of an issue. In fact, many of today's low-end smartphones are most of the way there in terms of a basic lifecasting rig. Battery power will remain an issue for anyone wanting to go live 24/7. But for a few hours at a time, several ounces of lithium-ion will keep the camera, compressor, and radio humming.

So what are the bourgeois, commercial applications?

• Conferences: organizers won't love broadcasts of the content, but, at least in tech, they are desperate to find some way to keep the shows compelling. I can see a lot of organizations sending one delegate to lifecast while others back home watch and interact, including talking to vendors, visiting hospitality suites, etc.
• Meetings: an interactive lifecast of a remote meeting would be a more productive way to participate than just a conference call or even a traditional web/videoconference.
• Social events: suppose your school reunion is far away and not nearly exciting enough to make the trek. But a friend who lives in the area goes, and you can ride along via lifecast? That could be a riot. And if it isn't, just close the browser.
• Education: how cool would it be sit in on some virtual flight lessons, tuned in to a lifecast from a CFI giving a real student a real lesson.
• Remote Personal Assistant: Instead of a worrying about wearable computers with smarts, you go about your business while a remote assistant tracks your lifecast. Need directions? a pickup line? a reservation? instant info on anything? Your assistant, sitting somewhere comfortable with easy access to all things cyber can do the virtual legwork and send you what you need in real time.

The monetization platform is already here, as early adopters have jumped out ahead. Phone hardware (which will serve as the workhorse for the system, just as it does now for millions of phone-cam snapshots, videos, and mms messages) is moving at a rapid pace. That just leaves a few more parts to design and sell to complete the picture. With the amounts of VC money flowing today, I don't see that last part as a problem.

LOC Counts to Compare Individual Productivity? No Way.

"Measuring programming progress by lines of code is like measuring aircraft building progress by weight." - Bill Gates

The other day, someone on my project team proposed ranking developers by their lines-of-code committed in Subversion. I really hope the suggestion was meant to be tongue-in-cheek. But lest anyone take LOC measures too seriously, it's worth pointing out that LOC is a bad, bad metric of productivity, and only gets worse when one tries to apply it across different application areas, developer roles, etc.

Here are a few reasons why LOC is not a good measure of development output:

• LOC metrics, sooner or later, intentionally or unintentionally, encourage people to game the system with code lines, check-ins, etc. This is bad enough to be a sufficient reason not to measure per-developer LOC, but this is actually the least bad of the problems.
  
• LOC cannot say anything about the quality of the code. It cannot distinguish between and an overly complex and bad solution to a simple problem, and a long, complex, and necessary solution to a hard problem. And yet it "rewards" poorly-thought-out, copy-paste code, which is not a desirable trait in a metric.
  
• In software development, we desire nicely factored, elegant solutions to a problem -- in a perfect world, we want the least LOC solution to a problem that still meets requirements such as readability. So a metric that evaluates the opposite -- maximum LOC for each task -- is counterproductive. And since high LOC certainly doesn't necessarily mean bad code, there isn't even a negative correlation to use from the measurement.
  
• In general, spending time figuring out the right way to do something, as opposed to hacking and hacking and hacking, lowers your LOC per unit time. And if you do succeed in finding a nice compact solution, then it lowers your gross LOC overall.
  
• Even in the same application and programming environment, some tasks lend themselves to much higher LOC counts than others, because of the level of the APIs available. For example, in a Java application with some fancy graphics and a relational persistence store, Java2D API UI code probably requires more statements than persistence code leveraging EJB 3 (based on Hibernate) based on the nature of the API. Persistence code using straight JDBC and SQL strings will require more lines of code than EJB 3 code, although it’s most likely the “wrong” choice for an application for all sorts of well-known reasons.
  
• In the same application and environment, not every LOC is equal in terms of business value: there is core code, high-value edge-case code, low-value edge-case code. To imagine that every line of every feature is the same disregards the business reality of software.
  
• You may have read that the average developer on Windows Vista at Microsoft averaged very few lines of code per day (from 50 down to about 5 depending on who you read). Is Microsoft full of lazy clueless coders? Is that why the schedule slipped? I doubt it. There were management issues, but Microsoft also worked extremely hard to get certain aspects of Vista to be secure. Do security and reliability come with added lines of code? Unlikely – in fact, industry data suggest the opposite: more code = more errors, more vulnerabilities.

But no one on our team would ever write bad code, right? So we don’t need to worry about those issues.

Not so fast… Developers, even writing “good” code, generally make the same number of errors on average per line (a constant for an individual developer). So if I write twice as many lines of code, I create twice as many bugs. Will I find them soon? Will I fix them properly? Will they be very expensive sometime down the line? Who pays for this? Is it worth it? Complex questions. Never an easy “yes.” Or, as Jeff Atwood puts it, “The Best Code is No Code At All.”

And beautiful, elegant, delightful code is expensive to write (because it requires thought and testing). The profitability of my firm depends on delivering software and controlling costs at the same time. We don’t fly first class and we don’t use $6000 development rigs, even if we might offer some benefit to the client or customer as a result. And we don’t write arbitrary volumes of arbitrarily sophisticated code if we can help it.

Ok, so why, then, is the LOC metric even around? If it’s such a bad idea, it would be gone by now!

Here’s why: while LOC is a poor measure of developer output, it’s easy to use, and it’s a (primitive but functional) measure of the overall complexity and cost of a system. When all the code in a large system is averaged together, one can establish a number of lines per feature, a number of bugs per line, a number of lines per developer-dollar, and a cost to maintain each line for a long, long time into the future.

These metrics can be valuable when applied to estimating similar efforts in the same organization under similar constraints. So they’re worth collecting in aggregate for that reason.

But for comparing individual productivity? I don’t think so.

Sun CTO: JavaFX Mobile Stack Aims to Clean up J2ME Disaster^H^H^H^H^H^H^H^H Issues

It's only mild hyperbole to call J2ME a disaster. If you've depended on ME to make a profit, it might not be hyperbole at all.

But good news may be somewhere in the pipeline. This morning I heard Sun CTO Robert Brewin speak at AJAXWorld. His talk largely concerned enterprise services, but when he mentioned that one argument in favor of Java is its ubiquity, and that Java runs on about two billion phones, I couldn't help but stand up and ask for the mic.

Since Robert was ready to acknowledge the existing hassles, my question was: does Sun plan to fix it -- e.g., by becoming more stringent about how devices are certified as Java-capable?

In his response, he punted on the J2ME aspect, suggesting developer could put pressure on device manufacturers to standardize their Java behavior. Since cellphone carriers are players in the equation, I'm awfully skeptical about that. But the cool part was that Mr. Brewin said that the place where Sun plans to really make an impact here is with JavaFX Mobile.

Since JavaFX Mobile, adapted from the Savaje Java-based OS acquired by Sun, is a full kernel-to-app stack, it should provide much better -- perhaps even 100% -- compatibility between devices.

Now let's keep our fingers crossed for full J2SE support on it.