Tuesday, December 30, 2008

On the Wide Range of Ruby Talent: Rails as a 4GL(?)

For a long time, I was puzzled by the extremely wide range of talent level among Ruby talent: more than any other language I could think of, "Ruby Programmers" seemed to range from the truly clueless to hardcore accomplished engineering veterans.

I had more or less chalked this observation up to the language and APIs themselves -- easy to get running with, highly intuitive ... and yet packing deep and powerful features that would attract the more sophisticated (easy C integration, metaprogramming, continuations, etc.)

On top of this was Rails with its famous screencasts showing database-backed websites automagically constructing themselves, a feat that got novices very excited, and reminded experts that they should be asking harder questions of any other framework they were using for the same kind of app.

But lately I've come up with another -- or perhaps stronger variant -- hypothesis.

Rails itself aspires to be a 4GL -- a DSL together with a framework, specializing in database-backed, RESTful web application development.

It appears that some programmers see (or want to see) just the 4GL parts (the DSL, the declarative bits, the "conventions" and the "magic") while others see a 4GL in close symbiosis with a 3[+]GL (Ruby) for doing procedural work.

In some sense, both groups are seeing accurately -- for certain problems. That is, apps that truly hit the "sweet spot" for which Rails was designed, and which do nothing else, can be treated as compositions of 4GL bits much of the time. Highest productivity, get your check, go home, watch a movie.

For other problems, additional logic and/or integration is required.

And here's where the two groups of programmers part company. The pure-4GL-ers want to search high and low for a plug-in, gem, or other black-box component to solve the problem. Even if the black-box component is harder to use, poorly documented or maintained, does not include a test suite verifying it does what it claims, this group of coders wants the plug-in. They'll search and argue and try different ones even if the functionality is 15 lines of Ruby code. But they won't write the functionality themselves.

The other group just writes the code. Perhaps they even bundle it into a plug-in and throw it on github. They also do things like submit patches to Rails itself.

Depending on the situation, either approach might be fine ... but: when something doesn't go right, in terms of logic or process or performance or data integrity or something else ... the first group hits the wall. All of a sudden it's a "Big Problem" ... whether or not it's really a big problem. That is the Achilles heel of all 4GLs: if you view them as "closed" systems that require pluggable components then anything outside their bounds becomes a "Big Problem."

And lately I've watched as some participants in the Rails community systematically leap tall buildings in a single bound when they encounter them, while others literally sit on their hands waiting for a plug-in to appear (with no track record or test cases), which they then gratefully throw into their app and hope for the best.

Monday, December 29, 2008

Fear of the Driver Disk

The problem of bloatware/crapware on retail PCs is well known -- to the extent that Apple makes fun of it, pointing out the absence of such software on new Macs, while PC tools exist just to clean it.

But bloatware has a less-famous, equally annoying sibling: all the garbage that brand-name hardware devices install off their driver or utility disk.

Pick up a peripheral -- printer, web cam, DVD drive -- from a major brand, and if you follow the automagical installer on the drive disk, you'll get a half-dozen apps that you may not need or like. In some cases, they're just bad apps (that have a habit of arranging to start at boot), while in other cases they can destabilize a perfectly-running system.

The problems are that

  1. in some cases you do need these apps, because some hardware features require "support software" to be present, and don't fully leverage the many built-in drivers and control panels available for free in Windows ...
  2. most hardware companies internally view the driver/utility software as an afterthought, writing it hastily, testing it inadequately, and staffing it with ... well ... whomever they can find.

There are two main remedies.

In many cases, getting an unbranded or "strange-branded" device is a smart idea (provided you know what you're getting). I've found these devices have straight-forward, minimalist support apps, make great use of built-in Windows drivers, and don't put any garbage on your system -- for the simple reason that they don't have the resources to write a bunch of half-baked apps, or to form "distribution partnerships" with people who do.

If I do have a brand-name product, I generally attempt to install it without its own driver disk, no matter what the instructions say. In many cases, the device is fully supported by Windows out of the box; in other cases, some features may not be available -- but I may not need them. (E.g., If I wanted to use my digital camera as a webcam, that would have required the vendor driver disk... but I have never wanted to use that feature of the device.)

And if that latter approach fails, it's pretty easy to uninstall the device or otherwise convince the PC it's never seen the device before -- so that you can go the RTFM route and use the supplied disk.

Tuesday, December 23, 2008

Stax Brings more Standard App Models to the Cloud, Marginalizes Custom Platform Further

Stax, which recently launched in private beta, is a cloud hosting/deployment/ops platform based on Java appservers. The coolest thing about Stax is that it offers many flavors of JavaEE-deployable applications, including Python apps (via Jython) and Rails (via JRuby) with ready-to-roll built-in templates.

Stax has a very AppEngine-y feel, not just on the website, but in terms of the SDK interactions, local development, etc.

This is good news for all of the popular platforms ... and bad news for those rattling around the corners with non-standard APIs. As the app-hosting industry continues to mature, the emphasis will clearly be on established players like Rails, ASP.Net, JavaEE, Pylons, et al. at the expense of guys like AppJet.

It's not about the language (JavaScript) but the about learning a set of APIs, patterns/practices, and sustaining a community ... based on a roll-your-own platform.

It is true that some of these built-for-the-cloud platforms were designed from the start to default to hash-style or big-table style storage -- popular for content-oriented cloud apps because of its easy horizontal scaling -- where the "traditional" platforms focus on relational stores and have a variety of bolt-on APIs for cloud DBs.

But now that there are so many standard alternatives, it is unlikely developers will pay any custom-platform-tax no matter how elegant that platform might be.

Thursday, December 18, 2008

We May Look Back Fondly on Our 68% Project Failure Rate

The report [PDF] referenced here -- focusing on failure originating in the business requirements realm and offering a "68% project 'failure'" headline -- inspired two thoughts.

First, it remains a head-scratcher why nearly every project, despite talk of risk analysis and mitigation, expects to be in the 32% success area. Even if a company has the best resources (and generally it will not), many causes of project failure originate in organizational factors -- friction, process, politics -- and externalities (e.g., no budget this quarter for the tools in the original plan).

Since these issues are rarely known -- and, I would argue, actively denied in the sense that whistle-blowers and problem-spotter are systematically excluded from planning (at best) or forced down or out -- the average "expected value" of the d100 role is way less than a 68.

In startups, I've heard the excuse that the whole company is a "long shot" -- as though that justifies taking on disproportionate risk in each project implementation.

In enterprises, it seems as though management is so used to failure (in a timeline or budgetary sense), that they have simply redefined success to mean anything that they can pull off in their tenure -- and if that means a system that kinda works, but no one likes, shipped in 200% time and 400% budget, well, that's just the "reality" (which, to be sure, it is once all other possible paths have been discarded).

This redefinition of success also has the side effect of removing accountability and pretty much assuring a nice bonus and making strict accountability impossible.

Another thought inspired by the report is how the gap between enterprise development and small / startup development is widening. On the one hand, large businesses could benefit from the high-productivity tools and agile approaches popular with startups; for a variety of reasons ranging from policies to personnel, they are not exploiting the latest wave of technology, and it's costing them.

On the other hand, what they need regardless of tech is solid analysis and estimation capabilities. Analysis and estimation are possible, but hard, and the agile camp has moved mountains to try and reframe the problem so that they can advocate punting on all of the hard parts of these disciplines. That works great for the hourly agile consultants, but inside large businesses and large projects, it just doesn't cut it. A business needs to be able to attempt to estimate and plan months or even years of a project (hence the prevalence of "fake" agile in companies that purport to use it).

The fact that the business does a horrible job with the estimation today does not mean that the organization (which, generally, is run by business people not developers) won't keep planning and targeting in the large.

The result of these two pieces (different tech, different process) is that enterprise and small development are moving farther and farther apart, which is a damaging long-term trend. Ideally, these two groups should be learning from each other. They should spend more time moving in the same world.

The enterprise learns that it probably doesn't need JavaEE and Oracle and a big planning process for myriad internal utility apps that could be done with Rails at a fraction of the cost and effort. The small company learns that relational integrity, transactions, estimation, and operations management are sometimes both necessary and profitable.

Even more importantly, individual employees in the industry can cross-pollinate ideas as they move between these environments over their careers, sorting fact from fiction and getting better at determining what will work.

The farther apart these groups are, the less appealing it will be for "startup guys" to work in an enterprise or a startup to hire on an "enterprise guy."

This trend -- since it keeps useful knowledge hidden -- can only help a 68% failure rate go up.

Friday, December 12, 2008

How is Google Native Client Faster Than -- Or As Safe As -- JVM or x86 VM?

When I saw Google's proposed native-code execution plug-in earlier this week, my initial reaction was: "Don't we already have those, and they're called exploits?"

I decided to mull it over a bit, and I still don't like it. While the idea of sandboxed native x86 execution based on real-time analysis of instruction sequences makes for a great Ph.D. thesis or 20%-time project, it sounds like an awfully large attack surface for questionable benefit.

Here's what I'd like to know from a practical nuts and bolts point-of-view: how many "compute-intensive" scenarios could not be implemented effectively using either (1) the Java VM or (2) a plug-in based on widely-available open-source machine virtualization, running a tiny Linux snapshot.

While the JVM falls short of native code in some places, it can be as fast -- or even faster -- in other cases (faster because the runtime behavior provides opportunities for on-the-fly optimization beyond what is known at compile time). Yes, there are issue with clients not all having the latest Java version -- but that seems a small issue compared with the operational issue of deploying a brand-new plug-in or browser (Chrome).

Another approach is to use a plug-in that runs a virtual machine, which in turn runs a small Linux (started from a static snapshot). User-mode code should run at approximately native speed in the VM, which should cover the pure computation cases. In a rare situation where someone wants to run an OLTP file-based db (which would behave badly in a VM environment) or get access to hardware-accelerated OpenGL facililties, specific well-defined services could be created on the host, which code could access via driver "ports" installed in the VM image.

These approaches to the security boundary -- Java VM and x86 VM -- are well-known, long-tested and based essentially on a white-list approach to allowing access to the physical host. While Google's idea is intriguing, it sounds as though it's based on black-listing (albeit with dynamic discovery rather than a static list) of code. I'm not yet convinced it's necessary or safe.

Tuesday, December 09, 2008

Yes, VC May Be Irrelevant if it Continues Focusing on Weekend Projects

Where are Web 2.0's Amazon.com, PayPal, Google, or Travelocity? They were never funded.

Paul Graham and now Eric Schonfeld are extending the several-year-old "VCs don't know what to do with Web 2.0's super-low-cost startups" meme. The extension has to do with the recession, arguing that it will accelerate the decline of VC relevance, as VCs become more reluctant to fund these shoestring startups, and more entrepreneurs pull a DIY anyway and ignore VC.

Well... maybe. The VC problem with micro-cap micro-startups is real, in the sense that it's a real math problem where the VC fund size divided by the proposed investment size equals too many portfolio companies to interact with, and a kind of interaction that is a big break away from the old model.

But the easiest fix is for VCs to simply invest in more expensive businesses. The current predicament is a classic chicken/egg: after the dot-com crash, VC money was hard or impossible to get, so the businesses people started were these undergraduate-level 1 man-year (or couple-of-weekends!) efforts.

Small product, small team, small money. We got things like 'Remember the Milk.' Cute, sure. Useful, sure. But nothing too hard or too ambitious. Nothing a tiny shop -- or a bored college student -- couldn't hack out in their spare time. Indeed many were spare time or side projects.

Some of these apps got a lot of users, especially where network effects were involved, and eventually VC wanted nothing that wasn't viral, network-effect, social. Never mind that there was never big challenge or big value in those. Facebook is the biggest thing in Web 2.0 at the moment, and it's nothing but network effect and questionable monetization. "Not that there's anything wrong with that..." It's a fine, useful application. But in the absence of any real substance the model became more Hollywoodesque, more personality- and connection-dominated than it should have.

So where is the Amazon? PayPal? Google? Travelocity? Ariba? Netflix? Danger (maker of the Sidekick devices)? Those big projects that take tens of man-years, maybe hundreds? The projects that can't "launch in beta" after a few months and acquire tens of thousands of fanatical users because they're more than a glossy AJAX UI on a local database?

Where are the startups that drag whole industries whining and screaming into the 21st century and liberate billions in value, trapped in transactions that real people make every day?

Those big projects haven't been A-round darlings for a long, long time. VCs, terrified of risk, moving in a tight pack, loved the new ethos: let the entrepreneur build a product, get it launched ("beta"), get customers, get mirror-hall PR (blogosphere), then later drop in a few bucks. Less risk. But not easy money. Count the exits.  And the ad-only valuation has no more magic today than it did in 1998.

And no one even notices when hundreds or thousands of these little pownces and iwantsandys hit the deadpool.

It's no coincidence that many of the step-by-step tutorials for new frameworks and tools teach you to clone a blogger, a flickr, or a wiki farm in a sitting. After all, those are trivial undertakings, lacking only for a network-effect mob signing on. And we'd all have a good laugh about widgets one day... if we weren't laughing so hard already.

Can you imagine a tool vendor of the dot-com era giving an hour tutorial that produces a working Travelocity clone? a working PayPal clone? It's sketch comedy, or something sadder and more disturbing. Frankly, the most ambitious projects in all of web 2.0 are the tooling and infrastructure plays that are largely open source.

Investors, entrepreneurs, engineers and end users might all do well by hunting some bigger game.

Thursday, December 04, 2008

Approaches to the Silverlight Dilemma: Cashback? Vista SP? Win 7?

The jury's still out on Windows Live Search Cashback -- apparently there were some issues on Black Friday and, overall, things aren't up to goal ... but that could change.

In light of this cashback program, though, my proposal from last year that Microsoft simply pay people to install Silverlight seems astonishingly realistic.

The problem with Silverlight is that it doesn't have wide enough deployment -- so there is a chicken-and-egg problem deploying apps on the platform. Microsoft has released stats about how many people 'have access to a PC running Silverlight' -- but that's an odd definition of penetration (if a state university has 15,000 students, and 10 computers in a library lab have Silverlight, then presumably all 15,000 'have access to a PC with Silverlight'.)

So ... why not just pay folks a small amount to install? Maybe a $2 coupon(Starbucks/Amazon/PayPal/etc.) to install the plug-in in the user's default browser.

The WGA or XP key-checking components could be used to keep track of machines that have participated, making it impractical to game the system by performing extra installs. I'm sure a suitable equivalent could be used to verify installs on Macs and Linux boxes.

Another approach is to wait for a future Vista SP or Windows 7 upgrade cycle. Since Silverlight will presumably be included for IE, the trick is to hook the default browser selection and attempt to install a suitable version of Silverlight into the target browser (Firefox, Chrome) when the user selects it.

This move would surely be controversial -- even if there were a way to opt out -- and would bring back memories of the Microsoft of the 90s for some. But, since the web page itself decides what plug-ins to use, and no content is "forced" to render via Silverlight, it's only moderately different from making WMP or any other dynamically loadable object available.

And for those "open web" advocates who have nothing good to say about proprietary plug-ins ... consider that Silverlight does not really compete with the open web, but rather with Flash, which otherwise has a de facto monopoly on any RIA that cannot easily or cost-effectively be implemented in HTML/JavaScript. Breaking that monopoly could actually help the open web cause by creating a real platform debate in a space that doesn't really have any debate right now.

Tuesday, December 02, 2008

Microsoft Begins Posting "Proceedings" Docs for PDC Sessions

Microsoft has started posting an accompanying document for each 2008 PDC session, containing a detailed content summary, links, and indexes (into the video stream) for demos.

These docs, called "proceedings," had been part of the plan to publish the PDC content this year, and it's nice to see them showing up alongside the slide decks and full-length session video streams that had already been posted.

While not full transcripts, or even detailed outlines, the information value of these docs is far higher than the full-length videos (per minute of attention investment), so I highly recommend them if you have interest in any of the newest Microsoft technology.

Docs, transcripts, and outlines seem to be a more efficient way to present content for multicast consumption than live video. While full-length web video is cheap and easy to produce, it wastes an enormous amount of time on the viewer side. Instead of reading/scanning the content as fast as the reader likes, he or she must sit through the lower-information-density spoken content and -- what's worse -- is forced to do so in a linear fashion. The time loss is then multiplied by all of the viewers ...

If an employer is paying for your time watching videos -- whether they be technical sessions, product demo videos (all the rage now), or so-called webinars -- then perhaps it's nice to sit back, have a cup of tea, and go into watcher mode.

But if you're investing your own time/resource/productivity into acquiring some knowledge, it's nice to (1) be able to do it at your own reading pace and (2) be able to skim/scan sections of less value at high speed and pop out into 'careful reading mode' as necessary.