Ruby and Python as Cloud Lingue Franche; Ruby/Rails on 10gen

Not sure how this one slipped past me, but 10gen announced support for the Ruby language and most of the Rails framework APIs on their open-source cloud service last month.

This addition is great news for 10gen and for cloud computing (the hosted-application-platform flavor, not the hosted-hardware/datacenter flavor).

For 10gen, support for a well-known API and app model is a huge bonus, which makes it easy for people to move an app into the cloud without learning and coding to new APIs, and also lowers the perceived "lock-in" involved, should the move not work out.

Their original JavaScript platform approach, as I've written before, is problematic not only because folks are unlikely to have meaningful (for their business) apps lying around to try mounting in the cloud, but more so because there is no standard server-side JS API set. A half-dozen companies offer a JS app server or cloud and they all have different platform APIs for even the simplest things, such as reading HTTP request variables, or deleting a session.

10gen takes a big step forward, joining Stax, Heroku, and morph labs in supporting Ruby on Rails in the cloud.

This move also reinforces another emerging trend: Ruby and Python serving as lingue franche for cloud app stacks. While many cloud offerings support JavaScript or other languages, Ruby and Python seem to be emerging as the ones with broadest support: 10gen will support both; AppEngine supports Python and a language-to-be-named-later; Stax supports both; Azure will likely support IronRuby and IronPython (some Python apps can already work in Azure).

Of course, the language is only half of the battle -- there are the APIs to deal with as well, and issues will typically arise where the impedance mismatch is highest with cloud-related infrastructure. E.g., cloud databases are mostly non-relational and don't support SQL ... so an ActiveRecord or SQLAlchemy API won't work on 10gen's 'grid database' (a reasonable tradeoff for simpler scalability.)

Even so, it is starting to appear as though one could write a lot of core business logic using, say, Python, and expect it to run unmodified on most vendors' clouds. Not a bad position to be in for the Python folks.

On the Wide Range of Ruby Talent: Rails as a 4GL(?)

For a long time, I was puzzled by the extremely wide range of talent level among Ruby talent: more than any other language I could think of, "Ruby Programmers" seemed to range from the truly clueless to hardcore accomplished engineering veterans.

I had more or less chalked this observation up to the language and APIs themselves -- easy to get running with, highly intuitive ... and yet packing deep and powerful features that would attract the more sophisticated (easy C integration, metaprogramming, continuations, etc.)

On top of this was Rails with its famous screencasts showing database-backed websites automagically constructing themselves, a feat that got novices very excited, and reminded experts that they should be asking harder questions of any other framework they were using for the same kind of app.

But lately I've come up with another -- or perhaps stronger variant -- hypothesis.

Rails itself aspires to be a 4GL -- a DSL together with a framework, specializing in database-backed, RESTful web application development.

It appears that some programmers see (or want to see) just the 4GL parts (the DSL, the declarative bits, the "conventions" and the "magic") while others see a 4GL in close symbiosis with a 3[+]GL (Ruby) for doing procedural work.

In some sense, both groups are seeing accurately -- for certain problems. That is, apps that truly hit the "sweet spot" for which Rails was designed, and which do nothing else, can be treated as compositions of 4GL bits much of the time. Highest productivity, get your check, go home, watch a movie.

For other problems, additional logic and/or integration is required.

And here's where the two groups of programmers part company. The pure-4GL-ers want to search high and low for a plug-in, gem, or other black-box component to solve the problem. Even if the black-box component is harder to use, poorly documented or maintained, does not include a test suite verifying it does what it claims, this group of coders wants the plug-in. They'll search and argue and try different ones even if the functionality is 15 lines of Ruby code. But they won't write the functionality themselves.

The other group just writes the code. Perhaps they even bundle it into a plug-in and throw it on github. They also do things like submit patches to Rails itself.

Depending on the situation, either approach might be fine ... but: when something doesn't go right, in terms of logic or process or performance or data integrity or something else ... the first group hits the wall. All of a sudden it's a "Big Problem" ... whether or not it's really a big problem. That is the Achilles heel of all 4GLs: if you view them as "closed" systems that require pluggable components then anything outside their bounds becomes a "Big Problem."

And lately I've watched as some participants in the Rails community systematically leap tall buildings in a single bound when they encounter them, while others literally sit on their hands waiting for a plug-in to appear (with no track record or test cases), which they then gratefully throw into their app and hope for the best.

Stax Brings more Standard App Models to the Cloud, Marginalizes Custom Platform Further

Stax, which recently launched in private beta, is a cloud hosting/deployment/ops platform based on Java appservers. The coolest thing about Stax is that it offers many flavors of JavaEE-deployable applications, including Python apps (via Jython) and Rails (via JRuby) with ready-to-roll built-in templates.

Stax has a very AppEngine-y feel, not just on the website, but in terms of the SDK interactions, local development, etc.

This is good news for all of the popular platforms ... and bad news for those rattling around the corners with non-standard APIs. As the app-hosting industry continues to mature, the emphasis will clearly be on established players like Rails, ASP.Net, JavaEE, Pylons, et al. at the expense of guys like AppJet.

It's not about the language (JavaScript) but the about learning a set of APIs, patterns/practices, and sustaining a community ... based on a roll-your-own platform.

It is true that some of these built-for-the-cloud platforms were designed from the start to default to hash-style or big-table style storage -- popular for content-oriented cloud apps because of its easy horizontal scaling -- where the "traditional" platforms focus on relational stores and have a variety of bolt-on APIs for cloud DBs.

But now that there are so many standard alternatives, it is unlikely developers will pay any custom-platform-tax no matter how elegant that platform might be.

Rails and .Net Had One Thing in Common Before ASP.Net-MVC: Magic

It's possible now to build a very Rails-like website via ASP.Net MVC. And at RailsConf, John Lam showed IronRuby starting to run Rails.

But long before all this, these two very different platforms had one thing in common: "magic."

By magic, I mean the sort of implicit, we-solve-problems-you-didn't-even-know-you-had kind of magic that makes it easy for people who don't know the details, don't want to know the details, or know but don't want to be bothered by the details to build an app.

From ASP.Net's first iteration, where you could drop an item on a "web form," wire a server-side event handler, and talk to items on the page through a .Net server-side object interface, ASP.Net has been heavy on the magic. Rails has been the same way, from scaffolding up through ActiveRecord, you point Rails in the direction you want to go and it drives.

Of course we all know about the leaky abstractions... in .Net, you used to get some funky HTML and Javascript, plus a server-based page lifecycle that didn't always match the model of what you wanted to do with the DOM. In Rails, we've seen perf issues, and discussions of the "worst practices" [pdf] that can result from letting the autopilot do too much and your brain too little when writing for Rails.

But in any case -- as opposed to a framework like Django (which included a big feature push called 'removing the magic' to make functionality more explicit) -- Rails and .Net give you as much magic as they can.

Rails is "Slow," But Railshosting.Org May Be Scaring People Away Unnecessarily

Rails is slower (by many metrics) that a lot of web frameworks. But that doesn't mean it's not both great and cost-effective for many kinds of applications.

Here's a problem, though: if you're someone thinking about putting up a small-to-moderate-sized app on Rails, and you Google either "rails host, "rails hosting," "host rails," or "hosting rails," you will find RailsHosting.Org as search result #1 or #2.

RailsHosting has a section called "What is the right plan/stack for you?" and it basically paints such a terrifying picture that it might scare a lot of newcomers away. Is that hyperbole? You tell me: according to rh.org, a shared hosting plan with FastCGI is only good for 1,000 hits per day. For 2,000, you should be using a virtual dedicated server or an actual dedicated server along with Mongrel, Pound, and/or mod_proxy_balancer. 5,000+ hits/day? You need a dedicated server.

What? what?! whaaaat?!?!? If I'm a developer in an enterprise and I see numbers like this, it's game over, forget it.

Happily, these numbers aren't right. Based on my testing with actual shared hosts, it should be possible to run a site at least up to 10,000 hits/day (assuming that load ranges from 0.33x to 3x at different times of day) with a decently configured and run shared hosting plan -- even with a non-optimized app, little or no caching, etc. If the average host can't handle that, it's not a Rails problem, it's a shared hosting provider problem.

Now, if you want 10,000 hits/hour with lots of writes per page into the same database, and peak rates higher, than you're gonna need a different approach. Rails may still do it for you, but it won't be plug-and-play on a shared host.

In any case, server technology, virtual datacenter technology, and "point-click-buy" Mongrel and dedicated memory has improved since RailsHosting first put this data online.

So, notwithstanding the fact that my Samsung Blackjack (running .Net Compact 3.5) could probably serve ASP.Net faster than the average hosting provider can serve Rails, the first information kiosk on the Road to Rails should offer a less pessimistic and more realistic view!

Application Models, Part 2

A little while back I described the idea of an "application model."

The application model is a key concept that keeps getting muddled and confused with partially overlapping concepts like languages (e.g., Paul Graham's discussions of "languages for smart people" vs. "languages for the masses") and platforms (a legacy platform vs. a "great new" platform, etc.)

So besides being a useful term for which I'm not sure there is an equivalent, what is the role of application models?

First, application models -- not languages or platforms or vendors -- argue, persuade, and sometimes win.

Whether it's Visual Basic (the original), or C++ and OpenGL, or Rails, the question is "How well does the application model solve the world's (ok, domain's) problems for the folks who actually do the solving?"

And the cool thing is that application models are very malleable in a way that languages and platforms aren't -- because the they include (well, really their complements or negations include) all kinds of APIs and libs that people cook up.

Huh? Like this: the application model for JSP/Servlet apps (including "model 2") stunk. So folks created all sorts of libs (Struts, Tapestry, etc.) and templating engines (FreeMarker, Velocity, etc.) in order to create a better-shaped "hole" -- a better application model for what were still fundamentally Java Servlet apps. Still a hassle. Nice underpinnings, great performance, needs some work. We got IoC containers and the application model became friendlier. For small, quick apps, Rails had a lot of appeal, so we got Groovy and Grails as options. In the .Net world, we have C# adopting elements of Ruby (yes, they're really Smalltalk or Lisp elements) and ASP.Net offering the "MVC Framework" which -- with LINQ -- creates a flavor of Rails.

There is no intellectual property violation going on here. Platforms and languages are adapting around good application models. Some of this adaptation is vendor driven, some is community driven, some involves new languages, some involves new tools, and some involves APIs. But the common thread is a desire to alter an application model in a particular direction.

Application models are what get mindshare and what make provide for the continuity of commercial application development patterns over time. More on this point later.

Before I get there, it's worth pointing out that the application model also bears heavily on the aspects of application development apart from core feature development. When we get into these aspects, non-functional requirements, or orthogonal (really non-collinear) concerns, the application model determines (or at least posits an optimal or easiest path for) integration. How the UI attaches, how app-level security is managed, how different services are integrated -- these all fall under the sway of the app model, and if we want to make these things better, the app model is what we should be talking about.

Over time, the adaptation of application models (by the all of the parties mentioned above) is the story of the evolution of commercial software. Commercial software evolution is often described as a sequence of machines, or OSes, or platforms, or languages. It most definitely is not these things, any more than it is a story of vendors fighting and dying and rising. The thread of continuity, and the real underlying story, is about people adapting application models, piece by piece, over the years.

Path of Least Resistance == Most Rails Apps Send Your Password In the Clear

The bulk of developers will follow the gradient down hill, even many of the ones who think they're rebels (I'm pretty sure this scene from Life of Brian was about Steve Jobs and the cult of iConform).

Too bad for them, right? Well, no. Too bad for us as a software industry with a reputation for troubled delivery. And too bad for us as citizens and consumers, who suffer from bad software not only in dramatic ways but in having our private data (passwords, SSNs, etc.) sprayed all over.

Elegant and minimalist principles are great; still, if you give most software developers a knife without a sheath, it's only a matter of time before the blood starts flowing from somewhere.

I was surprised when I realized that Rails has no built-in authentication module; more surprised when AWDR "teaches" you to roll your own authentication; and most surprised when I realized that acts_as_authenticated (nice, but intentionally not sophisticated) is de facto standard for auth in Rails.

Hmmm... I wonder how many of those Rails sites I use all the time actually bother to protect my credentials? If the originating or following page isn't kept in SSL, you can't tell by looking for the little "lock icon," but you can tell be having a look at where those login forms are posting to.

I looked at the first two pages (24 apps) on happycodr.com, a site that claims to be "the unambiguous showcase for sites designed with Ruby on Rails," and which is linked directly off of the Apps page at rubyonrails.org.

Eight of these sites are content-only and don't offer logins. Of the remaining 16, fully 12 send their passwords in the clear. Two of the four which don't are AOL properties, required by AOL to use a combination of AOL SNS or OpenID, and SSL.

Of the 14 Rails sites built by non-AOL companies and taking a password, 12 (85%) just send all their users' passwords in the clear.

Some of these organizations think that because their services may not contain sensitive data, it doesn't matter. But most people use a very limited set of passwords (or just one!) for all online sites. When someone keys a password into a site I build, I protect it because I assume that is also their online banking password, or their email password, etc.

To be fair, it is no secret that HTML forms are vulnerable in transmission, and many Rails resources point out that the easiest solution to this is to employ SSL. But clearly the message is falling on deaf ears.

And these developers aren't even trying to cleverly roll-their-own system. They're not hashing the username/password on the client (vulnerable to MITM, replay, impersonation, and other problems but better than nothing); they're not using a nonce and hashing (still broken but better). They may not even realize there's a problem.

To be fair to Rails, it's not the only platform in this fix. I was stunned when Microsoft rolled out a nice set of prebuilt User/Role/Profile/Login components in ASP.Net 2.0, without building any security in. I figured these widgets would get dropped into a lot of sites without a second thought.

It's a little "wider" issue for Rails, though, because, Microsoft has been strong in enterprise development where there are IT departments and policies that may wedge SSL in there before an app makes it to production. Whereas Rails is used more widely in smaller organizations where there is not likely to be anyone with the authority to impose security on a development team.

For ASP.Net, where many developers never look under the hood of the components, and both the client-side script and server-side post-back handling is accessible inside the ASP.Net stack, I would recommend loading the login widget with the best possible pseudo-security using JavaScript crypto for non-SSL scenarios, and then still requiring the developer to actively add a web.config setting to allow running in non-SSL mode. This has the added benefit that, when the app is moved to a production environment for the first time, with a new web.config, the error will pop up again as a reminder right at deployment time.

For Rails, things are a little more complicated. It's going to come down to better educating, and perhaps a little more structured thinking about what I call the "application model" (not platform nor architecture). Which I will write about next time.

A Couple of Tiny Rails SSL Helpers

Here are a couple of helper methods for ensuring forms are set to submit via https in the production environment (but not in dev), and for redirecting back out of SSL afterward. Since SSL can be resource intensive on the server, it's usually good to hop back out into cleartext unless the nature of the application (e.g., financials) warrants encrypting the whole session.

To create a form that uses, SSL, just replace form_tag with form_tag_using_SSL_in_production.

In application_helper.rb:

def production? 
ENV["RAILS_ENV"]=='production'  
end

def form_tag_using_SSL_in_production form_args, &block 
form_args[:protocol], form_args[:only_path] = 'https', false if production?  
form_tag form_args, &block  
end

form_for is the preferred helper ... if it's actually a "form for" a model object, which this one was not. I'll leave the analogous form_for_using_SSL... as an exercise for the reader.

When you're done with the relevant action processing, any content rendered is going to get sent back under the SSL connection. At some point (in my case, immediately) you want to redirect out of SSL. Just use redirect_and_drop_SSL the same way you would use redirect_to.
In application.rb:

def redirect_and_drop_SSL destination 
destination[:protocol], destination[:only_path] = 'http', false if request.ssl?  
redirect_to(destination)  
end

You might be thinking this stuff is too trivial to post about, and anyone who needs to use SSL knows this stuff already. Unfortunately, that's not entirely the case, as I'll write about in my next post.

In Which I Try to Save Someone from a Rails/MySQL BLOB Debugging Hassle

I was recently debugging an app and spent a bunch of time on a MySQL error symptom. I'd like to save the next developer from the same searches, since there are lots of documented MySQL connection issues with Rails, and few (any?) posts mentioning the following particular gotcha:

If you're getting the Lost connection to MySQL error, or its cousin, MySQL server has gone away (depending on whether you're using the Ruby driver or the Ruby-C driver), and you happen to be working with BLOBs (images, file uploads, streaming docs from a DB, etc.), there is a reasonable chance that the problem is you're sending too large a "packet" to MySQL. The "server has gone away" doc mentions this and refers to more detail at the "packet too large" error detail page.

In the current MySQL version, the default max packet size is 1 MB and by packet, they don't mean the TCP packet you might be using under your connection, they mean "a single SQL statement sent to the MySQL server, a single row that is sent to the client, or a binary log event sent from a master replication server to a slave." So it's actually quite easy to hit the default limit if you're working with BLOBs.

The limit can be raised in various ways (command-line, service config, etc.) and the doc points out that raising the limit should not in itself incur a performance penalty, as the additional memory is allocated only when needed.

Before cranking the max_allowed_packet setting up and calling it a night, though, it's also worth a quick sanity check as to whether you really mean to be reading and writing BLOBs this big. If you're storing videos or RAW data from a DSLR, then maybe yes.

In my case, I was using a heuristic for converting and compressing smallish photos and logos. I didn't intend to create a larger image than what the user was uploading (the largest test item was around 250kb), so the 1 MB default limit should have been fine. On closer inspection, a bug in my logic flow was incorrectly deciding to convert some 250K JPEGs into 4MB PNGs. Doh!

Rails Hack: Combining Acts_As_Auth with Long-Lasting Anonymous Sessions

In one of my Rails apps, I have two kinds of users: anonymous visitors, who do not have to sign up at all to use the app's functionality, and "vendor" users, who need to sign up and create some content in order to get value from the site.

The requirement for sessions was simple:

Anonymous visitors should have a session that persists for a long time (e.g., months) via a cookie. Since there is no information at all collected from these visitors, there is no reason to clear them out of the browser, and they would appreciate the convenience of being able to pick up where they left off when they come back.

Vendor users, however, have more data at stake, and so should have the typical login/password/remember-me?/logout pattern, allowing them to decide if their sign-in should persist, and letting them explicitly logout to kill their authentication.

I dropped in acts_as_authenticated, and it does a fine job with the second case (vendors). For the first case, I then added sliding_sessions, which is elegant and has now been added to the relevant wiki page.

The problem is: once session cookie durations are extended in general (at the application or controller level), aaa's authenticated users will be "remembered" upon return to the site, whether they want to be or not -- because their user info is already in their session (i.e., the "login from cookie" is not necessary and is never called). Timing out a session/auth on the server side is a good practice, but doesn't solve this issue -- if the user closes the browser and walks away, someone might come up right behind them and come back to the site, where the session/auth has not timed out.

Here's my solution. It seemed quick and easy; if anyone has criticism or improvements, let me know:

There are three cases -- anonymous user; vendor asking to be "remembered"; and vendor not asking to be "remembered" -- of which the first two worked fine. So I add a cookie upon login, set to timeout at the end of the current session, which will help me separate out the third case.

The details:

AAA adds a couple of "account controller" actions to your app. In login, I added the new cookie (see line 11). Then, in application.rb, I added this method as a before_filter. It checks to see whether you are an authenticated user (i.e., not an anonymous visitor). If so, it resets the session unless it sees the vendor cookie (meaning you've signed in in this browser session and so should not be cleared out yet regardless of "remember me" choice) or the auth_token cookie (created by aaa, and indicating the the "remember me" persistent login choice).

So if you authenticate and do not choose "remember me" then you have no auth_token cookie and, when you close the browser, you lose your vendor cookie. When you come back to the site, your session (and user info) are cleared.

It's About the API

Ok, at the risk of being almost timely, Zed's Rails Rant has inspired me to finish a point I've been wanting to make for a while now: Platforms (and languages too) are about the developer experience (API). Not the community, although that can help. Not the implementation -- that can be changed, redone, ported, etc.

What is nice about Ruby and Rails is that they're "fun" or "pleasant." In general, when programmers say this, what they mean is

1. Working with the language / API does not go against the grain of how their mind works. Since programmers have varied ways of thinking, simply avoiding an "against-the-grain" experience across a set of non-trivial cases is a real accomplishment. Ruby does this in spades, Rails almost as much once you get used to a particular "Rails way" of doing something.
2. It's at the right level of abstraction most of the time, when used in the kind of work for which it was intended. RoR is fabulous at this. It does the "super-high-level-language" thing when it reads your data schema, does :has_and_belongs_to_many, and makes a very sensible attempt to match URLs, actions, and views by default. It goes low-level by working with clean, trivial HTML, which is easy to write and debug by hand, while ignoring the temptation to autogen all sorts of decorated HTML/CSS/JS.
3. When you need to read the core sources or other people's code, it's not a terrifically painful experience that leaves you sitting there saying,"You've got to be kidding" over and over again.

Now, apart perhaps from a bit of #3, these are descriptions of a programmer's experience interacting with the language/platform/framework/tools.

They are API issues, not implementation issues. If the implementation is bad enough, the API experience will be overshadowed. But the developer experience sells the platform.

So... what about the MRI, and the Rails core team, and Mongrel and horrific perf issues, and all that?

The current state of affairs needs to be seen as a reference implementation. It shows off the specs under discussion, proves they can be implemented, provides all the right output to the definitive test suite, etc. Sun releases new Java APIs with a reference implementation. Some of those RIs have been good enough to use in production, some haven't. Sun hoped that third parties would create commercial implementations -- in some cases, this happened. Where an API was popular and important, a "real" version -- or several -- became available, based on market demand.

There are already a bunch of alternative implementations of Ruby either finished, or in the works. Rails has yet to see an alternative implementation, so far as I know, but it seems plausible that both open source (e.g., modify or redo the Rails code to provide multithreading support, etc. and run on top of JRuby) and proprietary (Rails as a set of ASP.Net assemblies) implementations will emerge.

When Java offered a great developer experience compared to the alternatives, it picked up adherents even when the adequate performance (heck, even existence) of various APIs was shaky. When .Net offered a great developer experience, it brought more shops into the Microsoft fold than 20 years of, shall we say, other approaches to gaining market share.

Some failed platforms suffered because of business issues (e.g., Delphi and C++ Builder), but most fail just because they don't do #1-3 above as well as the next guy on the block. My point is that Zed's rant is a bit beside the point, even if it's all true. Ruby and Rails have already accomplished the end of influencing industry thought and behavior in a broad kind of way. We'll need to be a little more patient (or make some serious contributions) before we get the followup implementations.

Loving RoR with NetBeans 6

I've switched to NetBeans 6 for Ruby coding, and I'll never go back to those bogus text editors.

Some time ago, I wrote about IDEs for Ruby. I'm not sure I did a great job pointing out out that for Ruby (like JavaScript and other dynamic languages) some of the basic features we'd like from an IDE (code completion, refactoring) are non-trivial problems. The Sapphire in Steel blog has some great articles on working those problems.

The milestones for NetBeans were promising; the other free contender, the Eclipse bits formerly known as RadRails, are now part of the "Aptana IDE." Aptana is a reasonable tool for JavaScript, but last time I checked, Aptana was in "1.0" and the Ruby support still crashed and failed in interesting ways. The fine print points out that Ruby/Aptana/RadRails isn't 1.0. Bummer.

I'm very excited about this NetBeans release. For a detailed discussion of features, check out Roman Strobl's review (part 1 and part 2). Once the app starts and gets warmed up (which admittedly takes longer than booting the OS on my Windows server box), it does a pretty good job of the hard stuff, like refactoring. And it's rock-solid on the rest, like interactive debugging.

Unfortunately, Rails 2.0 became official at about the same time NetBeans 6 did. And Rails 2 changes a few things, like filenames/extensions on erb templates, that NetBeans doesn't know about. But it's easy to work around those things, and I'm sure an update will be forthcoming.

Amazon SimpleDB isn't Astoria... it Could Be, but Does it Need to Be?

A while back I wrote about Microsoft's Astoria REST-based relational data store in the cloud (or in your data center, if you want it there).

With Amazon's SimpleDB, we're a step closer to making this vision a reality. Now we're almost on track for competition (and sooner-than-later commoditization) of the new world where you don't even need MySQL to store your stuff.

Why almost? Because SimpleDB is not a full RDBMS, but is looks more like a flavor of triple store. Now, a typical (i.e., SQL-style) RDBMS can be built on top of a triple-store fairly easily. So we could will see a SQL processor, JDBC drivers, and the like, from the community pretty soon.

Another way to look at that "top layer" is to take a REST API like those used by Astoria or ActiveResource [PDF link] and simply implement that. Not as expressive as hardcore SQL, but easier, and probably enough for many applications.

What I don't see -- in the long run anyway -- is applications developing against thin wrappers specific to the Amazon triple store service itself. There's nothing fundamentally flawed in doing so ... it's just that, for a variety of reasons, data storage has evolved very slowly. The relational model is going on 40 years old, but still reigns supreme in terms of popularity, even if it has conceptual or technical flaws when put to work in today's applications.

Given the brilliant data storage alternatives that have fallen flat time and again, I doubt Amazon SimpleDB will change the way people talk about storing structured data. So SimpleDB doesn't need to be SQL but it will probably need to at least be RESTful.

Beware the Hail Mary Meta-Hyper-Framework

A while back I spoke with the CEO of a midsize enterprise software company. He invited me to work with a small team building a framework for customers to create applications that would run with his company's app.

I spent some time with the team, looked at the framework, and declined. As diplomatically as possible, I explained what I saw as the key problems with the initiative:

• The term "architecture astronauts" barely touched these guys. I think they actually used the term 6GL in conversation. They were architecture hyperspace/time-travelers.
• At the same time, I'd call them "infrastructure spelunkers" as well. The team trusted nothing, and was reinventing everything. They were rebuilding so far down the stack I started wondering where they built their own hardware. Then I realized that was silly -- after all, they would mine their own copper before they'd use someone else's wires.
• The initiative had been running for several years with no concrete delivery. There was no deadline for a future delivery either. Given that the predominant tools and platforms were evolving underneath this team faster than the team was moving, I suggested I did not expect they would ever release.
• No real world app -- either inside the company or among its customers -- was using it yet. That is, there was no dogfooding or validation of any kind that the framework was learnable or made the relevant problems easier to solve than existing approaches.

The CEO appreciated my frankness, and said that from his point of view, it was a big long-shot bet:

• For him, the 15+ man-years that it cost to work on this framework were a non-issue. With plenty of revenue and people, if this initiative went nowhere, it would not be a significant loss to the company.
• The framework was not critical-path for anything else, so he didn't anticipate any major downside.
• Potential upside was, he felt, huge. If this framework somehow succeeded, he could grow his company much faster than he could any other way he could envision.

Well, I thought, I wouldn't make the same choice -- I would probably look to gain the same leverage using a radically simpler framework or mechanism -- but then I wasn't looking to take on an executive role, so it wasn't really my call.

DHH, the man behind the Rails web app framework, wrote about frameworks when he explained "Why there's no Rails Inc":

"... Rails is not my job. I don't want it to be my job. The best frameworks are in my opinion extracted, not envisioned. And the best way to extract is first to actually do. ... That's really hard if your full-time job is just the extraction part since you now have to come up with contrived examples or merely live off the short bursts of consulting. For some that might work, but I find that all my best ideas and APIs come from working on a real project for a sustained period of time."

Nicely put.