Stored Procedures and Code in the Cloud

For modest-sized Internet applications, the allure of cloud services has two elements.

First, there's simplicity of implementation and maintenance -- the hassle of real-world ops is the sort of problem startup CEOs dream of having, while startup engineers (and engineering budgets) are ill-equipped to deal with it. Second is the promise of easy scalability -- another problem the CEOs dream of, and the engineers secretly hope will become Somebody Else's Problem.

Storage in the cloud is conceptually easy. Especially with ActiveRecord patterns that ignore (at their own peril, but that's an article for another day) 35 years' worth of learnings about data integrity in the relational model. And for those who need to write things more complicated than 37signals' latest masterpiece, true structured data services in the cloud are coming.

What about application logic, though? There's raw EC2, which works on the level of provisioned VM images, and makes you design for clustering, manage your instances (while they're up), and keep your dynamic data somewhere else. Fabulous infrastructure but non-trivial to use.

Folks like heroku have value-added application-level services above EC2, which offer the elasticity with less hassle.

But what about going even higher level, and defining a unit of work, or a service module that can be deployed into a scalable container, preferable "nearby" the data it needs?

Real world example: in a recent project, I needed to be able to run Dijkstra's algorithm on big (250,000+ nodes) graphs in a persistent store. It would be great to use SimpleDB or SSDS (Astoria) for storage, but what about running the algorithm? It's not practical to extract a representation of the graph over the network, and then run Dijkstra on it just to find some interesting nodes each and every time. Changing the algorithm or using a different one? Maybe ... But what I really wanted to do was create a small module that I could ship over to the data, and run there. Even better, I'd like to be able to compute on the data "in place" in a storage facility, rather than extract. Conceptually a bit like a stored procedure.

I believe that the solution -- and an easier way to start shipping computing into a cloud facility -- is to create a module definition that one can code to, and then just upload. I think Python or Ruby would be ideal languages as they are popular, truly cross-platform, and not encumbered with a IP issues. Plus the modules would be provided as source, so that they could be scanned for, e.g., insecure or computationally intensive uses of stuff like eval. In fact, given Google's investment in Python and in interesting tooling in general, they may already be most of the way there.

I just need a better name than "stored procedures" -- that one's not getting any points for cool.

Well, That's Embarrassing (VASGen Bug Fix)

The other day I was using VASGen, my free tool for generating ActionScript 3 code from UML via Violet, and realized the getters and setters on generated properties had the wrong visibility.

Doh!

Not sure how I missed that one before, but it was a trivial fix. Visit the VASGen project page to get the latest (0.2.1) code.

Rails is "Slow," But Railshosting.Org May Be Scaring People Away Unnecessarily

Rails is slower (by many metrics) that a lot of web frameworks. But that doesn't mean it's not both great and cost-effective for many kinds of applications.

Here's a problem, though: if you're someone thinking about putting up a small-to-moderate-sized app on Rails, and you Google either "rails host, "rails hosting," "host rails," or "hosting rails," you will find RailsHosting.Org as search result #1 or #2.

RailsHosting has a section called "What is the right plan/stack for you?" and it basically paints such a terrifying picture that it might scare a lot of newcomers away. Is that hyperbole? You tell me: according to rh.org, a shared hosting plan with FastCGI is only good for 1,000 hits per day. For 2,000, you should be using a virtual dedicated server or an actual dedicated server along with Mongrel, Pound, and/or mod_proxy_balancer. 5,000+ hits/day? You need a dedicated server.

What? what?! whaaaat?!?!? If I'm a developer in an enterprise and I see numbers like this, it's game over, forget it.

Happily, these numbers aren't right. Based on my testing with actual shared hosts, it should be possible to run a site at least up to 10,000 hits/day (assuming that load ranges from 0.33x to 3x at different times of day) with a decently configured and run shared hosting plan -- even with a non-optimized app, little or no caching, etc. If the average host can't handle that, it's not a Rails problem, it's a shared hosting provider problem.

Now, if you want 10,000 hits/hour with lots of writes per page into the same database, and peak rates higher, than you're gonna need a different approach. Rails may still do it for you, but it won't be plug-and-play on a shared host.

In any case, server technology, virtual datacenter technology, and "point-click-buy" Mongrel and dedicated memory has improved since RailsHosting first put this data online.

So, notwithstanding the fact that my Samsung Blackjack (running .Net Compact 3.5) could probably serve ASP.Net faster than the average hosting provider can serve Rails, the first information kiosk on the Road to Rails should offer a less pessimistic and more realistic view!

Large File Uploads with ASP.Net Require a Config Setting

I'm just putting this out there to save someone else an extra search: last week, a colleague and I were discovering that an ASP.Net app was throwing exceptions when we tried to upload files around 5 MB in size. Turns out that one's by design: here's an article that covers two relevant points:

1. There's a configuration setting (detailed in the article) which limits uploads to 4MB by default.
2. The uploaded file needs to fit in the memory used by the worker process that's handling that particular HTTP request -- all at once; apparently, it can't be configured to 'stream' to a file object or the like. ASP.Net "health management" of the worker processes can get in the way of really large uploads because it gets anxious if the worker process memory consumption exceeds certain thresholds. There are additional config params around this feature that you can tweak if you need to.

Giving the Right Stuff Away with Your Freemium / Free Trial

Freemium maybe a neologism, but the model has been around since the days of shareware, and the free trial concept is prevalent not just in 'web 2.0,' but in everything from end-user software to enterprise software to dev tools.

After all these years, it's interesting to see how far off some vendors end up when they try to decide what exactly they want to give away and why.

Let's look at "why" first: there lots of reasons to offer the free SKU, among them

1. The obvious: get users hooked somehow and get them to buy the paid SKU
2. Build a really big user base with an appealing free product, never mind how many actually upgrade. Variant (a): your ability to give stuff away is limited by specific costs (e.g., bandwidth, burn). Variant (b): you are intentionally burning investor cash to give away more than you can "afford" to in order to build your user base.
3. Build mindshare by giving away something (else) that has exchange value (i.e., is worth real money). This is the most unusual, but examples include sites giving away iTunes song tokens, or software companies giving away a license to a tool or server product if you join one of their sites and do some activities.

Once you know why you're giving something away, you can use that knowledge to inform what and how.

Online, there is a big difference between giving away space (e.g, remote storage of documents) and time (a trial period that ends). If you offer a time-limited trial of an online service, you're discouraging your users from making a commitment of their time or energy, or from storing their valuable data. Why? Because their initial assumption is that if they don't pay, then at the end of the trial period they will not be using your product anymore. Paradoxically, this initial "pessimistic assumption" makes them unlikely to commit enough that they really get hooked and decide the service is worth paying for.

Free space is a much better (and more popular) model: it tells users to relax, get comfortable ... as long as they don't exceed a certain allotment, they can use your service for free forever. Something about free forever makes folks more comfortable. Eventually they really love -- nay, need -- the service, but their free space allotment is full. It's a smaller cognitive leap to toss in a few bucks for some more space, bandwidth, or logins at that point.

There's a big difference between a long trial period, a short trial period, and a "number of uses" period in an offline product. A short trial period makes it difficult for people to settle in before having to take the leap. If they project-switch a lot, they could end up spending a big part of the trial period working on something unrelated and never getting to really eval the product.

A long trial will pick up some casual users who treat it as their own licensed product. Especially if, a couple of months after the trial expires, they can expect to get a beta trial of your next version. A number-of-uses period fixes some of the "short period" problems, but can discourage users from getting comfortable by launching your app just to play around with it.

The idea that the trial period should be purely trial, and not enough for the user to accomplish a real task, is a poor idea. The trial should allow a well-intentioned user to get something done so that he or she can ascertain that the product actually solves a real problem.

There's also a big difference in how you "degrade" functionality for a free version. Online, an ad-encrusted free version is nice because it is very clear what the core product does, even if it may encourage more parasitism than you ultimately want. Removing features, on the other hand, can make the core product look weak and unappealing.

In the offline software world, degradation may take some thought. One time, I was testing video conversion products for Windows. The DirectShow architecture notwithstanding, some software is robust at processing video, even when there are issues in the video or the DShow filters, and some software explodes violently whenever anything unexpected happens. So it's necessary to thoroughly "trial" any app that claims to convert a big range of formats.

Some trial packages added a watermark or logo to the output. Other trials didn't add a watermark, but limited the length of the video clip you could process. Still others had calendar-time-limit features.

Of these approaches, the "limited-length clip" approach was a total loser: I could see the beautiful output quality produced in a few cases, but by making it impossible to convert a lot videos for real viewing in the trial, the authors made sure I couldn't learn whether the app works in the general case.

The watermark apps were somewhat better -- I could use these for a lot of "fooling around" projects and find out if they worked or not. And the calendar-limit apps were the best: within the limit, I used them as though they were my solution ... until one or another demonstrated through failure that it was not.

In the end, the obvious tip is to think hard about your trials. But more to the point: if you're doing free trial or freemium software or services, the free/trial experience should really be designed into the customer lifecycle experience from day one.

Application Models, Part 2

A little while back I described the idea of an "application model."

The application model is a key concept that keeps getting muddled and confused with partially overlapping concepts like languages (e.g., Paul Graham's discussions of "languages for smart people" vs. "languages for the masses") and platforms (a legacy platform vs. a "great new" platform, etc.)

So besides being a useful term for which I'm not sure there is an equivalent, what is the role of application models?

First, application models -- not languages or platforms or vendors -- argue, persuade, and sometimes win.

Whether it's Visual Basic (the original), or C++ and OpenGL, or Rails, the question is "How well does the application model solve the world's (ok, domain's) problems for the folks who actually do the solving?"

And the cool thing is that application models are very malleable in a way that languages and platforms aren't -- because the they include (well, really their complements or negations include) all kinds of APIs and libs that people cook up.

Huh? Like this: the application model for JSP/Servlet apps (including "model 2") stunk. So folks created all sorts of libs (Struts, Tapestry, etc.) and templating engines (FreeMarker, Velocity, etc.) in order to create a better-shaped "hole" -- a better application model for what were still fundamentally Java Servlet apps. Still a hassle. Nice underpinnings, great performance, needs some work. We got IoC containers and the application model became friendlier. For small, quick apps, Rails had a lot of appeal, so we got Groovy and Grails as options. In the .Net world, we have C# adopting elements of Ruby (yes, they're really Smalltalk or Lisp elements) and ASP.Net offering the "MVC Framework" which -- with LINQ -- creates a flavor of Rails.

There is no intellectual property violation going on here. Platforms and languages are adapting around good application models. Some of this adaptation is vendor driven, some is community driven, some involves new languages, some involves new tools, and some involves APIs. But the common thread is a desire to alter an application model in a particular direction.

Application models are what get mindshare and what make provide for the continuity of commercial application development patterns over time. More on this point later.

Before I get there, it's worth pointing out that the application model also bears heavily on the aspects of application development apart from core feature development. When we get into these aspects, non-functional requirements, or orthogonal (really non-collinear) concerns, the application model determines (or at least posits an optimal or easiest path for) integration. How the UI attaches, how app-level security is managed, how different services are integrated -- these all fall under the sway of the app model, and if we want to make these things better, the app model is what we should be talking about.

Over time, the adaptation of application models (by the all of the parties mentioned above) is the story of the evolution of commercial software. Commercial software evolution is often described as a sequence of machines, or OSes, or platforms, or languages. It most definitely is not these things, any more than it is a story of vendors fighting and dying and rising. The thread of continuity, and the real underlying story, is about people adapting application models, piece by piece, over the years.

The Pecking Order of Free Email Services

Why is it that hotmail.com and yahoo.com email addresses are viewed with a certain disdain by technical folks, but gmail.com has a different and higher status?

Is it that Google is (or was?) more respected than Yahoo or Microsoft?

I have actually seen services that specifically say that yahoo addresses are not allowed but gmail is ok. Gmail obviously isn't less anonymous, so it's not about pretending one address is somehow traceable and another isn't.

Is it a spam filter issue? Is the Yahoo spam filter so bad that folks' emails get lost in there, when they should have landed in the Inbox? Or does it have to do with other people spamming/spoofing from real or pretend Yahoo accounts? It's hard to imagine there is a server-based solution to that problem that gmail can implement but Yahoo Mail cannot, given the open standards for email and history of the spam problem.

There's also a fashion aspect. It seems perfectly respectable for a pro geek to use a gmail address for work, while a yahoo or hotmail (is that Windows Live now?) address somehow smells a little suspect. In particular, hotmail.com is below yahoo.com in respect, and other anonymous inbox services fall even further down.

There's definitely some groupthink at work here... if you have ideas as to what/why/whence it is, please comment!

Thinking ahead. Not thinking ahead.

For some reason I thought about DirecTV and downtime the other day.

As in, I don't think I've had a minute of downtime on my DirecTV/TiVo in over 5 years. More reliable than the electricity where I live, the cable, the DSL, cell service, even the analog landline, which tends to stay up.

One time I stayed at a vacation spot during a huge winter storm and the DirecTV stopped working. I went outside with a broom and knocked a few inches of snow off the dish and it came right back up (the dish still had a decent covering of snow and ice, just not as much).

Satellite thing works pretty well. The client box has never crashed, acted up, or rebooted. And they haven't bricked it, even temporarily, with an update.

On an unrelated and more disappointing note, I've gotten two emails from Microsoft Exchange users at two different companies now, where the email includes a link whose target looks something like http://go-to-exchange-and-get-a-redirect-to/here?url=http://place-i-really-want-to-go

Which fails as soon as you send a link like this to someone whose web browser is outside the firewall and can't connect to the "go-to-exchange-and-get-a-redirect" part. Of course you can copy the destination and edit it yourself, but the idea of a link is it should be clickable.

Surprisingly bad behavior. Even if it's optional or the result of poor Exchange admin, it should be "easy to do the right thing" and apparently it's easy to do something odd instead when configuring outgoing HTML email in Exchange.

Application Model, Part 1

As programmers, we participate in an evolving discourse around the tools, languages, platforms, and APIs that we can use. Sometimes this discourse is purely academic or reflects personal preferences; often, though, the goals are broad: higher quality software, developed more efficiently, with more reliable outcomes than what we do today.

But one critical and useful piece of terminology seems to be missing from the conversation. I propose the term 'Application Model' to describe the essential 'shape' of how a basic program would be arranged, given a particular platform, libraries, tools, and possibly even architecture.

One way to think about this is to imagine a full stack, with a non-trivial Hello World program. The non-trivial Hello World program is a basic program that exercises the core purpose of the stack in the way that the stack intends. So, for example, a simple blogging or e-commerce app might serve as the non-trivial Hello World for a web application programming stack. For a big, high-level graphics library (like WPF), maybe rendering Hello World in a specific font, extruding, rotating to an arbitrary angle, and texture mapping with a streaming video source is the non-trivial Hello World.

Now, remove the platform, libraries, etc., and, assuming it's constructed the way the stack intends, look at the shape of what's left -- of the actual Hello World app. That arrangement or shape of the code is the Application Model.

Another way to think about it in general is to consider the set-wise complements of the API/platform/language stack, in a collection of 'typical applications' running on that stack.

The complement will be a little different for each app, but across the collection, a certain shape should emerge at the boundary.

This boundary, the edge between 'app developer' code and 'stuff that app developer code uses' is different from the formal interface mechanisms (linking/loading; object interfaces) because it is shaped by the practices, patterns, and intents of the platform.

Let's describe a few examples in case it's all too abstract so far:

What is the 'application model' for a C app meant to run in a console? A main function where the execution starts, #includes to access additional platform/API/library capabilities, and a routine to set up signal handling. Compile/link/run. Those characteristics make up the application model. From there on out, it's features and "business logic."

How about a COM Win32 raw (non-MFC) C++ app with a little GUI?  Well, that's a lot more complicated. You need the basic WinMain stuff, the GUI and message loop stuff if you want a real window not just a ShowMessage, then you need the OLE/CoInitialize stuff, and the basic COM dance (CoCreateInstance, QueryInterface) to get the COM object you want. Then you can do myObj -> HelloWorld. Compile/link/run. With the bonus that you don't need to link DLLs (including COM objects).

Win32 via .Net and C#? You're back to void just a main function and a couple of using directives. Compile, don't link, run.

Raw J2EE Hello World? not a ton of code, but you need a lookup service for your other services, you need to do lookups, create bindings, and ask for HelloWorld from whatever service(s) you are demo'ing. Reference other code using import and classpath references. Compile, don't link, but put together descriptors, and subpackages, wrap in an EAR file, deploy, possibly configure app server, run.

J2EE using an IoC container and libraries? access other Java code the same way, but a lot of the other steps drop out. Maybe the packaging work is even automated (and requires less inside anyway), depending on which framework and/or appserver you're using.

The idea is that the 'application model' represents (1) the typical development experience on the platform as well as (2) the core knowledge that a developer will need to have or acquire to be productive.

Different combinations of platform, language, and supporting frameworks create different application models -- so it's a matter of comparing entire stacks and looking at the shape of their photographic negative -- that's the space that will be inhabited by developer work.

Public Domain Reprints Offers a "Transactional Web" Mashup

Some time ago I wrote about how the "programmable web" has really turned out to mean the queryable web, because real transactional APIs have not been readily available.

Public web services (with a few exceptions, mainly around payments and online file storage) are a way of retrieving state. Using web services to execute useful "write" transactions into other systems has turned out to be the business of enterprise SOA, not late-night hacker camps.

The concern, of course, is over users "getting it wrong" when insufficient semantic data is available. Although that's a bogus argument (many industries and services have solid semantic models that are readily accessible from web service descriptions), most folks seem to be waiting for the legendary semantic web to come along.

But "most" isn't "all," and I was glad to see O'Reilly write about Public Domain Reprints, a transactional mashup that submits books to print-on-demand services where you can later order a copy.

Although Public Domain Reprints is just formatting and "preparing" the book -- you still need to order a copy yourself from the service of your choice -- the mere adding of an item into a print-on-demand service's catalog is fairly exciting given how rare these "commit data"-type open web integrations (mashups) are.

At this stage, it doesn't really matter what sort of integration mechanism is used -- it needn't be a web service; it could be emailing a PDF into a dropbox, or simulating a file upload from a web form. Obviously, having a "strong" interface (including some kind federated authentication, so that a mashup can act on my behalf using a restricted token and not my password) will make more services feasible.

I continue to hope that we will see more services that don't just pull down a bunch of data and create a nice report, but actually reach out to services I use and improve my life by doing a little bit of work for me.

Path of Least Resistance == Most Rails Apps Send Your Password In the Clear

The bulk of developers will follow the gradient down hill, even many of the ones who think they're rebels (I'm pretty sure this scene from Life of Brian was about Steve Jobs and the cult of iConform).

Too bad for them, right? Well, no. Too bad for us as a software industry with a reputation for troubled delivery. And too bad for us as citizens and consumers, who suffer from bad software not only in dramatic ways but in having our private data (passwords, SSNs, etc.) sprayed all over.

Elegant and minimalist principles are great; still, if you give most software developers a knife without a sheath, it's only a matter of time before the blood starts flowing from somewhere.

I was surprised when I realized that Rails has no built-in authentication module; more surprised when AWDR "teaches" you to roll your own authentication; and most surprised when I realized that acts_as_authenticated (nice, but intentionally not sophisticated) is de facto standard for auth in Rails.

Hmmm... I wonder how many of those Rails sites I use all the time actually bother to protect my credentials? If the originating or following page isn't kept in SSL, you can't tell by looking for the little "lock icon," but you can tell be having a look at where those login forms are posting to.

I looked at the first two pages (24 apps) on happycodr.com, a site that claims to be "the unambiguous showcase for sites designed with Ruby on Rails," and which is linked directly off of the Apps page at rubyonrails.org.

Eight of these sites are content-only and don't offer logins. Of the remaining 16, fully 12 send their passwords in the clear. Two of the four which don't are AOL properties, required by AOL to use a combination of AOL SNS or OpenID, and SSL.

Of the 14 Rails sites built by non-AOL companies and taking a password, 12 (85%) just send all their users' passwords in the clear.

Some of these organizations think that because their services may not contain sensitive data, it doesn't matter. But most people use a very limited set of passwords (or just one!) for all online sites. When someone keys a password into a site I build, I protect it because I assume that is also their online banking password, or their email password, etc.

To be fair, it is no secret that HTML forms are vulnerable in transmission, and many Rails resources point out that the easiest solution to this is to employ SSL. But clearly the message is falling on deaf ears.

And these developers aren't even trying to cleverly roll-their-own system. They're not hashing the username/password on the client (vulnerable to MITM, replay, impersonation, and other problems but better than nothing); they're not using a nonce and hashing (still broken but better). They may not even realize there's a problem.

To be fair to Rails, it's not the only platform in this fix. I was stunned when Microsoft rolled out a nice set of prebuilt User/Role/Profile/Login components in ASP.Net 2.0, without building any security in. I figured these widgets would get dropped into a lot of sites without a second thought.

It's a little "wider" issue for Rails, though, because, Microsoft has been strong in enterprise development where there are IT departments and policies that may wedge SSL in there before an app makes it to production. Whereas Rails is used more widely in smaller organizations where there is not likely to be anyone with the authority to impose security on a development team.

For ASP.Net, where many developers never look under the hood of the components, and both the client-side script and server-side post-back handling is accessible inside the ASP.Net stack, I would recommend loading the login widget with the best possible pseudo-security using JavaScript crypto for non-SSL scenarios, and then still requiring the developer to actively add a web.config setting to allow running in non-SSL mode. This has the added benefit that, when the app is moved to a production environment for the first time, with a new web.config, the error will pop up again as a reminder right at deployment time.

For Rails, things are a little more complicated. It's going to come down to better educating, and perhaps a little more structured thinking about what I call the "application model" (not platform nor architecture). Which I will write about next time.

A Couple of Tiny Rails SSL Helpers

Here are a couple of helper methods for ensuring forms are set to submit via https in the production environment (but not in dev), and for redirecting back out of SSL afterward. Since SSL can be resource intensive on the server, it's usually good to hop back out into cleartext unless the nature of the application (e.g., financials) warrants encrypting the whole session.

To create a form that uses, SSL, just replace form_tag with form_tag_using_SSL_in_production.

In application_helper.rb:

def production? 
ENV["RAILS_ENV"]=='production'  
end

def form_tag_using_SSL_in_production form_args, &block 
form_args[:protocol], form_args[:only_path] = 'https', false if production?  
form_tag form_args, &block  
end

form_for is the preferred helper ... if it's actually a "form for" a model object, which this one was not. I'll leave the analogous form_for_using_SSL... as an exercise for the reader.

When you're done with the relevant action processing, any content rendered is going to get sent back under the SSL connection. At some point (in my case, immediately) you want to redirect out of SSL. Just use redirect_and_drop_SSL the same way you would use redirect_to.
In application.rb:

def redirect_and_drop_SSL destination 
destination[:protocol], destination[:only_path] = 'http', false if request.ssl?  
redirect_to(destination)  
end

You might be thinking this stuff is too trivial to post about, and anyone who needs to use SSL knows this stuff already. Unfortunately, that's not entirely the case, as I'll write about in my next post.

Whee! AOL Leaps Into the Whirling Saw Blades

You know that cliché about horror movies where the ingénue ends up off by herself in creepy surroundings, and the audience is saying "What are you thinking? Don't go in there!" ?

Well, I'd like to start a MST3K takeoff where the "film" is a tech show off of G4TV or a clip reel from the vlogosphere, and we have, say, a VC, a computer scientist, Ted(!), and a couple o' engineers who have done their time at Initech in the audience.

They could all yell "sure, why not go in there... looks like fun" when someone (like AOL this week) does something colossally foolish, like try and build yet another abstraction platform so everyone can write little apps that run on every cellphone, everywhere, like magic.

AOL says:

"The new open platform will help stimulate innovation by providing developers with ready access to the tools and source code they need to build and distribute applications across all major mobile device platforms and operating systems including BREW, Java, Linux, RIM, Symbian, and Windows Mobile."

Anyone who's ever done anything with cross-platform mobile knows that this is a really hard problem, and not the software kind (where solid R&D might yield results), but the business/social kind, where even the resources of megacorp like AOL are a drop in the ocean.

That's ok, because they have a plan for exactly how it's gonna work. If you're playing along at home, stop reading. Close your eyes and guess the innovative architecture for pulling this rabbit out of the hat. Ok, now you can look:

"The platform will consist of three components:

• an XML-based, next-generation markup language;
• an ultra-lightweight mobile device client;
• and an application server."

I think the secret sauce must be the next generation markup language. See, the next generation markup language will become sentient, take what you write, process it into extraordinarily cogent Powerpoint, email it to all the VPs of all the mobile carriers, device manufacturers, and mobile OS vendors in the world, and then wait on a queue until they all agree to change their businesses so your stuff will work.

I've written lots of times before about the mobile app ecosystem and why, for now anyway, this kind of idea is just a complete non-starter. At least startups fail fast, since they have vaguely limited resources. Then you have players like Yahoo! that find themselves in a deep hole, and say "hmm... this is so big and so deep, we can roll an excavator down here and start the real digging!" They do things like deploy an implementation of J2ME in .net bytecode, so that people can run a mediocre (I'm being polite here, guys) MIDP app on their brand new Blackjack or HTC Touch.

In my new MST3K show, I'll keep the video clips short, so we can watch, laugh, and get on with our lives.

Quick Poke: Look at MS Live Labs' Volta

If you haven't seen Microsoft's Volta project, it's worth a look.

Volta is about two big things:

(1) Blurring the lines between applications tiers to allow declarative or even automated tier-splitting (yes, there are lots of issues with this, but the MS team insists it's aware of the historical legacy of the distributed computing "leaky abstraction" and won't muck things up by pretending it's not an issue).

(2) Automagic cross-compilation between, say, .Net IL and JavaScript.

Whoa, what's that last bit?!

After the initial flurry of press coverage, the team insists that running arbitrary .Net IL on any old JavaScript-capable browser is no big deal, just a convenient demo, don't focus on that part... But, seriously folks, a couple of months on since the project was made available to the general public, the tier splitting is looking less and less interesting, and the cross-compilation is looking more and more interesting.

Once upon a time, crazy people did crazy JavaScript stunts. Like this demo that emulates the hardware of an entire 80s-era game system complete with game, or this similar 6502-based emulator.

Emulating a modern VM like the JVM or CLR is not practical from a performance point of view. But JavaScript runtime as a compilation target is becoming increasingly popular: first we had GWT and Script#, which compiled Java and C# respectively to JS. But because they only compiled your code supplied minimal bits of the respective class libraries (Java API/.Net BCL), their use was restricted.

So here's a wicked end run around that: take anything that compiles to IL -- or any IL binary! -- and run it in a browser. According to the Volta team, it has been done in such a way that all of the object semantics from the original language are preserved. Unless they also port the Win32 API to JavaScript, it's not like we're going to see WinForms apps running on Safari. But this is being positioned as a fallback for, say, places where Silverlight isn't installed.

Go here and look at the demos. The perf is awful -- but that's not the point of the demos. So far as I can tell, this is the first commercial implementation of what we all kind of knew might be coming.

Using Google 'My Location' to Troubleshoot AT&T's Broken Network

The AT&T Wireless coverage in the San Francisco area is poor. It's not that the "more bars" commercials or the beautiful coverage map aren't technically accurate -- pushing enough buttons on a handset sooner or later results in bars on the display. Having calls go through and not get dropped, or having the data service behave, is another story entirely.

But this post isn't a rant about AT&T. None of the wireless carriers are perfect, and making changes in an asset-and-operations-intensive business is hard to do. (Think about all the legacy airlines biting the dust -- there are a lot of parallels in terms of the economics.)

This post is about looking at Google mobile maps' "my location" feature, which shows the location of your current cell tower, to take a few guesses at what might be going on.

The first thing I noticed when playing with "my location" is that, on freeways, there seems to be  consistent and stable handoff between towers. Not every spot on the freeway can see a tower, which might explain the fact that within 5 miles of SF there are major freeways with dark spots, where calls predictably drop. But in the covered areas, all is orderly and the service (both voice and HSDPA) is good.

Then the trouble begins: in some suburban and downtown (SF financial district) areas, it seemed as though the phone flits between towers willy-nilly, even if I'm standing still. The results are predictable: service is mediocre, and calls can drop even in downtown. The data service becomes intermittent deep the "3G coverage area" as the phone becomes confused whether 3G is available, whether to fallback to EDGE, or even GPRS. By the time the phone has done the appropriate renegotiation, it gets new information and changes its mind.

Cell phone networks are famous for sporting sophisticated hysteresis mechanisms to manage tower handoffs. Too much handoff, and the switching overhead and side effects dominate... too little handoff and, well, you drop calls. When a handset isn't moving, there is little reason (aside from site congestion) to do a handoff. But how do you know the handset isn't moving? It's a Catch-22, because without external input, the network can only guess by measuring changes in reception from nearby towers. If reception fluctuates, the client and network need to figure out if the phone is really moving, which way, and how fast.

The result is so bad in some cases, that it's almost worth having a button marked "I'm in my office" or "I'm at home" so that the user can explicitly tell the network what's happening. At my house, I'm hopping between three towers (one on the other side of a small mountain, with enough S/N to grab my phone, but not enough to even initiate a call) with the result that the phone is more or less a brick. There's no consistent data or voice service (20 minutes from downtown SF!), so the only possible change is "better."

Here's the interesting part: a lot of the AT&T network problems don't seem to occur on Sprint or Verizon (I've been on all the networks, and was an "early adopter" of digital in the U.S. back when Sprint didn't have any coverage at all in large places like, ah, Chicago.) I do not believe the difference is because Sprint or Verizon have more towers, or superior engineers. I have a feeling this is due to (1) differences between GSM and CDMA (CDMA seems to be technically superior, but business factors make GSM dominant) and (2) the fact that AT&T is a historical rollup of lots of other networks (e.g., a former TDMA or AMPS tower setup may or may not be the ideal place for a GSM tower and I doubt they were all relocated) and supports too many disparate protocols (GPRS, EDGE, HSDPA, UMTS).

Every time the phone changes its mind between HSDPA, EDGE and GPRS, it seems to have to renegotiate its presence on the network. Perhaps there's a software fault in there too, because once the phone starts changing its mind, your network connectivity is shot. Sometimes until you reboot the phone. Whereas with Sprint and Verizon, it was EV-DO all the way. Some mechanism ensured the phone never even tried to fallback, and the result was an overall smoother ride.

I actually use the Google maps data sometimes to figure out whether I'll have coverage, or whether I need to reboot my phone. Probably not what was intended, but a lot more useful than the little bars.

Ruby Hack: In-Memory Files with Metadata and Better .open() Support

PDF::Writer is probably the closest thing there is to "the" library for generating PDF docs in Ruby. But there are some other nice, tiny libraries for generating a quick PDF doc, like Ruby FPDF.

The one thing I needed recently from FPDF that it didn't offer was the ability to add an image from a in-memory data blob (the FPDF Image method reads from a file). Making FPDF read a memory stream was a fun bit of Ruby meta-style hackery, which I offer here for streaming images into your PDFs and also as a neat example of how you can usefully and easily change a module's behavior without touching its source code.

The basic plan:

1. Since FPDF wants to open a file, and I have a String, StringIO is my friend, since it's very file like.
2. Since StringIO doesn't have file metadata, like a filename, and FPDF looks at the filename to decide how to parse the data, I'm gonna need to make StringIO seem like it has metadata.
3. Since FPDF uses only the raw Kernel.open, it's looking so a file path. I need to make it a little more cosmopolitan by giving it the Kernel.open in open-uri, which among other things calls open on openable objects. Like StringIO.
4. Lastly, since StringIO.open behaves like "new," and I want to pass an object to open that already exists, I need to change the way open behaves.

First, let's make FPDF use the enhanced 'open':

FPDF.module_eval { require 'open-uri' }

Now, we'll define a function that takes a String of binary data and a pretend filename, and produces a hacked StringIO object (description of what's going in is in the comments):

def in_memory_file data, filename 
  #load up some data
  file = StringIO.new(data)

  #tell the class that it knows about a "name" property,
  #and assign the filename to it
  file.class.class_eval { attr_accessor :name } 
  file.name = filename 

  #FPDF uses the rindex and [] funtions on the "filename",
  #so we'll make our in-memory file object act like a filename
  #with respect to these functions:
  def file.rindex arg 
    name.rindex arg 
  end 

  #this same pattern could be used to add other metadata
  #to the file (e.g., creation time)
  def file.[] arg 
    name[arg] 
  end  

  #change open so that it follows the formal behavior
  #of the original (call a block with data, return
  #the file-like object, etc.) but alter it so that
  #it doesn't create a new instance and can be
  #called multiple times (rewind)
  def file.open(*mode, &block) 
    self.rewind 
    block.call(self) if block 
    return self 
  end 

  return file 
end

In this case, I had the FPDF source so I could see exactly what I needed to do. With Ruby, I'm likely to have the source pretty much all the time (unless the classes are pre-compiled to target another VM, e.g. CLR). But an interesting next step is to add hooks/instrumentation to a "black box" library, and use that output to try and make adjustments to the behavior of the library.

Blackjack / WinMo 6 Upgrade: More Baby Steps Toward Smartphone-as-Computer

I upgraded my Samsung Blackjack to Windows Mobile 6 via the new official ROM, and the upgrade pointed out a few things indicative of where the mobile industry is going.

I was actually surprised that Samsung released the upgrade -- it had been promised for the better part of a year without showing up -- rather than just force customers to buy the new Blackjack II. Eventually I realized why they had gone through with it, and why doing a full-version upgrade (not an update) of the OS on your smartphone will become commonplace: This wasn't done for me. It was done for enterprise customers who bought (or committed to buy, or were thinking of buying) hundreds or thousands of the devices.

Consumers -- especially the kind who buy smartphones -- are interested in getting a new phone every year or two. The whole free-with-contract-extension system helps that keep that flowing.

Businesses, however, buy their handsets differently (often via a more complex deal) and account for the costs of maintaining a "fleet" of mobile devices (management, email, VPN support, etc.) differently as well. For a business, cycling the devices every year or two could well be prohibitive because of the overhead, not just the equipment cost itself.

I've never been tasked with executing a smartphone rollout for a large business. But if I were, I would probably want to create a system where the average device had a lifespan from 2 to 4 years: 2 years for the high-level execs and the techies who need more power, up to 4 years for employees who travel but are not particularly sophisticated or needy when it comes to mobile email. Beyond 4 years or so, the hardware gets so outdated it's not worth the legacy hassle; it's time to buy something new.

So a regimen of solid, easy-to-install full-version OS upgrades becomes a sine qua non for device makers (Samsung) and software companies (Microsoft) who want to sell into the 3-4 year lifespan device market. RIM figured this out a long time ago, and have pushed the envelope when it comes to keeping old models (e.g., Blackberry 7250) alive for years, offering not just OS-level upgrades, but software radio protocol upgrades: an early-model 1xRTT 7250 can become a 3G EV-DO rocket with just a firmware update.

Windows Update ... on Mobile 6

Now, with Windows Mobile 6, Microsoft can also deploy updates to our phones via Windows Update. So they can be more proactive than the hardware maker or the carrier when they need to fix a serious bug or security hole, or release a new feature.

Although other devices have provided updates before, the Windows Mobile Update feature is breaking new ground because the OS runs on so many devices made by so many companies ... and on nearly every carrier on earth. The updates had better not brick too many phones; on the other hand this could be a strategic advantage in the smartphone wars.

Other Stuff

There's lots more to say about WinMo 6 and device evolution ... from Internet Sharing to more Bluetooth support to the GPS "intermediate driver" ... and those were covered well when the OS was initially release.

In Which I Try to Save Someone from a Rails/MySQL BLOB Debugging Hassle

I was recently debugging an app and spent a bunch of time on a MySQL error symptom. I'd like to save the next developer from the same searches, since there are lots of documented MySQL connection issues with Rails, and few (any?) posts mentioning the following particular gotcha:

If you're getting the Lost connection to MySQL error, or its cousin, MySQL server has gone away (depending on whether you're using the Ruby driver or the Ruby-C driver), and you happen to be working with BLOBs (images, file uploads, streaming docs from a DB, etc.), there is a reasonable chance that the problem is you're sending too large a "packet" to MySQL. The "server has gone away" doc mentions this and refers to more detail at the "packet too large" error detail page.

In the current MySQL version, the default max packet size is 1 MB and by packet, they don't mean the TCP packet you might be using under your connection, they mean "a single SQL statement sent to the MySQL server, a single row that is sent to the client, or a binary log event sent from a master replication server to a slave." So it's actually quite easy to hit the default limit if you're working with BLOBs.

The limit can be raised in various ways (command-line, service config, etc.) and the doc points out that raising the limit should not in itself incur a performance penalty, as the additional memory is allocated only when needed.

Before cranking the max_allowed_packet setting up and calling it a night, though, it's also worth a quick sanity check as to whether you really mean to be reading and writing BLOBs this big. If you're storing videos or RAW data from a DSLR, then maybe yes.

In my case, I was using a heuristic for converting and compressing smallish photos and logos. I didn't intend to create a larger image than what the user was uploading (the largest test item was around 250kb), so the 1 MB default limit should have been fine. On closer inspection, a bug in my logic flow was incorrectly deciding to convert some 250K JPEGs into 4MB PNGs. Doh!

What Does Pownce Have to Do With Bogus Internet Movie Rentals?

Maybe my point about Pownce got lost in a larger, broader, and more academic argument. That's fine; I'm glad to see Ted Dziuba (yes, uncov) pretty much made the same point about Pownce. He has a larger audience than I do. My larger point, of course, is that we're not always the hiking-in-redwoods, biking-to-work, inventing-computers-in-the-garage, tossing-whole-industry-paradigms-out-the-window-with-our-wicked-code folks that our narrative says we are. We're as ridiculous, self-absorbed, and self-referential as the folks we take swipes at on our blogs.

What does this have to do with Internet movie rentals? Well, Internet movies -- and VOD in general -- is one of those always-coming-never-here technologies. Even if does arrive, it'll be 10-15+ years into the (U.S.) broadband-penetration era, meaning it will kind of dribble out the exhaust pipe of innovation as an incremental use case long after its revolutionary potential has disappeared. But Netflix (with LG) and Apple are back to the baccarat table for another round of tech's most pleasant losing gamble.

While Apple and Netflix have made little gains, big ol' Safeway is now selling a wad of recent hit films at 2-for-$6. On DVD. With a non-revocable license to watch in perpetuity, carry around, re-encode (if you're "careful" how you do it), and pass on to one's heirs for the next 1000 years. Or you can just rent the films for $0.99 - $1.49. The studios aren't going to hassle Safeway about "$3=watch forever" pricing (and heck maybe Safeway is paying $2 for those DVDs), because it's a retail supermarket. They understand it and it makes sense in their world. The same reason Joe Teenager can sign up with a record club and, if he works his deals optimally, pay a better per-track price than Apple does (maybe 30% of retail).

The house will have the advantage as long as tech keeps thinking like a player (and believing its own mythos) and not like the casino (the other that it tells stories about). Who's the "house" ? ... well, studios, producers, broadcasters, cable networks, telcos, CE makers (guess what, they don't think of 'tech innovation' the way software folks do), even government ... and that's just a start.

I made this point before. Now I'm gonna take it a step further, and talk about one way to move forward.

If you're an innovator in one of these areas, and you need to get a foot in the door, take on another perspective in addition to your own. Try and find a place to live in the other industry's model of the world (their value chain/web/network). I don't mean philosophically, I mean literally. Think of it as hacking their worldview, if it makes you feel better.

Here are some examples:

• When ecast needed legal clearance to get running with a digital library of songs in 1999, "MP3" and "Internet download" were conversation killers with record labels. ecast needed to be a jukebox, don't worry about the gears and levers inside, nothing to see here, move along. There were some hiccups in licensing until they shifted to the latter approach. Record labels know what a jukebox is and how they make money. Problem solved.
• Mediabolic ended up taking a different course, but at one point when the firm was working on  VOD with a set-top box, they looked into subleasing excess digital broadcast spectrum time for moving their data. What does that mean? It meant they would have been in some real sense a broadcaster using TV spectrum to reach customers. Which is a whole different conversation with content owners from the "renegade startup conversation."
• When Skip Interaction focused on travel data management, and mobile transactions, I advocated becoming a travel agency. While Skip would not likely have made much (any?) money on bookings, it changes the conversation with airlines, travel agencies, and companies whose employees travel a lot. Skip would have appeared as a "known" entity in the industry. All of a sudden, instead of requiring custom arrangements to do anything, Skip would have had a huge pile of boilerplate data and access to work from.

These examples are just from my personal experience -- I'm certain that this strategy is not at all uncommon. But we don't hear about it a lot, because it doesn't fit the plotline we want.

Looking For: a Reverse-Search Web Service (aka Semantic Analysis the Easy Way)

For one of my projects, I'm looking for a web service that does search "backwards" to reveal page semantics. If anyone can point me in the right direction, I'm all ears!

What do I mean by backwards? A normal web search service (Google/Yahoo/MSLive API) takes a set of search terms and other conditions and returns the web pages that best match.

A reverse search takes a URL and returns the search terms for which this page scores well. I call this Semantic Analysis the Easy Way because, strictly speaking, it doesn't require actually understanding the content of the page -- yet you can get semantic data out. Of course the better your content analysis engine can understand the content, the better your search engine will work, so the hard problem figures in a little bit too.

The big search engines certainly have the data -- I'm sure it plays a big role in ad-placement mechanisms like AdSense. Just as the search engines expose their search APIs, confident you can't (or won't be allowed to) steal their search results and pretend they're your own indefinitely, they could theoretically expose the reverse search data too. But sadly, I haven't found one yet that does.

I did the requisite scan of the major players, looking through programmableweb, etc. No dice. (Although programmable has a link to an interesting "hard way" semantics service that did a nice job of analyzing the text I threw at it from real pages.) The closest I've come so far is the del.icio.us URL API, which will tell you the top tags associated with a specific URL -- valuable data indeed, but not the same thing.

Little help? Anyone?

Thanks!

Post-structuralism for Dollars in Silicon Valley

On the eve of one of our grand theatrical spectacles, featuring one of our industry's undisputed showmen as emcee, let's pause for a minute and imagine that we've got our mental map of the tech world all wrong. We've been drawing lines not just in the wrong places, but where the model isn't about lines at all. As a result, we're misunderstanding ourselves and a lot of other stuff. And, since the economy doesn't care about our degree of self-awareness, it means we all -- entrepreneurs, investors, engineers, designers, marketers, journalists -- are leaving big money on the table.

I'm hoping that last sentence might catch someone's attention, especially that of the VCs who have a lot to gain or to lose based on their understandings or misunderstandings.

Where do we draw lines by mistake? We live in a structuralist mythology that we make up because it's easy and it's a shared set of beliefs that we can use to argue (in circles sometimes):

• The "new" versus the "old" is of course the big daddy duality
• How about Client / server, Startup / enterprise
• Geek / management, Engineering / marketing
• East Coast / West Coast, Stanford / Berkeley
• Open source / proprietary
• Content / code
• NorCal / SoCal, Hollywood / Silicon Valley
• Entertaining / Functional

These just scratch the surface -- you can probably think of fifty pairs of your own now, along with examples of how they are convenient, self-reinforcing, and yet ultimately invalid.

They are a cliche, but they own the conversation. I want to talk about the last few pairs: think of the narratives we make and live inside, as regards Northern California vs. Southern California, Hollywood vs. Silicon Valley, and Entertaining vs. Functional.

Now look at Steve Jobs, an entertainer who could get his fans to poke out their own eyes and get in line for an iPatch. Apple is theater, the Apple store a set, a Disney experience for people who feel superior making snarky comments about Disney. Do you really think computer geniuses work at the Genius Bar? "Space may be the final frontier / But it's made in a Hollywood basement" -- and in this case the fans have already reserved tickets to the sequel.

But, wait! I'm not bashing Apple or Steve -- that would be buying into the very dualism I'm pointing up here...

Now consider also: we have another award show, the Crunchies, coming up. Before you laugh, remember this event isn't intended as a joke. And donating money to charity doesn't make you serious, just generous.

We have our gossip writers, our A-lists and even our stunts that get a third of the Valley looking self-consciously at the floor, embarrassed at having to wonder whether Carly Fiorina or Leah Culver has set feminism back further, a third watching our great-standup-with-the-dirty-mouth pretending they all get the joke, and another third just saying the heck with it and jumping on.

Ya know, maybe we're not so far away from our SoCal cousins as we think. And just to keep it clear: I'm not saying there's a middle position, a compromise, where we're all gonna end up. I'm saying that the duality itself is a fiction. So there is no middle. Until we get this, we're going to continue to have a heck of a time getting some things right. Or making money with them!

Like what? Like things that aren't quite code and aren't quite content. We never got SVG right, because it didn't fit into the categories we thought made up the world.

Is a Flash media code or content? When my mom gets an e-card, she thinks it's content, not an app.

We don't have a cross-platform vector graphics standard in 2008 (!) ... because practical vector graphics in the real world means some logic as well as geometry.

Why can music acts sell (via the carriers) all manner of wallpapers and ringtones for cold hard cash to technically unsophisticated folks, and we still can't get the average person to install or run a mobile app on their phone for free? We're playing that game wrong and we can't see it.

We have every manner of "media center" -- hardware, software, open, closed, expensive, free. Even Microsoft has been in on the act for five years now. And I still don't know any regular folks at all who use any of them.

We have tons of great ideas on the shelves because we can't understand 'the other' well enough to make the deals we need on content licensing. (They don't understand us either.)

Three quarters of "Web 2.0" is not about any kind of functionality at all; it's not even about interaction design. It's about the glossiest thinnest veneer of user experience. It's about buttons and realistic smoke ... the kind of thing that some folks refer to as "production values." Moreover, three quarters of it is free and ad-supported. Not unlike most radio and television. Unlike radio and television, its reach is minimal. Ask your aunt in Duluth what del.icio.us is. Before we spend another session making fun of the studios for not realizing where their business really is, we might think about what an adjustment in ad rates will do to our own.

One last time for the folks in the cheap seats: it's not about right or wrong, steak or sizzle. We're just not where we like to think we are. We're somewhere else. To get a better idea where we are, we need to do some demolition on our usual Monday morning narratives. It's uncomfortable. The good news is, there's a ton of opportunity. And eventually it feels good to realize we've outgrown the notion that the world is flat.

Now go to Macworld and enjoy the, uh, show.

Rails Hack: Combining Acts_As_Auth with Long-Lasting Anonymous Sessions

In one of my Rails apps, I have two kinds of users: anonymous visitors, who do not have to sign up at all to use the app's functionality, and "vendor" users, who need to sign up and create some content in order to get value from the site.

The requirement for sessions was simple:

Anonymous visitors should have a session that persists for a long time (e.g., months) via a cookie. Since there is no information at all collected from these visitors, there is no reason to clear them out of the browser, and they would appreciate the convenience of being able to pick up where they left off when they come back.

Vendor users, however, have more data at stake, and so should have the typical login/password/remember-me?/logout pattern, allowing them to decide if their sign-in should persist, and letting them explicitly logout to kill their authentication.

I dropped in acts_as_authenticated, and it does a fine job with the second case (vendors). For the first case, I then added sliding_sessions, which is elegant and has now been added to the relevant wiki page.

The problem is: once session cookie durations are extended in general (at the application or controller level), aaa's authenticated users will be "remembered" upon return to the site, whether they want to be or not -- because their user info is already in their session (i.e., the "login from cookie" is not necessary and is never called). Timing out a session/auth on the server side is a good practice, but doesn't solve this issue -- if the user closes the browser and walks away, someone might come up right behind them and come back to the site, where the session/auth has not timed out.

Here's my solution. It seemed quick and easy; if anyone has criticism or improvements, let me know:

There are three cases -- anonymous user; vendor asking to be "remembered"; and vendor not asking to be "remembered" -- of which the first two worked fine. So I add a cookie upon login, set to timeout at the end of the current session, which will help me separate out the third case.

The details:

AAA adds a couple of "account controller" actions to your app. In login, I added the new cookie (see line 11). Then, in application.rb, I added this method as a before_filter. It checks to see whether you are an authenticated user (i.e., not an anonymous visitor). If so, it resets the session unless it sees the vendor cookie (meaning you've signed in in this browser session and so should not be cleared out yet regardless of "remember me" choice) or the auth_token cookie (created by aaa, and indicating the the "remember me" persistent login choice).

So if you authenticate and do not choose "remember me" then you have no auth_token cookie and, when you close the browser, you lose your vendor cookie. When you come back to the site, your session (and user info) are cleared.

$30 and $60 at Fry's

I guess it's a story of manufacturing in volume: at Fry's, I recently bought my wife this "Periscope Hardcover Booklight"

It's a piece of (fake?) leather and two LEDs, made in China, and priced (by Fry's) at $30. Less than ten feet away was MicroATX motherboard (not this model, but similar), with socket 775 support for Core 2 Quad procs and all the bells and whistles. For $60. Likely made in China.

Fallacious reasoning to be sure, but still a big delta in what you can get for another $30.

Who Considers These Folks the Elite Again?

I just read David Megginson's "Programming languages of distinction" in the context of Jeff Atwood's great post on The Shiny. The Scala cartoon cracked me up. But in real life, what truly "elite" programmers are going to move from Megginson's "Step 1" (too many riff-raff using their current language) to "Step 2" (look for a new, little-known language with fewer annoyances)?

At the end of the day, all of these trends/fads/fashions fall somewhere between two well known poles:

• Assembly language, as specified for typical mass-produced hardware (x86, 680x0, ARM, etc.)
• Lisp, and in particular Scheme

If you start with Assembly as your deck of cards (stacks and registers and jumps), and start shuffling and doing slight-of-hand tricks, you can get C, C++, Java, et al. If you start with a Scheme deck (lambda expressions), you can shuffle to get Haskell and Erlang.

If you throw all the cards on the floor and mix-and-match from both decks, you get your Ruby and your Scala and your "JavaScript 2/ES4" (well, maybe).

The reasons for religiously sticking to one pole or the other are not compelling for coding at the "application level." That's a fairly trivial statement. Once we specify the sort of typical end-user or business application that makes up the bulk of development, we evade the issue of whether a primitive imperative language (C/Assembler, close to the metal as far as specifying instructions) or a functional one (Haskell/Erlang/etc., potentially easier to parallelize on multi-core hardware) will perform better.

In the application world, we have hybrids like C#/Java/Ruby/ECMAScript. Sure, Scala. At that point, all sorts of other considerations come into play, from design to cost to maintainability to security, which have more bearing than whether a curly brace is a bad thing or /: is evil as an operator.

And guess what? With ANTLR and some open source code you can go make your own half-a-cup of Assembler-half-a-cup-of-Scheme language quite easily. You can use Kanji for the keywords, Roman numerals, a keyboard with a special key that generates Unicode chars above 0xFFFF for declarations, and maybe change the text direction to indicate which way functions are evaluated. What a niche! You'll be the leading authority! You can write a book. You can entice mediocre engineers and terrorize bad ones. You can entice mediocre engineering managers and terrorize bad ones. Or you can work on software that solves real problems for real people.

It's About the API

Ok, at the risk of being almost timely, Zed's Rails Rant has inspired me to finish a point I've been wanting to make for a while now: Platforms (and languages too) are about the developer experience (API). Not the community, although that can help. Not the implementation -- that can be changed, redone, ported, etc.

What is nice about Ruby and Rails is that they're "fun" or "pleasant." In general, when programmers say this, what they mean is

1. Working with the language / API does not go against the grain of how their mind works. Since programmers have varied ways of thinking, simply avoiding an "against-the-grain" experience across a set of non-trivial cases is a real accomplishment. Ruby does this in spades, Rails almost as much once you get used to a particular "Rails way" of doing something.
2. It's at the right level of abstraction most of the time, when used in the kind of work for which it was intended. RoR is fabulous at this. It does the "super-high-level-language" thing when it reads your data schema, does :has_and_belongs_to_many, and makes a very sensible attempt to match URLs, actions, and views by default. It goes low-level by working with clean, trivial HTML, which is easy to write and debug by hand, while ignoring the temptation to autogen all sorts of decorated HTML/CSS/JS.
3. When you need to read the core sources or other people's code, it's not a terrifically painful experience that leaves you sitting there saying,"You've got to be kidding" over and over again.

Now, apart perhaps from a bit of #3, these are descriptions of a programmer's experience interacting with the language/platform/framework/tools.

They are API issues, not implementation issues. If the implementation is bad enough, the API experience will be overshadowed. But the developer experience sells the platform.

So... what about the MRI, and the Rails core team, and Mongrel and horrific perf issues, and all that?

The current state of affairs needs to be seen as a reference implementation. It shows off the specs under discussion, proves they can be implemented, provides all the right output to the definitive test suite, etc. Sun releases new Java APIs with a reference implementation. Some of those RIs have been good enough to use in production, some haven't. Sun hoped that third parties would create commercial implementations -- in some cases, this happened. Where an API was popular and important, a "real" version -- or several -- became available, based on market demand.

There are already a bunch of alternative implementations of Ruby either finished, or in the works. Rails has yet to see an alternative implementation, so far as I know, but it seems plausible that both open source (e.g., modify or redo the Rails code to provide multithreading support, etc. and run on top of JRuby) and proprietary (Rails as a set of ASP.Net assemblies) implementations will emerge.

When Java offered a great developer experience compared to the alternatives, it picked up adherents even when the adequate performance (heck, even existence) of various APIs was shaky. When .Net offered a great developer experience, it brought more shops into the Microsoft fold than 20 years of, shall we say, other approaches to gaining market share.

Some failed platforms suffered because of business issues (e.g., Delphi and C++ Builder), but most fail just because they don't do #1-3 above as well as the next guy on the block. My point is that Zed's rant is a bit beside the point, even if it's all true. Ruby and Rails have already accomplished the end of influencing industry thought and behavior in a broad kind of way. We'll need to be a little more patient (or make some serious contributions) before we get the followup implementations.

HD or Blu? Format Wars are Over, Winner is x264

Holiday sale $99 refurb HDDVD players couldn't win the war for the HD camp, and the PS3 still shows no sign of ending it in BluRay's favor.

Meantime, the de facto standard online for HD disc content, legit or not, is x264.

After DivX/XviD/MPEG-4 replaced MPEG-2 as a generally practical format for video content from DVDs, it took some time before gadget makers released decent players that could load up a DivX stream off of any CD-R or DVD-R and play it (reliably).

My money says we'll have a much shorter wait this time. In fact, in the end-of-year spirit of goofy tech prognostication, I'll wager that by Christmas of next year

1. The "official" format war between BluRay and HD-DVD will still not be meaningfully resolved
   
2. But 200 bucks will get you a DVD player that can load up an x264 stream and play it back, upsampled and/or at HD resolution
3. That same player (or maybe the $300 version) will either have a USB port so you can plug in your 1TB external enclosure and play HD movies that you've, um, obtained, without using a disk at all, or else it'll have a network port.
4. Game over.
   

Loving RoR with NetBeans 6

I've switched to NetBeans 6 for Ruby coding, and I'll never go back to those bogus text editors.

Some time ago, I wrote about IDEs for Ruby. I'm not sure I did a great job pointing out out that for Ruby (like JavaScript and other dynamic languages) some of the basic features we'd like from an IDE (code completion, refactoring) are non-trivial problems. The Sapphire in Steel blog has some great articles on working those problems.

The milestones for NetBeans were promising; the other free contender, the Eclipse bits formerly known as RadRails, are now part of the "Aptana IDE." Aptana is a reasonable tool for JavaScript, but last time I checked, Aptana was in "1.0" and the Ruby support still crashed and failed in interesting ways. The fine print points out that Ruby/Aptana/RadRails isn't 1.0. Bummer.

I'm very excited about this NetBeans release. For a detailed discussion of features, check out Roman Strobl's review (part 1 and part 2). Once the app starts and gets warmed up (which admittedly takes longer than booting the OS on my Windows server box), it does a pretty good job of the hard stuff, like refactoring. And it's rock-solid on the rest, like interactive debugging.

Unfortunately, Rails 2.0 became official at about the same time NetBeans 6 did. And Rails 2 changes a few things, like filenames/extensions on erb templates, that NetBeans doesn't know about. But it's easy to work around those things, and I'm sure an update will be forthcoming.

Amazon SimpleDB isn't Astoria... it Could Be, but Does it Need to Be?

A while back I wrote about Microsoft's Astoria REST-based relational data store in the cloud (or in your data center, if you want it there).

With Amazon's SimpleDB, we're a step closer to making this vision a reality. Now we're almost on track for competition (and sooner-than-later commoditization) of the new world where you don't even need MySQL to store your stuff.

Why almost? Because SimpleDB is not a full RDBMS, but is looks more like a flavor of triple store. Now, a typical (i.e., SQL-style) RDBMS can be built on top of a triple-store fairly easily. So we could will see a SQL processor, JDBC drivers, and the like, from the community pretty soon.

Another way to look at that "top layer" is to take a REST API like those used by Astoria or ActiveResource [PDF link] and simply implement that. Not as expressive as hardcore SQL, but easier, and probably enough for many applications.

What I don't see -- in the long run anyway -- is applications developing against thin wrappers specific to the Amazon triple store service itself. There's nothing fundamentally flawed in doing so ... it's just that, for a variety of reasons, data storage has evolved very slowly. The relational model is going on 40 years old, but still reigns supreme in terms of popularity, even if it has conceptual or technical flaws when put to work in today's applications.

Given the brilliant data storage alternatives that have fallen flat time and again, I doubt Amazon SimpleDB will change the way people talk about storing structured data. So SimpleDB doesn't need to be SQL but it will probably need to at least be RESTful.

How Bay Area Subprime Mortgages Relate to High-Tech Startups

Earlier this week, Tom Campbell, Dean of Berkeley's Haas School of Business, gave a long interview about "the mortgage crisis" on KCBS' in-depth segment [MP3]. At one point, the conversation turns to the general paucity of housing in the Bay Area (relative to demand, anyway), and the notion that landlords may be beneficiaries as folks suddenly discover that (1) they can't afford $975,000 for that 3-bedroom house and (2) it isn't worth $975,000 anymore either.

But aren't these b-school types the ones who are always telling us to "think outside the box"? The best Tom can come up with is a kind of housing trust or co-op, that lets you own, say, 25% of an expensive house, while you still get to live in it (an investor group owns the rest).

That's a really cute way of pumping up housing prices and speculation even further, by letting investors who would never do the real-estate transaction themselves pool their investments and then spread them across a bunch of properties (doesn't this sound a little like the mortgage problem we just saw?), while leaning on loan guarantees to cover their downside.

The cost of housing and difficult commutes are a big brake on the tech industry; they consistently rank at or near the top of Silicon Valley business leaders' concerns about the growth of their companies and the success of the region.

In the positions I've held over the last 4 years or so, I've been responsible for hiring engineers. Not just punch-the-clock engineers, but wicked smart, willing-to-build-it-from-scratch-but-wise-enough-not-to startup-minded engineers. But it is brutally hard. Even with robust salaries, it is difficult to get applicants in the door, let alone hired. And, no, it's not due to a shortage of U.S. geeks. Housing costs and impractical commutes seem to be the primary limit on the realistic pool of applicants.

Over time, if we don't do something about it, we'll only have the old-guard geeks (who bought homes a long time ago or live in rent-controlled places in SF or Berkeley) and immediate college grads (who are up for an adventure and happy to have roommates).

The problem is, when your only tool is sprawl, and you run out of farmland to sprawl on, you either throw in the towel (Silicon Valley) or you sprawl farther away (American Canyon, Fairfield, Tracy, even Monterey). When your only tool for traffic congestion is building more lanes, that's what you try to do, even though latent demand means that more freeway creates more traffic in the long run, not less.

If I'm so clever, what am I suggesting? We need to use a different tool, namely smart growth.

Among other things, we need significantly higher density, infill development, and more mixed-use development (residential, commercial, and light-industrial uses in the same or nearby buildings). This isn't a speculative proposal: where it has happened locally, it has been popular. Look at Santana Row, San Mateo, downtown San Rafael, even South of Market/Mission Bay SF (although arguably the planning there didn't go nearly far enough, leaving too few housing units to make any dent in affordability).

Luckily, the Bay Area does not share America's poorly grounded prejudice against living in towns. So communities like the ones I propose, on the peninsula and in the East Bay (Hayward, anyone?) would likely be embraced. And infill opportunities abound: Every time you see a big-box store in the Bay Area, whether you love them or hate them, imagine a few stories of residences on top and additional businesses lining the enormous "blank" sides of the store at street level.

Will this generate so much housing that no one will be tempted to take on debt at crazy terms they can't afford? Of course not. But if it can help keep that housing-cost-to-salary ratio from growing quite so fast for the folks we want (and need!) to work with, while bringing the ancillary benefits of denser communities, it seems like a no-brainer.

No Uber-Soft Launches and No Stealth Mode

Uber-Soft Launches and Stealth Mode are two common practices that are usually big red flags of impending trouble.

To be clear, an Uber-Soft Launch is not a classic, small-scale launch where you release a decent version (maybe beta) of your product, but don't blast every PR trumpet you can find until you get a first round of feedback and some perf stats. There's nothing wrong with that; it borders on a best practice.

On the contrary, an Uber-Soft Launch is when the CEO or entrepreneur starts hedging about whether this launch is really the product or really the big launch he's been working toward and talking up. "We're going to just try this out and see what happens ... "

That statement is legit if it's intended as faux-modest understatement from a guy (or gal) who's clearly going for broke to make the product succeed. But if the firm or leader is really this wishy-washy and diffident about the launch, forget it. It's game over.

But then, in that case it doesn't matter because what the entrepreneur is really saying is that he doesn't expect to succeed so he's simply covering himself so he doesn't look silly after the failure. And that CYA attitude is one of the key things that indicates impending failure. It's a symptom of lack of conviction, a fear of failure that drives systematic bad decision-making.

Stealth Mode is a little less black-and-white, as there are a few cases where it may pay off. A few. Meaning not many. Luckily, Web 2.0 seems to involve far less "stealth mode" than Web 1.0 so it's less of a problem.

When might stealth mode be useful?

(1) A company has a specific physical or algorithmic invention (no, not Amazon one-click), and intends to patent it and to defend the patent vigorously (= has the massive cash to do so). Company wants to make sure it's documented and filed before anyone else files. If this is you, then you'd better be working toward the patent filing as fast as you can, no excuses. And get a good lawyer. If you're not ready and planning to defend, then stealth mode doesn't matter. Someone else can implement your technique if they want, and even patent it. You'll win or lose on execution and customer acquisition.

(2) A company's value is going to be based on a "network effect" play rather than a "hard-to-duplicate" play, and already has big the PR for the launch you lined up. In this case, since you know you're not "hard to duplicate," you don't want to spill the beans until you can fire off the giant PR cannons, at which point, you'll either grab a big enough chunk of network effect to sustain you, or else you'll drift. An example of this is Ning, which was in stealth mode for a long time. Marc Andreessen's celebrity and connections were the big PR blast, timed to match Ning's actual launch. But what if you're not Marc Andreessen or Kevin Rose and you're not going to make any headlines with your launch? Then you're not going to get a big "pop" when you come out of stealth mode, so really you're just:

Afraid of someone stealing your idea

But that's not a good reason for stealth. There are very few new ideas. If an entrepreneur thinks he or she has one, it almost certainly indicates insufficient research to find the people with the same or very similar idea before (and now), and consequently ignorance of why they failed (or might fail).

If this is you, get over yourself. Make it part of your "leadership agenda" to systematically find the previous incarnations of your idea -- or related ones. Analyze the heck out of them. Do better. Or be more popular. (Pick one or both).

But here's the twist, and if you're introspective then you saw this coming: you're not really afraid of someone stealing your idea, you're really

Afraid of someone not liking your idea

and you don't want to deal with that. You think that by waiting until you have perfect execution you will stun disbelievers with the beautiful product. That's just a delay/procrastination tactic. The underlying fear (of the product falling flat) will just make you want to delay and delay, making the product more and more "mature," so as to defeat nay-sayers.

Doesn't work. There will be nay-sayers. Embrace them. Love them. If you have money, make them into a focus group and pay them! Separate the whining pessimists from the ones with specific advice. Don't waste your time on the former, but realize that the latter are creating value in your company for free.

They are doing what your best product managers and designers should be doing -- finding and clearing roadblocks to adoption. Listen to them and verify what you think they're saying, by checking with other real people. Then you have a real bug list to work on, not getting the that drop-shadow AJAX doodad the right shade of pink.

As an entrepreneur, you're already drinking enough Kool-Aid by necessity; if you hide from naysayers, it just leads to a Kool-Aid overdose death-spiral.

I've been involved with companies that have committed both of these sins (and many more) so of course my perspective is warped by that. But don't take my word for it. Find the companies and entrepreneurs that you look up and want to learn from. Read their stories or go talk to them (being careful to filter out the Spiel and the 20/20 hindsight). Whatever you do, don't hide out convincing yourself you've invented cold fusion.

MozyHome Backup Fails to Backup Designated Files

I have been testing out Mozy's MozyHome remote backup product, and have found that it sometimes ignores new or changed files in the backup set marked to be backed up. Sometimes it "discovers" these files (or changes) days or weeks later and backs them up; other times, if I modify a file "nearby" (relative to the way my backup set is constructed), it will suddenly discover all the other changed files and back them up. Still other times, no poking or prodding seems to make it back up these files.

This is a serious problem. After all, the raison d'être of the product is backup. Imagine that a home user or a business installs this solution, marks files to be backed up, observes that the backup process is running successfully and then -- after a data disaster -- learns that, well, some of the files are backed up and some simply are not.

If Mozy were a fly-by-night outfit, one might say that better due diligence is required in choosing a backup provider. But with its recent acquisition by storage giant EMC and its global contract with GE, Mozy appears to be a solid company.

The software, though not perfect, basically works. The backup and restore are straightforward. You can encrypt locally with your own key so that no one but you could ever decrypt your data in the case of a breach (although interestingly, the file names are not encrypted, so don't count on hiding the existence of my_illegal_off_balance_sheet_transactions.xls, or gifts_to_my_mistresses.doc).

But we're talking about the number one, sine qua non, only -- really -- important use case for backup. It has to back files up. Or at least, if it doesn't, it needs to tell you what failed, when, and why.

When I first discovered this problem, I realized that publicizing it could have a negative impact on Mozy's business. So instead of blogging, I contacted them directly to learn more. Unfortunately, after a few back-and-forths, including my running their diagnostic tools and sending them the reports and explaining that I was not going to give their support personnel full remote access to my box without more information, they have gone radio silent.

As a software engineer, I am fully aware that this problem could be a strange corner case. Perhaps it is so narrow that it never affects anyone besides me. But I doubt it. And, in any case, this problem is severe enough that it warrants a little investigation to determine its breadth.

Why do I doubt it is an unusual corner-case failure? Simply because my configuration of the service is so "typical." I'm running XP SP2, NTFS on a well-maintained, modern machine, on a secure home network behind NAT, with no strange services or applications of any kind running (e.g., the kind that might hook and hack kernel file system operations, or leverage alternate data streams). The affected files are not under unusually named file paths, or have any funky attributes set on them.

The only things I am doing that are not defaults are using my own encryption key, and adding a few files to my backup set that aren't in the "My Documents" tree. My conclusion is that it is likely that whatever glitch is causing the software to miss files on my file system, is also missing files on other people's file systems. And they have no idea.

Perhaps this problem doesn't affect other users -- but without trying to verify the bug, how can Mozy know? In my last email to them, I specifically asked if their QA team had even attempted to replicate the bug. Had they tried and failed to reproduce it? Fair enough, maybe I could offer some help. But if they haven't tried, it makes you wonder what bug report could possibly be a higher priority? Maybe if their app runs off the rails and reformats your drive, that's a higher priority. But barring active destruction of your data, or a major security bug that could compromise you to a third party, I can't think of anything.

The ultimate problem here is not even an engineering problem. Yes, there's a bug in the software, but there are bugs in almost all software. Rather, it's a process problem. How does the QA process work? How does customer support work? What steps do you take if someone reports the Really Big Bug? Is the right thing to assume they're a crackpot? Can you afford to do that and not even look into it? (Hint: No.)

</end of regular post> <free:bonus>

Since I'm really not out to get these guys, but ideally want to help, I'm gonna offer a free first step: it's really easy to tell after the fact if this bug is manifesting, since the set of files actually backed up simply doesn't match the local description of the backup sets. So an easy diagnostic is to write a list of these two sets of files and diff them. If there are deltas, you've got a problem.

So... you push out an update to the client that creates a list of the files in the active backup sets and sends it over to QA as the last step in the online backup process. Then QA just has to generate a matching file list (the match is on the account id [email address] and either the date/time or the id number of the backup) from the Mozy meta-data store and compare.

Free ActionScript Code Generation: VASGen 0.2 Release

I finally got a new rev up of VASGen, my Violet UML ActionScript 3 Generator.

It is built against Alexandre de Pellegrin's enhanced version of Violet UML, which is designed to run either standalone, as an Eclipse plug-in, or via JNLP ("Java Web Start"). This version of Violet also has a much glossier look and feel, which make it particularly pleasant to use (thanks, Alexandre!)

     

The main functional enhancement to VASGen is in the area of package support. This version supports placing classes and interfaces (and nested packages) inside of packages, using the Violet modeling tool. When code is generated, the proper folders and qualified type names will be used.

If you don't want to package all of your types, or you are just sketching around inside what will end up one package, you can specify a default package by just dropping a package icon anywhere in the diagram and marking the name of it with a '+' (e.g., "+foobar").

I've tossed in a couple of bug fixes too. Details and the download are here.

If you uncover problems, I would love to hear about them and hopefully fix them.