Yahoo Mail Supports Free IMAP … But Only via Mobile

It’s hard to know what to say about Yahoo at this point. They have a lot of page views, but their internal organs seem to have been liquified from the inside by some flesh-eating plague.

Barely alive, they are of course responsible for their own continuing undoing. A great example is the failure to do something useful with the email user base. The purple crew has missed every opportunity to exploit the email product, from Google-style ads to Facebook-style social networking (back when they had a chance). And there’s the infamous CEO with her Napoleonic airs.

But I digress.

See, here’s the thing. On the Internet, unless you’re lucky enough to be a telco, you actually have to compete. Which means when Google gives away IMAP-enabled mail, and Windows Live gives away POP access and 25+ GB of online storage, you need to rethink your strategy of making your mail product harder to use while praying that the nine people on earth who haven’t heard of Google will sign up for your deluxe mail service (now with undercoating!)

But Yahoo has never been able to think one thing at a time. It’s not so much that they excel at multitasking than that they are schizophrenic. So while one group is committing ritual suicide with the email product, another realizes that to reach the mobile market, exposing mail via open protocols might help.

And hence they offer endpoints called something like imap.mail.yahoo.com and smtp.mobile.yahoo.com. Before you get all excited and ask me to check the hostnames and port numbers, though, I’ll drop the punchline: it appears Yahoo filters access to these services based on IP, and opens the service to mobile carrier address blocks (and carrier proxy addresses).

After seeing my phone successfully configure itself for these services, I couldn’t resist trying them from a desktop client. No joy. I guess if I were really hardy I could grab the public-net-facing IP address for my phone (by having it connect to my own server) and pretend to be that IP. But … really … is it worth it? No… with Yahoo still living in 2002, I’m afraid it’s just not.

Work 2 Different Logons (or Sessions) with Private Browsing Modes

Here’s a trick that’s fairly obvious to web developers but probably not so for everyone else.

Private browsing modes – such as Incognito (in Google Chrome) or InPrivate (IE 8) clear out locally stored browser cookies (snippets of tracking data) each time they are run, and do not share cookies with the main (non-private) browser tabs.

Since these cookies are the core mechanism by which web sites associate independent browser tabs/windows with a single user session, the private browsing modes can be used to interact with multiple different logons or accounts at the same time.

How does this work in practice?

Let’s say you want to interact with two totally different accounts (logons) at, say, Gmail, Yahoo, eBay, Orbitz, or some other site. You’ve probably noticed that if you are logged in to, e.g., Yahoo with one browser tab, and you open another window or tab in the same browser, and go to Yahoo, it will “know” who you are and allow you to interact with the same account.

In many cases, this is a desired behavior. But if you want to work with two different Yahoo accounts at the same time, it is quite tricky.

Here’s where the private browsing mode comes in.

Open a private browsing window/tab, and log on to Yahoo with the second account. Since the private tab doesn’t share cookies with the main tab, you now have two interactive sessions with the two different accounts, and they “stay separate.”

There are three catches though:

First, since the private browsing tabs are designed to discard cookies on shutdown, they will “forget” your logon when you close the browser even if you select “stay logged in” on a specific web site.

Second, it is possible (though unlikely) that this mechanism could fail if the site uses “Flash cookies” apart from regular cookies.

Third, while this trick works perfectly well, do not assume that the service you are connected to has no idea what you are doing. They probably don’t care. But if they do care, assume that they could guess that the same user was on both sessions (through IP addresses, NAT port assignment patterns, etc.) at the same time.

Two More Small Tales from the IP Front

There are a small number of areas in which I am an expert, and intellectual property is not one of them. For expertise, I like to refer folks to Mike Masnick and Techdirt.

Nonetheless, it’s hard to move through the technology world and not get bonked on the head by IP absurdities and incongruities calling out for some solution aside from lawsuits and Nancy-Reagan-style denials.

First up is the tale of two Comcasts, and two identical HD streams of Battlestar Galactica. One of these streams comes in via the HD DVR, and the other comes in via Bittorrent.

Actually, I’m leaving a bit of information out in the setup. See, the two HD streams from SciFi channel should be more or less identical modulo commercials. But they’re not.

Due to bandwidth allocation and network management issues, the “legitimate,” paid-for SciFi channel stream is full of bitrate-spike artifacts reminiscent of late ‘90s web video. So the HD channel, the HD DVR, the HDTV gets you … a pretty awful picture the second characters start fighting, running, or blowing things up.

Then you have the Bittorrent stream, ripped from SciFi, and carried in via the same Comcast coax line. This stream looks great, and makes a customer glad to have Comcast and a HDTV. In some strange respect, this latter stream may represent a problem in the “industry’s” opinion, even though it’s clearly what the customer wants and is paying to view, and even though the bandwidth (both instant and aggregate) is a small fraction of that required for the “broadcast” stream.

Next strangeness is Grooveshark, a free, crowdsourced, on-demand streaming service that appears to live on the razor-thin edge of legality if it has a claim on legality at all.

That said, the ability to send (or post/tweet/blog/…) a link that goes directly to a particular song is an extremely potent way to virally spread the music you like. And more music loved by more fans is the core asset base for any music industry, whether it resembles the 20th-century “legacy” record industry or not.

Without Grooveshark, there is always YouTube for sharing instant, no-membership, no-login-required tracks. But unlike YouTube, Grooveshark is structured in a way that encourages more exploration of an artist, album, etc.

So anyone wanna throw into a pool for how long ‘til this service gets shuttered? Wait, that’s not quite legal either. We’ll “just say no” and it’ll all just go away.

Suck Us All Into the Machine: Build FluidDB on Twitter and #Hashtags

I was reading today about a fairly amorphous, tag-based, public "database" concept called FluidDB.

To avoid the usual distractions, I will assume my audience knows all about relational and non-relational datastores, implications of tag-oriented metadata, etc. In fact, since my friends know about this stuff and they are most of my audience, I think I'm not too far afield making this assumption.

One reason I found the FluidDB concept interesting is vanity: about four weeks ago, I spent some time considering building just this kind of database ... on top of Twitter.

Why would one want to do that?

Precisely because Twitter already has a lot of meaningful data curated by humans and tagged with a well-known metadata scheme (hashtags).

In exchange for having very small or hypernormalized data records (since each atomic entry is limited to 140 characters minus the tags and any indexed keys), we get a strange merging of human- and machine-readable data.

Humans could read (and follow, search, etc.) data entities of interest.

And clearly the "goal" of automated (machine) participants (clients) would be to understand as much of the human content as possible, treating it as objects, tuples, logical inferences, or knowledge base "facts."

Moreover, the originator of a tweet, as well as any @-referenced recipients, are critical metadata. They are, actually, tags themselves in way which is linearly independent of the hashtags. That is, a from-@adbreind tweet (entity) marked #database is different from a to/ref-@adbreind tweet marked #database, while #database must be considered to (possibly) have a different sense than it does in, say, a from-@headius tweet. However, the same tools and semantic analyzers can be applied, essentially treating the writer and target of a tweet as special tags.

In this way, our twitter discourse, short enough to make machine understanding tempting even when the packed cultural references make such understanding impractical, merges us into the database and makes us "just another part of the machine."

AT&T “Fixes” My Phone by Downgrading It

I don’t want to be another corpse in the AT&T pile-on. AT&T has many problems, and they impact me, but let’s look at another technical clue about the network trouble:

I stopped in Tuesday at an AT&T store to see if they had any advice or fixes for the network issues plaguing my phone. The only thing the service rep said he knew of was to replace the SIM card. Sure, why not? Can’t hurt anything, can it?

Well the answer to that question apparently depends on whether ditching 3G access counts as “hurting.”

Since I got the new SIM, the phone spends most of its time on the EDGE and GPRS service, and only rarely gets a fix on the 3G service even when I’m in known-good 3G areas.

For a fleeting moment, I thought about taking the phone back and asking AT&T to do something to restore my 3G goodness.

But I quickly realized that the network’s inability to keep the phone on 3G was a big part of the problem. In other words, as so many iPhone users have discovered, staying on EDGE is better than flitting between EDGE and 3G and getting nowhere. Likewise in the case of EDGE and GPRS.

So for now, I’ll take my retro GPRS and its 50kbps or so of throughput. Since the setup and teardown of the data connection, as well as the latency in repeating a request, are the slowest part of the process, I’d rather use dependable 50kbps connection over an unstable, unusable 500kbps one.

Until I get a network that actually works, that is.

Link: An Illustrated Brief History of Augmented Reality

Check it out.

Plus who knew that Philippe Kahn invented the camera phone?

Apparently, a lot of people, just not me. The guy is so awesome.

Microsoft’s Real Punishment is Having to Fight with the Gloves On

By the time the Microsoft anti-trust cases wound down, it could be argued that the alleged damage was long since done to the industry.

In a curious reversal, it appears that Microsoft’s direct penalties from the case were only a tiny beginning.

The real penalties Microsoft pays are not in money, nor in shipping “K” and “N” SKUs. They are in product strategies unpursued because they would be too provocative. In other hands, such strategies  might be reasonable if aggressive, but for Microsoft they might look like “relevant conduct.”

Today’s announcements about Office 2010 (“14”) were … well … let’s just say if you didn’t read about it, the most interesting thing you missed was seeing critics point out that Microsoft, inventor of AJAX (literally, for the Outlook web client), is only bringing other apps online 10+ years later.

The reasoning here is not about selling client OS licenses. Microsoft could have moved to the cloud richer and faster and more profitable if it could take the gloves off.

Need to get Silverlight penetration up from its abysmally low numbers to where it can really compete with Flash and become a meaningful platform? Just ship it with Windows and make it a priority update to every Windows box in the world. Problem solved. Now we can get down to the real work writing apps. Or at least Adobe could actually face some competition. But Microsoft doesn’t dare do this.

Why not stitch cloud storage directly into the OS? I hate leaving files “on the other machine.” Right out of the box, anytime I see an “Open…” or a “Save As…” dialog box on Windows 7, I would like the default destination to be a secure folder on Windows Live SkyDrive. Using the provider pattern, other vendors could offer a similar service, and the end user could choose. But Microsoft doesn’t dare with this either.

It is quite possible that the long-term benefit to the industry of having Microsoft thus restrained far outweighs the “lost” value we could have had from Redmond. But let no one fool himself into believing that what we see from Microsoft these days is everything they have to offer.

VncSharp Rocks for Programmatic and Interactive Remote Access from .NET

VncSharp is a C# implementation of the VNC protocol together with a handy visual Remote Desktop .Net widget.

At first, the open-source VNC remote-access solution might seem like a surprising item to need in a Microsoft-based solution.

But once he gets you in the little room, VNC starts telling you that, in exchange for lower performance (than Microsoft’s own RDP), he can get you more flexibility, more features, no licensing issues, and access to remote Macs or *nix hosts. After these arguments (or is it the heat?) VNC looks a lot more persuasive.

Throw in the fact that you can run your VncSharp-enabled apps on Mono, and … well, it would be cool if that bought you a lot. But actually if you aren’t already focused on a Linux solution then the Mono angle is just another bullet.

VncSharp itself, though, works extremely well straight out of the gate.

When you see the documentation page – where the author essentially invites you to read the source to figure out how to drive it – you may be concerned. Or even start to form silent curse words with your lips.

Do not let that stop you. There are demo/sample apps that will show you what you need for basic use cases (e.g., popping open a “remote help” window will only require a line or two). And the source, if you need it, is elegant and straightforward to navigate.

A programmable VNC client is, perhaps, a niche product. VNC on Windows maybe more so. So it’s gratifying to see such a mature and streamlined OSS effort.

Can iPhone et al Drag Augmented Reality Into Non-Augmented Reality?

I’ve been pitching augmented reality apps in startup circles for a few years now, so it was exciting to see the AR startup crop grab some press coverage this week (VentureBeat and more VentureBeat and …even SF ABC 7).

With the iPhone hardware suite (and comparable devices like the Android and Pre), there’s no shortage of hardware for the core AR tasks:

• capture decent-resolution images
• recognize “target” areas in images
• contextualize the targets if necessary, by adding GPS data, solid-state compass data, and/or accelerometer (angle) data
• lookup augmentation data suitable to the target and the end-user via suitable web services
• employ “billboarding” or 3D rendering to composite a representation of the augmentation data on top of the target
• repeat as fast as possible without draining the battery (yeah, right)

Now for part two of the plan: this facility needs to run through a cool looking visor (a.k.a. "head-mounted display” or HMD). And neither the $6,000+ price tag nor the Silence-of-the-Lambs-night-vision look is appealing on these traditional high-tech units.

Happily, there are mass-market headsets designed for the gaming or personal entertainment market which are almost ready to go. A couple are even within striking distance of cool factor. Maybe an Apple logo would be enough to do it, at least for the Bay Area.

Even better, leaders such as Vuzix recognize the need to provide video and accelerometer data from the POV of the headset (vastly reducing the amount of computation needed contextualize the image). They appear to be planning these capabilities as optional clip-on modules to their newest “Fall ‘09” model visor.

Note the word “planning.”

Like smartphones themselves, we’ve been here before … a lot of times. The iPhone was easily the industry’s 10th attempt at a commodity handheld computer, so it’s not like the writing is on the wall. Unless it’s AR writing:

Workaround for Some Instances of Win 7 WiFi Problem

One of the very few real broken bits in the Win 7 RC is a WiFi problem. The current thread on Microsoft TechNet is here.

On at least a few laptops, the following is a dependable if annoying workaround.

1. Turn off the WiFi with the laptop hardware switch (“airplane mode”)
2. Open Start –> Computer –> Manage –> Device Manager –> Network Adapters
3. Right-click and disable the wireless adapter
4. Wait a bit and verify the OS has completely lost the wireless adapter (by watching the icon in the tray)
5. Turn on the WiFi with that hardware switch
6. Right-click and re-enable the wireless adapter
7. Wait … potentially a couple of minutes for Windows to find the appropriate network, connect, and recognize the route (if applicable) to the Internet

This maneuver is a hassle, to be sure. But it seems to work 100% of the time on some laptops (including mine) and it is much more convenient than rebooting, which is the only other dependable solution.

iPhone and Palm Pre – the Obligatory Post

I’ve had my paws on the Pre, and while I have not, of course, gotten hold of a 3GS, it doesn’t really matter.

See, getting my hands on a 3GS might convince me it has a better hardware/software experience. And since the 3G already has a better hardware/software experience than the Pre, I’m going to call it a “gimme” for the new 3GS.

The Pre, for all of its clever conceits compared to most phones, is still clunky, hiccup-y, and jittery next to even the current iPhone model. The graphics aren’t as smooth, the UI is harder to use, the physical keyboard is marginal, and on and on.

On top of that, it is hard to overstate how important the app ecosystem is to this “competition,” and Palm doesn’t even seem to be trying (they’re still saying “real soon now” on the SDK).

No matter how many apps in the App Store are just fart apps, and no matter how beautiful the bundled apps on the Pre are, there is no contest because these guys are playing different games.

Apple has succeeded in making the phone a general computing platform in the mind of the public – something I argued for 3 years ago – and you judge a platform not by its internal specs but by what you can run on it. Palm doesn’t seem to get that. They’ve got a decent bundle of specs but there’s nothing to run on it and there may never be much.

So with Apple still killing in the UX department, and Palm leaving their A-game at home (if they ever had one) as far as the app/platform/dev community goes … is there anything positive to be said for Pre in this contest?

Only this: AT&T’s network is so egregiously ill-behaved in so many prime metro areas that Sprint could actually pull a few people across the line.

I am one these last folks: I would much rather replace my current phone with an iPhone, but the thought of another two years of dropped calls, missed calls, bars-but-no-coverage, data connection unusable half the time … and I’m seriously considering the Pre.

Say what you will about Sprint (I’ve used every major carrier and none is perfect), where they have coverage, the devices just work. You can make or take a phone call. Which, ironically given that smartphones are bordering on augmented reality nowadays, is still the sine qua non for a phone.

Windows 7 RC, One Month In

A month ago, not long after the RC was released, I wiped my ol’ Server 2003R2 notebook (yes), and installed Win7 x64. Time to try a consumer OS on the machine again, finally go 64-bit, and jump without a net (I had a big client demo coming up and decided it would be sink-or-swim with 7).

First, a word about this laptop. Bought in the holiday sale period of ‘07 just for client demos and occasional web surfing, it would take a whole new, um, “lower end” category of Microsoft “Shopper” commercials to capture the spirit of this baby: for $299 I got a mobile Celeron (1.6 GHz, one core, no HT) and 512MB of RAM. Billed as “Vista Basic Ready,” it was being discontinued due to … not really being Vista Basic Ready. Well, duh. I had given up on Vista after my 3.6GHz desktop choked on it, so I put another GB in the laptop and loaded Server 2003R2 (using unofficial XP drivers I grabbed off the net).

So … Windows 7.

Upon install, all of the hardware was supported perfectly, which isn’t a huge surprise since 7 uses Vista-era drivers and this hardware ensemble was originally targeted for Vista.

Except for a compatibility issue with AVG Anti-Virus (which I’ve written about before), it has worked almost perfectly with everything I’ve thrown at it – ranging from Office and Visual Studio 2008 to Alfresco Enterprise (yes, a Java server app), Google Earth, and Ruby. Overall performance has been excellent and better than I would have expected. No, I wouldn’t play games on this machine, and the 5400-rpm hard drive can be a drag just as it is on most laptops.

The two big negatives I’ve seen are as follows and will hopefully be fixed by RTM time.

First, there is a well-documented issue with WiFi. On a cold boot, Windows does just what you want it to do with the WiFi. But, after waking from sleep or hibernate, or trying to switch networks a few times, it just cannot seem to sort itself out. Most of the time you are forced to reboot to get a working connection. This bug is all over Microsoft’s Win 7 feedback forums, I’m just too lazy to look up a link right now. So hopefully it will get a fix.

Second, the integration with other default browsers (I use Chrome) is broken in a bunch of places. Many of the cases where the shell is called to supply a browser to a specific URL do not work (e.g., menu options in twhirl, connections to the web from Office Live). In addition, Windows doesn’t want to associate local .html (or .htm) files with Chrome. Sometimes I can get these things fixed, but then they revert (perhaps partly a result of Chrome’s auto-updates).

These cases all work fine on XP and don’t require in-process loading (suggesting it’s not a x86/x64 issue). It’s inconvenient, and the anti-trust folks will be back after Microsoft if they don’t make other browsers first-class citizens soon. So this bug should get squashed.

Aside from those two admittedly very annoying issues, this is a really solid, fast, and elegant operating system. How serious is the “Microsoft tax” issue on OEM PCs? I don’t know, but if I do pay such an implicit tax on a new machine, I’ll be a heck of a lot happier if I can get a Win 7 license out of the deal and not a Vista license for my trouble.

Mark Process for x86 to Use Skype API or Topaz SigPlus API

If you happen to be integrating your Windows app with Skype's COM bridge, or with Topaz' SigPlus API for their biometric digitizing signature pads, those libraries won't play nice if loaded from an x64 process.

The easy fix -- assuming you don't have dependencies that mean you need x64 -- is to create a build configuration for your main app that specifies a target platform of x86 (instead of "Any CPU"). The resulting app will run in WoW (the 32-bit shim for 64-bit Windows) and works fine with these libraries.

The hard fix would appear to be using IPC to coordinate two different processes: your main app in x64, and a proxy/helper 32-bit process running in WoW that works these DLLs for you.

Those Little Shape Widgets in VS2008 Require Deployment

The Visual Studio 2008 SP1 control palette has a few "shape" widgets -- Rectangle, Oval, Line ... which come from a "Visual Basic PowerPack" library: Microsoft.VisualBasic.PowerPacks.Vs.dll

If you use these, you need to deploy the library manually with your application. That is, an up-to-date .Net 3.5+ machine will not have the library available globally, and Visual Studio will not mark this DLL to be copied into your output folder.

I haven't checked to see whether the VS Installer project will determine the dependency and include it -- I'm guessing it will. But if you are doing xcopy deploy, or just testing, you will need to bring this library along.

There is also an earlier version of this same library, that shipped on VS2008 but will not work (it is missing some components that were added to the namespace later). So if you need to build against the current version (e.g. if you are working on code that uses the shape components), make sure you update to VS2008 SP1, which updates this library.

Windows 7 and AVG == Unhappy CPU Usage

The title pretty much says it all.

Backstory is that I ran Windows 7 in a VM without any anti-virus while doing some evaluation, and I was impressed by how sparing it was of resources.

Then I installed the RC on my (underpowered old) laptop, figuring it would be a good replacement for Server 2003 (which is resource efficient but not exactly designed for laptops).

The RC used somewhat more memory -- it appears to be able to adjust its memory footprint depending on the host hardware, which is cool -- and a ton more CPU.

The CPU usage was suspicious because it's rare to find software that consistently pins a CPU with actual work, and because the usage was high priority -- coming from kernel space or a library tied tightly to kernel hooks. It was also suspicious because Microsoft plans to sell a version of 7 on netbooks, which are even more underpowered than my two-year-old bargain-o-matic laptop.

Turns out AVG, generally a fine anti-virus product, struggles with Windows 7 and often insists on every available CPU cycle, while the end user sits there wondering why context menus won't even open anymore.

After a bit of Googling, I removed AVG and installed Avast, and 7 is screaming along on the laptop now.

Note the retroactive Windows 7 hardware subsidy: this laptop was discontinued by the manufacturer in '07 because it shipped with Vista Basic but proved so slow as to be completely unusable in that configuration. The new OS basically injects value into the old machine.

Bonus Security Credit for Google Chrome’s Strange Install

Last fall, many folks including myself commented about Google Chrome’s unusual install behavior. Our best guess at the time was that it represented an attempt to accelerate adoption, by allowing non-administrative users to install Chrome.

It also allowed lower-privileged domain users in corporate environments to install and use Chrome unless their IT specifically blocked it.

With this recent Chrome vulnerability and rapid patch cycle, though, I’ve come to see the install in a new light.

Firefox requires an admin to initialize an update. This can be done through programmatic remote admin or right on the console, but still requires intervention. IE can be updated via Windows auto-update, but if auto-update isn’t set to run or if a specific patch needs to be applied, it requires intervention. Chrome, on the other hand, will update itself on the fly for each user’s install (it does require a restart, but only of the Chrome app) unless the installer is cracked to remove the GoogleUpdater component.

Given the cost of having an out-of-date browser version versus the risk of having Chrome updated without admin knowledge … I have to say I like this approach.

Notebarn Update

Notebarn, my Windows Mobile / Exchange sync notes app, definitely looks like an archaeological relic these days. Dating from early ‘07, before the iPhone era, and being a simple text utility, it is almost comic how it doesn’t resemble modern mobile apps.

That said, I still use it, and it turns out a lot of other people have been using it too. So when a user helped me reproduce a tricky timing bug that could cause data loss under certain circumstances on app initialization, I hopped back into the old (and quite small) codebase to fix it.

There is a little more info on the notebarn project page. Or if you just want to install the app you can install it over-the-air from here. If you already have the app it will automatically install in-place over your existing version. And since the “notes” are actually stored in an Outlook/Exchange Task, the install won’t affect existing data.

A word about backups: notebarn doesn’t have its own data backup mechanism. There are two main approaches to backing up and recovering data if you should lose it for any reason (e.g. problem with notebarn, problem with ActiveSync, accidentally deleting a note you needed, etc.)

One is to lean on whatever backup solution protects all of your Outlook/Exchange data, since notebarn data is really Outlook data. If you can go back to a backup snapshot of this data, even temporarily, you can simply grab the notes data from there. If that’s not practical, you can either manually or via a script back up the “My Notes” item from Outlook tasks, into another place in Outlook, the filesystem, etc.

Google “Similar Images” Roadmap

Ok, it’s not their roadmap, it’s my roadmap.

I was psyched to see the Similar Images announcement today, but I was underwhelmed by the results. That’s ok, it’s helpful, it’s free, and here is the post where I explained how to build the rest of it.

Oracle and Sun: Cui Bono?

Well, here’s a hint: it’s not Oracle, “Sun,” Java, or MySQL in the long run.

I’m thinking the Ruby, Python, and PostgreSQL worlds just got a shot in the arm, as this is minor calamity (at least) for Java, and a major one for MySQL. Ironic, since Java maturing like a fine wine and recovering from early-decade blunders; MySQL was already in trouble thanks to Sun.

As for benefits, it’s also not Google, who relies heavily on Java but could eventually find itself in an adversarial relationship with Oracle as enterprise computing moves to the cloud. Google does have enough sheer wo/manpower to exploit the OSS licensing on Java to take it in its own direction if necessary … but is that really a desirable way to go? or one the investors can live with?

I don’t think Microsoft minds this one bit either … since there was nothing that that Java, Oracle, and their communities (and users) couldn’t do before that they can now, while a number of scenarios (Java and open source databases/appservers in the enterprise) suddenly become just a bit murkier.

Atalasoft: Another Example of Gnarly DRM == Lost Sale

I’m working on a project that involves semi-automated document imaging. Scan, deskew, crop, re-arrange …

It’s on Windows, where every modern scanner hooks into both TWAIN and WIA out of the box, often without even needing a vendor driver, so I just needed a library/toolkit to do the lifting on the app logic side.

Enter Atalasoft DotImage imaging libraries. Does everything you need, works fairly well. Established presence in the market. We start heading in that direction. The Atalasoft bits we needed turn out to be pricey as components go, and we would need a runtime license as well as the development license – but this is a commercial project the success of which would not be diminished by the software costs. So we didn’t blink at the price.

We downloaded the dev SDK, implemented a few features … and we needed to show them to customers. In other cities on other machines. Well, the dev SDK is crippled and doesn’t allow that.

Atalasoft’s sales department generated a 30-day license for me, and sent me the instructions to install and deploy it. And … it half worked. Some machines could run the deployed app. Other machines, the app would crash when the relevant DLLs tried to load, despite deployment of the magic binaries, license files, and other DRM voodoo.

For a brief moment, I thought maybe my app is just broken … but, upon attaching a debugger, I saw that all of these crashes threw the same error. And, since it was .Net, the error was in plain English: Atalasoft’s licensing module was barfing and taking the whole app down.

At that point I could have spend more critical hours trying to navigate around these problems (I’m guessing their pre-sales tech support would have tried) … but … wouldn’t you know it, here is another company offering a similar library, much more agreeable terms, 30-day trial and a seemingly foolproof license key mechanism.

Download, type type build deploy. Success. Haven’t looked back.

Now it’s also convenient that this other product seems to work a little better, has more agreeable legal terms and costs less. But those were not dealbreaker criteria at this stage.

I would never have even gone down the list to this other vendor if Atalasoft’s DRM hadn’t broken my tight-deadline customer demos.

Facebook “Private” RSS Feeds Probably Don’t Leach Data…

Last year I experimented with private group microblogging systems via authenticated feeds. Didn’t go anywhere, because many of the biggest newsreaders don’t properly support authenticated feeds. And “obscure but public” feeds get indexed by aggregators like Bloglines, by design, making sensitive content much less obscure.

Enter feed access control, a several- (3-?) year-old RSS/ATOM extension that tells Bloglines, and anyone else who is listening, that this feed should be treated as private, even though it’s public.

Facebook’s feeds are intended to support this protocol:

Which seems reasonable enough.

There are a couple of issues though. First, this approach is based on a third-party’s positive action to prevent or “opt-out” of publishing and indexing, in a system that normally defaults to syndication, indexing, etc. So it’s easier for a glitch to expose data.

Second, the whole “fac” extension is a gentlemen’s agreement among parties that couldn’t even agree on making authenticated feeds work well. Perhaps they all make a best effort to isolate the marked content. But tomorrow, a startup with a rocking aggregator could simply ignore “fac” and expose all of the feeds it has.

In some sense, the same vulnerability exists with other systems – if you signed up with some random webmail provider, who’s to say they don’t expose your mail. But because RSS is public by nature, almost all feeds live utterly unprotected, and this extension is one vendor’s hack, it’s not quite the same.

All in all, probably not a big reason for concern. But when people tell me how private things can be on facebook (where you can sneeze and end up revealing your data because the IxD is tilted so heavily toward sharing everything) it always seems worth noting how your data (via your friends’ feed subscriptions) can slowly leach out into the open ocean of the indexed net.

Random Bit: Sysprep Re-Writes Boot.ini … Not Always Correctly

I discovered the hard way that Microsoft’s sysprep tool (for configuring machine images) re-writes (at least some of the time) the boot.ini file, the file which tells the Windows initial bootloader which OSes are installed on which devices and partitions.

The new boot.ini contains the same OSes as the old one, but it specifies a different default, and a zero timeout for the user to choose what to boot.

I can imagine some reasons why sysprep might want to do this, based on speculating how I might deploy enterprise images.

Only the thing is, if I were going to re-write boot.ini, I would at least check to see which OS was currently running and maybe make that the default. As it is, sysprep made a different OS the default – it picked the “first” OS in the device tree even though that is not the OS I was trying to sysprep. This behavior seems more like a bug than a feature.

In any case, if this happens to you, there was no long-term damage done -- you can just reconfigure the boot.ini file by hand and restart.

Enable “Modern” (Themed) Common Controls in Hybrid WPF/WinForms Apps

Here is a quick hint to save someone from a bunch of Googling:

If you are building a WPF app, you may find that you need or want to also use some Windows Forms windows. In my case, I was adding a form just to host a WinForms control, so there was no point in creating a WPF form just to host the WinForms Host container in order to add the control. A more common scenario is you want to invoke a built-in Windows dialog box, which is not natively a WPF object.

If you do this, it will work, but you will notice that some controls are rendering their old-fashioned look and behavior – you’ll be zapped back to the era of Win 2000 or the earliest .Net apps that lacked the benefit of comctl32.dll version 6. Square edges, no mouse-hover behaviors, etc.

The short answer for how to fix this is that you need to add a call to System.Windows.Forms.Application.EnableVisualStyles().

Add it once, somewhere early on. It’s ideal (though not always necessary) to do this before you start instantiating the WinForms objects.

Apparently the template code for WinForms projects contains this line, and depending on your POV, that’s either “low level boilerplate that an app developer shouldn’t have to care about” or “the kind of thing that kids nowadays just take fer granted with their magical IDEs and WYSI-whatnot, virtual memory and lazy programming habits.”

I was also particularly motivated to write this post because the most accurate (and earliest) Google hit I found on this topic was to one of those scam programmer support boards, where they wanted me to sign up for a trial with a credit card just to see the discussion thread on this issue.

Which is half insane if they could persuade me that had the right answer inside, but 100% insane since there was no way for me to know that their “answers” weren’t way off topic from clueless n00b who thinks a HWND is what you pull to keep the rain out of your office.

Visual and Context-Cued Semiotic Search Opportunity

Want a hardcore problem to work on? to fund? to stay-up-nights-only-to-see-Google-do-it? or maybe get-bought-by-Google?

We need a search engine that searches based on visual and contextual clues about the appearance of objects -- especially of signs and symbols -- rather than just based on words that (perhaps) describe them.

For example, if I see a bumper sticker around town, with a green star on a blue field, I might want to see if this represents some well known organization or cause. I could search for “green star" and “bumper sticker” or something similar. But I probably won’t find anything.

Moreover, when the elements of the design don’t have names (“star”, “stripe,” “field”), properly describing a complex design in a single search gets difficult. Imagine you saw the new Pepsi logo:

 

You don’t know what it is; for the sake of the argument, imagine you don’t have any cultural Pepsi associations to work from either. What do you type in to the search box? Circle? red? stripe?

Good luck.

How do we solve this problem?

I envision a search that consists of several stages. At the first stage, you can add descriptive words, or you can import a similar image, or even draw/sketch some cues right on the page. That may sound unlikely for less design-oriented folks, but many unknown visual designs consist of largely straight lines, simple geometry, etc. So it’s not unreasonable that I could sketch in a simple design, or even take a swing at the Pepsi logo above, with just a circle and 3 straight lines using an AJAX or Flash inline drawing tool.

From these inputs, the search engine draws a set of possible results – but it also generates a set of context-narrowing options that I can use.

It presents options to choose where I saw this design: web, billboard, tv, clothing, museum, public building (e.g., a capitol or courthouse), manhole cover, etc.

Perhaps knowing material is useful: was this printed? embroidered? leather? denim? engraved metal?

This is a challenging but eminently creatable piece of software.

I’ve actually had a lot of instances where I would have liked to use something like this – but, if it’s never happened to you, consider: when computer vision progresses beyond working with the local environment, objects and known patterns (people), the machine will need to take the next step. It will want to dereference symbols to find data and meanings in order to solve problems. And, in order to do this, it will need benefit from this kind of visual-semiotic search heuristic, which starts with a visual-context search like the one we are discussing.

Good Results So Far For Google RAM

A week ago I had 4GB of RAM die (well, part of matched pair anyway) in my main desktop PC.

I’m currently awaiting replacement under warranty from Corsair, but meantime it’s hard to run dev tools and big virtual machines with the measly amount of memory I have left. So I thought it was time to give the new Google network-attached RAM a try.

I had to flash the motherboard BIOS of course and upgrade the chipset driver and the on-board network controller firmware. Google RAM, just like wake-on-LAN, has to interact with the network card at a hardware/BIOS level. In this case, the purpose is to ensure that any OS I boot sees the new space just like local memory.

Then I rebooted and … nothing.

Where is my free 4GB of storage?

Then I remembered that Google’s revenue model for this product requires you to run a Windows service that in turn interacts with a Google-provided kernel patch for PAE.

In addition to providing checks in real time – as my machine accesses RAM – for any security threats, this service displays Google ads as 5 new icons on my desktop.

Apparently they are context-based, and determined by Google’s analysis of what I have in RAM at the time.

And they are surprisingly accurate. I had a picture of a Corvette open in Photoshop, and the G-RAM icons turned into links to car dealerships, new-car financing, and a discount oil change.

Google’s FAQ insists that it does not look at my clicks or the image file metadata – instead, its server analyzed the image in real time (since the network RAM is in their datacenter) and determined I was looking at a new Corvette.

The only downside was that my cable modem signal dropped out for a couple of minutes, and the local service warned me not to touch any processes using G-RAM until it could sync back up, or those apps would immediately crash.

No matter, overall it’s great technology, and I think my RAM replacement will arrive from Corsair tomorrow.

Quick Hit and a Deep Hit on Social Nets and Identity

This article from the WSJ is neither deep nor particularly novel, but I like it because if focuses a laser on propagation of identities and the history of identities in popular social networks. This is the most important metatopic for social networks.

If you have a few more minutes, this article by MIT Media Lab prof Judith Donath gets a lot clearer on the signals that make up identities online, and how the mechanics of those signals can function.

Harm Reduction in Windows 7

Guest mode … kid mode … whatever you want to call it, is brilliant.

But more than that, it’s an interesting admission that (1) you can’t fight the power of the darknet and (2) you might as well empower people to behave in a way that minimizes the damage, whether or not you approve of what they’re doing.

If I had a dollar for every individual who ever swore they never go near warez or pr0n or questionable media downloads, and ended up with a mucked up machine … or worse, a machine that transmits their passwords and SSN to a bad guy …

Even with an older OS, like XP, one can achieve a fair degree of isolation and protection by using a patched up Firefox or Chrome on top of a plain user (not admin) account. There are still holes by design; e.g., a user could fill up the hard drive or install software that persists in certain places. And I’m sure there are serious security flaws that allow code downloaded as user to escalate itself to admin … perhaps even coming from a “drive-by” Javascript source via Firefox/Chrome … but such threats seem to be pretty darned rare if everything is patched up and prophylactic protections are applied (e.g. Spyware S&D’s “immunization”).

Guest mode (and IE 8 “In Private” browsing) appears to close many of the remaining holes.

What we need now is an education campaign to convince people to segregate their online activities. But besides not knowing how to create these low-privilege accounts, a lot of people I know refuse to admit they ever visit the darknet. Or the visits are rare and they “hope for the best.”

Let’s pre-configure – by default -- a second account for ever power user (or admin) on a machine. At login time, offer the guest (more protected) account along with some description of when it might be a good idea to use it.

I’m not sure the best way to label the buttons, because it’s a bit hard to explain how the more secure, more protected mode is paradoxically for the more anonymous, more dangerous behavior; while the “less protected” mode is for normal operation which might involve vital personal data. I’ll let the UX wizards sort this part out.

(Semi) Portable Comet Framework

I meant to reblog this at the time of the announcement: Sun has released a first alpha of their atmosphere project. The project was about extracting useful comet-y bits from Grizzly and making a standalone pluggable kit for comet.

It is its own small framework, and autodetects where you drop it.

Here’s a nice article. It looks to be Jean-Francois Arcand’s project – follow his blog and the project’s twitter.

How Much Would You Pay to “Learn to Pitch Big [Failing] Newspapers”

If you’re in SF next week, and you don’t mind paying $15-20 for the privilege, you can come hear some people from the SF Chronicle and NY Times talk about how to pitch your (probably tech) company to them.

So they’ll write a glowing and informed article about you.

Wait, wait, wait … this is all wrong.

First, these are “reputable” newspapers, meaning they won’t necessarily write anything good about you. At most, they’ll theoretically assemble a balanced story, interviewing your competitors, talking to customers, maybe even your employees … or ex-employees.

Oh, wait, I’ve got this wrong again.

They aren’t going to anything like that … unless, maybe, you were already a big news-section story already. Else they will write something that’s like a watered-down blog post, without any specific expertise or authority, but with a couple of quotes. Newspapers like to quote because they can’t link. They’ll also mention twitter in the story, they can’t help themselves.

These papers do have a big circulation though, maybe that’s the appeal.

But it’s hard to tell their attention reach, or the “effective circulation” of your story buried in the tech or lifestyle section. How many people really read that? Are they influencers? Customers? Relevant at all to you?

It’s hard to tell. I can tell you that the people who are really interested might find the story … when it comes to them through the backdoor via some RSS feed or Google alert. But if they care enough to do the RSS thing and find you, then they’ll also have all the other, better, material about you that comes from all of the experts in your field who blog about you and also turn up in RSS and Google alerts. Ironic.

Next, these two newspapers are in dire financial straits. At this point, $20 probably keeps the Chron publishing for another couple of days. And why are they in trouble? Not just because people can read their content online for free – rather, it’s because in most areas of reporting, the big organs have no specific interest, capability, or credibility, and so no one cares what they write. The one thing they can do is send a foreign correspondent to Iraq or the White House, and maybe the correspondent has some credibility…

Wait, there I go again, the Times sold out on Iraq years ago, by their own admission, and so did most of the rest of the traditional press.

Ok, I give up.

I’m going to sponsor a meetup where newspapers can send people, who will each pay me $15-20, buying my attention long enough to tell me why I should care.

Silverlight Sound And Fury (You Know the Rest)

So as not to bore regular readers, I’ll skip the jeremiad.

Bottom line: despite the hoopla at MIX over Silverlight 3 – which is an incredible platform – there were still no meaningful penetration numbers presented.

And while it’s great to see the platform revving and maturing, the various version make a development decision that much harder. If you have a new idea for a Silverlight app, and you imagine your target audience will have the plugin or is able to install it, do you aim for v3? v2? v1?

I also haven’t heard a word about any explicit program to drive Silverlight client installs.

According RIAStats.com – perhaps not the best source of detected install info, but … wait, I guess if it’s the only source of information, that automatically makes it the best – Silverlight is on 22.3% of their observed clients.

Microsoft SDS Change Eerily Reminiscent of WinFS Fate

Last week Microsoft announced that they would be abandoning the ACE and dynamic entity (“property bag”) model for the SQL Server Data Services cloud data storage system. They would also switch from their REST data API (used in ADO.Net Data Services) to the old-school “Tabular Data Stream” wire protocol.

While Microsoft’s promise of more relational support was always a distinguishing feature of their cloud DB service, and while they tried to spin the news in that direction, it feels a lot more like when they abandoned WinFS and announced that, really, everything you could do with WinFS would work fine using NTFS and a whole heck of a lot of indexing. Maybe sorta true … but feels like a big step back.

Of course, big customers – large enterprises with SQL Server databases and lots of SQL code – would not want to see a change in their data layer and would prefer this move. But accommodating them is assuming that they are ready to become first-version customers of the data cloud at all. And I doubt this for two reasons.

First, any move to the cloud involves a trade-off of control which some companies are loath to make even if they are confident the system will work. Which is problematic because:

Second, anyone who has dealt with big databases knows that there is no magic. Despite the quest for automagic autoscaling self-tuning databases, no one, so far as I know, has made one that does all of this for really large enterprise applications. There are just too many application specific variables, not to mention poorly written app code that can cause trouble in proportion to the amount of resources you give it access to.

I do believe Microsoft has the engineering brainpower to try the problem, and are as likely as anyone to succeed. It’s just that I haven’t seen any evidence of a specific strategy or technology. Maybe if I were a bigger customer … but seriously, if Redmond had this problem solved (and it’s one of the biggest out there), they would either patent it or publish lots of white papers. Either way, it would be publicized and reviewed. A trade secret? maybe, but which Fortune 500 CIO is going to jump on that bandwagon and the cloud and the outsourced data stuff all at the same time?

To the extent that these large database apps could be made to behave without human intervention, there is likely to be a tradeoff in resources, and when you’re paying per GB or per compute-cycle, that equals a side order of more cost to go along with the entree of new greater risk.

The point is that the ACE/dynamic entity/REST model is well understood, performs, utilizes resources in a known manner. Not appropriate for every app. Not relational in the formal sense if at all. Not easy to migrate to. But it goes like the devil. So you’re getting something concrete in exchange for your risk and your dollars. Unlike a magical SQL Server instance in the sky.

Maybe there is magic in there, and I’ll be proven wrong. Or maybe 99% of the customers’ database needs are so small that it’s a non-issue, and Microsoft is really just competing with the thousands of hosting providers that will host actual individual SQL Server instances for you on a large server. But this change still seems to raise more questions than it answers.

Way to Compete, Guys…

Microsoft’s app store … sure, why not? But that’s not the bit they need to take on Apple.

Microsoft and the smartphone is really a funny/ironic/sad story depending on who you are.

They had a true next-generation mobile OS starting back in ‘01 … It was really easy to code for – like GUI-builder, point-and-click web services, run-your-regular-.Net-code easy. And they were outselling pretty much everyone in total device count a couple of years later. By ‘06 they even had consumer friendly devices, in the Moto Q series and then the Samsung Blackjack. They were poised to challenge RIM for the big shiny belt.

And then Apple came along and wiped the smirk off everyone’s faces. What’s surprising is that no ‘softie seems to have circulated an “Internet Tidal Wave” memo about mobile. Or, if they did, no one paid any attention.

In the last two years, we’ve seen a continuing proliferation of Windows Mobile devices, but no fundamental change – or even speed-up – on platform evolution. If anything, we’ve seen a slowdown, as Mobile 7 devices seem to be at least a year away, and the “app store” is going to launch on Mobile 6.5

In case anyone didn’t already notice, v 6.5 is a great OS if it’s 2005, but a non-entity in the iPhone era. An app store? well, maybe … but a store by itself has never been the magic sauce in mobile (remember Verizon’s “vending machine”).

And with a “logo validation” scheme for each app? Developers violating the logo cert guidelines is not the problem. The problem is that there are too many different form factors for Win Mo devices. Used to be, practically anything could run the OS. Around the 5.0 era, they reduced the number of supported screen configurations, and a few other things.

But there appears to be little escape from the compromise Microsoft made to be successful on the enterprise side: it’s really easy to code a simple utility/productivity/line-of-business app that will run great on almost any Windows Mobile device. And it’s equally hard to write anything really cutting edge, because there is simply too much variation in device capability and performance, and that genie's not going back in the bottle.

Perhaps Microsoft’s best chance lies in forking a “consumer” mobile OS, with stricter controls over the handsets. On the other hand, Apple is clawing into the enterprise, so an artificial separation of consumer vs. enterprise offerings may be hopeless at this point.

Adobe Time-Warps Half a Decade Back, Will Still Probably Defeat MSFT

Earlier this week, I went to see a couple of folks from Adobe present their latest progress on Flash Catalyst, Flex "Gumbo," and the "Spark" UI component framework.

As someone who does a bunch of Flex work, I liked everything I saw.

Especially since it was the second time around.

No, I didn't see this stuff at MAX, I saw it at Microsoft PDC in 2003 and 2005.

It was shocking how pleased Adobe seems with itself now that it's almost ready to release a design tool that generates XML and RIA code... since everything they showed -- and more -- was part of the earliest Microsoft Expression Blend alphas that I saw years ago.

The Microsoft product was code-named "Sparkle." But we won't get this confused with Adobe's "Spark" because (1) "Spark" refers to a different bit, Adobe's re-invention of lookless, templated controls, which Microsoft implemented in WPF and shared with the world at the time (around '04 or '05), and (2) because Expression Blend is already out in a 2.0 version, so unlike the Adobe products, it doesn't need a codename anymore.

Adobe even has yet another XML dialect to facilitate moving design assets through the workflow -- it's called "FXG." And it appears to supplement MXML quite well in specific areas, so that if you take MXML and add FXG, you get XAML. Not that XAML was de novo or anything -- the XUL and Java folks (desperate to stop writing Swing code) had been creating similar XML formats for a while. The Java community was especially fond of XML with tons of imperative programming constructs mixed in alongside data objects and calling it "simple and declarative." What XAML did was provide all the necessary power, while keeping it declarative.

Anyway ... Adobe should get credit for recognizing the right way to do this when they saw it. Namely, they realized which workflow tools were needed, embraced the idea of export from Photoshop and Illustrator to a vector markup with a visual editor with timelimes, and thence to an RIA build tool with a code-oriented IDE.

Now that they're finally getting this on track, Adobe is even more likely to trounce Microsoft in the RIA world. They have penetration numbers that MSFT can only dream of, and for a company that doesn't build real developer tools they're giving it the college try.

Which is kind of sad, since I believe Silverlight is a better technology with better language and tool support ... and not any less rather more open than Flash.

Want Help With Your Startup? Let It All Hang Out on Craigslist

It's awfully easy to go looking for folks doing stuff the wrong way ... and to find it. So it's nice to be surprised by someone doing something amazingly, shockingly, frighteningly ... right!

I was greeted by a craigslist ad in my RSS reader today, one of many startups looking for folks to, essentially, work for free. I've written about why this is a bad idea before, and it's still a bad idea.

But there's a little more to this ... the poster (the company's founder presumably) posts a link to a wiki. Maybe it's genius, maybe a trainwreck -- either way I had to look.

On the other side of this link is a company wiki. An explanation of what the company is building; where they are in the process; their calendar; UI mockups with notes and the comment stream by the creators; and other items.

This is absolute genius, and it's so rare. Plus it shows the guts that most entrepreneurs fancy themselves to have, but lack when tested. I'm not commenting on their specific business/tech idea, I haven't thought much about that to be honest.

But it is so refreshing to see someone out there on the beach letting it all hang out as it were.

I work with a lot of entrepreneurs and most of them think that they're the first ones to think up some genius idea, and the best way to be successful is to either keep it stealthy and secret, or to sign reams of NDAs and non-competes with you before disclosing (cue music) their subtle and delicate brilliance.

Just writing that last paragraph, it's a struggle to keep a professional tone. These folks are usually (97%, there are a couple of specific exceptions) complete fools. And truly, they are fooling themselves, unconsciously trying to avoid exposing their idea to someone who might not think it's so good, or who might point them to the dozen other people doing the same thing. Generally speaking, the secrecy ends up being a contributing factor to their failure. Which, since startups are highly failure-prone anyway, they will deny anyway.

That's why I was so thrilled to see this post. The founder is saying, "If you want to try and 'steal' my idea, you go ahead. But if you really believe there's a bunch of money in it, wouldn't you want to work with other people who believe the same thing and who have the will to execute? And if you go off with it and succeed anyway ... you're still helping me because you're establishing the category, while I plan to work nights and sweat blood to execute better and faster than you."

The ad is reproduced below. I was going to link it, but interestingly it has been 'flagged' for removal from craigslist. It's hard to imagine why -- the whole scenario seems rather more legitimate than the typical ad in the category. Perhaps the allusion to potential full-time work disqualifies it from the free "gig" listing ... but I think a startup seeking essentially non-paid volunteers in whatever capacity they can afford qualifies as a part-time or temporary arrangement.

Technical Wizard / Web Developer Wanted | Internet Startup (sunnyvale) An internet startup is seeking a highly talented web developer If you have experience with either: PHP/MySQL, Python, or Ruby we would love to talk with you. This a very exciting startup opportunity with massive potential. At this stage, we are looking to bring aboard those who are seeking equity share in the company. We simply do not have the capital to fund salaries. For more information, please have a look at: http://wiki.kunsoom.com All of the pertinent information will be included in the wiki page. Thanks for your interest in the project! We look forward to hearing from you.

Adobe and Microsoft Get Into It Like Children on the Playground

A week or so back, Adobe exec Mark Garrett got a bunch of attention for insisting that Microsoft's Silverlight effort has "fizzled."

Microsoft promptly screamed back that it wasn't so, pointing to the inauguration video stream, and a few other factoids.

What makes this truly schoolyard funny though is what happened today when Adobe "announced" it was bringing Flash 10 to phones. This seems just as dubious as Microsoft's oft-repeated plan (since as far back as '05, when it was WPF/e) to get Silverlight onto mobile phones ... by last year ... which obviously didn't happen.

Meanwhile, for years, Adobe has been pushing a weak technology called FlashLite for mobile ... and for a variety of reasons it has never been a usable option for content providers to deploy Flash content or apps.

For both Microsoft and Adobe, for both PC and phone applications, the critical metric is current "content-ready" penetration. How many devices are ready to run new Flash/Silverlight content off of the web today.

In this 2x2, the only square that's solidly covered is Adobe's Flash on the PC. "Ready" penetration of Flash 9+ is near 100%.

On PCs, existing install base is critical because of locked-down corporate networks that won't allow end-user installs. Microsoft needs to stop talking about download numbers, or numbers of people who "can access a PC with Silverlight," and start doing anything it can to get these ready penetration numbers up.

On mobile, the barrier is user confusion over configuration. Vendors could push the updates to phones, but in nearly 10 years of smartphones, only Apple has done much of this. Windows Mobile 6 has an updater ... and in over a year I don't recall it ever updating a darned thing.

Flash Lite trumpets a large "installed base," but these are strange installs, where the runtime (but not browser integration) is baked into the phone, and there's no reasonable way to get new Flash content onto the phone, either via web pages or download.

Both of these players are big on bluster and have been for a long time. Meanwhile, developers are left with few options for all of the smartphones in the world that don't an apple on the back.

Is It Too Early or Too Late for an Open RIA Design/Dev Toolchain?

I was playing with the Raphael JavaScript graphics library (a sort of script-based, cross-browser, implementation of SVG) and started thinking how helpful this library would be in creating a browser-based (as opposed to plug-in based) RIA.

That lasted for about 15 seconds before I remembered that creating large, non-trivial RIAs generally involves designers, and most designers don't like creating vector art by coding a set of "path" statements, or animations as a collection of key-value pairs and millisecond-based transition times.

That's why tools like Microsoft Expression, and Adobe Illustrator, Catalyst, and Flash exist.

And why Adobe and Microsoft are investing so heavily in the designer-developer workflow: the ability of designers to turn graphics and animations into app skins and interaction which are immediately available to coders.

In order for an open RIA solution to be competitive and realistic -- whether it's open in the pure-browser sense, using JS via dojo.gfx, or Rapael, etc., or whether it's via an open plug-in (Java/JavaFX seems like the closest, though it's not 100% open yet and may never be) -- this full toolchain needs to exist.

We need to be able to export vector art from mainstream design programs such that they can be incorporated as assets into the RIA. It doesn't matter if this is via SVG, XAML, AI/EPS, or something else entirely. What does matter is that the import/export is robust enough that designers -- whose jobs, after all, include making stuff look just right -- are confident that what they design is what end-users will see. The Microsoft and Adobe tools can do this. To date most OSS attempts cannot.

Next up, we need a truly usable, designer-friendly authoring tool for animations and interactions. It is often argued that some standard tools (*cough* Illustrator *cough*) are not paragons of usability themselves. No matter -- it's hard enough to get converts.

Happily, there seems to be emerging some consensus among the big vendors about how these tools should work (both on-screen and in terms of in intermediate data formats). That blueprint lowers the risk and challenge for an open source contender.

The biggest obstacle remaining is a classic open-source triangle-of-trouble:

1. The toolchain/workflow will not be viable until it is quite solid, since the commercial alternatives (Flash, mainly) are so entrenched.
2. It's hard to get enough contributor man-hours against such a huge project without an active user base.
3. Since the user base is not developers, the bootstrapping for #2 that makes many OSS projects work (devs are tolerant -- even excited -- about getting up on an 0.1 release) is unlikely.

The Second Law of Thermo-Specifications

A couple of days ago, Adam Milligan posted an eponymous "law" on the Pivotal Labs blog.

It includes a corollary stating, "The full definition of correct behavior of code exists in the tests for that code."

Now, there seemed to be something fundamentally off about this proposition ... and I wanted to figure out what that was.

Surely Mr. Milligan doesn't mean this in the trivial, tautological sense (define the spec to be no more than what test cases happen to exist at a point in time) ... even though Agile dogma often borders on that view, pretending it's some kind of paradoxical path to enlightenment like a Zen koan.

The comments to his post start to touch on "spec" vs. "test," and whether 100% test coverage is practical, desirable, or conclusive.

Of course even 100% code coverage, with a missing code path and a matching missing test and behavior spec equals ... test success, and wrong functionality.

In such a case, though, the problem has been externalized from the Agile context and put onto a faceless "business person" who "doesn't get" Agile because he or she actually wants to plan ahead and describe some certain specs that persist over time.

This is a neat trick. In physics, all sorts of magical things can happen if you look only inside of one context (or frame of reference) without looking at what's happening outside, or at what's happening to the frame itself. In finance, there can certainly be a free lunch ... if you can make its cost into an externality and remove it from your model.

Once you make the tests and code-based behavior spec a part of the application you're building -- and clearly, once it's a real cost center as well as a critical deliverable part of the project, you have done so -- then you are in a sense simply externalizing that troublesome human interaction (specification, functional analysis, planning). Yes, the tests match the code under test. But seen at a different level of abstraction, it's just another flavor of interface-driven development, or debugging.

Most engineers would love to realize the dream of self-describing, self-verifying code, whether that description be some kind of formal model, or a textual DSL (as with Microsoft's Oslo), or a set of tests and code-based behavior specs. And, indeed, these systems all improve the transparency of the code, propagating requirements from the outside in, and revealing when they are not met.

But even with the most "bought-in" business stakeholders, it is impossible to escape the the outermost specification layer, the one with the humans in it.

Windows 7 Keeps Suspending in VMWare? Change Your Power Settings

I'll admit, I fell right into this one.

My Win 7 VMs kept suspending while I was off working on another machine. I'd come back to the host machine and see VMWare Workstation idling with the Win 7 instance frozen in suspend.

Hmmm... Doh!

The default power configuration for Win 7 -- even when on the machine is "plugged in" -- includes dropping into sleep mode after half an hour.

This, thanks to VMWare, translates into a "Suspend Guest" operation.

Maybe Vista shipped a default power configuration like this. I don't remember it being the case, but, to paraphrase Pulp Fiction, "[Vista] may [save power] like pumpkin pie, but I'd never know 'cause I wouldn't [run] the filthy mother****"

Think About It: Do You Really Want Your Engineering Done "For Free"?

I enjoy reading the posts on craigslist, wherein naive entrepreneurs go looking for free software engineering talent. I know this sort of ad (no pay / equity only) bothers some people -- as though it somehow has real impact on the real jobs and pay in the industry -- but I think it's just fine. I mean, if you want a service for free, there's nothing wrong with asking. And if you are in the mood to work for free, it's nice to know where to find opportunities ...

Of course I am not referring to recruiters for genuine charity / volunteer work, where the goal is to provide services to a population that isn't served by the normal market mechanisms. Rather I'm talking about real for-profit, we're-gonna-be-the-1-in-100-successful-startup-and-make-a-boatload-of-cash kinda company.

The funny thing is that it's hard to imagine these wantrepreneurs looking for any other kind of professional services for free. Not because they feel coerced by social norms to offer money, but simply because they know better than to want what the "free professionals" have to offer.

I don't see these guys saying:

• I need heart surgery and I'm looking for a doctor who wants to keep their resume fresh. No compensation, but I'll thank you if I wake up.
• Going through a brutal divorce. Looking for a sharp lawyer to keep me from losing my shirt -- no pay, but I'll buy you a drink if we come out ok.
• Need a corporate lawyer and patent attorney. Equity only.
• Startup needs business travel on the cheap. Seeking a pilot and aircraft, not so concerned with FAA licenses or airworthiness.
• Building a new facility -- need architect, structural engineers, environmental compliance, project managers, and laborers. If we make a bunch of money, we'll pay you at some point in the future.
• Love my new car, but it needs engine work -- looking for a mechanic who'll work for free. Someone with little or no experience seeking to build up a resume is ideal.

Now, it is true that in the late 90s, during the dot-com bubble, landlords, attorneys and other did take equity. But they did so in addition to -- not instead of -- cash payment. And the odds of making bank on equity back then, while still long, were stunningly better than they are today.

I know what these guys are thinking. They're thinking "I just don't have a budget to pay for real development, and anything I can get for free is better than the nothing I can afford otherwise."

Well, in some cases, perhaps. But that's setting an awfully low bar. Likely too low to allow for a real chance of success. That is, the entrepreneur is theoretically pouring all his time, money, heart and soul into starting this business. But he's willing to gamble the whole thing on the questionable code he'll get out of someone who has nothing better to do to pad a resume while unemployed? Doesn't quite add up.

Or maybe he really wants a "partner" ... only it's unlikely this developer is going to see 50% equity. And anything less likely means the developer is along for the ride while the entrepreneur follows the same hare-brained decision process that led him to advertise the non-paid gig in the first place. In any case, the developer has no real clout when push comes to shove. He's a silent partner, just along for the ride.

True, there are a few brilliant technologists out there who simply don't need any (more) money. They have cashed out of a startup, or they run another business on the side that covers their expenses. And if you can hire one of these guys, then bully for you. But I think they are more likely to spend their time where they can make an impact on a real business, product, open-source project, or non-profit.

As for the entrepreneurs who get someone to bite ... they don't know what they're in for: a few months or more down the line, they have (perhaps) some kind of a half-baked system. They need the loyalty of that unpaid assistant to modify and improve the code, while the assistant has now seen through all the "about to receive funding" promises presented at the outset. And as the limitations of the half-baked system slowly become clear, the founder may wish to replace the developer, or throw the whole thing out and start over with a new iteration.

But unless the original developer was a real sucker, all that equity means unrecoverable dilution in the cap structure. The more equity involved and the earlier the start (meaning smaller valuation), the worse this situation turns out to be. It could even come to pass that a potential investor one day passes on the deal because of these problems in the cap structure.

Is that Service Really a Scalable Cloud or Just Full-Service Web Hosting?

A lot of cloud stacks, or cloud app platforms promise scalability for your app, "With a little EC2 in every box!" (TM). There is a big catch and a little catch, though, and if your app gets big, then either or both of these may be a deal-breaker.

First, and most important: Running a vanilla RDBMS (e.g. MySQL) in a VM somewhere does not make it magically scalable. Read that sentence one more time.

Some cloud offerings integrate tightly to the traditional sort of DB instance you might attach to your web app on a single server. Examples include Heroku, which applies your Rails migrations to a PostgreSQL instance, and Stax, which offers MySQL.

The great thing about these environments is that they don't require significant changes to your standard app built on their supported platforms (mostly Rails and Java variants). Upload, minimal admin, and IJW (it just works).

That's turn-key, full-service web hosting, right there. It's beautiful -- in fact, in an OO and Rails course I wrote, I chose Heroku for deployment as a way to let students get something up and running on the web without getting into the operations/deployment/tuning aspects of Rails which deserve their own course.

But if your app gets large -- or just uses large datasets -- the database is rapidly going to be a bottleneck. Scaling out an app logic tier to a dozen EC2 instances automatically may sound good, but it won't do a thing for a DB-bound app (it may make it worse). And these databases don't scale out without a little architecture, planning, configuration -- all of the things which these cloud platforms are designed to avoid. And which, on some platforms, you cannot do at all.

For example, so far as I can tell on Heroku or Stax, there is no way to even configure multiple servers and replication, which is just a minimum starting point for scaling a DB to multiple machines. Stax may allow for a logical sharding setup, but it's not clear how one would control which VMs and disks the databases run on. Rightscale seems like the kind of firm which would specialize in the management scripts / meta-API that one would need to automate sharding, but the sharding option doesn't appear in any of the models on their website. With replication, which Rightscale does offer (though they're not exactly an app platform, more an infrastructure play), you get to this, still limited, picture:

Other cloud platforms offer datastores specifically designed to scale out, including Google App Engine, 10gen, and others. These platforms offer a non-relational or pseudo-relational datastore, with different data access APIs and a variety of restrictions relative to what you may be used to. These datastores are architected to scale easily, but there are real tradeoffs that must be considered. In fact, if you don't know these tradeoffs cold, you are not the right person to be making this platform decision. Get on craigslist and hire (or borrow) someone who knows the stuff.

The other catch is that whichever approach you choose, these vendors are offering you convenience, some outsourced operations management, and (in some tiers) elasticity and scalability ... but they are not offering cheap compute cycles. That is, if you know you'll need a large, predictable amount of raw compute time, then know also that you're paying a premium to do that computation in one of these environments.

A friend who has designed, built and operated feature film renderfarms for a number of studios confirmed that he has, on a semi-regular basis, analyzed the costs of remote VM-based datacenters (e,g. EC2) compared to their physical ones. Because the studios use these machines intensely, and are consistently consuming raw compute power, the local physical servers have always made more sense.

What does this have to do with your web app and datastore? Well, suppose you have designed your app to leverage a scalable datastore. These may not be tunable, may not perform fast, and may require you to do certain operations in code which traditionally are done in the DB. You may never see these slow queries or operations ... until they show up in your bill. That is, if the system is truly elastic and scalable, it will apply resources as needed to handle your work. If your query or sort or filter takes a lot of CPU cycles, the cycles will be made (almost) instantly available, so the user always sees your app perform well. And then you'll pay for all those cycles or instances at the end of the month.

Either way, there is no free lunch on the data persistence side. Which is not in itself a reason to avoid cloud environments. But it should be a bigger part of the conversation than it is today. And it absolutely must be part of the conversation, if larger businesses are going to move their services into the cloud.

Using AppEngine -- Or Similar Datastore -- To Integrate Complex Legacy Data Formats

I gave a lightning talk last night at the SF Bay Area App Engine Developers, showing some work I've been doing to represent gnarly legacy records in AppEngine so as to maintain source fidelity, minimize upfront analysis, and make them easy to integrate with other systems.

I had started with an XML record that I wanted to parse and represent in the datastore -- without knowing which tags and structures would be present, since this format had, ahem, evolved to obscurity over time, as often happens with real-world legacy records.

Before I talk about my approach, here's why I thought this effort might be interesting to the group: a lot of data structures have a tree structure in common with XML. From C structs and file blocks that include a header, telling which types to cast the next n bytes to (and so on inside of those) ... to mainframe "structured data" records I've encountered which consist of nested records, parsed recursively, with their meanings occasionally opaque, lost to history, or belonging to some partner company.

My approach -- which is simply to create a mapping of how to assemble and disassemble the records -- enables a record to be stored in a single App Engine record. But not as a block (or blob) -- rather with fine-grained addressable fields that are easy to talk to using the GAE Datastore API.

In my case, since my original was XML, I created a mechanism similar to a tiny subset of XPath describing the sequence of tags where a data element lived -- but with the characters changed so that it would be Python and GAE-friendly. That is, instead of "/foo/bar[2]/baz" I used _Foo_Bar__2_Baz.

This let me "flatten" the XML into a set of key-value pairs, while allowing that the XML might contain arbitrary structures injected by others ... and that I might want to inject my own extra structures. This arrangement is perfect for the Expando models in App Engine Datastore, or any similar store (e.g. Hypertable, which is modeled after BigTable, or Microsoft SQL Data Services which uses SQL 2008's sparse tables to similar effect).

So now I can store and retrieve my records. Any fields/subrecords which I understand and care about, I can easily work with from other systems, by mapping to the appropriate "key" in the stored record.

For example, if I'm storing a bunch of catalog data, and another system just cares about enumerating each "Product" with "Name" and "Price," then I can create a facade or wrapper in GAE that maps, say, Price to _Strange_Old_Way_To_Represent_Current_Price, and we're all set.

To be sure, there could be performance issues if you tried to use this to create arbitrary queries and reports against the data. That's not really the purpose and, in my experience, if there are no "shortcuts" to processing these legacy records, then the business folks are not used to being able to make an OLAP cube out of them either. (They probably have a batch or offline extraction process.)

Nonetheless, it's another tool in our chest when we need to work with systems and data that have been out in enough real-world battles to come home scarred with lots of cruft.

Twitter's Underwherlming (Former?) Architecture Problem

I recently came across this post from May 2008 comparing Twitter traffic and the Options Price Reporting Authority data feed. Needless to say, the stock market feed is many orders of magnitude larger, at 700,000+ messages per second(!)

It's also not the fairest comparison in the world on its face, for a variety of reasons: the OPRA data system was planned (Twitter met success more or less by accident), Twitter is minimally funded, etc.

A more relevant comparison, in my opinion, is that provided by newzwag, which presented its performance challenges, triumphs, and secrets at a recent SF Ruby meetup.

newzwag's site and trivia game is built on Rails, started small, and had to grow to meet traffic driven by Yahoo and the Beijing Olympics to 9 million pageviews per hour (using a total of a half-dozen machines or so). And lest you think this is a content site served out of a cache, most of the traffic consists of data writes by game players that then need to be ranked, published, etc.

As far as I can tell, that's somewhat larger than Twitter, even considering that Twitter has grown 3-4x since last May's stats.

newzwag's solutions, which they share here, are a study in sanity, reasoned problem solving, and smart efficient architecture.

Without the timelines or resources of a stock-market app, newzwag produced a nice solution that -- at least in hindsight -- appears drama-free.

Interestingly, a newzwag - Twitter comparison can be enlisted to support a variety of different startup social narratives.

One narrative is that an amateur-hour effort yields amateur-hour results, and aspiring startups shouldn't fool themselves into thinking that they won't need old-time Architecture and Sophistication to scale.

A different narrative says it doesn't matter -- if Twitter's success is your worst-case scenario, you still win. That is, build it fast, get it out there where people can try it, and you should be so lucky as to need a real re-arch to fix your scaling problems. In this model, both Twitter and newzwag played it right -- newzwag because they knew the Olympics would provide a narrower time window to showcase their system, so they managed risk against that stricter goal.

And yet another narrative says if you accept these two stories, you still wouldn't want your brokerage transaction flowing through a system built to "see what sticks," and hence Web 2.0 startup methodologies stare at mission-critical business apps from across a huge chasm.

I see this last story as persuasive but also as a big opportunity: there is a chasm, to be sure, but it needn't be quite so big. There are legacy mainframe apps that can speak webservices. Every manager in a big company wants their product to be "100% critical" even if they could create more value by admitting that a lot of nice-to-have two-nines business apps are the real bricks in the wall. If enterprises can get better at separating their Twitters from their OPRAs, they can make more money and have both.

New iPhone App Store Rules Take a Step Closer to Scriptable Apps

A lot of folks commented today on newly-approved web browsers appearing in the App Store. Or, more precisely, a handful of apps using the existing web browser widget to offer a slightly tweaked browser experience.

While iPhone apps could include the UIWebView component before -- and indeed this has proven a popular route to getting hybrid native/web apps up and running quickly -- today's change is about allowing apps that "duplicate" a built-in feature of the phone. And one of the fundamental characteristics of any web browser nowadays is that it is thoroughly scriptable.

If you build an app this way, it already includes a scripting environment ... so the question (since scripting and dynamic apps are verboten on un-jailbroken iPhones) is how far one can let the scripting go and still pass muster with the App Store overlords.

Using stringByEvaluatingJavaScriptFromString we can inject script into the browser ... including script that pulls data back out.

And although the JavaScript bridge is "one directional" compared to the OSX desktop API, there are workarounds such as registering a protocol handler to receive scripted "requests" from inside the page ... or by hooking decidePolicyForNavigationAction with a script-initiated navigation request (disclaimer: I haven't checked to see if this is in the phone API, but it seems plausible) to signal the availability of data.

So native code becomes effectively scriptable. Or, for an even less controversial but perhaps equally powerful route: just inject a bunch of JavaScript API libraries into the browser and keep the scripting (and more of your app) in Safari. That's not too different from pointing a browser at a web site (where the page loads various scripting libraries) ... except that underneath it all we are in native-caps mode ...

Unless I'm missing something here, a somewhat ambiguous situation has gotten thornier with the admission of this new class of general purpose browser apps.

Windows 7 Product Name is Missing a Feature

I didn't feel strongly one way or the other about the Windows 7 product name (i.e. "Windows 7") ... until recently when I wanted to troubleshoot the Azure SDK on Windows 7. (Apparently Azure on 7 has worked with the M3 build for at least one intrepid forum poster, but it's not behaving with the beta build for me at the moment.)

I started searching newsgroups, forums, blogs, etc., and realized that "Windows 7" is not a great search term.

On an engine like Google you can put quotes around it, specifying exact phrase, but some other full-text search systems don't seem to want to keep the Windows and the 7 together. Or perhaps they have an index by single words, and they link the results together to match your phrase later, but once you throw in other terms like SDK and Azure, the matching engine becomes a little more promiscuous, offering you a "promising" combo of Azure, SDK, and Windows ... or SDK and 7 ... as a higher-ranked match. Making it, in any case, rather harder to find what you want.

One-word product names, like "Silverlight," "Vista," and "XP" work a lot better for this kind of search.

Which is perhaps a reason that folks include the release name with the version number on products such as Ubuntu (Hardy Heron, Intrepid Ibex, etc.)

So ... what would be a good nickname to put next to Window 7?

Ruby and Python as Cloud Lingue Franche; Ruby/Rails on 10gen

Not sure how this one slipped past me, but 10gen announced support for the Ruby language and most of the Rails framework APIs on their open-source cloud service last month.

This addition is great news for 10gen and for cloud computing (the hosted-application-platform flavor, not the hosted-hardware/datacenter flavor).

For 10gen, support for a well-known API and app model is a huge bonus, which makes it easy for people to move an app into the cloud without learning and coding to new APIs, and also lowers the perceived "lock-in" involved, should the move not work out.

Their original JavaScript platform approach, as I've written before, is problematic not only because folks are unlikely to have meaningful (for their business) apps lying around to try mounting in the cloud, but more so because there is no standard server-side JS API set. A half-dozen companies offer a JS app server or cloud and they all have different platform APIs for even the simplest things, such as reading HTTP request variables, or deleting a session.

10gen takes a big step forward, joining Stax, Heroku, and morph labs in supporting Ruby on Rails in the cloud.

This move also reinforces another emerging trend: Ruby and Python serving as lingue franche for cloud app stacks. While many cloud offerings support JavaScript or other languages, Ruby and Python seem to be emerging as the ones with broadest support: 10gen will support both; AppEngine supports Python and a language-to-be-named-later; Stax supports both; Azure will likely support IronRuby and IronPython (some Python apps can already work in Azure).

Of course, the language is only half of the battle -- there are the APIs to deal with as well, and issues will typically arise where the impedance mismatch is highest with cloud-related infrastructure. E.g., cloud databases are mostly non-relational and don't support SQL ... so an ActiveRecord or SQLAlchemy API won't work on 10gen's 'grid database' (a reasonable tradeoff for simpler scalability.)

Even so, it is starting to appear as though one could write a lot of core business logic using, say, Python, and expect it to run unmodified on most vendors' clouds. Not a bad position to be in for the Python folks.