Rejected for Individual Health Insurance Coverage in CA? Meet the MRMIP

I keep this blog to tech commentary (and to areas I have expertise in), or software I've cooked up. But I'm writing today about a California-specific health insurance program that guarantees coverage for, among others, self-employed geeks who seek individual health coverage and get rejected.

This is a common phenomenon for those who stray outside the white-picket-fence world of employer-sponsored group coverage. Last weekend I came across another smart, savvy tech pro who got rejected for individual coverage and didn't immediately find info about this critical program. I'm not a big SEO guy, no doubt to the detriment of my traffic volume, but this information really needs to make its way up the Google rankings: for the heck of it I ran a number of searches myself, and info is just as obscure today as it was four years ago when I went looking for help on this issue.

So here's the executive summary:

• California's "Major Risk Medical Insurance Program" or MRMIP is a state-administered program that guarantees certain groups of individuals can buy a health insurance plan from a specific set of private-sector plans
  
  
• This program is mainly valuable to folks who need to obtain individual coverage (i.e. are not eligible for group coverage through an employer, union, etc.)
  
  
• You are eligible if you have been formally rejected for coverage in the last year or if you have been offered coverage at a more expensive rate than the current MRMIP program rates
  
  
• The program addresses accessibility of coverage, not affordability. I.e., it will help you get insured; it will not help you pay for it. Sadly, that means you more or less need some flavor of middle-class income to get your HMO on.
  

Considering this program lives at the unholy intersection of (a) government bureaucracy and (b) health insurance bureaucracy, it is well documented, easy to apply for, and easy to work with. YMMV, but I am speaking from personal experience, having struggled with the "I have no group coverage" problem and being happy with the solution MRMIP provided.

Ironically, once you know the magic keyword (MRMIP) it's easy to Google the rest of the data. But here are some direct links anyway:

2007 program brochure, which includes details on eligibility, carriers, coverage, prices, and the application form. The original is on the state's web site. Here's the program home page.

The program is administered under the auspices of the "Managed Risk Medical Insurance Board." If you're interested in the politics, policy, and future of this program, these notes are interesting.

Two last things worth noting:

First, there is an annual coverage cap of $75,000 on these plans. That seems like a lot of money, but in the US's insane healthcare economy it is very possible to run through that amount if something bad happens. For that reason it is still worth looking at which program offers the most potential care for $75k. I have a hunch that would be Kaiser, though I don't have all the data to back that claim up.

Second, although insurance companies seem to be getting more diligent about investigating your medical history, there may be the temptation to falsify or conceal various bits in order to get coverage, or to get a better rate. Doing so is hazardous.

If you get away with it in the short term, you may get some cheap doctor visits. But if you have a serious illness or injury, where the insurance company is looking at paying big bills, you can bet they'll put a fraud investigator on the case. If they can turn up evidence that you lied on your application, they'll deny your claims and probably try to terminate your coverage altogether. You could fight, but it would be a long uphill battle and not the kind you want to take on when you're already sick or hurt.

Bite the bullet, tell the truth, and if they tell you to get lost, programs like MRMIP can help.

Notebarn Update: Dial Phone Numbers from Notes

Once I gut a bunch of phone numbers in my Notebarn notepad, I started getting frustrated that I couldn't dial a phone number in a note just by viewing the note and clicking on the number. So I've updated the app to include this capability.

If you want this feature, and don't want to read any more geek ramblings, you can go right to the project page and download the app.

I missed this feature acutely because I used to use a Blackberry. On the Blackberry, whenever you come across an email address, URL, or phone number, the device recognizes the format of the text and creates a sort of smart tag. The trackwheel menu gets new tasks related to the data. If you move the cursor over a phone number, the wheel menu will include Call, Send SMS, and, if the number is recognized as matching someone in your address book, clever things like "Send an email to [name]", "Invite to meeting", etc.

It's not accidental that most Blackberry apps support this functionality: the folks at RIM supply classes in their platform API (net.rim.device.api.ui.component.ActiveRichTextField and net.rim.device.api.ui.component.ActiveAutoTextEditField to be specific) that do pretty much all the work for you. Use these classes, and you get this functionality for free.

Despite many strengths relative to the Blackberry platform, the .Net Compact Framework on Windows does not include such a class (at least as of 2.0).

Making this kind of an implementation harder, the Smartphone version of a multiline textbox has a "clever feature": In order to allow multiline textboxes to fit nicely onto a small screen, play nice with directional tabbing, and still accept a lot of text, they have two states. The base state shows one line of text and a little "expand" triangle. You can edit the text, the control responds to events, etc. If you click "Enter", triggering the expansion, the textbox switches into a fullscreen state where you can put in all the text you like.

The problem is that in fullscreen mode the component does not respond to all of its events (it doesn't even fire an event when switching modes), making it very hard to do context-sensitive stuff, such as "look at where the cursor is, extract the nearby phone number, and allow the user to dial it." Which is what I really wanted to do.

I implemented two workarounds instead. When notebarn finds one phone number in the note (using a RegEx that looks like a U.S. number), it adds a "Dial 415-555-1212" command to the main app menu. If there are multiple numbers, it opens a new window with a list showing each number and some of the context around it (so you can tell which number is which). Pick the one you want, and click to dial.

Happy calling!

TechCrunch vs. The Maker Faire

Mike Arrington wrote one of those shot-heard-round-the-world posts today. It wasn't news, but Mike Arrington writing it on TechCrunch was the news. The Valley in a troubling state? Indeed.

It would bum me out a lot more if I hadn't spent this weekend at the Maker Faire. The fair was a beautiful thing. The people and projects looked great; the companies mostly silly. The bigger they tried to look (Yahoo!) the sillier they looked. The more they focused on doing cool stuff (Microsoft... sorta kinda) the better they looked. But really it was DIY anything and everything. The essence of the geek thought process was there, the thought process that makes Silicon Valley work decade after decade: one part science, one part "I bet if I monkeyed with this a little more, it would be really damn cool," and one part "that is really damn cool -- you need a hand with that?"

Robert Scoble already made this connection. But I want to hammer on it a little more. Spend 15 minutes browsing this flickr stream and you'll feel right as rain. It's meatspace stuff, mostly, some fire and robots and yarn along with the software. But the idea is the same, and the membrane between online and offline has never been thinner.

On Saturday, we spent 20 minutes looking for parking and ended up in the far reaches of a dirt lot where some cargo trailers were parked. The event was well attended.

The peninsula has its own cash-driven strain of lycanthropy, never more than a full moon away, but a lot of people here always want to sit down with a soldering iron or scissors or a blank text editor window and put something new and cool into the world.

No Signature Required: It's the New Cash

Banks, credit card companies, startups, mobile phone companies, and whole bunch of other folks have jumped onto "future payment systems" in recent years -- focusing mostly on how to make small day-to-day payments more frictionless. So we've seen a dozen wireless phone wallets that, in the U.S. anyway, aren't very useful. RFID payment experiments at Mobile or McDonalds, and credit cards with chips of dubious value embedded in them.

When I recently found myself hoarding cash because it made certain purchases easier than using a credit card, I thought a little bit about the ease-of-payment problem. I came up with the following equation:

Ease(CreditCardPaymentNoSignatureOrPIN)
>= Ease(LoyaltyCard)
>= Ease(CashWithNoChange)
>= Ease(CreditCardWithSignatureOrPIN)
>= Ease(CashWithChange)

In English? The easiest game in town is swiping your card (or handing it over) and walking away (well hopefully you get it back). No signature, no PIN, no touchpad. Where does this happen? Starbucks and 7-Eleven are among the merchants supporting the Visa No Signature Required program for amounts under $25. It's a beautiful thing -- I can't imagine any easier way to pay, and I used to work on some of those RFID systems. At best they match this experience, but they can be much worse. The credit card gives me near perfect fraud protection, an itemized bill, and a short-term zero-interest loan.

A lot of businesses (Peets, Subway, etc.) haven't hopped on the bandwagon though, I suspect because the fine print is higher pricing on the back end. They'd love to avoid and card fees if they can, and they're happiest when they carry your cash for you. So they sell you a stored-value card and in exchange for letting them keep your interest and unspent funds, you get a swipe-and-run experience. This approach also has the drawback of requiring an extra card for each vendor, more of a hassle than pulling out a single Visa card.

After this, a cash transaction where no change is involved is fast and painless. You can pay and run, and cashiers love not having to count or give out change. Pricing in round numbers (think parking garages or ballparks) eliminates most change. Much is made of the supposed "anonymity" of cash. But that argument applies only to illicit transactions. Any transaction you do in person with an established business is bound to be on camera and timestamped these days, so you can be tracked even if you pay in cash.

Then you have credit card purchases with a signature or PIN -- my principal method of payment. This sounds easy until you're handed a card and a receipt to sign and return at a parking garage on a windy day. Aaaargh.

Lastly there's old fashioned cash with change: pain for all involved.

The big opportunity here is for banks and other credit-card issuers to cement their lead in the payment industry by pushing hard to expand the No Signature program. As it is, there is vastly improved data transmission infrastructure compared with what existed when credit card protocols were designed. The Obopays and Speedpasses, not to mention alternative credit card entrepreneurs are all over it. But the same data infrastructure could let Visa vastly extend No Signature without the fraud risk this would once have entailed.

Flight Data Button 0.5 for Outlook

I've spent a little time working on an Outlook add-in for putting flights in my calendar. A few weeks back, I took a trip and wanted to be able to pull up flight status on my phone, without keying in an address and surfing, or installing an app like Skip. (Skip is great, but was never fully implemented for Smartphone.)

So I put my flight into Outlook as a calendar item, and put a link in it to FlightStats Mobile, for my specific flight. FlightStats is the premier flight data provider in the U.S. -- many airport flight boards actually get their data feeds from FlightStats. Now, when my phone syncs with Exchange, the flight item comes onto my phone along with a quick-click link for status.

I decided this is a useful thing to automate, since any system that syncs Exchange to a device (DirectPush, Blackberry Enterprise Server, Moto/Good) would carry the link. But I was frustrated that I needed to key in the flight times, and I realized that should be automated too. To top it off, I foolishly ended up booking an appointment during time I needed to be driving (but which showed "free" in my calendar) -- so I figured my "automation" widget ought to add the travel time to and from the airport, and the time waiting at the airport before departure.

Using OpenKapow, I built a web service that scrapes flight schedule data. Assembling a FlightStats link is almost straightforward (getting to the individual segment within a flight number turned out to be tricky). And integrating to Office is delightfully simple thanks to great templates that take all the work out of it so a lazy sod like me can just write business logic.

I've called the addin "Flightify" since, to use it, you select an appointment in your Outlook calendar (where you've previously stuck some basic flight info), click the button, and the plug-in "flightifies" the appointment.

Here's an example: you start with basic info like this:



Click the button, and select the additional calendar items you'd like:



And a few seconds later your calendar shows the sad reality of your travel day:



You can't see them, but the mobile links to realtime flight status are embedded in the body of the flight item, so it's ready-to-click on your phone.

Here's the project page with the download/install if you want to try it. Full source code is there too.

I wanted to make this more of a learning experience and use the Outlook Addin templates that ship as part of Visual Studio Orcas Beta 1. I could try out more of Orcas and easily produce both an Office 2003 and Office 2007 version of my button.

Unfortunately, the Orcas Beta includes project templates, but not all of the Interop Assemblies you need to build the project (Office COM libs are required as well). So to get it building I would have needed to install Office on the Orcas test machine (in my case a VM with limited resources, meant for testing VS CTPs). My recommendation to Visual Studio team? Ship stub DLLs that implement the relevant COM classes (perhaps they just log calls to a file) and all of the Interop Assemblies. That way, the machine where Orcas is running doesn't need to have Office installed and can still successfully build these Office project types.

A few final thoughts on the plug-in:

• Since the web service calls run in the main thread, Outlook will freeze for a second or two while they're being made. Considering that Outlook freezes the UI thread all the time while doing its own built-in tasks, I didn't put a lot of time into a proper workaround.
  
  
• I don't write a lot of Windows Forms apps, so I was pleasantly surprised that the snap-band positioning logic in the form designer allowed my dialog box to appear the same on both a standard and high-DPI display without my having to do anything special.
  
  
• The automagical "Setup Project" builder was also great. I didn't really have to do anything, and it produced a nice exe/msi install, that includes registering my addin with COM via RegAsm.
  

OpenKapow Fun: Scraping an AJAX Site

I'm interested in web services which represent operations that make a persistent change in the non-web world. Like buying a ordering food or checking in for a flight. These capabilities exist as human-powered web workflows today, but rarely as remixable web services. Sooner or later, that's going to change.

Many of these services resist initial attempts at scraping and remixing because they contain AJAX elements such as scripts that rewrite the DOM. Scraping such a site is more than a little challenging, since you either need to analyze exactly what the scripts do and try to grab and run just the scripts you want, or else you need to emulate a Javascript enabled browser and then scrape the screen when the script is finished and the DOM is in the "right" state.

One tool that seems promising in the quest to tame these sites is the openkapow service host and its Robomaker IDE, which both work by hosting a Javascript-enabled browsing engine. The IDE combines this engine with a set of tag-finding and flow-control tools so that you can point-and-click your way to a script that automates the target site.

I've been looking for an excuse to build something transactional with Robomaker, and I hacked up REST services that execute a checkin or an offload ("un-checkin") for a passenger on a United Airlines flight. It worked for me on a couple of flights. But without having a large sample of itineraries that I could abuse in the testing process, I was a little uncomfortable with how the robot might handle some multisegment flights. The source (".robot" XML files) are available here though if you want to try it out or tinker. Seat selection would be an interesting and nontrivial feature to add...

So I reined in my ambitions a little and created a service to get flight schedules, in order to add automation to a small Office 2007 tool I'm working on.

The Airwise flight schedule page seems simple enough at first glance, but turns out to be one of those pages that uses script to write the DOM data that the user ultimately sees. That makes it a perfect candidate for ... RoboMaker!

It was straightforward to configure my robot to enter travel dates and airports into the form, click "go," and find the table with the results. What I wanted to do was look through the table rows, create XML snippets for the data, and package them up into a response.

But here's where my inexperience with RoboMaker and impatience got the best of me: once I got beyond the point-and-click part of automating the web page, I wanted to just write some imperative code to pack up my XML. Since RoboMaker is a Java app, I wished I could just write a micro-plugin for this stage of my robot in Java. I found the deep spelunking in dialog boxes, squirrelly regular expressions, and mediocre help docs to be frustrating.

Maybe the idea is to give me a taste of Kapow so I'll license their enterprise technology, which isn't free. Not sure about that. But I did know that I could dump the entire flight schedule table as the service response and deal with it on the client. So I selected to return the whole table content as "advanced structured text" (which basically translates into plaintext with newlines in between every table cell).

Although on the client it's trivial to parse the resulting data set and find what I want, I feel a little guilty about the ugliness of the XML response. You can view/use/download/edit the REST robot on the openkapow portal, and if you run it with the defaults you can see the mildly embarrassing chunk of XML produced (in the browser, the newlines in XML are ignored, so it looks even worse). A REST call looks like this.

I got over my issues with the service, though, and moved on to using the data for my Office plugin, which I'll post about soon.

Interaction Sequence Diagram for Google Web Toolkit

I've been doing some work with Google Web Toolkit. GWT is based on a cool model where the client-side and server-side code are both written in Java. The client-side code follows certain API guidelines which allow it to be compiled to optimized Javascript by GWT. Among the APIs that can be used is a pattern for asynchronous remoting, so that AJAX calls can be made and received in Java, with Java objects as arguments and return values (a bit like RMI). The marshaling and unmarshling, JSONizing and the like is done automagically. Nikhil Kothari has a similar technology on the .net side that lets you code client-side Javascript via C#. It is called Script#.

This model has a number of major strengths. But one weakness is that is not common, so it is not immediately familiar to developers. The interaction pattern can seem a little mysterious even with Google's doc set, debugging tools, etc. It gets more interesting when you consider that you might also be generating the original pages (the ones with AJAX interactions on them) via Java. (GWT does not require a dynamic page generation scheme at all, and can use static HTML.) In order to make this clearer, I wrote a sequence diagram which I'm posting because it may be valuable to other developers getting their heads around GWT.

The diagram shows three interactions. First, I'm generating my initial pages using the FreeMarker templating system. There are lots of other ways (like JSP for example). But here I'm using FreeMarker. This all takes place before GWT gets involved in the pages.

Next, I show how the GWT scripting gets running and loads the page specific code which will rewrite the DOM. At the end of this, the page is rendered the way the user will see it in the browser.

Last, I show how an AJAX interaction flows, including a call to an external web service on the back end -- e.g., if there is a "Load My Favorites" button on the page that results in a call to an external service and asynchronously returns with data to bind into the DOM.

I hope it's helpful. If I've left anything key out, let me know.

Click to enlarge / view:

Lunchrtime on Mobilecrunch

I was psyched to fire up the newsreader tonight, pull the latest mobilecrunch stories and see my own "mobile 2.0" food ordering site, lunchrtime, as the top story.

Photoshop notwithstanding, I can't resist including a screenshot while this is still the latest mobilecrunch post.



What? Why? ... check out the about page. Is this going to be the next twitter? Probably not, but if the AdWords revenue covers the cost of the web services, I'll be pretty happy. If you like it, have fun, and maybe add in some restaurants near you.

Meantime, if you're up for some geekery, one of the things I loved about building the site was leveraging so many pieces of the asp.net platform to make it a little less code, a little more action with a very small amount of effort.

The entire app includes only around 300 lines of code and some markup. The total time to build, including the user-facing functionality, some quick admin bits, and the micro cms that I use to upload and serve pages with scanned menus like this? something like 70 hours. Total. It doesn't look as slick as it might, but if you've read this far you know I'm a developer and not a designer.

Here are the (mostly) asp.net pieces I used so I could be lazy, do my day job, and get this running:

• Built in Users and Roles infrastructure
• Built in Login widgets and automagic password reset by email for above
• Built in User profiles
• Asp.net Master pages
• Asp.net AJAX
• Mobile Web Forms adaptive rendering technology (needed a tiny bit of massaging, but there's an easy pattern for that too)
• Practically-writes-itself code to mash in SOAP services
• and REST services
• HTTP handlers for serving arbitrary data or acting as AJAX service endpoints
• SubSonic Data Access Layer, aka ActiveRecord on .net, so no database code
  
• FreeTextBox WYSIWYG HTML editing (I can't wait for 4.0 with the legendary Safari support)
• Flash
• Yahoo Maps Javascript API
• Asp.net hosting with SQL Server 2005 for under $10/month

It's a blast -- I really don't feel like I need to do much besides wave the baton, and all this stuff just comes together and starts playing in Visual Studio.

There are a bunch of solid web platforms out there today aside from asp.net. But anyone who thinks .net is somehow cumbersome, un-agile, un-fun, or not suitable for rapidly building and modifying a modern web app should really log out of the groupthink.

Minor Update to Notebarn

Sorry for the hassle, but I have a slightly new version of Notebarn up on the project page now, which you may want to download if you have the original.

There was small focus-related bug that would manifest when the user returns to Notebarn after an Exchange sync. I thought it had passed a test earlier in the week (before the initial release) but I must have gotten distracted and messed up the test, because the bug was back. It passes the test now, and focus works the way it's supposed to. Honest.

It allows over-the-air update-in-place ... that is, you just run the new CAB file from the project page on your device, and Windows Mobile replaces the old version with the new one.

Free Smartphone "Notes" with ActiveSync/DirectPush Support

A former Blackberry user, I'm now running Windows Mobile Smartphone on a Blackjack. I love most things about Smartphone. But I miss the Blackberry "notes" integration with Exchange and Outlook. I used the notes feature constantly, managing all sorts of unstructured lists, and I liked being able to work on them on the desktop and know the changes would be propagated to the Blackberry and vice versa.

So I decided to write a little notes application that would sync with Exchange. If you like this idea and want to try the app, but don't want to hear any more geekspeak, you can get the app and instructions straightaway from here.

If you're still reading, here's what I did... The Pocket Outlook Object Model, which is the first stop for manipulating Outlook/Exchange data on the Smartphone, does not support note items. I wasn't sure if the Windows Mobile MAPI APIs covered notes, and I didn't really want to go that route as a MAPI novice, having heard many MAPI integration tales, none of the pleasant.

The Outlook Web Client accesses notes via AJAX, hence there is a web service endpoint of some flavor that can manipulate notes (Outlook can obviously get this data via HTTP too, since it can be instructed to operate in that mode). I could have sniffed/scraped that interaction ... but I wouldn't the get DirectPush I was jonesing for.

I took an easier route, and chose to store my notes as delimited strings in the body of a Task called "My Notes"

Since Exchange, Outlook, ActiveSync, and DirectPush take care of syncing my Tasks folder, I don't have to worry about that. And Tasks are first-class entities in the Pocket Outlook Object Model, so reading or writing one in C# is about 3 lines of code.

This approach worked well. I built a UI for the Smartphone using TextBoxes, put in the parsing logic, and I was set. On the Outlook side (or Outlook Web), I just open up the task called "My Notes" and there are all my notes, delimited by the same string. I can edit them, save, and the changes are propagated.

Only two wrinkles showed up.

The first is that Outlook 2007 is lazy about syncing the Tasks folder with Exchange, at least when it's set up to connect over HTTP. I cut my app out of the loop completely when testing this, by running Outlook 2007 in one window, and Outlook Web Client in another window. Web client read and committed changes to tasks immediately, while in order to refresh tasks in Outlook, even restarting Outlook did not always work. So, bummer... for now, it's an FYI.

The other interesting bit was the syncing logic in the phone client. The goal was to have the data saved and restored (when updated from the Exchange side) as automagically as possible. I've commented the relevant code, most of which lives in the Notebarn_Activated event handler, so you can take a look at the source if you like.

Any time you switch away from the app, or edit a note in full screen mode, your notes are saved, and any time you switch back, if Exchange has updated the Task data but you haven't, your notes are reloaded from the Task. I've tested various approaches and this one has worked the best for me.

I hope it works for you too. The project page (with source) is here.

When "All-You-Can-Eat" is the Wrong Plan: a Plea for Metered-Rate Software

Why can’t I license productivity software on a metered-rate basis?

I had a mini-project that required vector-art manipulation. The assets I was using were Illustrator files, and I felt Illustrator would be the best tool for the job. But I didn’t have a license for Illustrator. I’ve always wanted Illustrator, but not enough to lay out the big bucks … after all, I would only fire it up a couple of times a month. Ditto for things like Photoshop, QuarkXPress, Premier and lots of other neat tools. If only I could install them and license them on a usage basis!

Without this sort of licensing, the options include:

• Use some substitute app which might not be as good, as useful, or industry standard
• Obtain a cracked/pirated copy of Illustrator
• Find someone with a licensed copy and go use his
• Install a new trial copy into a virtual machine (kinda gray-market, and a bad option for heavyweight apps anyway)

With metered licensing, here are potential advantages:

• I get to use Illustrator, becoming (hopefully) more productive
• My company benefits from my productivity and from lower licensing costs than if I needed to buy a full license
• Adobe benefits because I develop my skills on their app, strengthening its hold in the market, and I don’t go install my 90-day trial of this
• Adobe receives revenue from me where otherwise they would not

The metered approach is a total no-brainer: I can have all the apps I want installed permanently (not 30-day trials) on all my computers. At a cost of few cents or a couple dollars, it’s easy to pay as I go… and if start using the app heavily, the bills add up to the point that it’s clearly cheaper to buy. Maybe the publisher is even nice enough to apply some rent-to-own logic, so once I’ve dropped a few hundred buck, I am offered a discount to buy a permanent license.

It makes no sense to charge the same amount for Premier or Final Cut Studio regardless of whether the customer only edits a couple of videos a year, or is a professional editor who makes a living with it. One approach is for a publisher to create lots of SKUs at different price points for different user levels (Visual Studio Express Editions, Standard, Professional, Team System, Tools for Office) but that puts a huge burden on the publisher. Just let the users install the Ferrari version, and charge ‘em for what they do with it!

The infrastructure is already there — most of these apps are available on some kind of trial basis, and the entire install can be downloaded from the publisher’s site.

The per-copy tracking infrastructure is there for many publishers — activation codes are commonplace for expensive software.

Micropayments? Aggregate the charges on my “Microsoft” or “Adobe” or cross-industry-pay-per-use account, and bill me every quarter or whenever the total is high enough to justify it.

Tracking my activity in the app? That’s easy enough (LexisNexis has done this for a long time). Don’t charge me per minute or session, that encourages me to shut down the app, and makes it really expensive for a newbie to learn. Charge by use case: if I do something that every freeware clone also does, it’s cheap… but if I’m hitting the killer differentiating feature, whatever that is, make me pay.

This is free money for software publishers. Folks who absolutely need the app will already have it (hopefully legally), while pirates just bittorrent a crack. The metered system works for all of us in between, and I think that’s orders of magnitude more people than the publishers even imagine.

O'Reilly Web 2.0 Expo: Thoroughly Unimpressive == Very Encouraging

I spent some time at the Web 2.0 Expo yesterday, and as tech shows go, it was pretty weak:

• The show floor was obviously, painfully small, even for a one of the smallest rooms in Moscone
• Some of the BigCo sponsors seemed detached and irrelevant
• The "food and libations" where hard to find, hard to pry away from vendors, and generally in short supply
• The genuine web 2.0 plays seemed like they weren't sure why they were here; they weren't adept at engaging those of us who wanted to learn stuff that's not obvious from 5 minutes on the website
• The tooling companies got a ton of attention, and were not prepared to leverage the attention... I went back to one tooling booth no less than four times to try and get a demo and conversation, but they were in long one-on-ones and had no bandwidth and no "backup chatters" (the ones who typically try to engage you at a show, triage you, and then if you seem important go the extra mile to find you the Chief Architect or the VP of Foobar to talk to)
• The companies trying to sell services (consultants) have not figured out how to get attention and relevance

Wow. That sounds real bad. But I was elated, actually, and not because I'm working on a secret new thing and I wanted to feel superior to some competitors.

Instead, I realized that web 2.0 and the traditional tech conference are at odds conceptually. They don't make sense together, and the fact that this was obvious so fast means the web 2.0 ecosystem has not in fact been corrupted by so much froth that it starts looking like 1999.

I was really thrilled to see that by this yardstick there's a whole lot more runway for web 2.0 innovation before this plane floats up and away.

Lemme run through that list another time and tell you exactly what I liked:

• Small show floow? Who needs a show floor? Web 2.0 exists on the -- gasp -- web! It's good to talk to the humans at these companies, but booth square footage is unimportant.
• Big corps seemed irrelevant? well, many of them are irrelevant ... a web 2.0 meme is they became irrelevant as this wave of innovation took hold: Pricey license to get started? Whatever. Enterprise web stack with more classes in it than my app has lines of code? (I'm not exaggerating, this is a true story!) Get real. Proprietary SDK with a restrictive license agreement? No thanks, buddy.
• Not enough free food and drink? This means there isn't hot and cold running money just yet, and the folks who absolutely must have hors d'oevres aren't there. Web 2.0 is more about Promax bars and BevMo (or Costco) than about catering.
• Real web 2.0 plays seem flatfooted in the "traditional conference setting" -- of course they do. The traditional tech conference is really a marketing-oriented event. These companies were on the cluetrain from the beginning -- their marketing is via their users, and their web forums and wikis (with employee participation) are the best part of a tradeshow everyday, without the nonsense. Their schwag is the free service you can use right off their website.
• Tooling is going to be increasingly important, and the best tooling companies will get their stuff together as the market matures. I think these companies got caught a little by surprise, and still have to work out a way to charge enough to stay in business, without charging too much to... stay in business.
• Lastly, the consultants. I believe there is a place for consulting in the new web world (after all, I work for a consulting firm) but the traditional pitch is going to have to change. There are a number of themes I think make real down-to-earth sense around web 2.0 consulting, but they'll have to wait for another post.

When JavaOne turned the corner (in a bad way) and started looking like this show, it was clear the excitement was overwith. Java is an enterprise-style ecosystem. (Not that you can't build small agile things in Java if you really want, but seriously, do you really want?) It thrived on enterprise-scale players making huge investments with money, hardware, teams, code. Tons of mid-size software shops flourished, selling apps that run $100 up to $800 a seat, and there was money in it because it helped build even bigger enterprise systems. Lots of money, lots of free beer.

When JavaOne went small, it was a symptom of a serious illness. But then the virus took up a home in web 2.0 like some kind of endosymbiont and here we are.

Out of the Naming Shadows

When I saw the link to Silverlight (the artist formerly known as WPF/e), I was thrilled that Microsoft is apparently starting to move toward more compelling product names. Remember the firestorm maybe 20 months ago (publicized by Scoble, though he was really just the messenger) over “Why can’t Microsoft products have cool names?” … Folks talked about the constant confusion over .net being followed by even more boring names like “Windows Presentation Foundation” and asked why “Sparkle” had to become “Expression Interactive Designer.”

At first it seemed that the Microsoft branding group got defensive, citing trademark issues, cultural sensitivity, etc. as reasons to steer away from stronger names. But now it looks like they’ve come around: first we found out that Expression Interactive Designer would be called “Blend” – this doesn’t redline my engine, but it’s definitely a step up.

Now “WPF/e = Windows Presentation Foundation Everywhere” has the much sexier name “Silverlight” which makes me want to go grab a wizard’s cloak and conjure something. That’s a pretty encouraging reaction for a technology like this one. Besides the fantasy association, there’s also the sound of “silver screen” cinematics or a spotlight, which works for this product as well.

GrandCentral Heralds VoIP Apps (Fer Real This Time)

I've been using GrandCentral and Gizmo for about a month now, and I recommend everyone check them out. Why Gizmo and not Skype? mainly, because GrandCentral integrates directly to Gizmo without going out to a "PSTN call in number" and back.

GrandCentral is rockin'! OK, their "Beta" is a genuine beta -- I've had some calls drop, calls not go through, voicemail never pick up... they have some bugs in the software and some issues in ops. But it's still a fabulous service. And when something does go awry there's a real live human on the other end of a chat widget. She's helped me a couple of times and is a real pro. Not to mention the longer the beta goes on, the longer you can enjoy the full power of GrandCentral, including unlimited calling into the PSTN, for free!

The "one phone number to rule them all" concept has been around a long time. The question is when to stop being a skeptic and when to start believing. Photorealistic graphics, good color printing, network-based apps with rich interactivity... all were promised several times and several years before becoming everyday reality. I'm not perfect at figuring out just when "this time around" becomes the time it really works. If I were, I'd be a VC instead of a developer. But I believe the time is now for VoIP and advanced apps for VoIP. Why? Besides Skype, and corporate adoption of VoIP, I think when Verizon sues Vonage over questionable patents in order to put the brakes on, and people actually care, then the time has come. Just like mobile apps had clearly crossed the chasm when everyone worried about NTP shutting off their Blackberries.

I tried to go "all in" by making the GrandCentral number the only one on my business card (due to a clerical error it didn't completely work). With my desk and my cell phones both ringing for business calls a client or coworker would be guaranteed to find me whether I'm in a conference room, at my desk, or out for a coffee -- plus I wouldn't need to check my office voicemail remotely, or force a client to deal with "I got voicemail... I'd really love to get feedback on this issue now, but is this really important enough to call his cell?"

For me, the remaining question isn't whether these services can be hits. They can. The question is how to manage an enormous number of communications accounts each with its own fees. I have a landline (originally for DSL, now because it's still cheaper for local by nearly an order of magnitude than cell or VoIP calls); a cell phone with minutes and a seperate data high-end data plan; a Gizmo phone that offers some free calls, but mostly charges 1.9 cents per minute (if you're keeping score, that's lower than a cell phone but higher than a landline); and GrandCentral, which is free in beta but plans to charge for connections into the PSTN. And this doesn't even include the cable modem account that gets IP to my house.

Each of these services plays a specific role and they all dovetail together to keep me connected to pretty much whatever I want. But there has to be some way to simplify this picture for mass adoption. Unfortunately, the only players in a position to do this are large telcos or cable companies, and they aren't known for innovation or customer service. In the Bay Area, for example, Comcast -- which offers "bundles" of cable Internet, TV, and VoIP -- bought radio time to hawk photo sharing as their great new thing. This is San Francisco in 2007, for godsakes, gimme a break. I'm not taking sides with AT&T either, who would also probably be happy to take $200 a month from me to sell me around half of what I need.

The other question is how number portability applies to the new services, in case this time around turns out to be more like the last time around.

ASP.net DB-backed Micro CMS in 50 Lines and 5 Minutes

I was working on a small web app where one area was intended to allow users to publish their own content. This area was not going to be ready for a while. In the meantime, I thought, there should be some way for users to put pages up on the site if they are inspired to do so. Images, too.

There are great lightweight CMS apps just for this, such as wikis. But with asp.net and SubSonic, I put this micro-CMS up in about 5 minutes. If you need a little area with WYSIWYG editable web content and binary uploads (images, PDFs, etc.) you might find it useful. n.b.: for my site, the created pages are meant to be be publicly readable and writable. If you want to restrict access, you will need to make some adjustments for that purpose. So here's how to brew it up:

Step 1: Create a table in SQL Server 2005

CREATE TABLE DataObject (
 [ID] [uniqueidentifier] NOT NULL CONSTRAINT [DF_DataObject_id]  DEFAULT (newid()),
 [TextContent] [ntext] NULL,
 [ImageContent] [image] NULL,
 [MimeType] [nvarchar](50) NULL,
 [CreatedOn] [datetime] NULL,
 [CreatedBy] [nvarchar](50) NULL,
 [ModifiedOn] [datetime] NULL,
 [ModifiedBy] [nvarchar](50) NULL,
 [IsDeleted] [bit] NULL CONSTRAINT [DF_DataObject_IsDeleted]  DEFAULT ((0)),
CONSTRAINT [PK_DataObject] PRIMARY KEY CLUSTERED ( [ID] ASC ) )

You'll notice a number of convention-over-configuration moves that let SubSonic do some of my work: Created/Modified On and By as well as IsDeleted.

The main data fields are TextContent as ntext (i18n clob), ImageContent (blob), and MimeType. The PK and ID is an autogenerated GUID.

Step 2: Create a web page for editing and uploading content. Grab FreeTextBox for WYSIWYG HTML editing. Then toss in controls:

    <FTB:FreeTextBox ID="FreeTextBox1" runat="Server" Width="600px" />
    <asp:Button id="SavePageButton" runat="server" Text="Save" 
OnClick="SavePageButton_Click"/>
    <asp:FileUpload ID="FileUpload1" runat="server" />
    <asp:Button id="Upload" runat="server" Text="Upload" 
OnClick="Upload_Click" />
    <asp:Label ID="LabelForNewURL" runat="server">(none)</asp:Label>
    

Add line breaks and formatting to taste.

Step 3: Add some code-behind to save the page.

    protected void SavePageButton_Click(object sender, EventArgs e)
    {
        object id = Session[Globals.PAGE_EDIT_ID_CONSTANT]; // add some code in
           // Page_Load to grab this out of the request, 
           // and load it into the FreeTextBox instance
        DataObject o;
        if (id == null)
            o = new DataObject();
        else
            o = new DataObject(id); // this is SubSonic foo... 
               //read all about SubSonic, which rocks!
        o.MimeType = "text/html";
        o.TextContent = MyHtmlUtilities.CleanUpHTML(FreeTextBox1.Xhtml); 
           // Remove anything we don't want, like scripts
        o.Save(User.Identity.Name);
        LabelForNewURL.Text = "URL is http://www.mysite.com/Object.ashx?id=" 
           + o.ID;
        Session[Globals.PAGE_EDIT_ID_CONSTANT] = o.ID;
    }
    

Add some more code-behind to save an uploaded file.

    protected void Upload_Click(object sender, EventArgs e)
    {        
        if (IsPostBack && FileUpload1.HasFile &&
            FileUpload1.PostedFile.ContentLength < MAX_LENGTH)
        {
            String fileExtension = System.IO.Path.GetExtension
                                     (FileUpload1.FileName).ToLower();
            if (allowedFileTypes.ContainsKey(fileExtension))
            {
                DataObject o = new DataObject();
                o.MimeType = allowedFileTypes[fileExtension];
                o.ImageContent = FileUpload1.FileBytes;
                o.Save(User.Identity.Name);
                LabelForNewURL.Text 
                  = "URL is http://www.mysite.com/Object.ashx?id=" + o.ID;
            }
        }
    }

This code omits the definition of MAX_LENGTH, a limit to the length of uploaded files, and allowedFileTypes, a Dictionary<string, string> that maps allowed file extensions to MIME types, like so: allowedFileTypes[".png"] = "image/png";

Now you have file-upload, database storage, and WYSIWYG editing in around 20 lines of code and 5 markup tags!

Step 4: Create the Object.ashx handler that serves these pages and objects. Add a new asp.net request handler like so:

<%@ WebHandler Language="C#" Class="Object" %>

using System;
using System.Web;

public class Object : IHttpHandler {

    public void ProcessRequest (HttpContext context) {
        DataObject obj = new DataObject(context.Request["id"]);
        if (string.IsNullOrEmpty(obj.MimeType))
            context.Response.ContentType = "text/plain";
        else
            context.Response.ContentType = obj.MimeType;
        
        if (!string.IsNullOrEmpty(obj.TextContent))
            context.Response.Write(obj.TextContent);
        else
            context.Response.BinaryWrite(obj.ImageContent);
    }
}

That's it! Of course if you wanted to build in more features here, it's easy to do all sorts of stuff, from adding an attribute for a document/file name, to sorting or searching the database table to generate product thumbnails, automagic menus or site maps, etc. Ok, including all the SQL, asp.net markup, and the C# (except braces) that's at least 51 lines.

Addendum: I had to monkey with line breaks to keep the code readable outside of RSS readers, so never mind the line counts, I think the code is fun and the point is made anyway.

Mobile RIAs with FlashLite - Evaluation

Flash Lite is a promising route to RIAs on mobile devices. Adobe has recently strengthened their commitment to reaching mobiles, including strong device support in CS3, and planning for 1 Billion Flash-Enabled Handsets. The chart in that post also suggests 2007 is the year when "Adobe addressable handsets" clearly cross the 50% mark. Commitment to move the mobile platform toward a Flex-capable and ActionScript-3-capable runtime has been announced as well.

Wow. That's great. So how well and how easily does it actually work? I set out to do a quick evaluation of getting an interactive network connected app up and running on Flash Lite.

For my target platform, I picked Flash Lite 2.1, 320x240, on Windows Mobile, for these reasons:

1. Since this spec lies toward the high end of the Flash Lite ecosystem, I knew that if major things failed on FL2.1 and a 200MHz+ processor, they would be very unlikely to work on downversion FL

2. Real HW-level emulators are freely available for Windows Mobile, something that's not true for many mobile phones

3. I own a compatible device, and when working with devices, at a certain point there's just no substitute for the real hardware

For my "application," I wanted to see if I could implement a couple of use cases for a mobile 2.0 site for ordering food. This seemed like a reasonable real-world use for a Flash Lite application outside of entertainment; the use cases involve retrieving an order list from the web site, letting the user choose one, and then retrieving relevant destination restaurants where the order could be sent.

First I sketched out a simple Flash app which used the MX components for the onscreen text boxes, lists, and buttons, as well as for the SOAP web service client. I targeted Flash 7 (FL 2.x is essentially Flash 7), and built. After debugging on the PC, I tried it in the
WM 5 Smartphone emulator (best enjoyed splashed over this SDK).

No joy. The movie starts to play, but then seizes up and the standalone Flash Lite player gives this error message: "ActionScript stuck" in both the emulator and real device. The culprit seems to be the mx.controls.List class, which is either too complex for the runtime to handle, or else triggers some low-perf detection code meant to ensure movies stop running altogether before they run badly. My web service connection via mx.services.WebService and SOAP was causing a similar problem.

This wasn't looking good, since those components date back at least to Flash MX 2004, and the hope was that a FlashLite 2.x app could be authored more or less like an MX 2004 app. Googling around, I found that the MX UI components do not, in general, work on Flash Lite 2 (a few do, but the kit as a whole cannot be assumed to work). This leaves the old-school approach of custom writing the UI widgets as part of the movie (not a bad approach for something so small) or finding widgets that would work. Since I wanted to complete this evaluation quickly, I looked for a widget kit for FL, and found Jesse Warden's Shuriken Components (see also his presentation here). For my purposes, this was perfect. Easy to use, and ran without a hitch on FL 2.

Next I needed web service connectivity. By adding this

<webServices>
        <protocols>
            <add name="HttpGet"/>
        </protocols>
</webServices>

to an ASP.net app's web.config, you can turn on (Lo)RESTful behavior in addition to SOAP. (This switch enables not just GET but a simplified XML response format.) Next I used XML.load in ActionScript and gave it the appropriate URL to GET. This worked great on the emulator and device.

Then I added a couple of lines of "business logic" in AS, to allow additional dynamic interactions, so that I could bang on it without worrying about caching in the Windows Mobile HTTP stack or in Cingular's WAP-gateway-proxied network.

So far, so good. These patterns (movie-based or Shuriken UI + XML.load for web services) should work well on the high end of FlashLite (Windows Mobile devices range from about 166 MHz to 520 MHz; I was using a 220MHz OMAP). The next step will be to find some more constrained implementations that still support a the FlashLite "application" content type (e.g. a Nokia S40 device with FL 2.0) and try it out.

Samsung Brings Software Game?

I've started using a Samsung Blackjack recently, and it's a great device. I'm not gonna review it here, because there are tons of solid reviews out there, and whole blogs and fansites just on this product.

I want to talk about the interesting partnership to be observed among the companies bringing this product out.

Samsung brings fabulous hardware to the device, which is pretty much what we expect from Samsung. Cingular brings the marketing and enough data bands (GPRS/EDGE/UMTS/HSDPA) to allow limited simultaneous voice and data, which is a genuine differentiator and makes an impression the first time you receive and respond to an email while you're on a voice call. Microsoft brings WinMo 5 Smartphone AKU 3+.

Samsung, though, has gone beyond the usual OEM manufacturer role. In addition to the promotional web site and the basic drivers for the hardware, they've bundled a set of apps carefully chosen to complement the Windows Smartphone OS and improve the value and usability of the phone.

User feedback has identified a number of weaknesses in Smartphone, some of which are addressed now in WinMo 6, some not. For example, in the past, users have screamed that the core app set is missing some basic PIM apps like a notepad. Never mind that you can get notepads online -- that's a step beyond a lot of consumers. So Samsung hops in with a mini productivity suite to fill in the holes: an enhanced filesystem browser, a notepad, calculator, and bunch of other stuff.

The stock Smartphone home screen has taken a lot of heat, so Samsung codes up a handful of alternates with different information densities and usability features. Need to read PDFs or Office docs? Samsung has licensed and supplied a version of Picsel Viewer -- frankly the video quality and smooth experience on this product is better than the Smartphone shell; maybe Microsoft should buy those guys!

The partners seem to be bringing their A game to the overall ecosystem around this device too: according to CrunchGear, Cingular will unlock this GSM device for free; Microsoft is working with Cingular to release an OS upgrade to WinMo 6 this year; and Cingular jumped all over early battery complaints with a pretty straightforward process for snagging a free extended battery.

Without being overly optimistic and wondering if a major shift is happening in the smartphone world, why does everyone seem to be trying so much harder than usual in this particular collaboration?

Why Does Software Suck? Why Do You Think?

My friend Andy over at Security Retentive sent me this link to Cigital discussing a twist on the meme of "software does snazzy stuff but it's built like crap."

I want to add two words to the "software sucks" discussion: organizational dynamics. If you start every conversation about software quality, process, security, predictability, etc. with these two words, you'll be on the right track.

Look at technology as just one competency of an organization, then zoom out and look at the whole organization. Now ask "do I believe that in an effort demanding some amount of precision, control, and predictability, this organization can execute?" With many organizations it's immediately obvious that the answer is "no."

If that question seems too vague or unfair, then zoom back in and take a look at specific capabilities inside the company. Look at product management. Look at marketing. Look at HR. Can you keep a straight face while they talk their talk?

Put it this way: think of a company where you have observed "software is crap" in action, whether it's a software product they sell, some line-of-business software they rely on (managing their type of widget or service), or a horizontal product they use to deal with the world (CRM, accounting). Now think of their customer service, or marketing or another department. If you can't take their marketing seriously, or they preach "great customer service" while delivering something else (maybe of great value, but in any case accompanied by awful customer service), then as an organization they are fostering cognitive dissonance at a group level.

The ability to tolerate cognitive dissonance is necessary for most organizations in order to allow flexibility and to prevent entropy from creating constant crises. But more than the "magic amount" of such tolerance just translates to being too comfortable with corporate doublespeak and BS.

In any case, dissonance and software is a bad thing because the machines that run software are remarkably anal. Compilers are adamant about not tolerating dissonance in the scope of a specific application. You can slip some shenanigans in when you're working in a scripting environment, but eventually the gap will manifest somewhere. And when an "enterprise app" is made up of various subsystems and databases, organizational cognitive dissonance leads to "HAL Syndrome" -- the ailment of the computer in 2001 which eventually starts doing undesirable things because it has been secretly given two sets of logically incompatible inputs.

Basically, a necessary (not sufficient) condition for software not to suck is that the organization producing it must have the capability to know when it is being hypocritical or producing conflicting messages, to recognize there is a problem, and to set some priorities which can resolve organizational discontinuities. Without that self-reflective and real self-modifying capacity, the software output may sometimes work, but has no chance of being predictable, reliable, high quality.

NetBeans 6 m8 and Ruby: Get Yours Now

As of yesterday, NetBeans 6.0 milestone release m8 is out.

Having known NetBeans since its very early days, I'm surprised and excited that it seems to be a serious come-from-behind competitor in the contest of world-class IDEs.

Besides the core Java IDE elements, the platform is being pushed in a bunch of ways. My favorite at the moment is the Ruby support (which works with JRuby or the regular Ruby interpreter). The team has feverishly added a bunch of debugging features over the last few weeks, and the latest Ruby-support builds snap in nicely with the m8 release. (Some of the recent IDE builds didn't snap together quite so well.)

Here's a quick screenshot showing debugging stopped at a breakpoint, with all the goodies you'd expect -- stack trace, local variables, etc.



Fun stuff! At least for IDE junkies like me.

I had been watching to see where Ruby IDEs would come out... the "free Ruby IDEs" are, well, no offense, but they're awful compared to stuff like Eclipse and Visual Studio. Arachno Ruby is better, but I didn't love it when I tried it.

I've been using a trial of the SapphireSteel's Ruby In Steel, which is built on Visual Studio. Ruby In Steel is amazing, and I consider it the first really world class Ruby IDE. But it requires VS2005 Standard edition or above, which means money and a Windows box, both of which I believe will be impediments to longterm success. The money thing is sad, because the SapphireSteel guys seem wicked smart, and it's a bummer that we're living in an age where they probably can't/won't get away charging $200 or $300 per seat for this package. But with Eclipse, NetBeans, and Sun Java Studio Creator being free, and Microsoft's Empower prorgam giving away the farm (a.k.a. MSDN) for under $400 to legitimate startups, it's rough.

This NetBeans add-on is at the head of the pack. It's a really nice piece of work and clearly (read the changelogs) there's a ton of energy going into it. Nice work!

Another ASP.NET Adaptive Rendering Tip

For the background to this adaptive rendering thread, you may want refer to this previous post.

It turns out that in .net 2.0, the base "Mozilla" rendering profile sets the cookies capability to false. Why would it do this? Well, specific mozilla-derived browser profiles turn the setting back on, so it's not like asp.net thinks Firefox won't support cookies. Here's a clue, in the comments at the top of the mozilla.browser config file, showing some sample user agent strings that would match:

<!-- sample UA "Go.Web/1.1 (UP.Browser/3.1-UPG1 UP.Link/3.2; Mozilla/1.0; RIM957; Elaine/1.0 )" -->
<!-- sample UA "Mozilla/1.0[en]; Go.Web/1.1 (compatible; MSIE 2.0; HandHttp 1.1; Palm)" -->

Some old school mobile browsers were the targets here. Unfortunately, if users come to your site on, say, a Samsung Blackjack or another new device running WM5 and PIE, they end up in the same bucket. You can sniff this out by logging Request.UserAgent, Request.Browser.Browser, and Request.Browser.Type. I highly recommend doing this at least on one "entry" page for any mobile site. You may also want to log the computed capability for an attribute your app needs, like cookies (Request.Browser.Cookies).

Even though asp.net will automagically try to do cookieless sessions, I want to force cookie-based authentication on my mobile site so that users don't have to sign in every time they visit (those extra key presses are brutal on a phone). I know that more or less every mobile browser out there can support cookies now, so I'm not too worried about this decision.

The fix? Similar to the others, we'll just override this one attribute in our supplement.browser file:

<browser refID="Mozilla">
  <capabilities>
    <capability name="cookies" value="true" />
  </capabilities>
</browser>

Another free bonus is that with cookies, asp.net doesn't have to do the URL mangling that is involved in cookieless techniques. So your users get URLs that make more sense in their history and work properly as bookmarks!

Yahoo Mail API Misses the Point

Within the last day, Yahoo has officially released the web service API for mail they demo'd at Hack Day last year. Earlier in the week, mailbox storage limits were removed. Everyone seems to be cheering, and talking about the obligatory Flickr mashup.

Before we get too drunk on champagne, let's take a look at the details. The API only allows retrieval of the mail message body for Yahoo Mail premium subscribers. This correlates to their policy of allowing POP/IMAP access only for premium subscribers too. They are missing the whole point of an open API here. There's only so much I can build if I know that Yahoo mail basic users (i.e., most people) will have degraded functionality.

The Yahoo motivation is, of course, to drive people to the web site to get their mail, where they can be exposed to ads, which keep Yahoo in business providing free mail. Ok, cool. But why not use this opportunity to explore some alternatives that keep the revenue coming, but allow Yahoo to really open the data service up, instead of having this silly $20 tax (i.e. "premium") on people too lazy to switch to a more open free service? (User inertia is a dangerous business model because it exhibits "tipping point behavior" -- one day, everyone's too lazy to switch and the next day all your best users are gone.)

So what are the more imaginative approaches Yahoo could have taken? Well, for one thing they could continue to insert text ads, or even image ads into mail sent or retrieved. They could specify terms of service that require another app (say a mail client or another web site) to display certain ad items when using Y! Mail content, or else risk losing developer key access.

They could innovate further by creating a loyalty program across Yahoo properties that informs whether a user gets certain service levels: for example, if I have lots of My Yahoo page views, or I click a lot of ads there, or I buy a ton of stuff from Yahoo Stores businesses, shouldn't that count for something? Certainly this sort of user is different from a freeloader who hypothetically never views a Yahoo web page, and just sucks his mail down into Outlook. How about if I have a full demographic profile in Yahoo, and I leave myself logged in with a persistent cookie, and I use Yahoo search a bunch ... thereby providing Yahoo a wealth of data that should raise the value of ads shown to me?

C'mon guys ... if you don't do it, someone else will.

Opera Mini as Abstraction Layer

Opera Mini is a free J2ME-based browser from the folks at Opera. This FAQ tells you how it works and this demo lets you try it. I've spent some time using it on my old Blackberry 7250 and it's a pretty impressive browser &em; far more capable than the built-in browser on the Blackberry (although that's not saying a whole lot). How does it work its magic? The J2ME client handles user I/O and screen rendering (including a fabulous small antialiased font that is built into Opera); the page rendering, on the other hand, is offloaded to Opera's servers.

Since latency is the bane of even the fastest 3G networks, having extra proxying in the path makes the actual go-to-page browsing way slower than usual for a mobile. On the other hand, you have that warm feeling that practically any page, even moderately AJAX-y ones, will actually render attractively and usably when they do show up. Contrast this to the Blackberry browser experience, where the data starts coming down very quickly, but the device spend 30 seconds thinking about how to render it, and then ultimately produces something barely usable. Since subsequent navigations seem to perform better than initial ones, I suspect the Opera server is chasing forward links and preparing them while you read the first page. Not a bad plan.

I started wondering: Would Opera Mini serve well as a "base platform" abstraction layer for mobile web apps? If I code to O.M., and I can expect to be able to run unmodified on handsets all over the world, there's significant value in that.

What are the costs of such an approach? Hmm... let's see...

• Like all abstraction layers (Win32/POSIX/.net/...), the end user has to get it up and running. Sounds easy, but ask anyone who's ever wanted to deploy a "lightweight" .net or Java app and assumed users would already have the runtime ready to go...


• Opera has a done a lot of heavy lifting, but what if they change their browser capabilities, and now there's another fragmented set of clients, just a layer up the stack?


• Like all abstraction layers, there's a performance hit. In this case it's twofold: the memory and CPU usage of the Java runtime hits some devices pretty hard, and the proxy-through-Opera network access is painful even on EV-DO
  

In the end I think I'll take the weaselly way out for now. I wouldn't go all in, build an app, and say to the user "Opera Mini is a pre-requisite for this app. If you get it, you're golden; if not, no support for you!" I'd target a base of browser set with some kind of adaptive rendering. But I might well make Opera Mini the first-tier tech support response for any native browser problems.

The Eagle has Landed at 601 Townsend St.

I spent last night at Adobe's Apollo Camp. The Apollo Camp was about exposing a group of developers to the (almost) latest build of Apollo; introducing some partner apps and the insights and clever tricks these apps have already spawned; and letting us meet, chat, question, and occasionally argue with the guys who build Flash, Flex, and Apollo.

Since this is a commentary blog, I'll let the tech news blogs cover all the great content that was presented last night. Much of it is probably in the blogosphere by now; a lot more is due to be released by Adobe via Labs very soon.

Apollo is going to be an important and prominent platform. It's not perfect (Adobe certainly doesn't claim it is right now either), and the truth is it doesn't actually do all that much. But it is a kind of fabulous connective glue for the front end. Metaphorically it reminds me of OLE/COM/ActiveX in its desktop ambitions. It's not defined to be a desktop object bus per se, but it's way easier to use, and it's cross platform. It is also reminiscent of web services and RSS in the middleware realm, in that it provides a connective mechanism simple enough but expressive enough to throw the door open for all sorts of creative mixing and mashing and integrating fun.

It's interesting that we're at this juncture with Adobe/Macromedia poised to leap into the RIA lead with this tech. Credit is due to some long-range strategic thinking. For example, we were told that Flash player has for years now been designed as an express install vector for Apollo. And moving to from AS2 to AS3 reminds me of the bold but necessary move from VB6 to VB.net that Microsoft took in '01.

More than a little is owed to luck, or lucky timing, as well: without the evolution of Agile and TDD, leading to effective development practices with dynamic languages (and popular day-to-day AJAX, Ruby, and Flash apps as a result), it would be hard to imagine a bright future for desktop apps built on ActionScript. But these things have come to pass, and AS3/Flash9/Flex/Apollo/Tamarin has shown up at just the right place, and just the right time.

Making Mobile Browsers Work Better with ASP.NET

ASP.NET is architected with a robust scheme for adaptive page rendering. This scheme allows "server controls" or metamarkup to be rendered appropriately to various browsers, and becomes particularly important when using the mobile web controls.

For an overview of how this works, take a look at ASP.NET Web Server Controls and Browser Capabilities (from MSDN/Visual Studio '05 docs)

There are also numreous ways of overriding this framework, from code (Page.ClientTarget) to XML config files, to defining new browsers, or altering attributes of the browser configurations that ship with the framework.

Modification is critical for mobile web development, because the device data that ships with ASP.NET 2.0 doesn't include any of the wireless devices shipped in the past two years or so. (For a list of what is included, check out this list and walk down memory lane)

Since most users of mobile applications will be carrying a device built in the last 2 years the built-in browser profiles will not cover current customer devices. The good news is that the browsers in these newer devices are more advanced than the earlier browsers, and make a credible attempt to process xhtml and javascript. So less divergence from a "basic desktop browser" profile is needed to address these clients.

Here is what I did to handle some problems with newer Motorola and Nokia devices:

If the framework cannot come up with a positive match for the User Agent string to either an individual browser definition or a family of browsers, it is bound to the "default" profile and identified by the name "Unknown"

All major desktop browsers are correctly identified, whereas many (maybe most) late-model mobile browsers are not identified. So if the browser is identified as "Unknown", we'll assume it's a new mobile device.

The framework will render for a base desktop browser, which works pretty well. But certain postbacks cause a processing error which can be circumvented by altering the "default" profile and adding the attribute "requiresPostRedirectionHandling" right into the default definition. Rather than editing the files that ship with ASP.NET (which may be impossible in a shared hosting environment anyway), the preferred approach is:

1. Add the special "App_Browsers" folder to your ASP.NET 2.0 app
2. Add .browser file to that folder -- mine is called Supplement.browser
3. Add the attribute into an XML element that specifies this should modify the existing default definition. Here's how the whole file looks:

<browsers>
 <browser refID="Default">
  <capabilities>
   <capability name="requiresPostRedirectionHandling" value="true" />
  </capabilities>
 </browser>
</browsers>

The refID attribute refers to the Default definition that you can find in [WINDIR]\Microsoft.NET\Framework\[VERSION]\CONFIG\Browsers

For my application, this fix got my app running great on the MOTO RAZR devices, where I had experienced a number of problems before.

4. I also wanted to target newer Nokia phones. They have a family of browsers which are identified even when the individual phone model is not. So I added this element inside the 'browsers' tag to fix the Nokias:

<browser refID="Nokia">
 <capabilities>
  <capability name="cookies" value="true" />
  <capability name="preferredRenderingMime" value="application/xhtml+xml" />
  <capability name="preferredRenderingType" value="xhtml-basic" />
  <capability name="isColor" value="true" />
  <capability name="requiresPostRedirectionHandling" value="true" />
 </capabilities>   
</browser>

As you can see, I noticed that these devices are xhtml capable, but the Nokia default profile was sending them WAP content. I also pointed out to ASP.NET that these devices take cookies, are all color, and handle redirects differently from desktops.

I'm sure I'll find more issues and more fixes, but these have made a big difference once I figured them out.

Incidentally, save yourself a lot of pain and do not rely on either Motorola's ADK emulator or Cingular's online (ActiveX) emulators to test web content on these devices. Many difficult bugs appeared with these emulators, but on the actual devices, everything worked great. Nokia's emulator/SDK on the other hand was pretty solid and very helpful.

I Have a New 6-lb. Cellphone

I was traveling last week with a pretty decked-out Thinkpad T60p, and decided it would make a great smartphone. It's got a huge screen, a 512MB 400MHz ATI graphics card, 2 GB of RAM, happily runs a few VMs, development tools, databases, and DVD-quality movies all at the same time. It runs 5 hours on a charge while doing that, can idle in sleep mode for days, and stays on the network with integrated WiFi and EV-DO.

Last time I checked, smartphones don't do any of that stuff, and if you continually use them to access data, their batteries will not even last 5 hours.

So crank up Skype or Gizmo Project and it's a heck of a phone.

And yet ... at 6.5 lbs, and a foot across, I'll get some odd looks when I clip it to my belt.

Where am I going with all this? there are two things we need here:

First, a new BIOS-level powersave mode that lets me receive calls -- routed to a Bluetooth headset -- when the laptop is sleeping, maybe even with a satellite UI ... wait, someone had this idea, it's called Vista Sideshow. So I'm asking for a Vista Sideshow VoIP phone gadget.

Second, a more flexible way to link multiple telephony devices to a single phone line. I want my Sideshow PC phone and I want a $49 cellphone that I can carry around when I don't want to carry the PC -- and I want them on the same number. And I'd like to link my Blackberry onto that number too, so I have my choice of high productivity, high voice quality, or a compromise. I'll bring the hardware and the subscriptions. Wait! My second wish is already coming true. It's called GrandCentral.

Now let me get this all rigged up and I'll report back.

BitTorrent's paid movie rentals are just silly

Not the concept. But these offerings are so predictable, and so predictably bad, in their pricing that each one is just another big delay before a service can finally be produced that offers real value to consumers, studios, content creators, and publishers/distributors.

Here are a couple of examples. First up, this Bittorrent service: from them, for $4, I get a 24-hour viewing period on a media file playable in some devices, with Windows Media Player.

Now, behind door #2, I have a video store where for a little over $3, I can get a physical disk that will play in far more devices, for a period of several days, most likely with better video and sound quality.

Maybe this demo (males 15-35) is too cool to go to a video store? I doubt it, but you also have Netflix. A conservative calculation (cycling 2 disks per week on a 3-at-a-time sub) yields about $2 per rental. Plus I can keep the disk as long as I like, and I have the luxury to not watch it in 24 hours if I’m busy. Although I do pay another convenience cost in that I am dealing with the Netflix queue, not an on-demand selection, this cost is essentially paid to Netflix; that is, it is a virtual subsidy of the Netflix operational model. The studios get no benefit from that at all, since the relative physical scarcity of disks is not in their model (they fix the original disk price and press as many as they can sell), but only enters into Netflix’ model (Netflix can only reasonably acquire, use, and then dispose of a modest number of disks for each film).

There’s also on-demand films from satellite/cable. Cost: $4. Terms of use? basically old school TV on VHS rules: I can record the show to my TiVo and make suitable personal copies (e.g. with my DVD recorder), which last indefinitely and which I am allowed to watch whenever I want. Downside? NTSC quality video.

Bittorrent is selling me a strictly inferior good at a higher price. Economics says they’re not going to succeed, and the studios will claim it’s because viewers are all crooked.

Not to pick on Bittorrent in particular – without doing a rundown of all of these work-alike online stores, let’s look at one more: Amazon Unbox made headlines for its onerous Terms of Service as well as its implausible pricing. Here are a few couple of typical price matchups all from amazon.com:

The Departed:
Unbox restricted download: $14.99;
actual DVD, widescreen with extras: $15.99;
BluRay high def 1080p disk $23.95

The Devil Wears Prada:
Unbox restricted download: $14.99;
DVD new $13.89;
DVD used ("very good condition") $8.76

Babel:
Download: $14.99;
DVD new: $14.24;
HDDVD or BluRay: $27.95

In some cases there are shipping charges, but there are numerous ways to avoid shipping charges on Amazon. For reference, iTunes new releases are around $12.99 and are also heavily restricted.

The point here is that not only do these ventures refuse to concede that more restrictions on media make it less valuable to the consumer, but they actually imagine that they are somehow innovating in a way that will let them charge more than the baseline cost (physical DVD/CD and accompanying rights are the baseline).

Nothing here suggests that the media should or must be "free" – only that Bittorrent president/cofounder Ashwin Navin and the studios are all yanking our chains when Navin says, "We're really hammering the studios to say, 'Go easy on this audience' ... We need to give them a price that feels like a good value relative to what they were getting for free."

MSFT apologia

Ok, it's not a huge secret that I'm a Microsoft apologist (that is to say defender). Not that Microsoft hasn't made its share of mistakes and done some things wrong. But yesterday a friend, not a developer but a power user, lightheartedly referred to Bill et al. as "software bozos" and I felt obliged to point out a few things...

Microsoft produces great products under an unbelievable set of constraints. Customers want Microsoft stuff to work seamlessly on everything from cell phones to PCs to set-top boxes to web servers to XBox 360s; they want it to make sense to everyone from CEOs to doctors to my mom; they want it to be localized (support local language, culture, currency, calendar, phones) everywhere in the world, and to be accessible to the handicapped and to be secure even when an extremely unsophisticated user tries to do really dumb things.

They also want it to be inexpensive and to work on any cheap hardware you buy off the 'net and install it on (unlike, say, Apple, where the OS is only legal and supported on the hardware they're in the mood to offer this month); oh and besides being a general purpose operating system, customers like it that Windows is one of the most advanced 3D gaming platforms, competing with dedicated gaming consoles that cost just as much to build as a PC and need do nothing except play games...

Oh, and also, unlike pretty much any other OS I'm familiar with, customers (especially business customers) need it to be perpetually backward compatible, so that when they put a new Vista machine together today it'll still run line-of-business apps that were written for DOS 4.01 in the 80s, and somehow magically these old apps will print reports on the new color laser printers attached to the computer, that were never even dreamt of when the apps were written. And mostly this actually works.

Now let's say you live in America and you buy a new/upgrade copy of Windows every 4 years for about $200, and a new copy of Office for about $400. You're paying about $12.50 per month. And you get the security updates, and browser updates, media player, Virtual PC, development tools (if that's your thing) and all kinds of other stuff for free (or included in your $12.50 per month admission price if that's the way you want to think about it.)

I'm not sure there's anything else I pay $12.50/month for that even tries to think about solving problems on this kind of scale, let alone succeeds.

Lastly, someone will be tempted to point out that Microsoft's enormous presence in the client OS and office productivity space may inhibit all kinds of other software ecosystems from flourishing. There are a number of open questions about this. First, it is reasonable to believe that standardization at one level in a stack enables massive innovation at the next level up the stack, which would otherwise have been impractical. This goes for any platform piece -- Ethernet, Windows, *nix, Java, HTTP...

More importantly, do not assume for a minute that the open PC architecture would even exist without the dominating historical presence of Microsoft Windows. The fact that you can even sit down with an assembler and start hacking a boot image and work your way up to running literally whatever you want on a readily available PC has never been a given. Considering the attitudes of more closed OS and hardware makers in other ecosystems (like cell phones), it is entirely possible that without Microsoft and the need for backwards compatibility, just running code on a cheap mass-produced box would long ago have required signed code, a crypto key from some industry licensing group, and more cash for membership and fees than any small company is ever going to have.

Grepping in PowerShell

I originally wrote this for the company wiki the other day, and thought it might be useful to a wider audience. The context is parsing and processing an iTunes library.xml file (just a one-off task), which I thought might a be a fun and educational opportunity to slice, dice, and ... how does that Ron Popeil commercial go? ... with PowerShell.

PowerShell is the new shell for Windows. New, and supported, but not "the official" in the sense that it doesn't ship with Vista, although I'm guessing it will ship in the Longhorn Server rev.

If you're used to Unix shells, then you'll probably be floored by the power of PowerShell and somewhat annoyed by the syntax, which, despite liberal aliases to familiar things like ls takes some getting used to.

.net framework integration means you can easily access any object in the .net base class library, and there are some special tricks that do some of this for you too. The canonical example seems to be this one, a quickie rss reader:


$wc = new-object System.Net.WebClient
$rssdata = [xml]$wc.DownloadString(‘http://foo.bar/rss.xml’)
write-host $rssdata.rss.channel.title
$rssdata.rss.channel.item | foreach { write-host $_.title }


Since the source file is xml, I had thought the XML parsing would come in handy, but it turned out that there was no real data model to the XML. Basically, there is just a big nested map structure (key-value pairs in blocks) in the item list. Sort of XML for the "takes-void*-returns-void*" crowd. So then grep looked promising because the keys and values (and their tags) were grouped on individual lines.

Grepping is a little counterintuitive with PowerShell because the pipeline between commandlets in PowerShell is filled with full-on objects not strings. If you just want text, you can use Get-Content, which provides its output as a bunch of string objects, 1 per line, which is convenient. Here's an example I came up with after struggling a little bit to get a grep type of functionality. I throw a sort and unique on here for fun:


Get-Content Library.xml | ForEach-Object { if ($_ -match [regex]"(?<=Artist\<.{13}).*(?=\<\/)" ) { $matches[0] }} | Sort-Object | Get-Unique | Out-File lib.txt


Many of these things can be abbreviated too, so if you want your script to read a little tighter, you can use


gc Library.xml | % { if ($_ -match [regex]"(?<=Artist\<.{13}).*(?=\<\/)") { $matches[0] }} | sort | unique

Isn't that sweet?

Since the regex uses zero-width lookahead and lookbehind assertions instead of extracting a marked subexpression, I'm curious if anyone has input on whether one approach is faster / better / shinier than the other.

My first guess is that they are similar, since my first cut at implementing lookahead + lookbehind would probably be to match the whole outer expression while naming the non-zero-width-bit in the middle, and assigning the value of that to the expression.

Yipes it's Y! Pipes

Super cool: there is no reason that a human should need to handwrite HTTP/XML/mashup/filtering logic for simple cases. Even with the highest-level toolkit, it still requires time, introduces bugs, needs to be hosted...
Systems like this are about moving toward a declarative specification for extracting semantics from web services (in this case RSS).

This particular implementation is a bit fancy on the graphics, which makes it run slowly, and it seems like it needs to extract data from RSS only. That is, if you try it out, it expects every URL "fetch" result to look like an RSS formatted collection of "somethings" ... which is nice, but it would be cool if you could also process XML from REST queries, or build SOAP queries as well. My first inclination was to ask for some kind of RegEx widget, but perhaps the Y! Pipes team intentionally doesn't want to allow us to go down that route ... over time they want more structure, not less structure in the data. They probably feel like RegEx has already been done in the HTML scraping world, although there is certainly lots more work to do there.

If you are interested in this stuff, check out some other approaches and flavors of this notion too:

- Dapper which tries to build web services on top of any web page as a data source. These guys have a "virtual browser" which lets you point and click your way through existing pages to build a service

- Kapow and OpenKapow -- enterprise and "free online" design tools for scraping, mixing, mashing and republishing the web

- QL2 an "old-school" enterprise software product used for industrial strength scraping, it implements a query language so that you can treat the web data sources that are being used as a virtual database (!) (frighteningly enough for an "unstructured data" query tool, this system is used in some large mission critical apps)

- YubNub: this souped-up version of wget lets you define "commands" (aka abbreviations) for issuing web queries, can substitute parameters, and pipe things together. It's arbitrarily extensible since you can always write a servlet/ashx/&c. to provide any data access or transformation you might want. On the other hand, it's more about plaintext (or human readable anyway) than XML