Thursday, January 31, 2008

Ruby Hack: In-Memory Files with Metadata and Better .open() Support

PDF::Writer is probably the closest thing there is to "the" library for generating PDF docs in Ruby. But there are some other nice, tiny libraries for generating a quick PDF doc, like Ruby FPDF.

The one thing I needed recently from FPDF that it didn't offer was the ability to add an image from a in-memory data blob (the FPDF Image method reads from a file). Making FPDF read a memory stream was a fun bit of Ruby meta-style hackery, which I offer here for streaming images into your PDFs and also as a neat example of how you can usefully and easily change a module's behavior without touching its source code.

The basic plan:

  1. Since FPDF wants to open a file, and I have a String, StringIO is my friend, since it's very file like.
  2. Since StringIO doesn't have file metadata, like a filename, and FPDF looks at the filename to decide how to parse the data, I'm gonna need to make StringIO seem like it has metadata.
  3. Since FPDF uses only the raw, it's looking so a file path. I need to make it a little more cosmopolitan by giving it the in open-uri, which among other things calls open on openable objects. Like StringIO.
  4. Lastly, since behaves like "new," and I want to pass an object to open that already exists, I need to change the way open behaves.

First, let's make FPDF use the enhanced 'open':

FPDF.module_eval { require 'open-uri' }

Now, we'll define a function that takes a String of binary data and a pretend filename, and produces a hacked StringIO object (description of what's going in is in the comments):
def in_memory_file data, filename 
#load up some data
file =

#tell the class that it knows about a "name" property,
#and assign the filename to it
file.class.class_eval { attr_accessor :name } = filename

#FPDF uses the rindex and [] funtions on the "filename",
#so we'll make our in-memory file object act like a filename
#with respect to these functions:
def file.rindex arg
name.rindex arg

#this same pattern could be used to add other metadata
#to the file (e.g., creation time)
def file.[] arg

#change open so that it follows the formal behavior
#of the original (call a block with data, return
#the file-like object, etc.) but alter it so that
#it doesn't create a new instance and can be
#called multiple times (rewind)
def*mode, &block)
self.rewind if block
return self

return file

In this case, I had the FPDF source so I could see exactly what I needed to do. With Ruby, I'm likely to have the source pretty much all the time (unless the classes are pre-compiled to target another VM, e.g. CLR). But an interesting next step is to add hooks/instrumentation to a "black box" library, and use that output to try and make adjustments to the behavior of the library.

Monday, January 28, 2008

Blackjack / WinMo 6 Upgrade: More Baby Steps Toward Smartphone-as-Computer

I upgraded my Samsung Blackjack to Windows Mobile 6 via the new official ROM, and the upgrade pointed out a few things indicative of where the mobile industry is going.

I was actually surprised that Samsung released the upgrade -- it had been promised for the better part of a year without showing up -- rather than just force customers to buy the new Blackjack II. Eventually I realized why they had gone through with it, and why doing a full-version upgrade (not an update) of the OS on your smartphone will become commonplace: This wasn't done for me. It was done for enterprise customers who bought (or committed to buy, or were thinking of buying) hundreds or thousands of the devices.

Consumers -- especially the kind who buy smartphones -- are interested in getting a new phone every year or two. The whole free-with-contract-extension system helps that keep that flowing.

Businesses, however, buy their handsets differently (often via a more complex deal) and account for the costs of maintaining a "fleet" of mobile devices (management, email, VPN support, etc.) differently as well. For a business, cycling the devices every year or two could well be prohibitive because of the overhead, not just the equipment cost itself.

I've never been tasked with executing a smartphone rollout for a large business. But if I were, I would probably want to create a system where the average device had a lifespan from 2 to 4 years: 2 years for the high-level execs and the techies who need more power, up to 4 years for employees who travel but are not particularly sophisticated or needy when it comes to mobile email. Beyond 4 years or so, the hardware gets so outdated it's not worth the legacy hassle; it's time to buy something new.

So a regimen of solid, easy-to-install full-version OS upgrades becomes a sine qua non for device makers (Samsung) and software companies (Microsoft) who want to sell into the 3-4 year lifespan device market. RIM figured this out a long time ago, and have pushed the envelope when it comes to keeping old models (e.g., Blackberry 7250) alive for years, offering not just OS-level upgrades, but software radio protocol upgrades: an early-model 1xRTT 7250 can become a 3G EV-DO rocket with just a firmware update.

Windows Update ... on Mobile 6

Now, with Windows Mobile 6, Microsoft can also deploy updates to our phones via Windows Update. So they can be more proactive than the hardware maker or the carrier when they need to fix a serious bug or security hole, or release a new feature.

Although other devices have provided updates before, the Windows Mobile Update feature is breaking new ground because the OS runs on so many devices made by so many companies ... and on nearly every carrier on earth. The updates had better not brick too many phones; on the other hand this could be a strategic advantage in the smartphone wars.

Other Stuff

There's lots more to say about WinMo 6 and device evolution ... from Internet Sharing to more Bluetooth support to the GPS "intermediate driver" ... and those were covered well when the OS was initially release.

Thursday, January 24, 2008

In Which I Try to Save Someone from a Rails/MySQL BLOB Debugging Hassle

I was recently debugging an app and spent a bunch of time on a MySQL error symptom. I'd like to save the next developer from the same searches, since there are lots of documented MySQL connection issues with Rails, and few (any?) posts mentioning the following particular gotcha:

If you're getting the Lost connection to MySQL error, or its cousin, MySQL server has gone away (depending on whether you're using the Ruby driver or the Ruby-C driver), and you happen to be working with BLOBs (images, file uploads, streaming docs from a DB, etc.), there is a reasonable chance that the problem is you're sending too large a "packet" to MySQL. The "server has gone away" doc mentions this and refers to more detail at the "packet too large" error detail page.

In the current MySQL version, the default max packet size is 1 MB and by packet, they don't mean the TCP packet you might be using under your connection, they mean "a single SQL statement sent to the MySQL server, a single row that is sent to the client, or a binary log event sent from a master replication server to a slave." So it's actually quite easy to hit the default limit if you're working with BLOBs.

The limit can be raised in various ways (command-line, service config, etc.) and the doc points out that raising the limit should not in itself incur a performance penalty, as the additional memory is allocated only when needed.

Before cranking the max_allowed_packet setting up and calling it a night, though, it's also worth a quick sanity check as to whether you really mean to be reading and writing BLOBs this big. If you're storing videos or RAW data from a DSLR, then maybe yes.

In my case, I was using a heuristic for converting and compressing smallish photos and logos. I didn't intend to create a larger image than what the user was uploading (the largest test item was around 250kb), so the 1 MB default limit should have been fine. On closer inspection, a bug in my logic flow was incorrectly deciding to convert some 250K JPEGs into 4MB PNGs. Doh!

Monday, January 21, 2008

What Does Pownce Have to Do With Bogus Internet Movie Rentals?

Maybe my point about Pownce got lost in a larger, broader, and more academic argument. That's fine; I'm glad to see Ted Dziuba (yes, uncov) pretty much made the same point about Pownce. He has a larger audience than I do. My larger point, of course, is that we're not always the hiking-in-redwoods, biking-to-work, inventing-computers-in-the-garage, tossing-whole-industry-paradigms-out-the-window-with-our-wicked-code folks that our narrative says we are. We're as ridiculous, self-absorbed, and self-referential as the folks we take swipes at on our blogs.

What does this have to do with Internet movie rentals? Well, Internet movies -- and VOD in general -- is one of those always-coming-never-here technologies. Even if does arrive, it'll be 10-15+ years into the (U.S.) broadband-penetration era, meaning it will kind of dribble out the exhaust pipe of innovation as an incremental use case long after its revolutionary potential has disappeared. But Netflix (with LG) and Apple are back to the baccarat table for another round of tech's most pleasant losing gamble.

While Apple and Netflix have made little gains, big ol' Safeway is now selling a wad of recent hit films at 2-for-$6. On DVD. With a non-revocable license to watch in perpetuity, carry around, re-encode (if you're "careful" how you do it), and pass on to one's heirs for the next 1000 years. Or you can just rent the films for $0.99 - $1.49. The studios aren't going to hassle Safeway about "$3=watch forever" pricing (and heck maybe Safeway is paying $2 for those DVDs), because it's a retail supermarket. They understand it and it makes sense in their world. The same reason Joe Teenager can sign up with a record club and, if he works his deals optimally, pay a better per-track price than Apple does (maybe 30% of retail).

The house will have the advantage as long as tech keeps thinking like a player (and believing its own mythos) and not like the casino (the other that it tells stories about). Who's the "house" ? ... well, studios, producers, broadcasters, cable networks, telcos, CE makers (guess what, they don't think of 'tech innovation' the way software folks do), even government ... and that's just a start.

I made this point before. Now I'm gonna take it a step further, and talk about one way to move forward.

If you're an innovator in one of these areas, and you need to get a foot in the door, take on another perspective in addition to your own. Try and find a place to live in the other industry's model of the world (their value chain/web/network). I don't mean philosophically, I mean literally. Think of it as hacking their worldview, if it makes you feel better.

Here are some examples:

  • When ecast needed legal clearance to get running with a digital library of songs in 1999, "MP3" and "Internet download" were conversation killers with record labels. ecast needed to be a jukebox, don't worry about the gears and levers inside, nothing to see here, move along. There were some hiccups in licensing until they shifted to the latter approach. Record labels know what a jukebox is and how they make money. Problem solved.
  • Mediabolic ended up taking a different course, but at one point when the firm was working on  VOD with a set-top box, they looked into subleasing excess digital broadcast spectrum time for moving their data. What does that mean? It meant they would have been in some real sense a broadcaster using TV spectrum to reach customers. Which is a whole different conversation with content owners from the "renegade startup conversation."
  • When Skip Interaction focused on travel data management, and mobile transactions, I advocated becoming a travel agency. While Skip would not likely have made much (any?) money on bookings, it changes the conversation with airlines, travel agencies, and companies whose employees travel a lot. Skip would have appeared as a "known" entity in the industry. All of a sudden, instead of requiring custom arrangements to do anything, Skip would have had a huge pile of boilerplate data and access to work from.

These examples are just from my personal experience -- I'm certain that this strategy is not at all uncommon. But we don't hear about it a lot, because it doesn't fit the plotline we want.

Monday, January 14, 2008

Looking For: a Reverse-Search Web Service (aka Semantic Analysis the Easy Way)

For one of my projects, I'm looking for a web service that does search "backwards" to reveal page semantics. If anyone can point me in the right direction, I'm all ears!

What do I mean by backwards? A normal web search service (Google/Yahoo/MSLive API) takes a set of search terms and other conditions and returns the web pages that best match.

A reverse search takes a URL and returns the search terms for which this page scores well. I call this Semantic Analysis the Easy Way because, strictly speaking, it doesn't require actually understanding the content of the page -- yet you can get semantic data out. Of course the better your content analysis engine can understand the content, the better your search engine will work, so the hard problem figures in a little bit too.

The big search engines certainly have the data -- I'm sure it plays a big role in ad-placement mechanisms like AdSense. Just as the search engines expose their search APIs, confident you can't (or won't be allowed to) steal their search results and pretend they're your own indefinitely, they could theoretically expose the reverse search data too. But sadly, I haven't found one yet that does.

I did the requisite scan of the major players, looking through programmableweb, etc. No dice. (Although programmable has a link to an interesting "hard way" semantics service that did a nice job of analyzing the text I threw at it from real pages.) The closest I've come so far is the URL API, which will tell you the top tags associated with a specific URL -- valuable data indeed, but not the same thing.

Little help? Anyone?


Post-structuralism for Dollars in Silicon Valley

On the eve of one of our grand theatrical spectacles, featuring one of our industry's undisputed showmen as emcee, let's pause for a minute and imagine that we've got our mental map of the tech world all wrong. We've been drawing lines not just in the wrong places, but where the model isn't about lines at all. As a result, we're misunderstanding ourselves and a lot of other stuff. And, since the economy doesn't care about our degree of self-awareness, it means we all -- entrepreneurs, investors, engineers, designers, marketers, journalists -- are leaving big money on the table.

I'm hoping that last sentence might catch someone's attention, especially that of the VCs who have a lot to gain or to lose based on their understandings or misunderstandings.

Where do we draw lines by mistake? We live in a structuralist mythology that we make up because it's easy and it's a shared set of beliefs that we can use to argue (in circles sometimes):

  • The "new" versus the "old" is of course the big daddy duality
  • How about Client / server, Startup / enterprise
  • Geek / management, Engineering / marketing
  • East Coast / West Coast, Stanford / Berkeley
  • Open source / proprietary
  • Content / code
  • NorCal / SoCal, Hollywood / Silicon Valley
  • Entertaining / Functional

These just scratch the surface -- you can probably think of fifty pairs of your own now, along with examples of how they are convenient, self-reinforcing, and yet ultimately invalid.

They are a cliche, but they own the conversation. I want to talk about the last few pairs: think of the narratives we make and live inside, as regards Northern California vs. Southern California, Hollywood vs. Silicon Valley, and Entertaining vs. Functional.

Now look at Steve Jobs, an entertainer who could get his fans to poke out their own eyes and get in line for an iPatch. Apple is theater, the Apple store a set, a Disney experience for people who feel superior making snarky comments about Disney. Do you really think computer geniuses work at the Genius Bar? "Space may be the final frontier / But it's made in a Hollywood basement" -- and in this case the fans have already reserved tickets to the sequel.

But, wait! I'm not bashing Apple or Steve -- that would be buying into the very dualism I'm pointing up here...

Now consider also: we have another award show, the Crunchies, coming up. Before you laugh, remember this event isn't intended as a joke. And donating money to charity doesn't make you serious, just generous.

We have our gossip writers, our A-lists and even our stunts that get a third of the Valley looking self-consciously at the floor, embarrassed at having to wonder whether Carly Fiorina or Leah Culver has set feminism back further, a third watching our great-standup-with-the-dirty-mouth pretending they all get the joke, and another third just saying the heck with it and jumping on.

Ya know, maybe we're not so far away from our SoCal cousins as we think. And just to keep it clear: I'm not saying there's a middle position, a compromise, where we're all gonna end up. I'm saying that the duality itself is a fiction. So there is no middle. Until we get this, we're going to continue to have a heck of a time getting some things right. Or making money with them!

Like what? Like things that aren't quite code and aren't quite content. We never got SVG right, because it didn't fit into the categories we thought made up the world.

Is a Flash media code or content? When my mom gets an e-card, she thinks it's content, not an app.

We don't have a cross-platform vector graphics standard in 2008 (!) ... because practical vector graphics in the real world means some logic as well as geometry.

Why can music acts sell (via the carriers) all manner of wallpapers and ringtones for cold hard cash to technically unsophisticated folks, and we still can't get the average person to install or run a mobile app on their phone for free? We're playing that game wrong and we can't see it.

We have every manner of "media center" -- hardware, software, open, closed, expensive, free. Even Microsoft has been in on the act for five years now. And I still don't know any regular folks at all who use any of them.

We have tons of great ideas on the shelves because we can't understand 'the other' well enough to make the deals we need on content licensing. (They don't understand us either.)

Three quarters of "Web 2.0" is not about any kind of functionality at all; it's not even about interaction design. It's about the glossiest thinnest veneer of user experience. It's about buttons and realistic smoke ... the kind of thing that some folks refer to as "production values." Moreover, three quarters of it is free and ad-supported. Not unlike most radio and television. Unlike radio and television, its reach is minimal. Ask your aunt in Duluth what is. Before we spend another session making fun of the studios for not realizing where their business really is, we might think about what an adjustment in ad rates will do to our own.

One last time for the folks in the cheap seats: it's not about right or wrong, steak or sizzle. We're just not where we like to think we are. We're somewhere else. To get a better idea where we are, we need to do some demolition on our usual Monday morning narratives. It's uncomfortable. The good news is, there's a ton of opportunity. And eventually it feels good to realize we've outgrown the notion that the world is flat.

Now go to Macworld and enjoy the, uh, show.

Wednesday, January 09, 2008

Rails Hack: Combining Acts_As_Auth with Long-Lasting Anonymous Sessions

In one of my Rails apps, I have two kinds of users: anonymous visitors, who do not have to sign up at all to use the app's functionality, and "vendor" users, who need to sign up and create some content in order to get value from the site.

The requirement for sessions was simple:

Anonymous visitors should have a session that persists for a long time (e.g., months) via a cookie. Since there is no information at all collected from these visitors, there is no reason to clear them out of the browser, and they would appreciate the convenience of being able to pick up where they left off when they come back.

Vendor users, however, have more data at stake, and so should have the typical login/password/remember-me?/logout pattern, allowing them to decide if their sign-in should persist, and letting them explicitly logout to kill their authentication.

I dropped in acts_as_authenticated, and it does a fine job with the second case (vendors). For the first case, I then added sliding_sessions, which is elegant and has now been added to the relevant wiki page.

The problem is: once session cookie durations are extended in general (at the application or controller level), aaa's authenticated users will be "remembered" upon return to the site, whether they want to be or not -- because their user info is already in their session (i.e., the "login from cookie" is not necessary and is never called). Timing out a session/auth on the server side is a good practice, but doesn't solve this issue -- if the user closes the browser and walks away, someone might come up right behind them and come back to the site, where the session/auth has not timed out.

Here's my solution. It seemed quick and easy; if anyone has criticism or improvements, let me know:

There are three cases -- anonymous user; vendor asking to be "remembered"; and vendor not asking to be "remembered" -- of which the first two worked fine. So I add a cookie upon login, set to timeout at the end of the current session, which will help me separate out the third case.

The details:

AAA adds a couple of "account controller" actions to your app. In login, I added the new cookie (see line 11). Then, in application.rb, I added this method as a before_filter. It checks to see whether you are an authenticated user (i.e., not an anonymous visitor). If so, it resets the session unless it sees the vendor cookie (meaning you've signed in in this browser session and so should not be cleared out yet regardless of "remember me" choice) or the auth_token cookie (created by aaa, and indicating the the "remember me" persistent login choice).

So if you authenticate and do not choose "remember me" then you have no auth_token cookie and, when you close the browser, you lose your vendor cookie. When you come back to the site, your session (and user info) are cleared.

Monday, January 07, 2008

$30 and $60 at Fry's

I guess it's a story of manufacturing in volume: at Fry's, I recently bought my wife this "Periscope Hardcover Booklight"

It's a piece of (fake?) leather and two LEDs, made in China, and priced (by Fry's) at $30. Less than ten feet away was MicroATX motherboard (not this model, but similar), with socket 775 support for Core 2 Quad procs and all the bells and whistles. For $60. Likely made in China.

Fallacious reasoning to be sure, but still a big delta in what you can get for another $30.

Who Considers These Folks the Elite Again?

I just read David Megginson's "Programming languages of distinction" in the context of Jeff Atwood's great post on The Shiny. The Scala cartoon cracked me up. But in real life, what truly "elite" programmers are going to move from Megginson's "Step 1" (too many riff-raff using their current language) to "Step 2" (look for a new, little-known language with fewer annoyances)?

At the end of the day, all of these trends/fads/fashions fall somewhere between two well known poles:

  • Assembly language, as specified for typical mass-produced hardware (x86, 680x0, ARM, etc.)
  • Lisp, and in particular Scheme

If you start with Assembly as your deck of cards (stacks and registers and jumps), and start shuffling and doing slight-of-hand tricks, you can get C, C++, Java, et al. If you start with a Scheme deck (lambda expressions), you can shuffle to get Haskell and Erlang.

If you throw all the cards on the floor and mix-and-match from both decks, you get your Ruby and your Scala and your "JavaScript 2/ES4" (well, maybe).

The reasons for religiously sticking to one pole or the other are not compelling for coding at the "application level." That's a fairly trivial statement. Once we specify the sort of typical end-user or business application that makes up the bulk of development, we evade the issue of whether a primitive imperative language (C/Assembler, close to the metal as far as specifying instructions) or a functional one (Haskell/Erlang/etc., potentially easier to parallelize on multi-core hardware) will perform better.

In the application world, we have hybrids like C#/Java/Ruby/ECMAScript. Sure, Scala. At that point, all sorts of other considerations come into play, from design to cost to maintainability to security, which have more bearing than whether a curly brace is a bad thing or /: is evil as an operator.

And guess what? With ANTLR and some open source code you can go make your own half-a-cup of Assembler-half-a-cup-of-Scheme language quite easily. You can use Kanji for the keywords, Roman numerals, a keyboard with a special key that generates Unicode chars above 0xFFFF for declarations, and maybe change the text direction to indicate which way functions are evaluated. What a niche! You'll be the leading authority! You can write a book. You can entice mediocre engineers and terrorize bad ones. You can entice mediocre engineering managers and terrorize bad ones. Or you can work on software that solves real problems for real people.

Thursday, January 03, 2008

It's About the API

Ok, at the risk of being almost timely, Zed's Rails Rant has inspired me to finish a point I've been wanting to make for a while now: Platforms (and languages too) are about the developer experience (API). Not the community, although that can help. Not the implementation -- that can be changed, redone, ported, etc.

What is nice about Ruby and Rails is that they're "fun" or "pleasant." In general, when programmers say this, what they mean is

  1. Working with the language / API does not go against the grain of how their mind works. Since programmers have varied ways of thinking, simply avoiding an "against-the-grain" experience across a set of non-trivial cases is a real accomplishment. Ruby does this in spades, Rails almost as much once you get used to a particular "Rails way" of doing something.
  2. It's at the right level of abstraction most of the time, when used in the kind of work for which it was intended. RoR is fabulous at this. It does the "super-high-level-language" thing when it reads your data schema, does :has_and_belongs_to_many, and makes a very sensible attempt to match URLs, actions, and views by default. It goes low-level by working with clean, trivial HTML, which is easy to write and debug by hand, while ignoring the temptation to autogen all sorts of decorated HTML/CSS/JS.
  3. When you need to read the core sources or other people's code, it's not a terrifically painful experience that leaves you sitting there saying,"You've got to be kidding" over and over again.

Now, apart perhaps from a bit of #3, these are descriptions of a programmer's experience interacting with the language/platform/framework/tools.

They are API issues, not implementation issues. If the implementation is bad enough, the API experience will be overshadowed. But the developer experience sells the platform.

So... what about the MRI, and the Rails core team, and Mongrel and horrific perf issues, and all that?

The current state of affairs needs to be seen as a reference implementation. It shows off the specs under discussion, proves they can be implemented, provides all the right output to the definitive test suite, etc. Sun releases new Java APIs with a reference implementation. Some of those RIs have been good enough to use in production, some haven't. Sun hoped that third parties would create commercial implementations -- in some cases, this happened. Where an API was popular and important, a "real" version -- or several -- became available, based on market demand.

There are already a bunch of alternative implementations of Ruby either finished, or in the works. Rails has yet to see an alternative implementation, so far as I know, but it seems plausible that both open source (e.g., modify or redo the Rails code to provide multithreading support, etc. and run on top of JRuby) and proprietary (Rails as a set of ASP.Net assemblies) implementations will emerge.

When Java offered a great developer experience compared to the alternatives, it picked up adherents even when the adequate performance (heck, even existence) of various APIs was shaky. When .Net offered a great developer experience, it brought more shops into the Microsoft fold than 20 years of, shall we say, other approaches to gaining market share.

Some failed platforms suffered because of business issues (e.g., Delphi and C++ Builder), but most fail just because they don't do #1-3 above as well as the next guy on the block. My point is that Zed's rant is a bit beside the point, even if it's all true. Ruby and Rails have already accomplished the end of influencing industry thought and behavior in a broad kind of way. We'll need to be a little more patient (or make some serious contributions) before we get the followup implementations.