Twitter's Underwherlming (Former?) Architecture Problem

I recently came across this post from May 2008 comparing Twitter traffic and the Options Price Reporting Authority data feed. Needless to say, the stock market feed is many orders of magnitude larger, at 700,000+ messages per second(!)

It's also not the fairest comparison in the world on its face, for a variety of reasons: the OPRA data system was planned (Twitter met success more or less by accident), Twitter is minimally funded, etc.

A more relevant comparison, in my opinion, is that provided by newzwag, which presented its performance challenges, triumphs, and secrets at a recent SF Ruby meetup.

newzwag's site and trivia game is built on Rails, started small, and had to grow to meet traffic driven by Yahoo and the Beijing Olympics to 9 million pageviews per hour (using a total of a half-dozen machines or so). And lest you think this is a content site served out of a cache, most of the traffic consists of data writes by game players that then need to be ranked, published, etc.

As far as I can tell, that's somewhat larger than Twitter, even considering that Twitter has grown 3-4x since last May's stats.

newzwag's solutions, which they share here, are a study in sanity, reasoned problem solving, and smart efficient architecture.

Without the timelines or resources of a stock-market app, newzwag produced a nice solution that -- at least in hindsight -- appears drama-free.

Interestingly, a newzwag - Twitter comparison can be enlisted to support a variety of different startup social narratives.

One narrative is that an amateur-hour effort yields amateur-hour results, and aspiring startups shouldn't fool themselves into thinking that they won't need old-time Architecture and Sophistication to scale.

A different narrative says it doesn't matter -- if Twitter's success is your worst-case scenario, you still win. That is, build it fast, get it out there where people can try it, and you should be so lucky as to need a real re-arch to fix your scaling problems. In this model, both Twitter and newzwag played it right -- newzwag because they knew the Olympics would provide a narrower time window to showcase their system, so they managed risk against that stricter goal.

And yet another narrative says if you accept these two stories, you still wouldn't want your brokerage transaction flowing through a system built to "see what sticks," and hence Web 2.0 startup methodologies stare at mission-critical business apps from across a huge chasm.

I see this last story as persuasive but also as a big opportunity: there is a chasm, to be sure, but it needn't be quite so big. There are legacy mainframe apps that can speak webservices. Every manager in a big company wants their product to be "100% critical" even if they could create more value by admitting that a lot of nice-to-have two-nines business apps are the real bricks in the wall. If enterprises can get better at separating their Twitters from their OPRAs, they can make more money and have both.

If Developers Love Speed, Why Is a Slow Laptop More Popular than a Fast Desktop?

When I was in college, there was an anthropology meme about how childbirth became riskier when the human pelvis adjusted for walking upright. So the ability to walk and run -- or the brain developments that caused humans to deal with problems by walking and running instead of some other way -- must have offered some massive evolutionary advantage that outweighed increased risk to mother and offspring in childbirth.

I'm not sure where this belief stands now -- whether it's established dogma or the anthro equivalent of an urban legend -- but there seems to be a funky analogy among developers and their dev machines.

I'm amazed by how many devs want to be mobile so bad that they use a laptop as a principal (or only!) development machine. I'm more amazed when these same people then get into a silly debate about "the best tools for the job," whether that's an OS debate, or IDEs, or something else.

Because, like walking upright in the story, a laptop offers mobility at a very heavy price.

I work machines pretty hard -- running server software, virtual machines, development environments / debuggers, lots of browsers, random other tools, some of which even use CPU cycles and not just memory. I appreciate the productivity and uninterrupted "flow" that a really fast machine offers.

Laptop performance is awful compared to desktop machines, and with every passing month a laptop (due to its limited ability for upgrade) falls farther and farther behind its well-maintained desktop counterpart. And a plain ol' $100 motherboard and $250 processor in a desktop will do things that make most laptops implode into a singularity, whether it's the 1333 MHz FSB, the 3+ GHz quad-core CPU, or a graphics card whose cooling pipe alone can't fit inside a laptop.

Even the top end, like the fastest Alienware gear, has some fundamental limitations that sound like desktops from a few years back: 2.8 GHz CPU / 800 FSB / 667 Memory ... and a price tag (with the best options) near $5,000!

It's not an OS thing either: the hyper-popular MacBook Pro, while one of the fastest "conventional/mass-availability/non-gaming" laptops, is a complete lightweight compared to the base 8-core Mac Pro tower (that runs the same $2,800 as a the top-end MacBook Pro).

Don't even get me started on the garbage laptops that most companies give their employees, machines which are optimized for durability, enterprise management, and running Office. Sporting things like 4200 rpm hard drives.

And for the occasional but real necessity of mobility, a $250 cheeseball laptop does a fine job for giving presentations, working with Office, and even a quick hack here or there, so it's not as though there's a huge sunk cost just in being able to bring a slide deck to a client.

Yet ... the ability to move around instead of sitting in one place offers -- or at least appears to offer -- some kind of power that compensates for all of these issues.

Can anyone clue me in on exactly what it is?

mod_ndb: Wicked REST for My[Giant]SQL Cluster

I caught a great presentation at CommunityOne a couple of weeks ago, and haven't had a chance to write about it until now.

In a nutshell, mod_ndb is an Apache module which allow a limited set of parametrizable MySQL queries to be automagically exposed as REST services via Apache. There are three things I left out of that sentence for clarity, that make this uber-cool:

1. The operations actually run against MySQL Cluster, a high-performance, shared-nothing (code for you don't need to manage a SAN and shared filesystem) scale-out system.
2. In exchange for not having full SQL capability, these services interact with MySQL Cluster using the cluster's native NDB API for maximum performance.
3. The latest version of MySQL Cluster, 5.1, has significant technical advantages over the earlier but still impressive 4.1 and 5.0. These include the ability to store more data on disk instead of dedicated RAM, and this collection of improvements. (In that doc, where you see references to 'carrier grade edition' note also that, according to the CommunityOne talk, future versions of MySQL Cluster will be on a unified codebase from that carrier grade version.)

I've liked MySQL Cluster since it's debut, and I'm thrilled to see this evolution.

As far as "let's download this right now and set it up," it is true that in the four years or so since that debut, the niche for scale-out clustering has narrowed:

On one hand, machines and storage have continued to get cheaper and faster, with the result that a data set and transaction load that might have a cluster of 2-4 dual-proc servers, and associated hassle, can now be handled by a single server with a couple of fat quad-core Xeons. The single OS and single filesystem simplify things vastly, to the benefit of databases like SQL Server, which have not invested in a scale-out strategy.

On the other hand, applications with a need for super-massive data that can live with some latency and a lightly structured, don't-call-me-relational data model can pay as they go for that scale-out capability with Amazon SimpleDB, Microsoft SSDS, etc.

Still, there is a well-defined middle section for a product like MySQL Cluster (and its arch-nemesis, Oracle RAC):

• big data sets and/or large-scale OLTP,
• plus a highly-structured relational data model. (Microsoft has asserted that SSDS will move toward full relational capabilities in the longer-term roadmap, but inasmuch as the initial service is still in beta, I don't count that.)
• legal or business requirements to keep the data in-house (not in the cloud)

Or you can flip it around the other way: if your app doesn't need a cluster like this (or something similar), what are you really doing, anyway?

Vista Performance: It Really Is That Bad

I try to stay away from the rant post, it's too easy and doesn't contribute a lot. But I feel myself succumbing, so at least I'll try and mix in a little useful material with my rant.

I just spent some significant time trying get Vista to behave on my wife's laptop. By behave, I mean not make it impossible for her to surf the web with Firefox because the CPU was pegged most of the time and the HDD never spun down.

First, some facts:

1. I'm as close to a Microsoft fanboy as you'll find in the Bay Area. They do great stuff. I'm an ecumenical kind of guy, so I also so like Ruby and Flex and Java and Linux (wait 'til my next MySQL post). I.e., I'm not married to Microsoft, but I think they have done some amazing engineering in the last 10 years and it's a pleasure to work with almost all of their late-model products.
2. This machine, while not blazing by 2008 standards is only a few years old, does have a 3 GHz HT proc, multiple gigs of RAM, nVidia graphics, plenty of hard drive space, etc. In fact, it ranks in the 4.x range on Vista's own "performance estimate" in every category except graphics. Which shouldn't matter because Aero Glass is turned off, and my wife isn't a gamer.
3. Just to cover all the bases, it's plugged in and not configured to ever step down the processor or anything like that to save power. And there's almost no software on it. It's basically Firefox and MS Office, Grisoft/AVG, whatever Windows Update considers critical, and not much else. My wife runs under a non-admin account because she neither wants nor needs to install or configure anything on it.
4. An early-ish adopter, I was forced to abandon Vista after seeing it run cripplingly slowly on a 3.5 GHz, 5+ rated desktop. As a developer, watching this machine churn while it tries to decide which UAC prompt to throw next wasn't acceptable, since I had XP on the same box (dual-boot) and could measure that XP was easily 25% faster. But the wife liked Vista's look and feel, actually preferred it to XP, so cool, didn't think perf would be a big issue for her.

Ok, so the performance was all gone to heck. I checked all the obvious background tasks that could be going nuts, virus scans, disk indexing and all that. Found a few services and tasks that had caused problems before and disabled all of them. Can't imagine why they're running by default. What ever happened to configuring services to run based on their actually being needed by something the user wants to do?

The killer this particular time was the 'network location awareness service' and the DNS caching service. First, NLA. According to MSDN (full info here), this service is 'vital for computers or devices that might move between different networks.' First, I'm not sure I believe that; so far, the computer seems to work better without it. The little icon (and service) that would take 5 minutes to realize the machine was actually on the Internet (you know the icon) is now dead, and apps connect right away. Funny how that works.

Just for the sake of argument, let's pretend that it really is 'vital for computers moving between different networks.' Here's a clue: (1) it doesn't take much monitoring for the machine to realize it's always on the same network in my house, so (2) shut this stupid service down, (3) if another network is detected or the old one is unavailable, maybe then spin this beast up and (4) monitor your own resource usage -- if the service starts using 50%+ of the CPU all the time, for whatever reason, and the network isn't ever changing, why not shut it the #$% down at that point?

I could make the same argument for DNS cache. Although it boggles the mind how a service this simple could ever be using a real percentage of a modern CPU. I turned it off. I guess Comcast will hate me now for making an extra 25 lookups a day.

I'm not going to get into the argument that there shouldn't be lots of services on the machine -- after all, most of the myriad services have little performance impact and at least in Vista are supposedly less likely to compromise security. Although I would prefer a 'configure to run after deciding it's useful and not harmful' self-management approach.

I just wonder, if a pristine machine like this one, well configured etc., scoring generally 4 in the Vista perf scale, and whose only sin is being 2005 vintage, gets mucked up this badly from such mild use, there are truly some problems with Vista, both for home users and businesses, that are more serious than I would have believed.

Ok, so I promised I'd try and offer a little useful info. Once again, for the sake of SEO

Some possible fixes for Really Bad Perf in a basic Vista machine:

• go into the task scheduler and get rid of tasks you don't want to run;
• look at the Network Location Awareness and DNSCache services (I'm not gonna say kill them unless that's what your particular machine needs ... but look at them);
• if you haven't quashed Aero already, turn it all off and see if that helps;
• and if you're running Vista Ultimate, and you don't actually ever use Media Center, go and kill all the tasks and services related to MC because Vista isn't smart enough to notice that you don't need 'em, and I've caught them hogging a ton of resources.

Last, not to get into the 'uphill to school both ways thing,' but a 700+MB working set with no user apps running??? You just want to say, "Man, have some self respect and get on a treadmill once in a while..."