Monday, January 19, 2009

Twitter's Underwherlming (Former?) Architecture Problem

I recently came across this post from May 2008 comparing Twitter traffic and the Options Price Reporting Authority data feed. Needless to say, the stock market feed is many orders of magnitude larger, at 700,000+ messages per second(!)

It's also not the fairest comparison in the world on its face, for a variety of reasons: the OPRA data system was planned (Twitter met success more or less by accident), Twitter is minimally funded, etc.

A more relevant comparison, in my opinion, is that provided by newzwag, which presented its performance challenges, triumphs, and secrets at a recent SF Ruby meetup.

newzwag's site and trivia game is built on Rails, started small, and had to grow to meet traffic driven by Yahoo and the Beijing Olympics to 9 million pageviews per hour (using a total of a half-dozen machines or so). And lest you think this is a content site served out of a cache, most of the traffic consists of data writes by game players that then need to be ranked, published, etc.

As far as I can tell, that's somewhat larger than Twitter, even considering that Twitter has grown 3-4x since last May's stats.

newzwag's solutions, which they share here, are a study in sanity, reasoned problem solving, and smart efficient architecture.

Without the timelines or resources of a stock-market app, newzwag produced a nice solution that -- at least in hindsight -- appears drama-free.

Interestingly, a newzwag - Twitter comparison can be enlisted to support a variety of different startup social narratives.

One narrative is that an amateur-hour effort yields amateur-hour results, and aspiring startups shouldn't fool themselves into thinking that they won't need old-time Architecture and Sophistication to scale.

A different narrative says it doesn't matter -- if Twitter's success is your worst-case scenario, you still win. That is, build it fast, get it out there where people can try it, and you should be so lucky as to need a real re-arch to fix your scaling problems. In this model, both Twitter and newzwag played it right -- newzwag because they knew the Olympics would provide a narrower time window to showcase their system, so they managed risk against that stricter goal.

And yet another narrative says if you accept these two stories, you still wouldn't want your brokerage transaction flowing through a system built to "see what sticks," and hence Web 2.0 startup methodologies stare at mission-critical business apps from across a huge chasm.

I see this last story as persuasive but also as a big opportunity: there is a chasm, to be sure, but it needn't be quite so big. There are legacy mainframe apps that can speak webservices. Every manager in a big company wants their product to be "100% critical" even if they could create more value by admitting that a lot of nice-to-have two-nines business apps are the real bricks in the wall. If enterprises can get better at separating their Twitters from their OPRAs, they can make more money and have both.

6 comments:

bob pasker said...

I am the original author of the post to which you refer in the first sentence, and I think you are confusing my point with comments added by others.

at the time, twitter was having tremendous problems which were ascribed to the volume. the debate was around whether twitter would ever scale. my post was an existence proof that set the least upper bound for twitter-like traffic volumes.

it was the commenters who brought up the "traditional architecture" point, not me.

Anonymous said...

burberry polo shirt
ed hardy clothes
Wholesale Handbags
babolat aeropro drive
cheap tennis racquets
cheap jacket
Puma basket
Cheap puma shoes
burbrery polos
ed hardy shirts

Los Angeles Computer Services said...

Wholesale Electronics from China Online Shop. We serve buyers around the world who are looking for China Wholesale Electronics at discount prices. You will find here clearance electronics, discount electronics, tv phones and more. We are strategically located in Shenzhen a city famous for electronics factories which is about 100km from Hong Kong. We offer the best service and the lowest wholesale prices from China china wholesale

Buying WOW Gold said...

The valuable metallic you will get from crafting is dependent over a person, specially in your objectives. There really are a selection related to developing professions that rake in a very massive quantity of valuable metal, for instance Blacksmithing.

Unknown said...

Regardless of what way you see it, developing a agency knowledge of the leather shoes for men following topic region relating to mossy pecan bikins as well as swimsuits for ladies may benefit an individual, even led faucet when it's just slightly

pubg mobile said...




Wonderful information! I found amazing information on the blogs i suggest everyone to follow the links below to get download files from this website.

gbwhatsapp apk
acmarket apk
freedom apk
live nettv apk
whatsappplus apk