Monday, January 19, 2009

Twitter's Underwherlming (Former?) Architecture Problem

I recently came across this post from May 2008 comparing Twitter traffic and the Options Price Reporting Authority data feed. Needless to say, the stock market feed is many orders of magnitude larger, at 700,000+ messages per second(!)

It's also not the fairest comparison in the world on its face, for a variety of reasons: the OPRA data system was planned (Twitter met success more or less by accident), Twitter is minimally funded, etc.

A more relevant comparison, in my opinion, is that provided by newzwag, which presented its performance challenges, triumphs, and secrets at a recent SF Ruby meetup.

newzwag's site and trivia game is built on Rails, started small, and had to grow to meet traffic driven by Yahoo and the Beijing Olympics to 9 million pageviews per hour (using a total of a half-dozen machines or so). And lest you think this is a content site served out of a cache, most of the traffic consists of data writes by game players that then need to be ranked, published, etc.

As far as I can tell, that's somewhat larger than Twitter, even considering that Twitter has grown 3-4x since last May's stats.

newzwag's solutions, which they share here, are a study in sanity, reasoned problem solving, and smart efficient architecture.

Without the timelines or resources of a stock-market app, newzwag produced a nice solution that -- at least in hindsight -- appears drama-free.

Interestingly, a newzwag - Twitter comparison can be enlisted to support a variety of different startup social narratives.

One narrative is that an amateur-hour effort yields amateur-hour results, and aspiring startups shouldn't fool themselves into thinking that they won't need old-time Architecture and Sophistication to scale.

A different narrative says it doesn't matter -- if Twitter's success is your worst-case scenario, you still win. That is, build it fast, get it out there where people can try it, and you should be so lucky as to need a real re-arch to fix your scaling problems. In this model, both Twitter and newzwag played it right -- newzwag because they knew the Olympics would provide a narrower time window to showcase their system, so they managed risk against that stricter goal.

And yet another narrative says if you accept these two stories, you still wouldn't want your brokerage transaction flowing through a system built to "see what sticks," and hence Web 2.0 startup methodologies stare at mission-critical business apps from across a huge chasm.

I see this last story as persuasive but also as a big opportunity: there is a chasm, to be sure, but it needn't be quite so big. There are legacy mainframe apps that can speak webservices. Every manager in a big company wants their product to be "100% critical" even if they could create more value by admitting that a lot of nice-to-have two-nines business apps are the real bricks in the wall. If enterprises can get better at separating their Twitters from their OPRAs, they can make more money and have both.

13 comments:

bob pasker said...

I am the original author of the post to which you refer in the first sentence, and I think you are confusing my point with comments added by others.

at the time, twitter was having tremendous problems which were ascribed to the volume. the debate was around whether twitter would ever scale. my post was an existence proof that set the least upper bound for twitter-like traffic volumes.

it was the commenters who brought up the "traditional architecture" point, not me.

huyuni said...

Thank you so much!!cheap polo shirts men'ssweate,Burberry Polo Shirts lacoste sweater, ralph lauren Columbia Jackets,ski clothing. Free Shipping, PayPal Payment. Enjoy your shopping experience on mensclothingus.com.You can find the father who desire fashionable, intellectual mens clothing simultaneouslyGod bless you!I really agree with your opinions.Also,there are some new fashion things here,gillette razor blades.gillette mach3 razor bladesfor men.As for ladies,gillette venus razor blades must the best gift for you in summer,gillette fusion blades are all the best choice for you.Fantastic!God bless you!Meanwhile,you can visit my ,we have the highest quality but the lowest price fashion products wholesale from China.Here are the most popular China Wholesale products for all of you.You can visit .Also the is a great choice for you.

huyuni said...

China Wholesale has been described as the world’s factory. This phenomenom is typified by the rise of buy products wholesalebusiness. Incredible range of products available with wholesale from china“Low Price and High Quality” not only reaches directly to their target clients worldwide but also ensures that China Wholesalers from China means margins you cannot find elsewhere and China Wholesale will skyroket your profits.

huyuni said...

nike shoes & Puma Shoes Online- tn nike,puma shoes,puma cat, baskets cheap nike shox, air max.cheap nike shox r4 torch, cheap nike air, nike running shoes air max, puma speed and more. Paypal payment.nike running shoes Enjoy your shopping experience on Nike & Puma Shoes Online Store.

huyuni said...

Lacoste Polo Shirts, , Burberry Polo Shirts.wholesale Lacoste polo shirts and cheap polo shirtswith great price. clothingol.com offers lot of 10ralph lauren polo lacoste polo shirts and lot of 20 Burberry Polo Shirts. clothingol.com offers classic fit polo shirts. polo clothingCheap Brand Jeans ShopMen Jeans - True Religion Jeans, burberry polo shirtsGUCCI Jeans, Levi's Jeans, D&G Jeans, RED MONKEY Jeans, Cheap JeansArmani Jeans, Diesel Jeans, Ed hardy Jeans, Evisu Jeans, Women JeansJack&Jones Jeans...

huyuni said...

Charlestoncheap columbia jackets. turned a pair of double plays to do the trick. spyder jacketsThe had at least one runner on in every inning but the first and outhit the RiverDogs by a 12-6 margin Lawal should be a focal point of the Yellow cheap polo shirts along with highly touted newcomer, 6-9 Derrick Favors, rated as the No. 1 power forward on the ESPNU 100. The Yellow JacketsThere are ed hardy shirts pretty ed hardy shirt for men,
ed hardy womens in the ed hardy online storedesigned by ed hardy ,many cheap ed hardy shirt ,glasses,caps,trouers ed hardy shirts on sale ,You can go to edhardyshirts.com to have a look ,you may find one of ed hardy clothing fit

huyuni said...

A study last yearnike tn, the author in the essay read NIKE , a reporter at the Shanghai headquarters in an interview, nike chaussuressee a pr in high school to read a league plans, employees interviewed told reporters in Beijing's streets, they children to interview, the children said, "truly understand them." tn chaussures These words, if the author touches product function, brand spirit and culture is to become part of the consumer of two basic methods

j said...

burberry polo shirt
ed hardy clothes
Wholesale Handbags
babolat aeropro drive
cheap tennis racquets
cheap jacket
Puma basket
Cheap puma shoes
burbrery polos
ed hardy shirts

Los Angeles Computer Services said...

Wholesale Electronics from China Online Shop. We serve buyers around the world who are looking for China Wholesale Electronics at discount prices. You will find here clearance electronics, discount electronics, tv phones and more. We are strategically located in Shenzhen a city famous for electronics factories which is about 100km from Hong Kong. We offer the best service and the lowest wholesale prices from China china wholesale

Auto Twitter Marketing Tools said...

Auto Tweet Generator Software!

Autopilot Twitter Marketing Just Through One Software!

Tweeting When You're Away From Your Computer Has Never Been Easier!

"Discover How This Twitter Schedule Tweets For You On Autopilot Mode"

Sit Back And Relax And Give Your Twitter Followers Valuable Content
While You're Away.... That'll Turn Them Into Raving Fans..

-> Manage your Twitter postings and schedule multiple posts in just seconds easily!

-> Set the time, edit and import messages in just a few clicks of a button!

-> Think about the time saved on making changes without having to log into Twitter!

-> Twitter marketing made easy because you can track your tweets through one software!

-> Gain rapid exposure through Twitter without having to physically be involved in tweeting all day!

-> It automatically submits your messages to Twitter while you focus on more pressing matters!

If You Can Click, Copy, and Paste, You Can Use "TwitterBuzz Auto Tweet Generator!

So go ahead! and Click Here to Download Your Software immediately...





Auto Tweeter Bot

Tweet Automator

Automated Twitter Tool

An Automated Tweeting Solution

Automatic Tweet

gaohui said...

The holidays are a time ed hardy of getting together with friends ed hardy shoes and family, attending elaborate ed hardy clothing parties, and other exciting events ed hardy clothes that involves dressing up in stunning ed hardy store wardrobes. If you ed hardy Bikini are pregnant during ed hardy swimsuits the holidays, it does not ed hardy Caps mean that you are unable buy ed hardy to look fabulous and ed hardy swimwear stylish. Now, an expectant ed hardy sale mother has many styles of chic ed hardy glasses maternity clothing that allows cheap ed hardy her to show off her baby bump Christian audigier while looking spectacular.

Buying WOW Gold said...

The valuable metallic you will get from crafting is dependent over a person, specially in your objectives. There really are a selection related to developing professions that rake in a very massive quantity of valuable metal, for instance Blacksmithing.

Michelle Ding said...

Regardless of what way you see it, developing a agency knowledge of the leather shoes for men following topic region relating to mossy pecan bikins as well as swimsuits for ladies may benefit an individual, even led faucet when it's just slightly