Da broken Mouse

The twitter problem, or physics still matters.

2008-07-04T01:44:12Z

If there is one lesson that really stuck from attending Google's Sydney Developer Day, it's that physics matters.

You see, one of the AppEngine engineers - Brett Slatkin (as seen in this YouTube video introducing AppEngine) had a wonderful slide on numbers that are important. The most important one to me? Disk seek time of 8ms means that you get approximately 100 disk seeks per second, per disk.

Why does this matter? It matters because each transaction you make your database do has at least one disk seek in it.

Let us look at the twitter problem. Twitter is melting down. This much is widely known. Pretty much every software engineer sits there and doodles out a solution using their favourite technology stack, and says "that'll work, gee the Twitter engineers must be stupid."

So what are twitter's numbers? Approximately 2.5 million users, 10 tweets a second (probably spiking higher given the fractal nature of these things), and a worst case estimate of 50,000 followers per user.

My understanding of twitter's architecture is that they are using MySQL (on a 32gig of ram sun box) as their transactional datastore. Given a worst case estimate of 10 tweets a second, each with 50,000 followers, we have half a million disk seeks a second to write to disk. To keep up with that, the database needs to be written to a SAN of 5,000 disks just to keep up with current load, let alone future expansion and disk failure.

And it is about here that I can easily say that most engineer's back of the napkin efforts at redesigning twitter will all have been failures, because their current technology stacks hide the cost of disk seeks from the developer. The abstraction is actually hiding something that needs to be front and center if someone is going to be able to correctly provision a solution to this problem.

The interesting thing that the AppEngine team are doing is that their architecture brings these costs into stark relief for the developers. It has caused a lot of pain on the mailing list as every man and their dog has whined about how AppEngine doesn't do feature X of SQL. But, for scaling, SQL kills you because you aren't aware of the cost of feature X, and thus you do it at a stupid time. Like, say, for each incoming request.

I like what AppEngine has done. Could I re-implement Twitter on top of AppEngine? Sure, if they let me chew up 5,000 disks in their cluster =)

Mmm

2008-07-02T05:54:46Z

Programmer Competency Matrix

2008-07-01T01:42:38Z

Programmer Competency Matrix wherein I figure out that I'm really not that competent a programmer. So much still to learn.

Just because it amuses me...

2008-06-30T12:11:48Z

A week with dad and guess what? I'm a tea drinker again. Something about his hopeful grin every hour - making me a cuppa? If ya can't beat 'em... =)

Christian The Lion

2008-06-30T11:54:14Z

Just stupidly cute. =)

There is a robot inside that stuffed toy...

2008-06-30T01:41:09Z

In Vestimentis Ursum. There's a robot beneath the fluff.

Gutted kid's toys. Kinda disturbing me.

SysAdmin dance

2008-06-29T13:33:08Z

The Website is Down. Too funny.

It's funny because it's true. All of it. Really. =)

I'm taking over the world.

2008-06-28T12:08:55Z

One programming language at a time. =)

Concurrency, Programmers, and masochism

2008-06-25T09:05:45Z

Programmers, by definition, like working with puzzles. In fact, the pecking order that programmers set up is based on each programmer's ability to understand and solve hard problems. This is a interesting result, and I am sure there is a psych paper or fifteen about this.

The part of this geek machismo that fascinates me is that it works against us. A lot of the time programmers tend to see hard problems as the reason they exist, as opposed to finding an easier way. Easier ways are for wimps. Which probably explains why most geeks run Linux or Windows instead of Mac OS X.

The current cause célèbre in the programming world is Concurrency. To break this down into laymans terms, a normal program is like a cooking recipe. A cook (the computer) follows the recipe (the algorithm) to produce dinner (the result data). Adding concurrency means that instead of cooking one dinner, the cook is cooking multiple dinners. Think busy chefs in a resteraunt. And you wonder why Gordon Ramsay swears a lot.

The problem us programmers are facing is that the common abstractions in use at the moment haven't progressed for thirty years. Locks. Semaphores. Signals. By the very names of the abstractions I think you can guess where the ideas for the abstractions came from. Train control systems, cannal systems, etc.

There is, however, a change flowing through the environment at the moment. A concurrency abstraction is starting to come into vogue that is actually inspired by something that the human mind knows how to deal with very well. The abstraction is called the Actor model, and in laymans terms it is probably best summed up as gossiping.

The actor model is apparent in Erlang, it has been implemented as a library in Scala, Google's phone handset Android's thread communication system looks as if it was inspired by the model, and I just found a byte code weaver named Kilim that introduces Actor capabilities to Java.

So what does it mean for you, the unsuspecting end public? In the first instance, not a lot. Most programmers are emotionally attached to what they already know. It cost them a lot of heart ache to learn locks, and they see other examples of failing multi-threaded code (my phone, for instance) as an excuse to feel good about what they know. A lot like what happened with the introduction of memory management, if I do say so myself.

In the longer term, I expect to see some web sites and end user applications that do things that the current generation of programmers can't achieve. And when that happens, they'll either adapt, or become niche players.

As always, I'll take the bleeding edge. I have software that I want to write that just can't be done with the current approach. It's going to be fun =)

Not your average superhero...

2008-06-20T12:12:26Z

The future is already here, it just ain't widely distributed yet...

2008-06-19T13:29:42Z

I have been shown the future. Group decision making via massively multi-player online spreadsheet.

I think I need a good stiff drink and a lie down.

Another book for y'all to read...

2008-06-16T05:22:58Z

I'm currently chewing into Norman Doidge's The Brain That Changes Itself: Stories of Personal Triumph from the Frontiers of Brain Science. So far it is a collection of stories of people who have changed themselves by using mental exercises. I found it out first hand doing the Dore work, where in I stretched what I was capable of handling through daily exercises.

Though, it is kinda odd reading bits where they point out people with the specific problem usually print instead of cursive write, because printing consists of short strokes to construct single characters, where as cursive writing requires constructing several letters at once, and thus overloads a specific brain center.

I gave up cursive writing in about year 5 because I just couldn't get it up to a decent speed without my brain exploding...

Boiling down the insanity to sound bytes

2008-06-13T21:11:10Z

Six Word Methods: Software methodologies in six word summaries. Here are a few of my favourites:

Waterfall: Nevermind backlog, requirements were signed off.
Agile: Avoid featuritis; outrun pesky business users.
Domain-specific languages: Compress every problem into one-liners.
Open-source: Bury the world in abandoned code.
Perl: Too busy coding to maintain anyway.

Google Apps Google IO videos

2008-06-13T06:55:29Z

From the videos of Google's IO day: Becoming a Google Apps Small Business Solution Provider.

Google are pushing for small business IT support companies to use Google's infrastructure (GMail, Google Docs, Google Apps) to support companies IT infrastructure needs.

Almost worth signing up to get access to their doco...

Nat Torkington is god.

2008-06-13T04:44:56Z

Nat re-frames a lot of net laws in terms that make sense for Enterprise usage in Web 2.0 Is From Mars, Enterprise Is Up Uranus:

Metcalfe's Enterprise Law: The security risk of a network is proportional to the square of the number of users.

Reed's Enterprise Law: The downtime of a network grows exponentially with the size of the network.

Moore's Enterprise Law: If you wait 18 months you can buy twice as much computational power for the same money, therefore you should never upgrade.

Torvald's Enterprise Law: Given enough eyeballs, all bugs are exploited.

Godwin's Enterprise Law: As an online discussion grows longer, the probability of a lawsuit due to someone being offended approaches one.

Brooks's Enterprise Law: Adding more people to a late software project is the only way to appear to be doing something about it.

Enterprise Definition of Social Software: software that wastes more time as more people use it.

Algal farming for Oil Production

2008-06-12T01:49:59Z

Endless Energy from Algae Biofuel: Closed-Loop Photo Bioreactor from Valcent (3 minute video)

I'm guessing these guys are still proving their ability to scale up to commercial quantities, but their ability to generate orders of magnitude more oil per acre per year than corn or palm plantations is going to give them a long run advantage. And while ever oil remains at $140US a gallon, these guys are running a money machine.

And now for something completely different...

2008-06-07T06:33:02Z

Chuppa has turned me into a LolCat. Rawk =)

Sometimes it's funny because it's true...

2008-06-07T00:04:53Z

I've had several people send me this xkcd comic...

Camera Tossing...

2008-06-06T01:55:03Z

DSC00369
Originally uploaded by QuakkauQ.

I just found out about a new wave in photography - camera tossing. Very cool images. There's a Flickr group, and there's even a mini-howto.

MetaGene, because C++ template metaprogramming is for the win

2008-06-05T00:20:17Z

MetaGene is an O'Caml subset that is translated to C++ templates for compile time meta-programming. This is idea is so full of win.

Or to be slightly less sarcastic, I can understand why they did this. But I can't understand why any sane programmer would use it. Or still be using C++ for that matter. Oh, hi Bill. ;-)

An OS X Subversion client

2008-06-04T22:05:46Z

Versions, a subversion client for mac.

If there is one thing i like about OS X, it's that the developers spend the time to create polished user interfaces. I hope this app allows version control to move outside the developer only circle...

Hmm, GMail is being stupid...

2008-06-04T11:24:03Z

Has any one else noticed that GMail is now defaulting to reply all when faced with a mass-mailed email? I fucking wish it wouldn't, as reply all is almost never what I want to do. Fucker.

I was an information junky once...

2008-06-04T10:40:59Z

Except my lj friends list mostly goes past unread, my google reader account has stupid numbers of unread articles, and after two hours of work, my unread email is down from 468 unread to 440 unread. I did actually get through a few more than 28 emails, but it just keeps coming in. Oh woe is me, my info addiction is going to be the death of me yet...

Pot shrinks brain, apparently.

2008-06-03T03:34:25Z

According to EuroGraduate.com, a study by Australian Researchers published in American Medical Association's journal Archives of General Psychiatry points to reduction in the size of the hippocampus and amygdala. The hippocampus regulates memory and emotion, while the amygdala plays a critical role in fear and aggression.

Mmmm. Suddenly a whole bunch of things become clear.

I'm having a good day...

2008-06-02T05:37:37Z

So, i worked all night last night, got into bed at about 7am, woken at about 10am by the project manager, go to the cafe to get breakfast, realise that the cow-orker didn't return my G3 card for my laptop, come home to finish a few things...

And now the fucking laptop is a brick. I hate windows.