« Catching a Game at Yankee Stadium | Main | meebo: Scaling Synchronous Web Apps »

September 18, 2008

Scaling Digg

Listening to Joe Stump speak at NYC Web 2.0 Expo about scaling digg.

A few notes as Joe is talking.

When scaling needs go up, normalization becomes less likely. Data often needs to be denormalized to spread it around.

Keep cache entries forever and explicitly expire. You may have some things that get used all the time, others that are used less frequently but that should still be left around until they get purged for space reasons instead of time.

MemcachedDB replaced MySQL in their "Green" badges project. Needed 3,000+ writes per second starting with 5 terrabytes of data growing at 7GB per day.

They are building IDDB, which has elastic partitions, eventual consistency, and variable partition types. Will continue developing and then hopefully open source it in a year or so.

Posted by mike at September 18, 2008 4:23 PM