April 24, 2007
Technology at digg.com
Listening to Elliott White III (Eli) and Tim Ellis (Time) talk about technology at the 2007 MySQL Conference. As expected, a packed session.
Eli is the Senior PHP dude, Time is the DBA.
Initially started with Apache 1.3 and PHP 4.x. Used MyIsam and MySQL full text search. Then moved to multiple servers, Apache 2.x, MySQL Innodb, PHP 5, and memcached. Are now using Lucene with solar for search. Currently around 100 machines, 30 databases and the rest serve other functions. 9 memcached machines running on the db slaves. Right now around 30 gigs of data. XSF for user interface stuff (millions of user images). ext3 for other things.
Many PHP servers behind a load balancer. Many MySQL slaves talking to a master. Randomize connections between PHP servers to MySQL connections.
Memcached is used heavily for caching chunks of content. You can't cache the entire page because it almost entirely customized for the user. There are issues to consider with memcached data how to failover and to handle things like when a memcached server goes down and then comes back up but has older data.
With MySQL, the first thing the DBA did was to bring in memcached to offload some of the works from MySQL. Then they started breaking databases into smaller ones (sharding) in order to get better performance. It makes it harder to do things in SQL because the data is in separate places.
Types of sharding:
- Table-based - spread out data based on tables (move user table to separate
- Date-based - spread data based by date
- Range-based - break up data by primary key
- Hashed - hash the primary key
- Partial (digg-created name) - based on what folks are accessing
This is like partitioning, but partitioning wasn't available at the time.
digg uses MySQL 5.0.32 on Debian. Their engine type is InnoDB (recoveres very quickly on a crash).
- the "buy more RAM" option is viable anymore
- rewriting queries
- convincing site architects to changes features for more scalable alternatives
- debug binary running in production
- schema cruft (stuff that isn't used anymore)
- Cacti (monitoring/alerting systems)
- Master is a single point of failure
- Why disks say things are on disk when they actually aren't.
Posted by mike at April 24, 2007 12:01 PM