« Visit to Krispy Kreme | Main | Our Linux Plan »

June 25, 2003

Building Redundancy

We're a pretty small shop, when I came here two years ago everything was on a single Sun Ultra60 (that was on a desk in an office). That included an instance of Apache for production, Real server, MySQL, and a development port running a minimal set of Apache processes for each developer.

A bad arrangement overall.

Over the last two years I have built that into a system with four machines. One webserver, one database server (with attached A1000 hardware RAID array), a test machine and a development machine (all Solaris). I recently stuck additional network cards in each machine and a switch between them for private data transfer between machines (primarily MySQL).

This system is far from ideal, the expectation these days is that under no circumstances should our service be unavailable. While our servers can usually handle the traffic, it is hard to ensure that services will be available because problems in a number of places could bring the system down for hours (or days).

I've been thinking a bunch about adding more hardware to provide redundancy, and how we'd go about that.

Areas where we're doing the right things with our system:
1. All machines have the OS installed on separate drive
2. All drives are RAID 1 (mirrored)
3. data is synced out to machines from central location (making a failed machine easy to rebuild)
4. Machines are in the Tufts "data center" which is physically secure, has UPS, provides nightly backups, and electrical feeds from two separate power grids (the building is on a town line and gets feeds from both towns).

Areas where we could easily get screwed:
1. Lone webserver machine
2. Lone database server machine
3. Each machine has only one connection to public network
4. Each machine has only one connection to private network
5. No backup switch for internal network

In addition preformance could be improved with:
1. Firewall appliance, removing burden from machine CPU
2. Gigabit private network between machines
3. Hardware SSL encryption, removing burden from machine CPU

So I'm writing a proposal to add machines and other pieces of hardware to our system to give us more redundancy and better performance. In general I have support to do this, we've even have some funds which could be applied to this. The plan needs to be built in a way that se can bite of chunks of it as funds are available. Most likely be something that is realized over the course of a few years.

I think this is an incredibly exciting project to be heading up. I enjoy researching, gathering data, getting new hardware, building new machines etc.

Posted by mike at June 25, 2003 2:33 PM