« tweaking mod_perl | Main | Object Oriented PHP »

January 14, 2003

moving from stunnel to private network

The project I've been working on for the past two years at Tufts University ran on one Sun Ultra 60 when I arrived (March 2001). In January 2002 we moved to a multi-server configuration. We split the functionality of one machine into three; web server, streaming server and db server. We run Apache (with mod_ssl and mod_perl) on a U60, stream RealVideo and FlashPix from a second U60, and run MySQL on an E250.

Doing this added some complexity, as data for the webserver needed to be pulled across the network from the db server. We were informed by our noc that our datacenter wasn't on a private subnet and that all traffic on the network was public. We tried both tunnelling through ssh and running a stunnel daemon and decided to go with stunnel. Over the course of the year we cringed from time to time when we thought of how much overhead we were using to encrypt all that data (we store almost everyything in MySQL, including images).

During the fall we secured a Cisco Catalyst 2950 and set it aside waiting for a rainy day to install it. Last week, when our load increased to back-breaking levels we decided it was time to put in the private subnet.

I worked all night on this. I started by installing the network cards. I had a Quad Fast Ethernet card I battled with for a while and ended up pulling it and using an SunSwift card. After the hardware was recognized it was as simple as adding an /etc/hostname.hme1 to each of the machines and adding an entry in /etc/hosts with the ip address and hostname. We didn't want to run an internal DNS server so we assigned the machines 10.0.0.x addresses and will have to live with using IP addresses in our MySQL connections from the webserver.

When I had the three machines up and running with the private network installed I discovered that the SunSwift cards were misnegotiating to 100Mbts *half-duplex*, and that running traffic over the internal network was slower than using stunnel. It was 4am and we desparately needed something working soon. As a temporary solution, while I determined the proper way to configure the second nic card to force full-duplex, I stuck a crossover cable between the webserver and db machine and found that traffic was moving at twice the speed as it did through stunnel over the public network.

With the pressure off for a bit I'm able to focus a little attention on configuring the switch (which we were doing injustice to by using as it was configured out of the box) and the hme1 interfaces to force the correct speed and duplex.

Story developing . . .

Posted by mike at January 14, 2003 12:04 PM