« AMD64 Gentoo Linux on Biostar K8 Motherboard | Main | Meetings with MIT on OpenCourseWare »

January 12, 2005

Replace Failed Mirrored Boot Drive on Sun 280R

There's nothing like working on a live, mission critical machine. Two weeks ago, our primary database machine failed to come up after some downtime for a UPS replacement in the data center.

I jumped on our console and determined we had a bad boot disk and had to boot from the mirrored disk. Apparently you can tell Solaris to do this automatically, but I think in most cases you want to bring it up manually.

I called Sun, who gave me the option of having someone come over and do it or sending the drive so I could do it myself. I opted to do it myself and came over to the data center today, after I had confirmed the drive had been delivered. The guy I spoke to at Sun pointed me to a tutorial on unixway.com, which describes resolving our issue in exact detail.

The 280R has hot-swap drives, which can be pulled with the machine running. Of course, you want to check and double check that you are pulling out the right drive (no, I've never pulled the wrong one).

Once you've replaced the failed drive with a working one, you partition the drive and then issue a series of DiskSuite commands to reset the RAID state database to start using the partitions on the new disk.

It is all pretty simple, but there's a good deal of stress. Before every step you hope that there isn't something that, in hindsight, will make sense for why that particular command caused the machine to go into some unforseen state. Yes, I've had that happen before . . . fortunately during a maintenance window.

Did I ever mention that my dream job is a trail maintenance worker for the forest service?

Posted by mike at January 12, 2005 11:00 PM