« August 2003 | Main | October 2003 »

September 30, 2003

Cool Image Illusions

Stumbled into this site during tonights weblog review. Interesting use of static images and colors that give different illusions.

Posted by mike at 9:51 PM

The Fine Art of Photo Restoration

When I first got out of college I was on a design path (graphic, web etc). Sometimes I miss those days, and always jump at the chance to dig out the old theories and tools.

Today was a perfect example, Pete took some photos on a trip to DC, several from inside the plane which looked like they could use some enhancing.

Here's Pete's original photo of DC from the airplane, next to the color corrected image


The original looks like someone put a grey transparency over it.

I opened the image in Adobe Photoshop where the Levels dialog (Image->Adjustments->Levels) indicates a lack of highlight and shadow colors, causing in the grey overcast.

For each of the RGB colors (selectable in the dropdown) I reposition the slider closer to the color range.
This redefines the color range for the image, which essentially redistributes the pixels over the range, making dark ones darker and light ones lighter.

The end result is an image which has the full range of colors.

A word of caution, Photoshop has an "Auto Levels" adjustment which I avoid, it gets close to doing the right thing, but rather than letting you ajust the range it determines the darkest point on the photo and lightest point and uses those to redistribute the pixels. This gets the color back, but in my opinion goes overboard, creating a fake look to the image. Better to use the sliders and carefully govern how to redistribute the pixels.

Posted by mike at 3:18 PM

September 29, 2003

Responding to a Nigerian Scam Offer

A few months old, but an excellent exploration in exactly what happens when taking up the Nigerian offer, "please let me deposit money into your bank account." Good for a few Monday laughs.

Posted by mike at 10:48 AM

September 28, 2003

Ampache - Setting PHP Execution Time

Installing a new version of Ampache today, a small service we run on the kruckenberg to let family and friends listen to shared mp3s. We have a nice stash of music on the server, and as I was importing (saving id3 info in MySQL) I started to see errors like:

Fatal error: Maximum execution time of 30 seconds exceeded in /newroot/home/kruckenberg/web/ampache/modules/id3.php on line 433
I went to php.ini suspecting a the limit was configurable and quickly found
max_execution_time = 30
Upping that to 60 seconds gives enough time to finish the import.

Ampache is a php frontend for managing a collection of mp3s on a server, allowing the user to click on links to generate .m3u or .pls playlists for artists, albums, songs and even customized playlists. It runs alongside mod_mp3, which is a module for Apache which will stream mp3 tunes to any capable player. Ampache and mod_mp3 are a great way to combine music libraries.

Posted by mike at 3:07 PM

September 27, 2003

Afternoon in Boston Harbor

Took an incredible trip today to Georges Island in Boston Harbor with the family. The island is one of 35 harbor islands. A $7 (round trip) ferry gets you to Georges Island and then a free shuttle service can get you to many of the other islands.

The ferry from Boston (Long Warf) is about 50 minutes. On the way out it was foggy, and hard to tell exactly how fast and far we were going. The three-floor ferry is passenger-only (no cars allowed on any of the harbor islands) and was relaxing.

Georges Island is home to Fort Warren, which was used in the civil, and both world wars (to differing degrees). Very cool brick and granite walls with large grassy fields on the inside. Much of the fort is left open, and in pretty good condition. We did the self-guided tour. One fascinating thing about the island is that the beaches are lined with millions of smooth, thin, perfect-for-skipping rocks. We spent a good chunk of time discussing geology (where did these perfect rocks come from?) as well as the physics of skipping rocks.

After being on the island for an hour the fog blew out and we enjoyed the remainder of the sunny afternoon playing on the grass, watching boats, locating and naming the 10 or so other islands within view.

By the time our ferry came to pick us up we realized that an afternoon on the island was just scratching the surface and we were going to need to come back at least a few more times. Hopes are that next time we will take an earlier ferry and use the free inter-island shuttle to spend some time on the other islands, maybe even pack some gear and spend the night.

(I'm asking myself over and over again why I still don't own a digital camera. Too attached to our SLR and a history in print photography.)

Posted by mike at 7:17 PM

September 26, 2003

Bulk jpegtran Update Script (with an XML check)

Thought I'd post the command I used to do the jpeg cleanup solving yesterday's problem:

for id in `find . -exec grep "<plist version" {} \; | awk -F. '{print $2}' | sed -e "s/\///"`; do jpegtran -copy none $id.jpg > $id-2.jpg; mv $id-2.jpg $id.jpg; echo $id" done"; done
The gist is to grep each image for the offending XML, and if it exists run it through jpegtran to remove the offending extra junk, saving the new image. Then move the new image over the original. Works like a charm (provided you're are in the directory with the images). My images are named 1.jpg, 2.jpg etc.

UPDATE: I discovered that a handful of images on this weblog had the problem, those have been fixed as well.

Posted by mike at 8:36 AM

September 25, 2003

IE Freezes Up Downloading Images

Turns out it's actually a Photoshop 7 problem . . . putting extra junk into jpg images. Read on for the whole story . . .

A few weeks back I worked on a problem with slowness on a site. One of the complaints was along these lines (from a user's email):

When I get to the site I only get the thumbnail pictures for the first ten items, but today also got the thumbnail for the twelve item. The browser then stops downloading with 28 items remaining.

I can select any of the items from the site, except that the picture does not download.

I could not duplicate the behavior no matter how many images I requested. I used ab to generate a load of 10,000 image requests over the course of ~60 seconds and served them all up without issue. Quite puzzling.

I thought maybe these users were on dialup so I got on the modem and did some more testing. It was slow, but the browser was able to get all the images.

It was then that I realized every machine I had fiddled with this was either a Mac or Linux so I fired up VNC to a Windows box at the office and discovered in IE 5.5 I could duplicate the behavior. Seems that on the first request of the page a handful of the images load up. After that no further images can be loaded until the browser is closed and reopened.

I put together a test page in HTML with 545 different image requests. The page loads all but 94 of the images (having loaded 451 images). Started with 20 Apache processes starting up, and bumped it up to 30. Turned off KeepAlive, turned up KeepAliveTimeOut, same results.

Some poking around the internet gets some interesting comments about images in IE, but it wasn't until I got down to the 19th item in the Google search that I found the answer.

Apparently there is a problem with jpeg images created by Photoshop 7. Photoshop embeds an XML preview into the jpeg, which causes IE to choke. I tracked down one of these images, and the xml (239 lines). The xml seems to be Apple's print formatting. It appears that even though these images were created with Photoshop and then resized with convert (of ImageMagick), the xml isn't cleaned out with convert.

1688 images (out of 3793) on the site have this XML. A simple command will get rid of it:
jpegtran -copy none <image> > <new image>

The none option makes sure no extra markers are copied from the source file. Cleaning up this image reduced the size from 21764 to 898 bytes.

Will have to be more careful using Photoshop 7, seems that the "save for web" option doesn't save the extra formatting in the image.

Posted by mike at 8:27 PM

Compile Modules into Apache Statically . . . Yes or No

Today I'm exploring trends in compiling modules statically into Apache. There are plenty of documents out there detailing the process, but haven't been able to track anything down that indicates what is considered best practice.

We statically compile all necessary modules into the Apache binary, but are making moves to implement Shibboleth for cross-institution authorization (so people elsewhere can get into our content). The Shibboleth modules require a DSO version of OpenSSL 0.9.7b.

Altering our build doesn't seem that difficult, recomiple Apache with everything statically compiled except ssl, and then add the LoadModule and AddModule directives to the configuration file.

The bigger question is do we run Sibboleth on our existing machines, or get separate machines to run the service. Seems like if there are different dependencies between the two services it would be better to have separate machines.

Posted by mike at 12:29 PM

September 23, 2003

Watching Simpsons for First Time in Months

Actually been awhile since I've watched any TV. Probably a grand total of 30 minutes over the summer. After a long day at work and an endless stack of requests piling up on top of each other I figure at least one episode is in order.

Wouldn't you know it the episode is one of those "collections" where Homer is in the hospital and the family is remembering old times. Bummer.

Posted by mike at 8:09 PM

September 21, 2003

Up to Speed with Gentoo

Over the past week I've rebuilt my first Linux box since deciding that my distro preference was most likely switching from Redhat to Gentoo. It's been a convincing experience.

I had ample time to get the machine rebuilt, and thought that at least the first time I should build from scratch using the stage 1 tarball. It took a long time to bootstrap (let it run overnight), but in the end it felt good to have built everything from source.

Just as rewarding as having complete optimization was getting the os running in such a bare-bones state, knowing exactly what was on the machine. I had to pause for a moment before installing additional applications to enjoy the simplicity of an initial install.

This is my first experience with a ports system, Portage is quite nice. Makes it incredibly easy to get up to date packages with all necessary dependancies. Was disheartened to see that installing emacs involved 35 dependancy packages, there goes the oh-so-clean-and-simple os.

I wonder how people who are using Gentoo for a production environment are managing the updates? Through Portage? Can I be sure it's always doing the right thing?

Gentoo's document makes the install process simple considering the user is doing most things from scratch. Just follow directions. Even if a person decides it's too much work I think it's worth at least one install just to get a good sense of everything that goes into installing and getting Linux running.

Posted by mike at 11:45 PM

September 18, 2003

Linux File System Standard

I have no idea why it took me so long to find an somewhat official document which outlines the standard for Linux directory structure. I haven't been seeking one actively because a few years back I couldn't find much and decided that it must be something that gets engrained with time and experience. I must not have been that good at Google back then or something, a search now gets a slew of docs.

I'm reading Linux: Rute User's Tutorial and Exposition (among other things) which includes a chapter with the Filesystem Heirarchy Standard. A document with clear explanation of the Linux filesystem and discussion of directory/file locations/purposed. Looks like a new version is being released on Oct. 1.

By now I'm pretty familiar with the conventions, but it's still interesting to read the rules and rationale. I know there are a few changes I'll be making in my future Linux configurations.

Posted by mike at 9:24 AM

September 17, 2003

Another Happy OS X User

A month ago I purchased a new printer. It's been sitting in the home office for some time, unused, primarily because we have an old Beige G3 300 which doesn't have USB and I wasn't terribly excited about getting the third-party USB PCI card up and running. Oh, and going to OS X on our office computer was not an option (or so I was told).

After spending one evening of trying to find the right software to get the USB card working in OS 9, and having the printer tools installation fail, I thought tonight I'd see if OS X would be any easier. I've been keeping an OS X partition on the office computer since the public beta*, but was only booting into it when I was using the machine.

I booted into OS X and found that after running the USB card installer, plugging in the printer, and running the printer software CD that I was printing, scanning and copying like no-ones business.

After showing my progress to Heidi she agreed to stay with OS X. Wohoo. It was easy to get all Heidi's stuff set up (email, web, Photoshop, word processing, spreadsheet) and going through it with her seemed to spark her interest. It's good news for me, I don't have to wait for the machine to reboot to use it. It's running 10.1.5, which isn't what I'm used to. Maybe at 10.3 we'll upgrade.

Glad to have another unix (or unix-like) machine around.

* Installing the public beta on a Beige G3 was quite an experience, had to develop a system of pulling off certain pieces of hardware to get the OS to install and then add them back later.

Posted by mike at 11:20 PM

Wasting Time with OpenSSH Upgrade

Since the buzz started yesterday afternoon about the OpenSSH vulnerability I've been working on getting machines updated.

I have four flavors of boxes that are running sshd; OS X, Linux, Solaris 8 with 32-bit libraries and Solaris with a mix of 32-bit and 64-bit applications/libraries. I figured I'd wait for Apple to release the OS X update and started on Linux . . . no problems. Creating a package for Solaris 32-bit was also a snap. Then I started on the 64-bit (around 8 last night).

Assuming everything would go as it had on my previous installs I configured, compiled, built and installed package for OpenSSH 3.7p1. When it was all running I attempted to ssh to the machine and after entering my password get a message:

Connection to finch.hsdb.tufts.edu closed.
I fiddled for a bit, turned on debugging and discovered one, not very helpful message in the debug dump of the daemon:
debug1: Received SIGCHLD.
Not terribly useful. Didn't find much on Google either, a few different threads about the shell sending the signal because corrupt libraries.

It was midnight and I had to be up early so I fiddled with the sshd_config file a bit and with the UseLogin set to yes I could actually get a shell going. Unfortunately the environment was all messed up, no X11 forwarding, PATH all messed up. But I figured it would do for the few people who need ssh on the machine. Supposedly, UseLogin is for old systems where sshd doesn't know how to authenticate or setup a login session.

During the night I thought about the corrupt libraries thing and first thing this morning checked out the libraries OpenSSH uses. zlib was 64-bit, OpenSSL was a few versions old and 32-bit. Grabbed a new version of OpenSSL and tried to build OpenSSH 3.7p1 64-bit, hoping the problem would be solved. No luck.

For a sanity check I went and grabbed OpenSSH 3.6p1, built it 64-bit using the most recent OpenSSL. Wouldn't you know it, works perfectly.

Note: Somewhere in the process I decided that it was probably annoying to users who might need to ssh into the machine if sshd kept going up and down, or was in debug mode so I started up a daemon on port 22 and then moved my testing to another port. Simple to do by setting Port in sshd_config and using ssh -p <> <>. Should have probably started there, but kept thinking "it's going to work this time."

Right now I'm glad I got SSH service restored on the machine, even though it is an older version (not as old as the 3.1p1 that was on the machine before I started).

I guess in some ways I'm back where I started . . . now got to figure out what exactly is causing the problem.

Posted by mike at 6:21 PM

Christmas Sightings

Was shopping at BJs today with the kids when my daughter yelled CHRISTMAS!!!

Sure enough there were two entire rows right at the front of the store with all kinds of trees, wrapping paper, blow up Santa Clauses, lighted lawn ornaments. Our favorite was a larger-than-life cartoonish blow-up Rudolph reindeer.

I always hear people complaining about how stores keep creeping Christmas buying earlier and earlier, never thought much about it. But in September, when summer is still lingering in the air? Just plain crazy.

Of course I must remember that we have several Christmas Tree Shops centers which have a handful of stores open all year round. One of them not to far from where I live and the parking lot is always packed.

Posted by mike at 3:04 PM

September 15, 2003

Back from Vacation, CPU in Permanent Spike

A nice welcome back from vacation. First thing I check this morning is the orca graphs and discover that the CPU was spiked on our primary webserver for the past 48 hours.

A quick ps -Af reveals that sometime on Saturday two requests came in for PDFs and attempts to generate them with FOP went out of control. After gathering a few pieces of data I killed the two FOP processes and we're back to normal.

We have yet to find which of the 340 PDF requests on Saturday sent the process running wild, ps will give us 82 characters of the command line request, which is too short to figure out the important details of the request.

Posted by mike at 10:19 AM

September 12, 2003

Seeing the Sun as it First Hits US Soil

Cadillac Mountain, in Acadia National Park, is the first place that the sun's rays hit in the US on most mornings of the year.

Pete and I climbed to the top of the mountain at 6am this morning to see the sun rise. There weren't many actual rays, the morning fog made ths sun look more like the moon.

Was worth the early morning.

Posted by mike at 11:54 PM

September 11, 2003

Battling with MySQL Indexes

Over the past week I got a stream of messages from users indicating that on one of the sites I admin there were some problems on certain pages. Of course I didn't have any details of the exact problem or which pages were failing so I started from scratch.

I intially thought the problem might be tied to lack of hardware (a bare-bones machine), but after setting up Zabbix it became clear that there was ample resources.

Spent an evening using ab to load-test pages and discovered most pages were normal except one, which pulls data from MySQL. I turned on MySQL's slow-query and found a multi-table join that was having problems. Went right to checking the indexes, which came up looking right with show index <table name>, but an explain <query> returned information indicating that the appropriate indexes weren't being used in the multi-table join. I couldn't figure out why the indexes weren't getting used.

After trying to decipher what the explain was demanding and failing multiple times I decided to show index on the individual tables, alter table drop index <index name> for each index and alter table add index (<column name>) for each column I wanted indexed. If I used syntax like alter table add index (<column name>,<column name>,...) I ended up getting several indexes all with the same name, which would create problems in my explain.

There is probably a good explanation for the index creation issue, but I found that creating indexes separately got rid of my slow query and reduced the page load times (under heavy load) from 59 sec to 1.2 sec.

I love indexes, especially when they work correctly.

Posted by mike at 11:38 PM

September 10, 2003

Using Zabbix for Performance Monitor/Grapher

I've wanted to get some sort of performance monitoring tool on the handful of Linux boxes I use. Nothing fancy, just something to graph vital signs (memory, cpu, etc).

I stumbled into Zabbix, which has a lot more functionality than I need, but the setup seemed simple so I tried it. Have put it on two machines so far and am quite pleased.

Zabbix is open source, comes with a number of C programs which monitor the system and sticks information in either MySQL or PostgreSQL, allowing the user to view and manage the monitoring in a PHP frontend. The setup takes less than 30 minutes (provided you are already running MySQL, Apache and PHP with GD).

The feature I like the most is being able to create custom graphs and screens. Zabbix allows me to plot any number of the checks on a single graph and then organize any number of graphs on a page.

My one complaint is that all the graphs default to 1 hour of history, which is useful in some cases, but most of the time I want to see more than that. I've created a set of 24-hour graphs by fiddling with the graph URL (can specify time period on the URL).

Feels good to have this running, should have done it a long time ago.

Posted by mike at 11:05 AM

September 8, 2003

RV and Yacht, Combined

No kidding.

Posted by mike at 3:49 PM

September 6, 2003

Sharing Dialup in Maine

On vacation with Pete and family for a week in Acadia (actually staying on Somes Sound). No cell phone coverage (or Verizon 1XRTT CDMA broadband) so the only connection to the outside is a dialup line.

After a small amount of fiddling we've gotten our VAN (Vacation Area Network) running. Using the 12" PowerBook's modem to dialup, and then enabling internet sharing (with Airport-equipped computers) which enables Pete to get online with his Sony VIO's wireless.

There was one hitch, Pete had problems connecting to a number of ports until I turned the firewall on the PB off.

Posted by mike at 11:31 PM

September 4, 2003

Problem with CVS Over SSH

A few weeks back I wrote about getting CVS set up over SSH between a few of our machines. Recently discovered a problem with it.

Each time you perform a checkout from cvs it creates a Root file in the CVS directory of the directory in the checked out module. The Root file contains the CVSROOT where the files were obtained. This is a problem when you set CVSROOT to something like this:

export CVSROOT=:ext:<username>@<machine name>:<cvsroot>
Why is it a problem? Because we have more than one person updating CVS on our boxes and if my username is in that CVS/Root file cvs thinks that it should connect to the server as me, and it prompts users for my password to connect to the CVS server. The environmental variable $CVSROOT is used only if the CVS/Root file doesn't exist.

There are a few ways to deal with this:

  1. The user updating CVS files uses the -d <other CVSROOT> flag which temporarily overrides the CVS/Root

  2. We set individuals CVSROOT to be something like (notice the missing username):
    export CVSROOT=:ext:<machine name>:<cvsroot>
    Which means that on any checkout or update SSH will use the person's login name on the machine, rather than a hard-coded one in the CVS/Root file.

We opted for number 2, and discovered there is no cvs method to reset the CVS/Root, so needed to rid ourselves of the Root files until the next complete checkout, at which time the userless CVSROOT will be used.

The remove command issued just inside the first dir of our CVS tree

find . -name 'Root' -exec rm {} \;

Posted by mike at 11:13 AM

September 3, 2003

OS X Crashes

I would have never expected this, but tonight I was attempting to connect to another machine at home (using "Connect to Server" in the Finder) and I got a greyed out screen with a message in it:

You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.
I did as instructed, the machine took forever to come back. I suspect doing some disk checking.

Apparently I'm not the only one who's seen OS X crash. Once in 6 months is pretty good, but would be better if never.

Posted by mike at 11:11 PM

September 2, 2003

New Apps Running on OS X

Over the past few days I've started running a few new applications on my laptop, thought I'd make a note about them.

1) A new Emacs. The version I had would only run in a terminal, and I wanted more. I was tempted to build from cvs, but the binary was too tempting. It's nice to have a fully functioning emacs on my local machine.

2) Adium (IM client). Am liking it much better than iChat. I'm a tabbed windows person. Adium also allows more control over alerting about new messages (screen names appear on dock icon). I got sick of iChat's organization of my screen names and only showing the first line of an IM until I clicked on the window. Of course I'd need to switch back if I decide to get an iSight.

3) I put MySQL on the laptop over the weekend. I had been attempting to run some intensive queries to get statistics from a few tables but running it on two different machines generated some slowness complaints from other users. I decided it was worth the effort to get my own instance of MySQL running with local copies of the tables (the data I'm gathering doesn't need to be completely up to date). MySQL install went as smoothly as expected.

4) Am trying PTHCPUMonitor for monitoring CPU and memory. I had been using X Resource Graph but wasn't terribly thrilled with it's look. PTHCPUMonitor is a bit more simplistic, but gives me what I need (and can run in the menu-bar).

5) No longer using NetNewsWire Lite. It is really a nice product, and gives me exactly what I'm looking for, but I'm not ready for an aggregator. I spend far too much time reading weblogs and news articles just because the aggregator indicates there are new messages. I suppose I need to learn to not treat it like an email client, or clean up the feeds. It definetely makes reading a large set of feeds easier. For now I'm back to visiting actual sites with the old-fashioned web browser, will have to update my links.

Not a lack of applications out there.

Posted by mike at 11:55 PM

September 1, 2003

Understanding Electical Wiring of 1928

Over the Labor day weekend we did a complete overhaul of our bathroom. For the most part it was just a lot of labor, but there was one piece I actually found quite challenging/rewarding—rewiring for a new vanity light.

Our house was built in 1928, so in tackling any remodel project we're never quite sure what will come up that will lead to hours of research online and several trips to Home Depot to rig up the new to work with the old.

Monday morning I was faced with putting on a new vanity light. I put in the box and pulled the wire through the wall, but when it came time to hook up the electricity I got stumped. In the 4 existing boxes (ceiling light, two vanity lights, switch box) there was a slew of wires coming and going to places. I did what seemed right theoretically, but ended up with lights that couldn't be turned off.

I called dad, who's an expert at wiring (built a museum about it at theschool where he teaches), and over the phone we used my multimeter to determine where the power was entering the bathroom and which wires were travelling from box to box. In the end we discovered that the wiring was running exactly opposite of intuition. We assumed the power went through the switch and then up through each of the light boxes. However, our tests indicated the power was coming into the ceiling light and then down through the vanity lights to the switch, exactly opposite of intuition.

In the end it was a rewarding experience in mapping flow of electricity. Once we figured out how the wires were set up I ended up solving the new-light problem in a way cleaner than my first approach.

Posted by mike at 10:59 PM