February 27, 2003
Reading other Weblogs
I've narrowed down my list of weblogs to the two I'm currently reading, I doubt anyone cares they were taken off my list. The original list was put together when I was spending a lot more time reading weblogs. Now that it's become more of my regular routine need to clean out and rebuild.
Posted by mike at 11:37 AM
Engrossed in Work
I have been so engrossed with an issue at work over the past few days that I have barely taken time to check my email or read any weblogs, which is unusual. It's been such a consuming process that once it was resolved I went home and didn't turn a computer on all evening. That hasn't happened in a long time.
Over the past few days I spent almost every waking moment researching the issue and trying to find concrete pointers to the problem, thinking about ways to resolve the problem, implementing a fix, or running load tests against our site to see if the problem had gone away.
The issue was that very sporadically one of our Apache/mod_perl processes would die during XSLT transformation of XML (using libxml2 and libxslt). The frequency of the issue was directly proportional to the length of time the process had been in memory.
One might think all I ever do here is fix problems . . .
So I found the problem:
I know that looks fine and dandy, but what can't be seen here is that the Twig was declared globally in one of our libraries that parses and gets the XML ready for transformation. Declaring the twig globally meant that for as long as an Apache process was in memory, all XML would be sent through the same Twig object. This was difficult to pinpoint because only a portion of our request use the XML tranformation, and additionally, the Twig only became "corrupt" after a certain combination of XML requests had been passed through it.
The good news is that the fix solved issues we were having in more than one place, which makes the effort even more worth it.
Now . . . back to doing some actual development.
Posted by mike at 11:26 AM
February 23, 2003
How many calories?
I rarely think twice about eating something, but in my workout I've been doing some cardio. After two weeks of exercise I've been able to get up to a 5 mile run in 30 minutes. The machine tells me I'm buring ~500 calories in that time. So I got to wondering, how much if 500 calories . . . was dissapointed to learn that one Little Debbie (which I eat at least one of during lunch regularly) is around 400 calories. Haven't had once since I learned that little tid bit, way too much work for one little treat.
I am glad that my typical can of Dew has only 140 cals.
Posted by mike at 6:28 PM
February 21, 2003
The number of problems brought to our attention during any given week varies, but rarely a day goes by without something coming up. The issues are typically entered into Anthill, a bug/feature tracking system, or in an emergency we might get a call, email or visit.
We've been discussing how productivity suffers from having a system set up where ongoing projects are stopped at the whim of incoming problems. I find that for the most part my projects are either small (one or two days), or that I break longer projects into small parts, knowing that I will undoubtedly have other issues that require a shift of gears. Another developer on our team argues that my approach inhibits the success of our ongoing projects.
He argues that not being able to dedicate a week (or more) without interruption on a project hurts the long-term development efforts. He proposes that without long chunks of time to work on a project you can't build momentum and the project suffers because the developer never gets fully immersed.
I can't decide what I agree with and why. I think I prefer to have a number of opportunities during a day or week to complete something. The reality is that with three developers doing sysadmin, programming and maintenance of a site with a large set of functions running on a handful of servers it's unrealistic to think there will be weeks of silence for us to work on the next big thing. That being said, it's important to prioritize the incoming issues and organize them such that one can spend more time focusing on a specific issue instead of thrashing between many.
Posted by mike at 1:44 PM
February 18, 2003
Today's Challenge: Debug Embperl
Today's challenge is debugging an Embperl file. Embperl is somewhat like PHP in that with some special syntax you can move between HTML and Perl with ease. We use Embperl on top of our Perl libraries, purely as an HTML templating engine.
The problem: on this particular page perl core dumps on every nth request, n being the number of Apache processes running. The file is ~500 lines with around 100 calls to Perl the libraries, primarily to get lists of objects to format in HTML. The core dump doesn't appear to be coming from the underlying libraries because we use them on all our other pages without issue, and the fact that Embperl is successful on most of the requests makes me think there is something within the Embperl.
I have hopes that this script, embpexec.pl, that comes with embperl will enable me to get deeper into the problem. I have attempted so far to debug over the web, but am getting useless information. I have tried a few things that we've had trouble with before (variable scoping, missing header variables) but to no avail.
Running the Embperl page on the command line shows no problems, which means that the trouble is executing the page in the Apache environment. Possibly some trouble with the request object. Will start at the top and work through block by block using exit() to end early and hopefully pinpoint the trouble block.
This has drug on for several days now, it took a full day just to get a test that failed in a predictable way. After that I turned on a buttload of logging with this option in httpd.conf:
PerlSetEnv EMBPERL_LOG $server_root/logs/embperl_log
PerlSetEnv EMBPERL_DEBUG 3584
This logged all of Embperls actions and flushed out to the log file on every action. This allowed me to pinpoint a few things that were happening in the Embperl. Bottom line, some experimental syntax had been used and it was causing Embperl to misparse the page.
Embperl allows you to specify a process block which will retain scope across multiple blocks using [* *]. We couldn't figure out exactly what was happening, but since these blocks could also be processed with [$ $] we switched them and the problem dissappeared.
I also found that in a few places Embperl variables were acting differently based on what was passed through the Apache process on the last request. Found myself making sure that all variables were declared explicitly before use to prevent problems with data hanging around from a previous request.
I both hated and liked the experience. Hated because it was a week wasted, liked it because a very troubling problem has gone away.
Posted by mike at 2:59 PM
February 13, 2003
Pete got the kruckenberg machine on Internet2 for research and development. So today I did a traceroute between my machine at work and the kruckenberg server and was pleased to discover that the traffic between these machines goes through the Internet2 pipe.
4 ATM10-430-OC12-GIGAPOPNE.NOX.ORG (184.108.40.206) 1.335 ms 1.443 ms 1.381 ms
5 220.127.116.11 (18.104.22.168) 6.351 ms 6.740 ms 6.850 ms
6 chinng-nycmng.abilene.ucaid.edu (22.214.171.124) 33.318 ms 26.915 ms 31.272 ms
7 iplsng-chinng.abilene.ucaid.edu (126.96.36.199) 35.471 ms 30.941 ms 30.517 ms
8 kscyng-iplsng.abilene.ucaid.edu (188.8.131.52) 39.755 ms 39.978 ms 39.824 ms
9 184.108.40.206 (220.127.116.11) 52.072 ms 49.962 ms 50.428 ms
10 dnvrng-dnvr.abilene.ucaid.edu (18.104.22.168) 50.187 ms 50.168 ms 50.235 ms
What does that mean? I don't notice significant speed differences between kruckenberg and other machines I use. I'm sure it would be more noticeable if the public internet was flooded with traffic. I don't believe Internet2 promises better speeds as much as it promises private bandwidth.
Posted by mike at 9:42 AM
February 12, 2003
Apple's Academic Pricing
I've put in my request (to the school) for one of Apple's new 12" PowerBooks (so sweet). In the past I've only perused Apple's education discounts under the "shop for yourself" scenario ($100 savings on the PB). I was suprised when I used the "shop for your school" that the discount was $300 on the PB. That is quite a discount.
The base model 12" PowerBook rings up at $1499.
But of course, instead of saving $ on it I'll jack up the ram, disk and other accessories.
Lest one should be tempted, I have purchased under Apple's education plan before and they do go through some checks to verify educational affiliation. Trying to remember, but I think I had to fax in my student id. Not sure if they called the school or not.
Posted by mike at 5:21 PM
Watching Streaming Nanog on Real Player for Linux
The other day I was working away and Pete sent me a link saying "I'm up next." The link was to a Real stream, I wasn't sure how well supported Real streams were on Linux (the server has been there for awhile).
I quickly did a search on Google and found the real player for Linux. Since time was critical, I grabbed the RPM, installed it, and without issues had the player installed.
The audio was clear. The video had remnants (esp when looking at the presentation slides from the laptop). There wasn't a lot of motion during the presentation so was about as good as I've seen from streaming Real over the internet. Didn't have any buffering problems (not sure what Nanog's streaming power is, but I'd imagine it's as good as any).
I am finding more and more how easy (and amazing) it is to bridge thousands of miles with a video camera and a microphone. I don't believe streaming could ever replace the real experience of being at a place, but it gets much closer than not being involved at all.
Posted by mike at 10:45 AM
Getting User Management off our Back with LDAP
I'm in the final stages of completing an LDAP integration project that has been well worth the programming time and migration efforts.
For years we have maintained a user table with personal information about each user (id,name,email, password etc). All the typical challenges of maintaining users applied, reset passwords, email address changes. The bigger challenge was getting the lists of users from the registrars office and creating all the accounts, getting passwords out etc. Big headache at the start of each school year.
So about a year ago Tufts rolled out "Enterprise Authentication," an LDAP service that tapped into each of the university's systems and contained "the source" of user information. We were anxious to get onto that bandwagon.
I spent a month or so modifying our code and creating a few modules that could be tacked on to our authentication process (which was hard coded to check against MySQL). After the modifications we were able to slip any number of modules into our configuration and the system would walk through each one. After this was done we slipped an LDAP module (which uses OpenLDAP) in front of our MySQL module and relieved ourselves of most of the user-management responsibilities.
The last piece of this is getting authorization information from LDAP. Up to now we've only verified that the user can authenticate on LDAP and let them into our system either as an existing user with an authorization set, or as a public user with access to limited resources. With this final piece we will pull information from LDAP that will allow us to assign a role to the user, which comes with a set of authorization permissions.
Posted by mike at 9:23 AM
February 10, 2003
It's been around 2 years since I've done any significant exercise. Oh I walk to the subway from time to time and climb a lot of stairs to get to my office but nothing like years back when I did a lot of training for alpine climbing (last summit was Mt. Rainier in Washington).
I realized that over the past 2+ years I have prioritized mental over physical. Perhaps 75% of my free time was spent reading some OReilly book or working on some project late into the night (ie. early morning).
So three weeks ago I found a coupon in a local paper for a week trial at the YMCA across the street from the medical school. Some of the YMCAs I've been to in the past were a little scary, but this one is newly remodelled and the membership seems to be somewhat normal (prehaps the downtown location brings in professionals).
The trial was for 1 week, after which I decided I could commit to the routine. Three days a week I get on the T at 5:30am, workout at 6 and am in the office around 7. It is quite envigorating (of course I am a morning person).
In conclusion, after a long, lathargic period, it really feels good to have sore muscles again and feel better about sitting at a desk all day.
Posted by mike at 7:55 AM
February 6, 2003
Kevin Mitnick - combat social engineering
Just read Pete's blog about Kevin Mitnick. The event's of Kevin's life leading up to his jail time were prior to my path in technology, so it's interesting to go back and read up on it now having some history.
I'm particularly fascinated with his comments on social engineering. I find myself going through a checklist of each person or physical barrier which would be a path into our machines and trying to determine how much work it would take for a person to get through. This story from Kevin's Slashdot answers provoked the thought:
On one occasion, I was challenged by a friend of mine to get his Sprint Foncard number. He said he would buy me dinner if I could get it. I couldn't pass up a good meal so I phoned customer service and pretended to be from the IT department. I asked the rep if she was having any difficulties with her computer. She wasn't. I asked her the name of the system she uses to access customer accounts, to verify I was working with the right service center. She gave it to me. Immediately thereafter, I called back and got a new service rep. I told her my computer was down and I was trying to bring up a customer account. She brought it up on her terminal. I asked her for the customer's Foncard number? She started asking me a million questions? What was your name again? Who do you work for? What address are you at? You get the idea. Since I did not exercise any due diligence in my research, I just made up names and locations. It didn't work. She told me she was going to report my call to security!
Since I had her name, I briefed a friend of mine on the situation and asked him to pose as the "security investigator" so he could take a report. He called back customer service and was transferred to the woman. The "security investigator" said he received a report that unauthorized people were calling to obtain proprietary customer information. After getting the details of the "suspicious" call, the investigator asked what information the caller was after. She said the customer's Foncard number. The "investigator" asked for the number. She gave it to him. Whoops! Case closed!
And wouldn't you know it, immediately after reading this I had a user support person approach me complaining that it was inconvenient to deliver a recently reset password in person and that it would be better if when the person called they just did it over the phone. My response was noticeably sharper than normal.
Posted by mike at 9:17 AM
February 4, 2003
Using Linux in a non-Linux Environment
About 6 months ago I stuck my neck out (at least it felt like it then). As with many universities, Tufts tries to keep some control over it's fleet of desktops (and understandably so). The Office of Information Technology pushes recommendations out which are used when ordering and building machines. These machines then can subscribe to updates from a central location.
But I wanted to run Linux (KDE desktop). I had used it at my previous job, at home, and wanted to have it at the office as well.
So I wiped my hard drive and installed my favorite flavor of Linux. As a backup I partitioned the hard drive so I could put the old OS on if it became critical. I was nervous at first, I wasn't sure just how tightly controlled the fleet was, and how much what could be transferred to the KDE desktop.
Over the past six months I've really only run into 2 issues:
1) From time to time I get asked to work on an application issue. During these times I had no choice but to reboot, not a big deal. An even better solution came along recently when I got an old machine and stuck it on a KVM. I also had a coworker bring me up to speed on VMWare, which I fiddled with but didn't stick with.
2) Browser verification is a little more work. 90+% of our users are on PCs with IE. It's only a few times a month that I make changes which affect the design of the pages. For the most part if the design is up to spec and Mozilla renders it, IE can get it close too.
Overall it has been well worth the move, and in retrospect there really wasn't much I was risking. So glad I did it.
Posted by mike at 11:11 AM
New Weblog Title
You know, when I started my weblog I thought it would be about web services, but looking down my entries, very little is about web services. So I'm changing the title to Mike Kruckenberg's Experiences and Observations: random thoughts about programming, sysadmin and other unrelated subjects
You are welcome to keep thoughts on just how dumb that sounds to yourself.
Note: I also rolled back some chances to my stylesheet. They looked good in theory, but then I saw how crappy it looked in an older browser.
Posted by mike at 10:17 AM