I had the need today to come up with an apache rewrite that would, in some cases, change the value of a query string parameter. So for example, for requests coming from anywhere but example.com with a URI beginning “/path” and with a query string parameter named “foo” with the value beginning “bar”, I needed to rewrite the value to be “baz”. I spent some time fooling around with backreferences in the RewriteRule, but I never came up with anything that worked. Eventually, I turned to RewriteMap, which lets you specify text files, hashes, or even external scripts whose output values will be inserted into the destination for the rewrite. So in my example, here’s the apache config:

RewriteMap partner_params prg:/path/to/script.pl
RewriteCond %{HTTP_REFERER} !(.*)example.com(.*)
RewriteCond %{REQUEST_URI} ^/path
RewriteCond %{QUERY_STRING} .*foo=bar.*
RewriteRule ^(.*)$ http://mysite.com$1?${partner_params:%{QUERY_STRING}} [R=301,L]

The RewriteMap line points to a script (source to appear below) that will be executed by the RewriteRule. The first condition specifies that the referring url does not contain “example.com”. The second condition specifies that the URI begins with “/path”. And the third condition specifies that the query string must have a key named “foo” with a value beginning like “bar”. The rule itself takes any matching request and diverts it to http://mysite.com with the same URI as the original request (so something beginning “/path”), then adds a question mark to denote a query string following. Then it passes the query string to the script that RewriteMap knows as “partner_params”. that script reads from STDIN and prints to STDOUT either a newline-terminated result or the four-character response “NULL”. If not NULL, the response is what gets substituted into the RewriteRule.

So now for the script:

#!/usr/bin/perl
$| = 1;
while (<STDIN>) {
if($_ =~/foo=bar/){
$_ =~ s/foo=bar/foo=baz/gi;
}
print $_;
}

Here we simply do a substitution, looking for foo=bar and replacing with foo=baz. And voila, we’ve got custom inline query string munging based on parameters available via pretty standard apache Rewrite data. My particular example probably describes a pretty rare need, but knowing how to have a rewrite call a script to do more complex parsing than is available via apache configuration directives could be handy in a number of ways.

I’m usually pretty leery of using online services that I don’t administer for things that matter to me. For example, I’ve resisted a number of times using Google’s calendar for work purposes because there’s potentially sensitive information being posted to the calendar. So not only do I not have control over leaks of the data, but I don’t have control over backups, uptime of the service, etc., and this seems a lot of liability for something I need to make sure I’ve got access to. (Honestly, though, I think the smart folk over at Google are probably generally more competent than I am to guarantee uptime, backups, etc. — comparative benefits packages would suggest as much, at least.)

I’m very satisfied with one aspect of Google’s online service, however, and I’m consistently able to put aside my paranoia to use Google Docs for collaboration. Now I’d never store an important sensitive prose/text document there, but for planning server maintenance, the spreadsheet application is hard to beat. You share a document with everybody who’s involved, and everybody can view and edit the document at the same time. This past weekend, I was tasked with taking another shot at setting up replication between some mysql servers. We’ve set this up in the past but have lost confidence in the validity of the replication. So a coworker and I made another go of it this weekend. In preparation, I made a punch list of our steps, from putting up downtime pages and blocking access to the database at the firewall to pasting in commands for dumping data and resetting meta-data. I was able to color-code the steps by server so that it was easy to tell at a glance on what hardware to perform a step. And then as we went through the steps, we could update columns describing who performed a step and when. Of course, we’re coordinating this in a chat window as we’re doing the work, but it’s neat to watch the spreadsheet being updated interactively as we go, and this method provides a really simple, nice way to collaborate and keep a record of the process. Since the data’s not terribly sensitive (provided you don’t put passwords in), hosting it elsewhere doesn’t give me the heebie jeebies, and it’s nice to have a centralized repository of past maintenance events to build on for future maintenance. If there were a version you could download and install on your own hardware, I’d do it in a heartbeat and even use the apps for sensitive data, but then how would Google watch your every move and deliver search results based on the documents you create?

Blogged with the Flock Browser

I recently upgraded an install of phpbb to phpbb3. Shortly thereafter, I moved the site that the forum runs on to different hardware after several days of downtime on the original hardware (and an unresponsive vendor). To move to the new hardware, I dumped the database to a text file, compressed it, and shot the database and all site files across the network to the new hardware. Then I uncompressed the database and slurped it into mysql. Simple enough. What I hadn’t considered in advance was the fact that I was moving from mysql4 to mysql5. Accordingly, some weird things started happening when I started testing the site on the new hardware. I googled around a bit to discover that some of the problems were a result of the mysql upgrade, and I finally found this script, which purports to solve the problems by modifying the database structure. The script seemed to work just fine. The problems I had seen went away, and I figured the migration was a success.

But then somebody in the forums pointed out that bbcode throughout the site was messed up. And sure enough, all posts that had been imported had weird extra characters appended to bbcode blocks, which kept the bbcode from being converted into the appropriate html. For example, a block of bbcode might look like this: [quote="username"scd]stuff[/quote:scd]. But the characters were never consistent across posts. A bit more googling turned up the fact that phpbb has a field called bbcode_uid that is supposed to allow eight characters, but either when moving from mysql4 to mysql5 or as part of that nifty script I ran (I’m not sure which), the field gets truncated to five characters, which lops off the last three characters of an eight-character bbcode_uid, which ultimately results in the weird display we found.

What’s going on is that parsing nested tags (e.g. “[quote][b][url][/url][/b][/quote]“) can become laborious for the server, especially when tags don’t get closed properly. To make it more surefire and to simplify the process, phpbb appends a bbcode_uid to any bbcode inserted. So when you type “[url]http://daryl.learnhouston.com[/url]“, what actually gets inserted into the database is something like “[url:d98cJ1pv]http://daryl.learnhouston.com[/url:d98cJ1pv]“. This makes it so that you’re not having to figure out arbitrary nesting, because every opening tag has a corresponding unique end tag; you don’t have to find a beginning tag’s mate by parsing a string recursively, in other words. It’s a really cool idea. Of course, to remove the bbcode_uids from posts as a page is built, you need to store the bbcode_uid associated with a given post, so that it can be stripped out once tags are matched to one another. This is the bbcode_uid field in the posts table. And this field has just been truncated to five characters by the database move. Which means that when phpbb tries to find the bbcode_uid value within a given post, it finds and strips out only the first five characters, which results in three weird characters being appended to bbcode tags and the improper display of bbcode. In every single post and every single signature of your forums, which in my case was nearly 200,000 posts.

The fix is rather daunting to implement. Basically, you have to script something that looks at every single post and every single signature, finds bbcode_uids therein, matches the first five characters to the bbcode_uid field in the posts table (just as a check), and then updates the bbcode_uid for each post to the match found (this is after altering your table to make the bbcode_uid column accommodate eight characters, of course). If you get this wrong, you’ve basically wrecked your whole database, and bbcode for posts in the past will never render correctly. Of course, if you’ve discovered this problem before anybody has posted to your site, then you can alter the database and reimport the data, but this isn’t an option if people have been using the site for a few days before the issue was reported. Luckily, I was able to come up with a pretty simple script to fix the issue. Of course I was terrified to push the start button, so to speak, but push it I did, and it worked.

If you’re having the same issue, you can try my fix at your own risk.

My sleep is seldom affected by being one of a few people at my company who spends part of his time doing system administration, but this week has been a sure exception. We moved our whole public server infrastructure to a new section of our data center (complete with new IP addresses and routing), implemented load balancing of two separate clusters of web front-end machines, migrated two database servers to new hardware, and set up database replication for our web-facing databases. And we did it in sort of a last-minute, pre-product-launch scramble with what shoestring planning we could cobble together, while working on other high-priority projects and with very limited down time and, as far as I can tell, very little in the way of experience among our staff with implementing any of these things in a production environment. I’m not sure it could have gone more smoothly had we planned it for three months. It’s inexplicable, really. Of course, helping to make all this happen necessitated my putting in long hours over the weekend and waking up at times like 1:00 a.m. or 4:30 a.m. before or after an otherwise full workday to minimize the impact of down time. We coordinated this with sysadmins in Germany, California, and Tennessee and a data center in Texas. With my dad coming into town this weekend and a pumpkin-carving planned for tonight, I aim to take off around lunchtime (having started work at 4:30 this morning after staying up late to watch the Red Sox take the second game of the World Series) unless somebody threatens to fire me for doing so.

I’ve been running Fedora Core 4 on my laptop for a year and a half now, and I figured it was about time to update to something newer. I had tried Ubuntu briefly on the laptop at the very beginning but wasn’t impressed. I’ve also had Ubuntu running as the OS on my backup machine for a while, but I never use that machine manually, so I haven’t done much testing. Here go some miscellaneous notes after a couple of days of using Ubuntu (Edgy Eft). This is on my Sony Vaio VGN-FS550, which I absolutely love.

The install was fairly easy, and for the first time in the history of linux on laptops for me, the wireless just worked out of the box with no poking around to find drivers and just the right config. Setting up the widescreen was also pretty easy, if a little different from on FC4. On the latter, I had to google around until I learned about a tool called 915resolution that helps you set up widescreen displays, and then I added some config mumbo jumbo to my /etc/rc.local file to adjust the screen resolution on startup. Ubuntu, it turns out, has a nifty package for 915resolution (so no more hunting around — just do “apt-get install 915resolution”), but it doesn’t have /etc/rc.local, so I had to find out how to make it run on startup. Ubuntu does this through /etc/default. When you install a package that may run at startup, it gets a file added in here by the package installer, and you add config to that file. So voila, within a matter of minutes, I had my widescreen set up (compare to hours of clueless googling and trial and error to get it set up originally in FC4).

The Alps mouse was a little more difficult. To fix this, you have to google around and find an xorg.conf config that’s close to what you need and then tweak it until it works. To find out what device to use, you can cat /proc/bus/input/devices and look for the mouse. In my case, it looks like this:

I: Bus=0011 Vendor=0002 Product=0008 Version=7321
N: Name=”AlpsPS/2 ALPS GlidePoint”
P: Phys=isa0060/serio4/input0
S: Sysfs=/class/input/input6
H: Handlers=mouse1 event5 ts1
B: EV=f
B: KEY=420 0 70000 0 0 0 0 0 0 0 0
B: REL=3
B: ABS=1000003

The “Handlers” line in theory tells you what to plug into the InputDevice section in xorg.conf (which in my case looks like this):

Section “InputDevice”
Identifier “ALPS”
Driver “synaptics”
Option “CorePointer”
Option “SendCoreEvents” “true”
Option “Device” “/dev/psaux”
Option “Protocol” “auto-dev”
Option “Protocol” “auto-dev”
Option “LeftEdge” “120″
Option “RightEdge” “830″
Option “TopEdge” “120″
Option “BottomEdge” “650″
Option “FingerLow” “14″
Option “FingerHigh” “15″
Option “MaxTapTime” “180″
Option “MaxTapMove” “110″
Option “ClickTime” “0″
Option “EmulateMidButtonTime” “75″
Option “VertScrollDelta” “10″
Option “HorizScrollDelta” “10″
Option “MinSpeed” “0.45″
Option “MaxSpeed” “0.75″
Option “AccelFactor” “0.020″
Option “EdgeMotionMinSpeed” “200″
Option “EdgeMotionMaxSpeed” “200″
Option “UpDownScrolling” “1″
Option “CircularScrolling” “0″
Option “CircScrollDelta” “0.1″
Option “CircScrollTrigger” “2″
Option “SHMConfig” “true”
EndSection

Googling around revealed to me that the “Device” line should use something like “/dev/input/event5″ where the event number matches what you found in /proc/bus/input/devices. The problem I encountered was that the event number varied from bootup to bootup. So one time, the hard-coded xorg.conf would work, and another it wouldn’t. I finally settled on using /dev/psaux, and it seems to work, though I haven’t rebooted since the last time it worked, and I’m not convinced that it’s not still a little flakey. The last time I started up, it wasn’t working, and then I did ctrl-alt-f1 to get to a different screen and then ctrl-alt-f7 to get back to the main screen, and I wonder if that isn’t what made it start working. It was after that switch that the touchpad started functioning fully. Who knows what’ll happen the next time I reboot?

I should note that most of the touchpad functionality was fine out of the box. It was simply the bit that lets you slide your finger down the right-hand side to scroll a window that was busted. I use that a whole lot, so it was pretty high priority to get it fixed. This did eat up a couple of hours of time over a couple of days, but it was the only frustrating part of the install.

The mouse is still a little flakey. Sometimes when I click something (a link or a taskbar item, for example), it just waits until there’s another click or a mouse movement to register the click. I suspect this is something I can tune in xorg.conf, but for the moment, it’s only a minor annoyance that hasn’t justified the further research yet.

Ubuntu comes with a nifty little program called Tomboy that lets you take notes easily. It’s basically a local wiki. If you want to add a note, you just hit alt-f12, and a little menu pops up from nowhere out of the task bar showing recent notes and letting you add one. You can easily link notes together, do some basic text formatting, etc. I’m not sure it’s something I’ll use a lot, but it does strike me as potentially useful.

I’ve had cause this morning to try to write an apache module in perl, and that meant compiling Apache::Constants against apache, which I had installed using apt-get. Which meant that there were no source files lying around that I could find. After some fruitless googling, I eventually checked the man page for apt-get and saw that you can get sources using “apt-get source [package]“. I did this but had troubles with the configure command (”error 77: C compiler cannot create executable”), which I resolved by doing “apt-get install g++”. So now I had apache sources ready to compile the perl module against, but it turned out that the module wants apache 1.3. Which meant that my labors were more or less a dead end, save that I learned how to get package sources in ubuntu. Someone in the ubuntu irc channel suggested as an alternative to downloading sources that I could just install the -dev package, though frankly, I’m not sure where ubuntu would put the sources for that either, so it may not be a valid alternative for my purposes. (While writing this paragraph, I learned that — duh — there’s Apache2::Const, which was already installed and does for apache2 what Apache::Constants does for apache.)

Bootup for this machine now takes about 14 years, and I read somewhere that they actually sped things up significantly in this release. Those who installed the previous releases I gather are still waiting for their systems to be ready for use. It also bugs me that I can’t see the status of the system as it boots up. I’ve always found it comforting to see what part of the bootup process the system was in. I suspect there’s a way to get to this view. Luckily for me on all counts, I shut down rarely.

The battery life is crappy now. Under FC4, I could get a solid hour or hour-and-a-half out of a fully charged battery. The other day, plugged in downstairs, I ran the computer upstairs on battery for a minute, and it almost immediately told me I had only 18 minutes of battery left. On the plus side, Ubuntu’s battery indicator seems to work at least marginally with this system, telling me at minimum whether I’m on AC power or battery. This feature alone would have saved me a number of accidental shutdowns in the past due to inadvertent lack of AC power.

Ubuntu’s UI in general feels just a little more sluggish and less polished than that of FC4, but in general, I’m satisfied with it so far.

The ubuntu online presence and community seem outstanding.

Dilemma: I had a bunch of rotated apache log files that I wanted to check traffic patterns in to see if some link changes I had made to a site were affecting traffic. Specifically, for the domain in question, I had tried to route requests to certain urls over to another domain by changing links in the html, and I wanted to see if it was actually impacting traffic. So I wrote the following little bash script to iterate over a set of log files and print the date and the line count for specified search strings.

#!/bin/bash

if [ -z "$1" ]
then
echo “Usage: $0 search_string [log_file_prefix] [log_directory]”
echo “log_file_prefix defaults to ‘access_log.’”
echo “log_directory defaults to ‘.’”
exit
fi

if [ -z "$2" ]
then
2=”access_log.”
fi

if [ -z "$3" ]
then
3=”.”
fi

FILES=$(ls $3)

for FILE in $FILES
do
EPOCH=${FILE##*$2}
DATE=$(echo $EPOCH|awk ‘{print strftime(”%c”,$1)}’)
COUNT=$(cat $3/$FILE | grep “GET $1″ |wc -l)
echo “Looking for $1 in log for $DATE: $COUNT”
echo “”
done

So say I execute the command as follows: “./log_parse /about access_log_www. www”. It would scan for requests beginning with “/about” in all log files whose names begin with “access_log_www.” in the directory “www”. The script assumes that rotated log files are suffixed with a timestamp, and it writes the time based on that timestamp. Output looks something like this:

[root@www logs]# ./log_parse /about access_log_www. www
Looking for /about in log for Thu 08 Jun 2006 07:00:00 PM CDT: 726

Looking for /about in log for Fri 09 Jun 2006 07:00:00 PM CDT: 681

Looking for /about in log for Sat 10 Jun 2006 07:00:00 PM CDT: 28

It’s certainly not a full-service solution for log analysis, but it makes a quick check of one-off traffic patterns over time pretty easy to spot.

I wrote earlier about getting a hard drive added to one of my Linux boxes so that I could do backups. I’ve been meaning to get backups in order for many years, but it’s been more urgent since about 10 months ago (yeah yeah, must not’ve been too urgent if it waited that long), when I lost all the data on my laptop and had to recover what I could from a backup I had manually made some months before. More precious than most of the data I might have lost in that case are the photos and videos we’ve accumulated of Lennie over the past two years. When I had to send Mleeka’s laptop in to the shop a few months ago, I made an effort at dumping her files onto a Linux fileshare, but it wasn’t terribly successful, it wasn’t easy, and it wasn’t a long-term, self-sustaining solution. My project tonight was to put such a solution in place.

Backing up from Linux to Linux is simple, and I’ve had that going for several days. You just run rsync as a daemon on the fileshare, create a module, put the client’s public ssh key on the server, and run a cron job on the client that pushes to the module. All I had to to today to make this work on the new disk was move the files from one disk on the server to the other, edit the module in /etc/rsyncd.conf to reflect the new path, and restart rsync. Nothing to it.

Backing up the Windows box has been somewhat more challenging. I spent a few hours wrangling with a couple of different options for running rsync clients and daemons on the Windows laptop. I kept running into problems with weird Windows path issues, Windows firewall issues, other connection issues, permissions issues, and so on. It was very frustrating. Finally, I read a suggestion that one just mount the Windows directory on the Linux fileshare and then rsync locally. Simple and brilliant!

Ubuntu Linux appears not to ship with smbfs enabled, so after getting errors with the mount, I issued “apt-get install smbfs” to get that module. Then I did the following to create a mountpoint and a share:

mkdir /mnt/laptop_mleeka
mount -t smbfs //server/sharename /mnt/laptop_mleeka

This is of course after right-clicking properties on the My Documents folder on the Windows laptop and sharing the folder as “sharename.”

Once that was mounted, I issued the following:

rsync -a /mnt/laptop_mleeka/* houston@localhost::mleeka_bak

In this case, “mleeka_bak” at the end of the command is an rsync module I set up that manages permissions and file locations for the sync. The initial backup is running as I type. I understand that samba mounts are pretty slow, so it’s taking a while to haul the 9 GB of data across even my local network. Luckily, rsync is incremental, so in the future, it’ll sync only differences between the file systems.

All that remains is to add the mount command to /etc/fstab so that the server tries to remount the drive in the unlikely event of a reboot and to toss the rsync command itself into a cron job so that the backup takes place nightly.

Dilemma: I’ve got an old desktop system with less than 20 GB of disk space. On that system, I wanted to store backups of my laptop and our photos of Lennie. I also wanted to use it as a machine that will allow me to build software from time to time (which can take up a few GB of disk space). Without even starting backing up the many GBs of photos yet, I’ve nearly filled up the 20GB, and that’s after deleting a bunch of stuff. I happen to have another desktop system that has been on its last legs for a while, but it’s got a 65 GB hard drive.

Objective: Put the 65 GB hard drive in the working desktop system and make it operational.

Dilemma 2: I don’t really know anything about hardware. I forged ahead anyway, though, and I got it working. Here’s how.

Disclaimer: I did this on a Ubuntu Linux box. I have no idea how it’d go on any non-Linux box (so don’t ask). If you try these steps and you or your computer blow up or become otherwise harmed or incapacitated, note well that by reading further, you’re asserting that you’ve proceeded at your own risk.

1. Remove hard drive from dying computer. (This is pretty straightforward and doesn’t merit more detailed instructions.)
2. Read the back edge of the hard drive (the part with the pins) to figure out how to make it a slave so that the computer doesn’t get confused when it tries to boot up with two drives, This usually involves pulling a little plastic tab that slips over two pins off and sliding it back down onto two other prongs. I learned this the hard way by installing the drive, having the computer fail to boot, and remembering that something like this had to be done, whereupon I consulted the back edge of the hard drive.
3. Install the hard drive in a free bay in the working computer. This is pretty straighforward and, like initial removal, will require a screwdriver.
4. Cross your fingers that you’ve got the necessary cables available in your working computer’s case. If you don’t, you should probably consult some other guide. I was lucky. One cable is a ribbon cable with a plastic component that will receive the long array of pins on the back edge of the disk drive. The other is a little bundle of colored cables capped by a white plastic piece. It’s got four or five holds in the back for receiving pins. These plug into the obvious spots on the back edge of the disk drive.
5. Now boot up. The computer should boot as it did previously.
6. Get a terminal window open. If you don’t know how to do this, you probably shouldn’t go any further.
7. Find out what the disk is named. The following command will give you some output about your disks and partitions. You should look for one that’s the same size as the disk you installed and one that’s not listed in the output of the command “df”. If you don’t see one, something went wrong with the hardware install and I probably can’t help you.sudo fdisk -l
8. Make a directory to mount the drive on. I used /bak: Sosudo mkdir /bak
9. Now give your user permission on that directory. You might have to do something more involved if you need to allow more than one user to access the disk. This works for me, though.sudo chown -R houston /bak
10. Now partition the disk using fdisk. This is weird if you don’t know what you’re doing. I’ll walk you through my steps, which may or may not be the best to have taken.
    1. First do the following command, which takes you into a little interactive prompt (where /dev/hdb is the disk; note that it differs from /dev/hdb1, which I guess represents the partition on that disk; also note that you should substitute your disk identifier as above):sudo fdisk /dev/hdb
    2. At the command, enter “d” and then “1″ to delete the first partition on the disk. If there are other partitions, delete them as well.
    3. Then press “w” to write the changes to the disk and exit. This’ll probably take a few seconds or minutes.
    4. Next do sudo fdisk /dev/hdb again (substituting your disk identifier)
    5. Press “n” to add a partition. Chances are that you want just one, in which case you can just accept the defaults in the ensuing prompts.
    6. Then press “w” to write the changes to disk and exit.
11. Now format the new disk as an ext3 disk. Note that the /dev/hdb1 below is the same as what I discovered in step 7 to be my new disk. Substitute your value there.sudo mkfs /dev/hdb1 -t ext3
12. Ok, now you’ve got an ext3-formated disk with one partition. All that’s left is to mount it so that it can be used. To do this, edit /etc/fstab and add the following line to the end (again, substituting the drive identifier and directory the disk should be mounted on in columns one and two):/dev/hdb1 /bak ext3 defaults 0 0
13. Finally, typing the following command should mount the drive so that you can begin writing to it (substitute your path if needed):sudo mount /bak
14. Since we added the entry to /etc/fstab, the drive should come back mounted on reboot as well.