Archive for the ‘code’ Category

Using RewriteMap for query string voodoo

November 17th, 2008 by daryl

I had the need today to come up with an apache rewrite that would, in some cases, change the value of a query string parameter. So for example, for requests coming from anywhere but example.com with a URI beginning “/path” and with a query string parameter named “foo” with the value beginning “bar”, I needed to rewrite the value to be “baz”. I spent some time fooling around with backreferences in the RewriteRule, but I never came up with anything that worked. Eventually, I turned to RewriteMap, which lets you specify text files, hashes, or even external scripts whose output values will be inserted into the destination for the rewrite. So in my example, here’s the apache config:

RewriteMap partner_params prg:/path/to/script.pl
RewriteCond %{HTTP_REFERER} !(.*)example.com(.*)
RewriteCond %{REQUEST_URI} ^/path
RewriteCond %{QUERY_STRING} .*foo=bar.*
RewriteRule ^(.*)$ http://mysite.com$1?${partner_params:%{QUERY_STRING}} [R=301,L]

The RewriteMap line points to a script (source to appear below) that will be executed by the RewriteRule. The first condition specifies that the referring url does not contain “example.com”. The second condition specifies that the URI begins with “/path”. And the third condition specifies that the query string must have a key named “foo” with a value beginning like “bar”. The rule itself takes any matching request and diverts it to http://mysite.com with the same URI as the original request (so something beginning “/path”), then adds a question mark to denote a query string following. Then it passes the query string to the script that RewriteMap knows as “partner_params”. that script reads from STDIN and prints to STDOUT either a newline-terminated result or the four-character response “NULL”. If not NULL, the response is what gets substituted into the RewriteRule.

So now for the script:

#!/usr/bin/perl
$| = 1;
while (<STDIN>) {
if($_ =~/foo=bar/){
$_ =~ s/foo=bar/foo=baz/gi;
}
print $_;
}

Here we simply do a substitution, looking for foo=bar and replacing with foo=baz. And voila, we’ve got custom inline query string munging based on parameters available via pretty standard apache Rewrite data. My particular example probably describes a pretty rare need, but knowing how to have a rewrite call a script to do more complex parsing than is available via apache configuration directives could be handy in a number of ways.

Phpbb3 import error: bbcode_uid truncation

January 17th, 2008 by daryl

I recently upgraded an install of phpbb to phpbb3. Shortly thereafter, I moved the site that the forum runs on to different hardware after several days of downtime on the original hardware (and an unresponsive vendor). To move to the new hardware, I dumped the database to a text file, compressed it, and shot the database and all site files across the network to the new hardware. Then I uncompressed the database and slurped it into mysql. Simple enough. What I hadn’t considered in advance was the fact that I was moving from mysql4 to mysql5. Accordingly, some weird things started happening when I started testing the site on the new hardware. I googled around a bit to discover that some of the problems were a result of the mysql upgrade, and I finally found this script, which purports to solve the problems by modifying the database structure. The script seemed to work just fine. The problems I had seen went away, and I figured the migration was a success.

But then somebody in the forums pointed out that bbcode throughout the site was messed up. And sure enough, all posts that had been imported had weird extra characters appended to bbcode blocks, which kept the bbcode from being converted into the appropriate html. For example, a block of bbcode might look like this: [quote="username"scd]stuff[/quote:scd]. But the characters were never consistent across posts. A bit more googling turned up the fact that phpbb has a field called bbcode_uid that is supposed to allow eight characters, but either when moving from mysql4 to mysql5 or as part of that nifty script I ran (I’m not sure which), the field gets truncated to five characters, which lops off the last three characters of an eight-character bbcode_uid, which ultimately results in the weird display we found.

What’s going on is that parsing nested tags (e.g. “[quote][b][url][/url][/b][/quote]“) can become laborious for the server, especially when tags don’t get closed properly. To make it more surefire and to simplify the process, phpbb appends a bbcode_uid to any bbcode inserted. So when you type “[url]http://daryl.learnhouston.com[/url]“, what actually gets inserted into the database is something like “[url:d98cJ1pv]http://daryl.learnhouston.com[/url:d98cJ1pv]“. This makes it so that you’re not having to figure out arbitrary nesting, because every opening tag has a corresponding unique end tag; you don’t have to find a beginning tag’s mate by parsing a string recursively, in other words. It’s a really cool idea. Of course, to remove the bbcode_uids from posts as a page is built, you need to store the bbcode_uid associated with a given post, so that it can be stripped out once tags are matched to one another. This is the bbcode_uid field in the posts table. And this field has just been truncated to five characters by the database move. Which means that when phpbb tries to find the bbcode_uid value within a given post, it finds and strips out only the first five characters, which results in three weird characters being appended to bbcode tags and the improper display of bbcode. In every single post and every single signature of your forums, which in my case was nearly 200,000 posts.

The fix is rather daunting to implement. Basically, you have to script something that looks at every single post and every single signature, finds bbcode_uids therein, matches the first five characters to the bbcode_uid field in the posts table (just as a check), and then updates the bbcode_uid for each post to the match found (this is after altering your table to make the bbcode_uid column accommodate eight characters, of course). If you get this wrong, you’ve basically wrecked your whole database, and bbcode for posts in the past will never render correctly. Of course, if you’ve discovered this problem before anybody has posted to your site, then you can alter the database and reimport the data, but this isn’t an option if people have been using the site for a few days before the issue was reported. Luckily, I was able to come up with a pretty simple script to fix the issue. Of course I was terrified to push the start button, so to speak, but push it I did, and it worked.

If you’re having the same issue, you can try my fix at your own risk.

Flexible Drupal surveys

May 14th, 2007 by daryl

A few weeks ago, my company needed to publish a survey with a pretty flexible layout. Had we been constrained to the one-field-per-row constraint that Drupal’s survey module allows for, we would have had a very long an ugly form when in fact what we wanted was a nice tidy grid of small form controls that was much less imposing for users to consider filling out. So I hacked our old version of the survey module (for Drupal 4.6) to add a “layout” field to the fields tab for a survey. In the layout field, those who can administer surveys can specify markup and drop form fields into the markup using numeric placeholders wrapped in curly braces. Surveys for which no layout is defined use the default layout with numeric field weighting. I’ve submitted a patch to incorporate this functionality, and you can find the bug report with attached patch here if you’re in need of such functionality before the patch gets review or if the survey module developer declines to integrate it. The patch is for Drupal 5.x.

On a related note, I created a patch for the forms module that lets you add HTML between form elements. This hack/patch arose out of a need to stick a quick text snippet between two fields, where the snippet couldn’t be contained gracefully in the “description” line of the topmost of the fields. I add a form field type “html” that spits out whatever HTML you specify. I wrote this code (or the old version of it; this patch too is for Drupal 5.x) when my company needed explanatory HTML between two fields but the need to manipulate the form fields themselves hadn’t arisen yet. This functionality is good for insertion of quick snippets of HTML, where the first patch I mention above is best for layout overhauls.

Both patches probably represent security risks on sites that allow non-administrators to create forms or surveys, as I’m not (as yet) filtering content, so anybody who can create a survey can add arbitrary HTML. So apply the patch with that in mind.

Styling Drupal 5.x search forms

May 9th, 2007 by daryl

I’m working on a project that requires me to apply a fancy pants style to a Drupal search form. I thought this would be simple enough, as it’s pretty easy to override default themes for pretty much everything else in Drupal, but it turns out either that I’m a dolt or that there’s not much clarity out there on this topic. After screwing around with a lot of things (e.g. poring over debug_backtrace() output, writing die() statements all over the place, temporarily hacking the search and node modules, etc.), I searched Drupal’s site and found a promising link that turned out not to be the solution I needed (it simply didn’t work).

At long last, I tried creating a theme function named mytheme_search_form(). The search module has a function named search_form(), but in all my hair-pulling, I never saw anything that indicated that you could override this function by prepending your theme name to it (I would have expected to find calls to “theme(’search_form’, $args)” somewhere). At any rate, I ultimately created the above-named function and gave it the following definition:

function mytheme_search_form($form){
        return _phptemplate_callback('search_theme_form', array('form' => $form));
}

Then I created a file in my template directory named “search_theme_form.tpl.php” and built a custom form.

Next up was adding additional search fields to the form. To do this, I looked at node.module, where the function node_form_alter() adds fields to the default form. It didn’t seem necessary to jump through that hoop since my form was mostly hard-coded anyway, so I just added some radio buttons with the name “category” so that I could filter search results by taxonomy. Simple enough. But the search never actually filtered on my results. Here I did more hair-pulling and weird debugging. Finally, I went back to the hook_form_alter() functionality, having noticed a “processed_keys” key in the $form array. So to make my form honor my category search, I added the following things to my template.php file. It’s not clear to me how much of this is necessary, and I rather suspect I’m doing something stupid here, but it seems to work and I’m on deadline, so I’m rolling with it.

function mytheme_search_keys($type = null){
        $keys = search_get_keys();
        if($type){
                $keys = search_query_insert($keys, 'type', $type);
        }
        if($_POST['category']){
                $keys = search_query_insert($keys, 'category', $_POST['category']);
        }
        return $keys;
}

/**
 * Taking over this function so that I can call mytheme_search_validate to do the advanced search.
 */
function blog_form_alter($form_id, &$form){
        if($form_id == 'search_form'){
                $form['#validate']['mytheme_search_validate'] = array();
        }
}

/**
 * Need to call this to add category to the processed_keys array item so that
 * the category actually gets searched in the mini-advanced form we generate
 * in mytheme_search_form().
 */
function mytheme_search_validate($form_id, $form_values, &$form){
        $keys = $form_values['processed_keys'];
        $keys = mytheme_search_keys($form_values['module']);
        form_set_value($form['basic']['inline']['processed_keys'], $keys);
}

function faq_search($op = 'search', $keys){
        switch($op){
                case 'name':
                        return t('content');
                default:
                        $keys = mytheme_search_keys('faq');
                        return node_search('search', $keys);
        }
}

function forum_search($op = 'search', $keys){
        switch($op){
                case 'name':
                        return t('content');
                default:
                        $keys = mytheme_search_keys('forum');
                        return node_search('search', $keys);
        }
}

function blog_search($op = 'search', $keys){
        switch($op){
                case 'name':
                        return t('content');
                default:
                        $keys = mytheme_search_keys('blog');
                        return node_search('search', $keys);
        }
}

Now for an explanation. The “mytheme_search_keys()” function is a helper that lets me make sure I’m limiting the search to a given node type. It’s not clear to me that this is absolutely necessary, but things seem not to work if I don’t add the “type:” string to the search, so I’m leaving it in. Note that this function also looks for $_POST['category'] and adds it to the keys. If I wanted to add other search fields, I’d add them here as well. I suppose that since mytheme_search_validate() calls this function to set keys, I could eliminate the extra function and just do the same work in mytheme_search_validate().

Next up, blog_form_alter(). The hook_form_alter() functions are associated with modules, so I chose one I knew my site would be using that didn’t have a form_alter hook defined already. It feels kind of hacky, but it seems to work. The idea here is that we need to make the form run a validation function in order to add the keys we’re pulling in from mytheme_search_keys(). It was when I added this code that the category filter actually started working, so it seems to be a crucial bit. The key seems to be adding the keys to the $form['basic']['inline']['processed_keys'] array item, which seems to handle adding the search criteria to the URL and to the search itself.

Finally, I added the three _search() functions, which again feel a little extraneous, but the thing doesn’t work unless I add them, so they’re staying put for now. All we’re doing in these functions is adding the node type to the keys being searched (the search code extracts things like “type:blog” and “category:3″ from the query to do advanced searches) and then executing the node_search() function with the revised $keys value.

So, there you have it, the long way around to having a custom-themed Drupal search form with additional filters based on node type and category. Hope this saves somebody a few of the 10 or so hours I spent staring incredulously at my screen as solution after solution failed to work.

PHP5

February 24th, 2006 by daryl

Brushing up a little on some of the things available in PHP5. Here’s something that made me weep happy tears. Given the XML file:

<people>
    <person>
        <name>Daryl L. L. Houston></name>
        <coolness>100%></coolness>
    </person>
    <person>
        <name>Steve Urkel></name>
        <coolness>-100%></coolness>
    </person>
</people>

And the code:

<?php

$people = simplexml_load_file('test.xml');
foreach($people->person as $person){
        print $person->name . ' (' . $person->coolness . ')' . "\n";
}

?>

You get the output:

Daryl L. L. Houston (100%)
Steve Urkel (-100%)

No iterating over arrays, no finding the right PEAR class and making sure you’ve got the right versions of dependent PEAR classes installed. It Just Works™.

I’ve more or less avoided PHP5 to date, as much of the code I write wouldn’t benefit a great deal from many of the things I understand PHP5 to provide, but this alone makes it worth a second glance.