Filter Builder Tutorial

This tutorial demonstrates how to use Filter Builder to view, search, and manipulate the mail.log sample file, test regular expressions, and build stand-alone filters.

Opening the Sample Data

To open the sample mail log file included with the Perl Dev Kit (PDK):

1. On the File menu, select Open|Input....

2. Browse to the Samples/Filter/ directory of the PDK installation, select sample.log, and click Open.

3. A dialog box is displayed prompting you to:

     - load the whole file,
     - load only the first 1000 lines, or
     - load only the last 1000 lines.

   As the sample.log is only 1900 lines long, select Load the whole file.

Interactive Searching

The sample.log file is a typical mail server log. Each line contains information about a particular mail transaction. Each message will generally show two lines: one containing from information (including the sender and relay), the other containing delivery information (including the to address).

Each line begins with a time-stamp, the server's hostname (samplegate), the mail transfer agent (some-mta), a PID (the number in square braces), and a message queue ID assigned by the MTA.

To search the sample file:

1. Choose Select Pane from the View menu (if it is not already visible).

2. On the drop-down list, click starting with.

3. In the adjacent text box, enter Sep 13 22

The Input Pane shows the matching lines highlighted in gray, and the Output Pane displays only matching lines with the matching part highlighted in a configurable color (yellow by default).

To enter additional search criteria, click the Add another selector button by the top right corner of the Select Lines from Input pane. Click the palette button to the right of the text box to select an alternative highlighting color to associate with this search term.

By default, multiple selections are applied to the input data with OR relationships. For example, adding an additional selector such as containing : pcsharp.org adds entries to the results rather than narrowing the search. Click the And radio button in the top left of the Input pane to switch to an AND relationship. This setting applies to all selections; to search with more flexibility (e.g. 'foo' OR 'bar' AND 'baz') use a regular expression.

Testing Regular Expressions

Regular expressions can be difficult to construct effectively and will often match much more (or much less) than you expect them to. Using Filter Builder to interactively view the results and see the highlighted matched part can make creating regular expressions much easier.

To use a regular expression in a filter, select matching from the drop list at the left of the selector.

This example will find all messages that match the following criteria:

  - sent on September 14th
  - from the uniperl.com domain
  - approximately 10k or greater in size

Since each line starts with the date, anchor the date string to the beginning of the line with the '^' metacharacter:

  ^Sep 14

The Output Pane displays only entries from September 14. We now need to narrow our search to messages from the uniperl.com domain by adding .*from<*@uniperl.com,:

  ^Sep 14 .*from=<*@uniperl.com,

Before you finish entering the expression, the missing output indicates that a mistake has been made and no lines match the expression. Without a '.' character modifying the '*' quantifier, the expression attempts to match the '<' character zero or more times (which it does) and does not allow for any characters in the user part of the email address. To correct this, add a '.' to match any character:

  ^Sep 14 .*from=<.*@uniperl.com,

Finally, add a stipulation to match all entries with 5 or more digits in the size= field:

  ^Sep 14 .*from=<.*@uniperl.com, size=\d{5}

The Output Pane displays 21 lines that match these criteria.

Building Stand-Alone Filters: fgrep and grep

From the File menu, select Open | Filter... Browse to the Samples/Filter/ directory of the PDK installation and select fgrep.pl. (Filter Builder will prompt you to save your previous changes. Click Yes or No as desired.) If they are not already visible, the Description and Variables panes will appear.

This perl filter functions like the Unix fgrep command. It takes the search string from the variable $string which can be specified at the command line:

  Usage: fgrep.pl <string> [<file>...]

Run the filter with sample.log using the command:

  perl fgrep.pl "j.parker@example.org" sample.log

This prints all lines containing the email address j.parker@example.org to stdout. This Output can also be redirected to a file:

  perl fgrep.pl "j.parker@example.org" sample.log > results.log

The fgrep.pl filter can only search for fixed strings. To modify it to use regular expressions:

1. Change the Select drop-box from containing to matching

2. From the File menu, select Save | Filter As...

3. In the Save Filter dialog, enter the name grep.pl and click Save

Open a console window and test that grep.pl works as expected by using the regular expression created earlier:

  perl grep.pl "^Sep 14 .*from=<.*@uniperl.com, size=\d{5}" sample.log

Substitutions

Filter Builder is also able to replace the strings it finds. To build a filter with similar functionality to the Unix command sed:

1. Starting with the grep.pl filter created above, click the Add another variable button at the top right of the Variable pane.

2. In the New Variable dialog box, enter the variable name $replace.

3. From the View menu, select Replace Pane.

4. Click the Add another replacement line button by the top right corner of the Replace pane.

5. In the first (search) field enter $string and in the second (replacement) field enter $replace.

6. From the File menu, select Save | Filter As...

7. In the Save Filter dialog, enter the name replacer.pl and click Save

This creates a stand-alone perl filter that can replace text based on the arguments given:

  Usage: replacer.pl <string> <replace> [<file>...]

Open a console window and test that replacer.pl works as expected by substituting uniperl.com with example.net:

  perl replacer.pl "uniperl.com" "example.net" sample.log

Multiple search and replace variables can be added as desired.

Additional Resources

More information on building regular expressions can be found in the Perl regular expression quickstart and the Perl regular expression tutorial .