NAME

perlfaq6 - Regular Expressions ($Revision: 10126 $)

DESCRIPTION

This section is surprisingly small because the rest of the FAQ is littered with answers involving regular expressions. For example, decoding a URL and checking whether something is a number are handled with regular expressions, but those answers are found elsewhere in this document (in the perlfaq9 manpage: "How do I decode or create those %-encodings on the web" and the perlfaq4 manpage: "How do I determine whether a scalar is a number/whole/integer/float", to be precise).

How can I hope to use regular expressions without creating illegible and unmaintainable code?

Three techniques can make regular expressions maintainable and understandable.

Comments Outside the Regex

Describe what you're doing and how you're doing it, using normal Perl comments.

        # turn the line into the first word, a colon, and the
        # number of characters on the rest of the line
        s/^(\w+)(.*)/ lc($1) . ":" . length($2) /meg;

Comments Inside the Regex

The /x modifier causes whitespace to be ignored in a regex pattern (except in a character class), and also allows you to use normal comments there, too. As you can imagine, whitespace and comments help a lot.

/x lets you turn this:

        s{<(?:[^>'"]*|".*?"|'.*?')+>}{}gs;

into this:

        s{ <                    # opening angle bracket
                (?:                 # Non-backreffing grouping paren
                        [^>'"] *        # 0 or more things that are neither > nor ' nor "
                                |           #    or else
                        ".*?"           # a section between double quotes (stingy match)
                                |           #    or else
                        '.*?'           # a section between single quotes (stingy match)
                ) +                 #   all occurring one or more times
                >                   # closing angle bracket
        }{}gsx;                 # replace with nothing, i.e. delete

It's still not quite so clear as prose, but it is very useful for describing the meaning of each part of the pattern.

Different Delimiters

While we normally think of patterns as being delimited with / characters, they can be delimited by almost any character. the perlre manpage describes this. For example, the s/// above uses braces as delimiters. Selecting another delimiter can avoid quoting the delimiter within the pattern:

        s/\/usr\/local/\/usr\/share/g;  # bad delimiter choice
        s#/usr/local#/usr/share#g;              # better

I'm having trouble matching over more than one line. What's wrong?

Either you don't have more than one line in the string you're looking at (probably), or else you aren't using the correct modifier(s) on your pattern (possibly).

There are many ways to get multiline data into a string. If you want it to happen automatically while reading input, you'll want to set $/ (probably to '' for paragraphs or undef for the whole file) to allow you to read more than one line at a time.

Read the perlre manpage to help you decide which of /s and /m (or both) you might want to use: /s allows dot to include newline, and /m allows caret and dollar to match next to a newline, not just at the end of the string. You do need to make sure that you've actually got a multiline string in there.

For example, this program detects duplicate words, even when they span line breaks (but not paragraph ones). For this example, we don't need /s because we aren't using dot in a regular expression that we want to cross line boundaries. Neither do we need /m because we aren't wanting caret or dollar to match at any point inside the record next to newlines. But it's imperative that $/ be set to something other than the default, or else we won't actually ever have a multiline record read in.

        $/ = '';                # read in more whole paragraph, not just one line
        while ( <> ) {
                while ( /\b([\w'-]+)(\s+\1)+\b/gi ) {   # word starts alpha
                        print "Duplicate $1 at paragraph $.\n";
                }
        }

Here's code that finds sentences that begin with "From " (which would be mangled by many mailers):

        $/ = '';                # read in more whole paragraph, not just one line
        while ( <> ) {
                while ( /^From /gm ) { # /m makes ^ match next to \n
                print "leading from in paragraph $.\n";
                }
        }

Here's code that finds everything between START and END in a paragraph:

        undef $/;               # read in whole file, not just one line or paragraph
        while ( <> ) {
                while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries
                    print "$1\n";
                }
        }

How can I pull out lines between two patterns that are themselves on different lines?

You can use Perl's somewhat exotic .. operator (documented in the perlop manpage):

        perl -ne 'print if /START/ .. /END/' file1 file2 ...

If you wanted text and not lines, you would use

        perl -0777 -ne 'print "$1\n" while /START(.*?)END/gs' file1 file2 ...

But if you want nested occurrences of START through END, you'll run up against the problem described in the question in this section on matching balanced text.

Here's another example of using ..:

        while (<>) {
                $in_header =   1  .. /^$/;
                $in_body   = /^$/ .. eof;
        # now choose between them
        } continue {
                $. = 0 if eof;  # fix $.
        }

I put a regular expression into $/ but it didn't work. What's wrong?

$/ has to be a string. You can use these examples if you really need to do this.

If you have File::Stream, this is easy.

        use File::Stream;

        my $stream = File::Stream->new(
                $filehandle,
                separator => qr/\s*,\s*/,
                );

        print "$_\n" while <$stream>;

If you don't have File::Stream, you have to do a little more work.

You can use the four argument form of sysread to continually add to a buffer. After you add to the buffer, you check if you have a complete line (using your regular expression).

        local $_ = "";
        while( sysread FH, $_, 8192, length ) {
                while( s/^((?s).*?)your_pattern/ ) {
                        my $record = $1;
                        # do stuff here.
                }
        }
        

 You can do the same thing with foreach and a match using the
 c flag and the \G anchor, if you do not mind your entire file
 being in memory at the end.

        local $_ = "";
        while( sysread FH, $_, 8192, length ) {
                foreach my $record ( m/\G((?s).*?)your_pattern/gc ) {
                        # do stuff here.
                }
        substr( $_, 0, pos ) = "" if pos;
        }

How do I substitute case insensitively on the LHS while preserving case on the RHS?

Here's a lovely Perlish solution by Larry Rosler. It exploits properties of bitwise xor on ASCII strings.

        $_= "this is a TEsT case";

        $old = 'test';
        $new = 'success';

        s{(\Q$old\E)}
        { uc $new | (uc $1 ^ $1) .
                (uc(substr $1, -1) ^ substr $1, -1) x
                (length($new) - length $1)
        }egi;

        print;

And here it is as a subroutine, modeled after the above:

        sub preserve_case($$) {
                my ($old, $new) = @_;
                my $mask = uc $old ^ $old;

                uc $new | $mask .
                        substr($mask, -1) x (length($new) - length($old))
    }

        $a = "this is a TEsT case";
        $a =~ s/(test)/preserve_case($1, "success")/egi;
        print "$a\n";

This prints:

        this is a SUcCESS case

As an alternative, to keep the case of the replacement word if it is longer than the original, you can use this code, by Jeff Pinyan:

        sub preserve_case {
                my ($from, $to) = @_;
                my ($lf, $lt) = map length, @_;

                if ($lt < $lf) { $from = substr $from, 0, $lt }
                else { $from .= substr $to, $lf }

                return uc $to | ($from ^ uc $from);
                }

This changes the sentence to "this is a SUcCess case."

Just to show that C programmers can write C in any programming language, if you prefer a more C-like solution, the following script makes the substitution have the same case, letter by letter, as the original. (It also happens to run about 240% slower than the Perlish solution runs.) If the substitution has more characters than the string being substituted, the case of the last character is used for the rest of the substitution.

        # Original by Nathan Torkington, massaged by Jeffrey Friedl
        #
        sub preserve_case($$)
        {
                my ($old, $new) = @_;
                my ($state) = 0; # 0 = no change; 1 = lc; 2 = uc
                my ($i, $oldlen, $newlen, $c) = (0, length($old), length($new));
                my ($len) = $oldlen < $newlen ? $oldlen : $newlen;

                for ($i = 0; $i < $len; $i++) {
                        if ($c = substr($old, $i, 1), $c =~ /[\W\d_]/) {
                                $state = 0;
                        } elsif (lc $c eq $c) {
                                substr($new, $i, 1) = lc(substr($new, $i, 1));
                                $state = 1;
                        } else {
                                substr($new, $i, 1) = uc(substr($new, $i, 1));
                                $state = 2;
                        }
                }
                # finish up with any remaining new (for when new is longer than old)