|
perlretut - Perl regular expressions tutorial
This page provides a basic tutorial on understanding, creating and
using regular expressions in Perl. It serves as a complement to the
reference page on regular expressions the perlre manpage. Regular expressions
are an integral part of the m//, s///, qr// and split
operators and so this tutorial also overlaps with
Regexp Quote-Like Operators in the perlop manpage and split in the perlfunc manpage.
Perl is widely renowned for excellence in text processing, and regular
expressions are one of the big factors behind this fame. Perl regular
expressions display an efficiency and flexibility unknown in most
other computer languages. Mastering even the basics of regular
expressions will allow you to manipulate text with surprising ease.
What is a regular expression? A regular expression is simply a string
that describes a pattern. Patterns are in common use these days;
examples are the patterns typed into a search engine to find web pages
and the patterns used to list files in a directory, e.g., ls *.txt
or dir *.*. In Perl, the patterns described by regular expressions
are used to search strings, extract desired parts of strings, and to
do search and replace operations.
Regular expressions have the undeserved reputation of being abstract
and difficult to understand. Regular expressions are constructed using
simple concepts like conditionals and loops and are no more difficult
to understand than the corresponding if conditionals and while
loops in the Perl language itself. In fact, the main challenge in
learning regular expressions is just getting used to the terse
notation used to express these concepts.
This tutorial flattens the learning curve by discussing regular
expression concepts, along with their notation, one at a time and with
many examples. The first part of the tutorial will progress from the
simplest word searches to the basic regular expression concepts. If
you master the first part, you will have all the tools needed to solve
about 98% of your needs. The second part of the tutorial is for those
comfortable with the basics and hungry for more power tools. It
discusses the more advanced regular expression operators and
introduces the latest cutting edge innovations in 5.6.0.
A note: to save time, 'regular expression' is often abbreviated as
regexp or regex. Regexp is a more natural abbreviation than regex, but
is harder to pronounce. The Perl pod documentation is evenly split on
regexp vs regex; in Perl, there is more than one way to abbreviate it.
We'll use regexp in this tutorial.
The simplest regexp is simply a word, or more generally, a string of
characters. A regexp consisting of a word matches any string that
contains that word:
"Hello World" =~ /World/;
What is this Perl statement all about? "Hello World" is a simple
double quoted string. World is the regular expression and the
// enclosing /World/ tells Perl to search a string for a match.
The operator =~ associates the string with the regexp match and
produces a true value if the regexp matched, or false if the regexp
did not match. In our case, World matches the second word in
"Hello World", so the expression is true. Expressions like this
are useful in conditionals:
if ("Hello World" =~ /World/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
There are useful variations on this theme. The sense of the match can
be reversed by using the !~ operator:
if ("Hello World" !~ /World/) {
print "It doesn't match\n";
}
else {
print "It matches\n";
}
The literal string in the regexp can be replaced by a variable:
$greeting = "World";
if ("Hello World" =~ /$greeting/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
If you're matching against the special default variable $_, the
$_ =~ part can be omitted:
$_ = "Hello World";
if (/World/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
And finally, the // default delimiters for a match can be changed
to arbitrary delimiters by putting an 'm' out front:
"Hello World" =~ m!World!;
"Hello World" =~ m{World};
"/usr/bin/perl" =~ m"/perl";
/World/, m!World!, and m{World} all represent the
same thing. When, e.g., the quote (") is used as a delimiter, the forward
slash '/' becomes an ordinary character and can be used in this regexp
without trouble.
Let's consider how different regexps would match "Hello World":
"Hello World" =~ /world/;
"Hello World" =~ /o W/;
"Hello World" =~ /oW/;
"Hello World" =~ /World /;
The first regexp world doesn't match because regexps are
case-sensitive. The second regexp matches because the substring
'o W' occurs in the string "Hello World". The space
character ' ' is treated like any other character in a regexp and is
needed to match in this case. The lack of a space character is the
reason the third regexp 'oW' doesn't match. The fourth regexp
'World ' doesn't match because there is a space at the end of the
regexp, but not at the end of the string. The lesson here is that
regexps must match a part of the string exactly in order for the
statement to be true.
If a regexp matches in more than one place in the string, Perl will
always match at the earliest possible point in the string:
"Hello World" =~ /o/;
"That hat is red" =~ /hat/;
With respect to character matching, there are a few more points you
need to know about. First of all, not all characters can be used 'as
is' in a match. Some characters, called metacharacters, are reserved
for use in regexp notation. The metacharacters are
{}[]()^$.|*+?\
The significance of each of these will be explained
in the rest of the tutorial, but for now, it is important only to know
that a metacharacter can be matched by putting a backslash before it:
"2+2=4" =~ /2+2/;
"2+2=4" =~ /2\+2/;
"The interval is [0,1)." =~ /[0,1)./
"The interval is [0,1)." =~ /\[0,1\)\./
"#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/;
In the last regexp, the forward slash '/' is also backslashed,
because it is used to delimit the regexp. This can lead to LTS
(leaning toothpick syndrome), however, and it is often more readable
to change delimiters.
"#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!;
The backslash character '\' is a metacharacter itself and needs to
be backslashed:
'C:\WIN32' =~ /C:\\WIN/;
In addition to the metacharacters, there are some ASCII characters
which don't have printable character equivalents and are instead
represented by escape sequences. Common examples are \t for a
tab, \n for a newline, \r for a carriage return and \a for a
bell. If your string is better thought of as a sequence of arbitrary
bytes, the octal escape sequence, e.g., \033, or hexadecimal escape
sequence, e.g., \x1B may be a more natural representation for your
bytes. Here are some examples of escapes:
"1000\t2000" =~ m(0\t2) # matches
"1000\n2000" =~ /0\n20/ # matches
"1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
"cat" =~ /\143\x61\x74/ # matches, but a weird way to spell cat
If you've been around Perl a while, all this talk of escape sequences
may seem familiar. Similar escape sequences are used in double-quoted
strings and in fact the regexps in Perl are mostly treated as
double-quoted strings. This means that variables can be used in
regexps as well. Just like double-quoted strings, the values of the
variables in the regexp will be substituted in before the regexp is
evaluated for matching purposes. So we have:
$foo = 'house';
'housecat' =~ /$foo/;
'cathouse' =~ /cat$foo/;
'housecat' =~ /${foo}cat/;
So far, so good. With the knowledge above you can already perform
searches with just about any literal string regexp you can dream up.
Here is a very simple emulation of the Unix grep program:
% cat > simple_grep
$regexp = shift;
while (<>) {
print if /$regexp/;
}
^D
% chmod +x simple_grep
% simple_grep abba /usr/dict/words
Babbage
cabbage
cabbages
sabbath
Sabbathize
Sabbathizes
sabbatical
scabbard
scabbards
This program is easy to understand. #!/usr/bin/perl is the standard
way to invoke a perl program from the shell.
$regexp = shift; saves the first command line argument as the
regexp to be used, leaving the rest of the command line arguments to
be treated as files. while (<>) loops over all the lines in
all the files. For each line, print if /$regexp/; prints the
line if the regexp matches the line. In this line, both print and
/$regexp/ use the default variable $_ implicitly.
With all of the regexps above, if the regexp matched anywhere in the
string, it was considered a match. Sometimes, however, we'd like to
specify where in the string the regexp should try to match. To do
this, we would use the anchor metacharacters ^ and $. The
anchor ^ means match at the beginning of the string and the anchor
$ means match at the end of the string, or before a newline at the
end of the string. Here is how they are used:
"housekeeper" =~ /keeper/;
"housekeeper" =~ /^keeper/;
"housekeeper" =~ /keeper$/;
"housekeeper\n" =~ /keeper$/;
The second regexp doesn't match because ^ constrains keeper to
match only at the beginning of the string, but "housekeeper" has
keeper starting in the middle. The third regexp does match, since the
$ constrains keeper to match only at the end of the string.
When both ^ and $ are used at the same time, the regexp has to
match both the beginning and the end of the string, i.e., the regexp
matches the whole string. Consider
"keeper" =~ /^keep$/;
"keeper" =~ /^keeper$/;
"" =~ /^$/;
The first regexp doesn't match because the string has more to it than
keep. Since the second regexp is exactly the string, it
matches. Using both ^ and $ in a regexp forces the complete
string to match, so it gives you complete control over which strings
match and which don't. Suppose you are looking for a fellow named
bert, off in a string by himself:
"dogbert" =~ /bert/;
"dilbert" =~ /^bert/;
"bertram" =~ /^bert/;
"bertram" =~ /^bert$/;
"dilbert" =~ /^bert$/;
"bert" =~ /^bert$/;
Of course, in the case of a literal string, one could just as easily
use the string comparison $string eq 'bert' and it would be
more efficient. The ^...$ regexp really becomes useful when we
add in the more powerful regexp tools below.
Although one can already do quite a lot with the literal string
regexps above, we've only scratched the surface of regular expression
technology. In this and subsequent sections we will introduce regexp
concepts (and associated metacharacter notations) that will allow a
regexp to not just represent a single character sequence, but a whole
class of them.
One such concept is that of a character class. A character class
allows a set of possible characters, rather than just a single
character, to match at a particular point in a regexp. Character
classes are denoted by brackets [...], with the set of characters
to be possibly matched inside. Here are some examples:
/cat/;
/[bcr]at/;
/item[0123456789]/;
"abc" =~ /[cab]/;
In the last statement, even though 'c' is the first character in
the class, 'a' matches because the first character position in the
string is the earliest point at which the regexp can match.
/[yY][eE][sS]/;
This regexp displays a common task: perform a case-insensitive
match. Perl provides a way of avoiding all those brackets by simply
appending an 'i' to the end of the match. Then /[yY][eE][sS]/;
can be rewritten as /yes/i;. The 'i' stands for
case-insensitive and is an example of a modifier of the matching
operation. We will meet other modifiers later in the tutorial.
We saw in the section above that there were ordinary characters, which
represented themselves, and special characters, which needed a
backslash \ to represent themselves. The same is true in a
character class, but the sets of ordinary and special characters
inside a character class are different than those outside a character
class. The special characters for a character class are -]\^$ (and
the pattern delimiter, whatever it is).
] is special because it denotes the end of a character class. $ is
special because it denotes a scalar variable. \ is special because
it is used in escape sequences, just like above. Here is how the
special characters ]$\ are handled:
/[\]c]def/;
$x = 'bcr';
/[$x]at/;
/[\$x]at/;
/[\\$x]at/;
The last two are a little tricky. In [\$x], the backslash protects
the dollar sign, so the character class has two members $ and x.
In | 
