|
perlretut - Perl regular expressions tutorial
This page provides a basic tutorial on understanding, creating and
using regular expressions in Perl. It serves as a complement to the
reference page on regular expressions the perlre manpage. Regular expressions
are an integral part of the m//, s///, qr// and split
operators and so this tutorial also overlaps with
Regexp Quote-Like Operators in the perlop manpage and split in the perlfunc manpage.
Perl is widely renowned for excellence in text processing, and regular
expressions are one of the big factors behind this fame. Perl regular
expressions display an efficiency and flexibility unknown in most
other computer languages. Mastering even the basics of regular
expressions will allow you to manipulate text with surprising ease.
What is a regular expression? A regular expression is simply a string
that describes a pattern. Patterns are in common use these days;
examples are the patterns typed into a search engine to find web pages
and the patterns used to list files in a directory, e.g., ls *.txt
or dir *.*. In Perl, the patterns described by regular expressions
are used to search strings, extract desired parts of strings, and to
do search and replace operations.
Regular expressions have the undeserved reputation of being abstract
and difficult to understand. Regular expressions are constructed using
simple concepts like conditionals and loops and are no more difficult
to understand than the corresponding if conditionals and while
loops in the Perl language itself. In fact, the main challenge in
learning regular expressions is just getting used to the terse
notation used to express these concepts.
This tutorial flattens the learning curve by discussing regular
expression concepts, along with their notation, one at a time and with
many examples. The first part of the tutorial will progress from the
simplest word searches to the basic regular expression concepts. If
you master the first part, you will have all the tools needed to solve
about 98% of your needs. The second part of the tutorial is for those
comfortable with the basics and hungry for more power tools. It
discusses the more advanced regular expression operators and
introduces the latest cutting edge innovations in 5.6.0.
A note: to save time, 'regular expression' is often abbreviated as
regexp or regex. Regexp is a more natural abbreviation than regex, but
is harder to pronounce. The Perl pod documentation is evenly split on
regexp vs regex; in Perl, there is more than one way to abbreviate it.
We'll use regexp in this tutorial.
The simplest regexp is simply a word, or more generally, a string of
characters. A regexp consisting of a word matches any string that
contains that word:
"Hello World" =~ /World/;
What is this Perl statement all about? "Hello World" is a simple
double quoted string. World is the regular expression and the
// enclosing /World/ tells Perl to search a string for a match.
The operator =~ associates the string with the regexp match and
produces a true value if the regexp matched, or false if the regexp
did not match. In our case, World matches the second word in
"Hello World", so the expression is true. Expressions like this
are useful in conditionals:
if ("Hello World" =~ /World/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
There are useful variations on this theme. The sense of the match can
be reversed by using the !~ operator:
if ("Hello World" !~ /World/) {
print "It doesn't match\n";
}
else {
print "It matches\n";
}
The literal string in the regexp can be replaced by a variable:
$greeting = "World";
if ("Hello World" =~ /$greeting/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
If you're matching against the special default variable $_, the
$_ =~ part can be omitted:
$_ = "Hello World";
if (/World/) {
print "It matches\n";
}
else {
print "It doesn't match\n";
}
And finally, the // default delimiters for a match can be changed
to arbitrary delimiters by putting an 'm' out front:
"Hello World" =~ m!World!;
"Hello World" =~ m{World};
"/usr/bin/perl" =~ m"/perl";
/World/, m!World!, and m{World} all represent the
same thing. When, e.g., the quote (") is used as a delimiter, the forward
slash '/' becomes an ordinary character and can be used in this regexp
without trouble.
Let's consider how different regexps would match "Hello World":
"Hello World" =~ /world/;
"Hello World" =~ /o W/;
"Hello World" =~ /oW/;
"Hello World" =~ /World /;
The first regexp world doesn't match because regexps are
case-sensitive. The second regexp matches because the substring
'o W' occurs in the string "Hello World". The space
character ' ' is treated like any other character in a regexp and is
needed to match in this case. The lack of a space character is the
reason the third regexp 'oW' doesn't match. The fourth regexp
'World ' doesn't match because there is a space at the end of the
regexp, but not at the end of the string. The lesson here is that
regexps must match a part of the string exactly in order for the
statement to be true.
If a regexp matches in more than one place in the string, Perl will
always match at the earliest possible point in the string:
"Hello World" =~ /o/;
"That hat is red" =~ /hat/;
With respect to character matching, there are a few more points you
need to know about. First of all, not all characters can be used 'as
is' in a match. Some characters, called metacharacters, are reserved
for use in regexp notation. The metacharacters are
{}[]()^$.|*+?\
The significance of each of these will be explained
in the rest of the tutorial, but for now, it is important only to know
that a metacharacter can be matched by putting a backslash before it:
"2+2=4" =~ /2+2/;
"2+2=4" =~ /2\+2/;
"The interval is [0,1)." =~ /[0,1)./
"The interval is [0,1)." =~ /\[0,1\)\./
"#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/;
In the last regexp, the forward slash '/' is also backslashed,
because it is used to delimit the regexp. This can lead to LTS
(leaning toothpick syndrome), however, and it is often more readable
to change delimiters.
"#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!;
The backslash character '\' is a metacharacter itself and needs to
be backslashed:
'C:\WIN32' =~ /C:\\WIN/;
In addition to the metacharacters, there are some ASCII characters
which don't have printable character equivalents and are instead
represented by escape sequences. Common examples are \t for a
tab, \n for a newline, \r for a carriage return and \a for a
bell. If your string is better thought of as a sequence of arbitrary
bytes, the octal escape sequence, e.g., \033, or hexadecimal escape
sequence, e.g., \x1B may be a more natural representation for your
bytes. Here are some examples of escapes:
"1000\t2000" =~ m(0\t2) # matches
"1000\n2000" =~ /0\n20/ # matches
"1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
"cat" =~ /\143\x61\x74/ # matches, but a weird way to spell cat
If you've been around Perl a while, all this talk of escape sequences
may seem familiar. Similar escape sequences are used in double-quoted
strings and in fact the regexps in Perl are mostly treated as
double-quoted strings. This means that variables can be used in
regexps as well. Just like double-quoted strings, the values of the
variables in the regexp will be substituted in before the regexp is
evaluated for matching purposes. So we have:
$foo = 'house';
'housecat' =~ /$foo/;
'cathouse' =~ /cat$foo/;
'housecat' =~ /${foo}cat/;
So far, so good. With the knowledge above you can already perform
searches with just about any literal string regexp you can dream up.
Here is a very simple emulation of the Unix grep program:
% cat > simple_grep
$regexp = shift;
while (<>) {
print if /$regexp/;
}
^D
% chmod +x simple_grep
% simple_grep abba /usr/dict/words
Babbage
cabbage
cabbages
sabbath
Sabbathize
Sabbathizes
sabbatical
scabbard
scabbards
This program is easy to understand. #!/usr/bin/perl is the standard
way to invoke a perl program from the shell.
$regexp = shift; saves the first command line argument as the
regexp to be used, leaving the rest of the command line arguments to
be treated as files. while (<>) loops over all the lines in
all the files. For each line, print if /$regexp/; prints the
line if the regexp matches the line. In this line, both print and
/$regexp/ use the default variable $_ implicitly.
With all of the regexps above, if the regexp matched anywhere in the
string, it was considered a match. Sometimes, however, we'd like to
specify where in the string the regexp should try to match. To do
this, we would use the anchor metacharacters ^ and $. The
anchor ^ means match at the beginning of the string and the anchor
$ means match at the end of the string, or before a newline at the
end of the string. Here is how they are used:
"housekeeper" =~ /keeper/;
"housekeeper" =~ /^keeper/;
"housekeeper" =~ /keeper$/;
"housekeeper\n" =~ /keeper$/;
The second regexp doesn't match because ^ constrains keeper to
match only at the beginning of the string, but "housekeeper" has
keeper starting in the middle. The third regexp does match, since the
$ constrains keeper to match only at the end of the string.
When both ^ and $ are used at the same time, the regexp has to
match both the beginning and the end of the string, i.e., the regexp
matches the whole string. Consider
"keeper" =~ /^keep$/;
"keeper" =~ /^keeper$/;
"" =~ /^$/;
The first regexp doesn't match because the string has more to it than
keep. Since the second regexp is exactly the string, it
matches. Using both ^ and $ in a regexp forces the complete
string to match, so it gives you complete control over which strings
match and which don't. Suppose you are looking for a fellow named
bert, off in a string by himself:
"dogbert" =~ /bert/;
"dilbert" =~ /^bert/;
"bertram" =~ /^bert/;
"bertram" =~ /^bert$/;
"dilbert" =~ /^bert$/;
"bert" =~ /^bert$/;
Of course, in the case of a literal string, one could just as easily
use the string comparison $string eq 'bert' and it would be
more efficient. The ^...$ regexp really becomes useful when we
add in the more powerful regexp tools below.
Although one can already do quite a lot with the literal string
regexps above, we've only scratched the surface of regular expression
technology. In this and subsequent sections we will introduce regexp
concepts (and associated metacharacter notations) that will allow a
regexp to not just represent a single character sequence, but a whole
class of them.
One such concept is that of a character class. A character class
allows a set of possible characters, rather than just a single
character, to match at a particular point in a regexp. Character
classes are denoted by brackets [...], with the set of characters
to be possibly matched inside. Here are some examples:
/cat/;
/[bcr]at/;
/item[0123456789]/;
"abc" =~ /[cab]/;
In the last statement, even though 'c' is the first character in
the class, 'a' matches because the first character position in the
string is the earliest point at which the regexp can match.
/[yY][eE][sS]/;
This regexp displays a common task: perform a case-insensitive
match. Perl provides a way of avoiding all those brackets by simply
appending an 'i' to the end of the match. Then /[yY][eE][sS]/;
can be rewritten as /yes/i;. The 'i' stands for
case-insensitive and is an example of a modifier of the matching
operation. We will meet other modifiers later in the tutorial.
We saw in the section above that there were ordinary characters, which
represented themselves, and special characters, which needed a
backslash \ to represent themselves. The same is true in a
character class, but the sets of ordinary and special characters
inside a character class are different than those outside a character
class. The special characters for a character class are -]\^$ (and
the pattern delimiter, whatever it is).
] is special because it denotes the end of a character class. $ is
special because it denotes a scalar variable. \ is special because
it is used in escape sequences, just like above. Here is how the
special characters ]$\ are handled:
/[\]c]def/;
$x = 'bcr';
/[$x]at/;
/[\$x]at/;
/[\\$x]at/;
The last two are a little tricky. in [\$x], the backslash protects
the dollar sign, so the character class has two members $ and x.
In [\\$x], the backslash is protected, so $x is treated as a
variable and substituted in double quote fashion.
The special character '-' acts as a range operator within character
classes, so that a contiguous set of characters can be written as a
range. With ranges, the unwieldy [0123456789] and [abc...xyz]
become the svelte [0-9] and [a-z]. Some examples are
/item[0-9]/;
/[0-9bx-z]aa/;
/[0-9a-fA-F]/;
/[0-9a-zA-Z_]/;
If '-' is the first or last character in a character class, it is
treated as an ordinary character; [-ab], [ab-] and [a\-b] are
all equivalent.
The special character ^ in the first position of a character class
denotes a negated character class, which matches any character but
those in the brackets. Both [...] and [^...] must match a
character, or the match fails. Then
/[^a]at/;
/[^0-9]/;
/[a^]at/;
Now, even [0-9] can be a bother to write multiple times, so in the
interest of saving keystrokes and making regexps more readable, Perl
has several abbreviations for common character classes, as shown below.
Since the introduction of Unicode, these character classes match more
than just a few characters in the ISO 8859-1 range.
-
\d matches a digit, not just [0-9] but also digits from non-roman scripts
-
\s matches a whitespace character, the set [\ \t\r\n\f] and others
-
\w matches a word character (alphanumeric or _), not just [0-9a-zA-Z_]
but also digits and characters from non-roman scripts
-
\D is a negated \d; it represents any other character than a digit, or [^\d]
-
\S is a negated \s; it represents any non-whitespace character [^\s]
-
\W is a negated \w; it represents any non-word character [^\w]
-
The period '.' matches any character but "\n" (unless the modifier //s is
in effect, as explained below).
The \d\s\w\D\S\W abbreviations can be used both inside and outside
of character classes. Here are some in use:
/\d\d:\d\d:\d\d/;
/[\d\s]/;
/\w\W\w/;
/..rt/;
/end\./;
/end[.]/;
Because a period is a metacharacter, it needs to be escaped to match
as an ordinary period. Because, for example, \d and \w are sets
of characters, it is incorrect to think of [^\d\w] as [\D\W]; in
fact [^\d\w] is the same as [^\w], which is the same as
[\W]. Think DeMorgan's laws.
An anchor useful in basic regexps is the word anchor
\b. This matches a boundary between a word character and a non-word
character \w\W or \W\w:
$x = "Housecat catenates house and cat";
$x =~ /cat/;
$x =~ /\bcat/;
$x =~ /cat\b/;
$x =~ /\bcat\b/;
Note in the last example, the end of the string is considered a word
boundary.
You might wonder why '.' matches everything but "\n" - why not
every character? The reason is that often one is matching against
lines and would like to ignore the newline characters. For instance,
while the string "\n" represents one line, we would like to think
of it as empty. Then
"" =~ /^$/;
"\n" =~ /^$/;
"" =~ /./;
"" =~ /^.$/;
"\n" =~ /^.$/;
"a" =~ /^.$/;
"a\n" =~ /^.$/;
This behavior is convenient, because we usually want to ignore
newlines when we count and match characters in a line. Sometimes,
however, we want to keep track of newlines. We might even want ^
and $ to anchor at the beginning and end of lines within the
string, rather than just the beginning and end of the string. Perl
allows us to choose between ignoring and paying attention to newlines
by using the //s and //m modifiers. //s and //m stand for
single line and multi-line and they determine whether a string is to
be treated as one continuous string, or as a set of lines. The two
modifiers affect two aspects of how the regexp is interpreted: 1) how
the '.' character class is defined, and 2) where the anchors ^
and $ are able to match. Here are the four possible combinations:
-
no modifiers (//): Default behavior. '.' matches any character
except "\n". ^ matches only at the beginning of the string and
$ matches only at the end or before a newline at the end.
-
s modifier (//s): Treat string as a single long line. '.' matches
any character, even "\n". ^ matches only at the beginning of
the string and $ matches only at the end or before a newline at the
end.
-
m modifier (//m): Treat string as a set of multiple lines. '.'
matches any character except "\n". ^ and $ are able to match
at the start or end of any line within the string.
-
both s and m modifiers (//sm): Treat string as a single long line, but
detect multiple lines. '.' matches any character, even
"\n". ^ and $, however, are able to match at the start or end
of any line within the string.
Here are examples of //s and //m in action:
$x = "There once was a girl\nWho programmed in Perl\n";
$x =~ /^Who/;
$x =~ /^Who/s;
$x =~ /^Who/m;
$x =~ /^Who/sm;
$x =~ /girl.Who/;
$x =~ /girl.Who/s;
$x =~ /girl.Who/m;
$x =~ /girl.Who/sm;
Most of the time, the default behavior is what is wanted, but //s and
//m are occasionally very useful. If //m is being used, the start
of the string can still be matched with \A and the end of the string
can still be matched with the anchors \Z (matches both the end and
the newline before, like $), and \z (matches only the end):
$x =~ /^Who/m;
$x =~ /\AWho/m;
$x =~ /girl$/m;
$x =~ /girl\Z/m; | 
