3.4. Regular Expression

Regular expressions provide powerful ways to match strings and beyond:

• Chapter 2.1: Regular Expressions, Chapter 2.1, Speech and Language Processing (3rd ed.), Jurafsky and Martin.

• Regular Expression HOWTO, Python Documentation

• Regular Expresions 101

Syntax

Grouping

Repetitions

Special Characters

Functions

Several functions are provided in Python to match regular expressions.

match()

Let us create a regular expression that matches "Mr." and "Ms.":

import re

RE_MR = re.compile(r'M[rs]\.')
m = RE_MR.match('Dr. Wayne')
print(m)

• #1: imports the regular expression library.

• #3: compiles the regular expression into the regex object RE_MR.

• #4: matches the string "Dr. Choi" with RE_MR and saves the match object to m.

The above code prints None, indicating that the value of m is None, because the regular expression does not match the string.

m = RE_MR.match('Mr. Wayne')
print(m)
if m:
    print(m.group(), m.start(), m.end())

• #1: since RE_MR matches the string, m is a match object.

• #3: true since m is a match object.

• #4: prints the matched substring, and the start (inclusive) and end (exclusive) indices of the substring with respect to the original string in #1.

<re.Match object; span=(0, 3), match='Mr.'>
Mr. 0 3

Currently, no groups are specified in RE_MR:

print(m.groups())

• #1: returns an empty tuple ().

It is possible to group specific patterns using parentheses:

RE_MR = re.compile(r'(M[rs])(\.)')
m = RE_MR.match('Ms. Wayne')
print(m.groups())
print(m.group())
print(m.group(0))
print(m.group(1))
print(m.group(2))

• #1: there are two groups in this regular expression, (M[rs]) and (\.).

• #3: returns a tuple of matched substrings ('Ms', '.') for the two groups in #1.

• #4,5: return the entire match "Ms.".

• #6: returns "Ms" matched by the first group (M[rs]).

• #7: returns "." matched by the second group (\.).

('Ms', '.')
Ms.
Ms.
Ms
.

RE_MR = re.compile(r'(M(?:[rs]|rs))(\.)')

print(RE_MR.match('Mrs. Wayne').groups())
--> ('Mrs', '.')

RE_MR = re.compile(r'(M([rs]|rs))(\.)')

print(RE_MR.match('Mrs. Wayne').groups())
--> ('Mr', 'rs', '.')

search()

Let us match the following strings with RE_MR:

s1 = 'Mr. and Ms. Wayne are here'
s2 = 'Here are Mr. and Mrs. Wayne'

print(RE_MR.match(s1))
print(RE_MR.match(s2))

• #4: matches "Mr." but not "Ms."

• #5: matches neither "Mr." nor "Mrs."

<re.Match object; span=(0, 3), match='Mr.'>
None

For s1, only "Mr." is matched because match() stops matching after finding the first pattern. For s2 on the other hand, even "Mr." is not matched because match() requires the pattern to be at the beginning of the string.

To match a pattern anywhere in the string, we need to search for the pattern instead:

print(RE_MR.search(s1))
print(RE_MR.search(s2))

• search() returns a match object as match() does.

<re.Match object; span=(0, 3), match='Mr.'>
<re.Match object; span=(9, 12), match='Mr.'>

findall()

search() still does not return the second substrings, "Ms." and "Mrs.". The following shows how to find all substrings that match the pattern:

print(RE_MR.findall(s1))
print(RE_MR.findall(s2))

• findall() returns a list of tuples where each tuple represents a group of matched results.

[('Mr', '.'), ('Ms', '.')]
[('Mr', '.'), ('Mrs', '.')]

finditer()

Since findall() returns a list of tuples instead of match objects, there is no definite way of locating the matched results in the original string. To return match objects instead, we need to interactively find the pattern:

for m in RE_MR.finditer(s1):
    print(m)

• #1: finditer() returns an iterator that keeps matching the pattern until it no longer finds.

<re.Match object; span=(0, 3), match='Mr.'>
<re.Match object; span=(8, 11), match='Ms.'>

for m in RE_MR.finditer(s2):
    print(m)

<re.Match object; span=(9, 12), match='Mr.'>
<re.Match object; span=(17, 21), match='Mrs.'>

You can use a list comprehension to store the match objects as a list:

ms = [m for m in RE_MR.finditer(s1)]
print(ms)

• #1: returns a list of all m (in order) matched by finditer().

[<re.Match object; span=(0, 3), match='Mr.'>, <re.Match object; span=(8, 11), match='Ms.'>]

ms = []
for m in RE_MR.finditer(s1):
    ms.append(m)

RE_TOK = re.compile(r'([",.]|n\'t|\s+)')
RE_ABBR = re.compile(r'((?:Mr|Mrs|Ms|Dr)\.)|((?:[A-Z]\.){2,})')
RE_APOS = re.compile(r'\'(\d\ds?|cause)')
RE_CONC = re.compile(r'([A-Za-z]+)(n\'t)|(gon)(na)|(can)(not)')
RE_HYPE = re.compile(r'(https?://\S+)')
RE_NUMB = re.compile(r'(\d+/\d+)|(\d{3}-\d{3}-\d{4})|(\d(?:,\d{3})+)')
RE_UNIT = re.compile(r'([$#])?(\d+)([km]g)?')

Natex Integration

The nesting example in Section 3.1 has a condition as follows (#4):

'{[{so, very} good], fantastic}'

r'((?:so|very) good|fantastic)'

It is possible to use regular expressions for matching in Natex. A regular expression is represented by forward slashes (/../):

transitions = {
    'state': 'start',
    '`Hello. How are you?`': {
        '/((?:so|very) good|fantastic)/': {
            '`Things are just getting better for you!`': 'end'
        },
        'error': {
            '`Sorry, I didn\'t understand you.`': 'end'
        }
    }
}

• #4: true if the entire input matches the regular expression.

S: Hello. How are you?
U: So good!!!
S: Things are just getting better for you!

S: Hello. How are you?
U: Fantastic :)
S: Things are just getting better for you!

S: Hello. How are you?
U: It's fantastic
S: Sorry, I didn't understand you.

You can put the expression in a sequence to allow it a partial match:

transitions = {
    'state': 'start',
    '`Hello. How are you?`': {
        '[/((?:so|very) good|fantastic)/]': {
            '`Things are just getting better for you!`': 'end'
        },
        'error': {
            '`Sorry, I didn\'t understand you.`': 'end'
        }
    }
}

• #4: the regular expression is put in a sequence [].

S: Hello. How are you?
U: It's fantastic!!
S: Things are just getting better for you!

S: Hello. How are you?
U: I'm so good, thank you!
S: Things are just getting better for you!

Variable

It is possible to store the matched results of a regular expression to variables. A variable in a regular expression is represented by angle brackets (<..>) inside a capturing group ((?..)).

The following transitions take the user name and respond with the stored first and last name:

transitions = {
    'state': 'start',
    '`Hello. What should I call you?`': {
        '[/(?<FIRSTNAME>[a-z]+) (?<LASTNAME>[a-z]+)/]': {
            '`It\'s nice to meet you,` $FIRSTNAME `. I know several people with the last name,` $LASTNAME': 'end'
        },
        'error': {
            '`Sorry, I didn\'t understand you.`': 'end'
        }
    }
}

• #4: matches the first name and the last name in order and stores them in the variables FIRSTNAME and LASTNAME.

• #5: uses FIRSTNAME and LASTNAME in the response.

S: Hello. What should I call you?
U: Jinho Choi
S: It's nice to meet you, jinho . I know several other choi .