arrow-left

All pages
gitbookPowered by GitBook
1 of 1

Loading...

3.4. Regular Expression

How to use regular expressions for matching in Natex.

Regular expressions provide powerful ways to match strings and beyond:

  • Chapter 2.1: Regular Expressionsarrow-up-right, Chapter 2.1, Speech and Language Processing (3rd ed.), Jurafsky and Martin.

  • Regular Expression HOWTOarrow-up-right, Python Documentation

hashtag
Syntax

hashtag
Grouping

Syntax
Description

hashtag
Repetitions

Syntax
Description
Non-greedy

hashtag
Special Characters

Syntax
Description

hashtag
Functions

Several functions are provided in Python to match regular expressions.

hashtag
match()

Let us create a regular expression that matches "Mr." and "Ms.":

  • #1: imports the .

  • #3: the regular expression into the RE_MR.

circle-info

A regular expression is represented by r'expression' where the expression is in a string preceded by the special character r.

The above code prints None, indicating that the value of m is None, because the regular expression does not match the string.

  • #1: since RE_MR matches the string, m is a match object.

  • #3: true since m is a match object.

Currently, no are specified in RE_MR:

  • #1: returns an empty ().

circle-exclamation

What are the differences between a list and a tuple in Python?

It is possible to specific patterns using parentheses:

  • #1: there are two groups in this regular expression, (M[rs]) and (\.).

  • #3: returns a of matched substrings ('Ms', '.') for the two groups in #1.

The above RE_MR matches "Mr." and "Ms." but not "Mrs." Modify it to match all of them (Hint: use a non-capturing group and |).

The non-capturing group (?:[rs]|rs) matches "r", "s", or "rs" such that the first group matches "Mr", "Ms", and "Mrs", respectively.

Since we use the non-capturing group, the following code still prints a tuple of two strings:

hashtag
search()

Let us match the following strings with RE_MR:

  • #4: matches "Mr." but not "Ms."

  • #5: matches neither "Mr." nor "Mrs."

For s1, only "Mr." is matched because match() stops matching after finding the first pattern. For s2 on the other hand, even "Mr." is not matched because match() requires the pattern to be at the beginning of the string.

To match a pattern anywhere in the string, we need to for the pattern instead:

  • search() returns a match object as match() does.

hashtag
findall()

search() still does not return the second substrings, "Ms." and "Mrs.". The following shows how to substrings that match the pattern:

  • findall() returns a list of tuples where each tuple represents a group of matched results.

hashtag
finditer()

Since findall() returns a list of tuples instead of match objects, there is no definite way of locating the matched results in the original string. To return match objects instead, we need to the pattern:

  • #1: finditer() returns an that keeps matching the pattern until it no longer finds.

You can use a to store the match objects as a list:

  • #1: returns a list of all m (in order) matched by finditer().

circle-exclamation

How is the code above different from the one below?

What are the advantages of using a list comprehension over a for-loop other than it makes the code shorter?

Write regular expressions to match the following cases:

  • Abbreviation: Dr., U.S.A.

  • Apostrophe: '80, '90s

hashtag
Natex Integration

The nesting example in has a condition as follows (#4):

Write a regular expression that matches the above condition.

It is possible to use regular expressions for matching in Natex. A regular expression is represented by forward slashes (/../):

  • #4: true if the entire input matches the regular expression.

You can put the expression in a sequence to allow it a partial match:

  • #4: the regular expression is put in a sequence [].

circle-info

When used in Natex, all literals in the regular expression (e.g., "so", "good" in #4) must be lowercase because Natex matches everything in lowercase. The design choice is made because users tend not to follow typical capitalization in a chat interface, whether it is text- or audio-based.

hashtag
Variable

It is possible to store the matched results of a regular expression to variables. A variable in a regular expression is represented by angle brackets (<..>) inside a capturing group ((?..)).

The following transitions take the user name and respond with the stored first and last name:

  • #4: matches the first name and the last name in order and stores them in the variables FIRSTNAME and LASTNAME.

  • #5: uses FIRSTNAME and LASTNAME in the response.

0 or 1 repetitions

??

{m}

Exactly m repetitions

{m,n}

From m to n repetitions

{m,n}?

Any whitespace character

\S

Any non-whitespace character

\w

Any alphanumeric character and the underscore

\W

Any non-alphanumeric character

#4: matchesarrow-up-right the string "Dr. Choi" with RE_MR and saves the match objectarrow-up-right to m.

#4: prints the matched substring, and the startarrow-up-right (inclusive) and endarrow-up-right (exclusive) indices of the substring with respect to the original string in #1.

#4,5: return the entire match "Ms.".

  • #6: returns "Ms" matched by the first group (M[rs]).

  • #7: returns "." matched by the second group (\.).

  • What if we use a capturing group instead?

    Now, the nested group ([rs]|rs) is considered the second group such that the match returns a tuple of three strings as follows:

    ,
    'cause
  • Concatenation: don't, gonna, cannot

  • Hyperlink: https://github.com/emory-courses/cs329/

  • Number: 1/2, 123-456-7890, 1,000,000

  • Unit: $10, #20, 5kg

  • [ ]

    A set of characters

    ( )

    A capturing group

    (?: )

    A non capturing group

    |

    or

    .

    Any character except a newline

    *

    0 or more repetitions

    *?

    +

    1 or more repetitions

    +?

    ^

    The start of the string

    $

    The end of the string

    \num

    The contents of the group of the same number

    \d

    Any decimal digit

    \D

    Any non-decimal-digit character

    Regular Expresions 101arrow-up-right
    regular expression libraryarrow-up-right
    compilesarrow-up-right
    regex objectarrow-up-right
    groupsarrow-up-right
    tuple
    grouparrow-up-right
    tuplearrow-up-right
    searcharrow-up-right
    find allarrow-up-right
    interactively findarrow-up-right
    iteratorarrow-up-right
    list comprehensionarrow-up-right
    Section 3.1

    ?

    \s

    RE_TOK = re.compile(r'([",.]|n\'t|\s+)')
    RE_ABBR = re.compile(r'((?:Mr|Mrs|Ms|Dr)\.)|((?:[A-Z]\.){2,})')
    RE_APOS = re.compile(r'\'(\d\ds?|cause)')
    RE_CONC = re.compile(r'([A-Za-z]+)(n\'t)|(gon)(na)|(can)(not)')
    RE_HYPE = re.compile(r'(https?://\S+)')
    RE_NUMB = re.compile(r'(\d+/\d+)|(\d{3}-\d{3}-\d{4})|(\d(?:,\d{3})+)')
    RE_UNIT = re.compile(r'([$#])?(\d+)([km]g)?')
    import re
    
    RE_MR = re.compile(r'M[rs]\.')
    m = RE_MR.match('Dr. Wayne')
    print(m)
    m = RE_MR.match('Mr. Wayne')
    print(m)
    if m:
        print(m.group(), m.start(), m.end())
    <re.Match object; span=(0, 3), match='Mr.'>
    Mr. 0 3
    print(m.groups())
    RE_MR = re.compile(r'(M[rs])(\.)')
    m = RE_MR.match('Ms. Wayne')
    print(m.groups())
    print(m.group())
    print(m.group(0))
    print(m.group(1))
    print(m.group(2))
    ('Ms', '.')
    Ms.
    Ms.
    Ms
    .
    RE_MR = re.compile(r'(M(?:[rs]|rs))(\.)')
    print(RE_MR.match('Mrs. Wayne').groups())
    --> ('Mrs', '.')
    s1 = 'Mr. and Ms. Wayne are here'
    s2 = 'Here are Mr. and Mrs. Wayne'
    
    print(RE_MR.match(s1))
    print(RE_MR.match(s2))
    <re.Match object; span=(0, 3), match='Mr.'>
    None
    print(RE_MR.search(s1))
    print(RE_MR.search(s2))
    <re.Match object; span=(0, 3), match='Mr.'>
    <re.Match object; span=(9, 12), match='Mr.'>
    print(RE_MR.findall(s1))
    print(RE_MR.findall(s2))
    [('Mr', '.'), ('Ms', '.')]
    [('Mr', '.'), ('Mrs', '.')]
    for m in RE_MR.finditer(s1):
        print(m)
    <re.Match object; span=(0, 3), match='Mr.'>
    <re.Match object; span=(8, 11), match='Ms.'>
    for m in RE_MR.finditer(s2):
        print(m)
    <re.Match object; span=(9, 12), match='Mr.'>
    <re.Match object; span=(17, 21), match='Mrs.'>
    ms = [m for m in RE_MR.finditer(s1)]
    print(ms)
    [<re.Match object; span=(0, 3), match='Mr.'>, <re.Match object; span=(8, 11), match='Ms.'>]
    ms = []
    for m in RE_MR.finditer(s1):
        ms.append(m)
    '{[{so, very} good], fantastic}'
    r'((?:so|very) good|fantastic)'
    transitions = {
        'state': 'start',
        '`Hello. How are you?`': {
            '/((?:so|very) good|fantastic)/': {
                '`Things are just getting better for you!`': 'end'
            },
            'error': {
                '`Sorry, I didn\'t understand you.`': 'end'
            }
        }
    }
    S: Hello. How are you?
    U: So good!!!
    S: Things are just getting better for you!
    S: Hello. How are you?
    U: Fantastic :)
    S: Things are just getting better for you!
    S: Hello. How are you?
    U: It's fantastic
    S: Sorry, I didn't understand you.
    transitions = {
        'state': 'start',
        '`Hello. How are you?`': {
            '[/((?:so|very) good|fantastic)/]': {
                '`Things are just getting better for you!`': 'end'
            },
            'error': {
                '`Sorry, I didn\'t understand you.`': 'end'
            }
        }
    }
    S: Hello. How are you?
    U: It's fantastic!!
    S: Things are just getting better for you!
    S: Hello. How are you?
    U: I'm so good, thank you!
    S: Things are just getting better for you!
    transitions = {
        'state': 'start',
        '`Hello. What should I call you?`': {
            '[/(?<FIRSTNAME>[a-z]+) (?<LASTNAME>[a-z]+)/]': {
                '`It\'s nice to meet you,` $FIRSTNAME `. I know several people with the last name,` $LASTNAME': 'end'
            },
            'error': {
                '`Sorry, I didn\'t understand you.`': 'end'
            }
        }
    }
    S: Hello. What should I call you?
    U: Jinho Choi
    S: It's nice to meet you, jinho . I know several other choi .
    RE_MR = re.compile(r'(M([rs]|rs))(\.)')
    print(RE_MR.match('Mrs. Wayne').groups())
    --> ('Mr', 'rs', '.')