What are regular expressions?

Imagine you’re helping organize a community event and you’ve got a big text document full of messages from volunteers. Everyone sent their phone numbers in different formats:

Call me at 555-123-4567

My number is (555) 123 4567

5551234567

555.123.4567

You need to pull just the phone numbers from this mess, but ideally without spending hours scrolling and copying by hand. The usual ‘find’ tool won’t work here, so what do you do? Enter regular expressions!

Regular expressions

A regular expression (regex for short) is a series of characters that describe a pattern in a string of text according to a standardized set of rules. This pattern gets compared to the string and either it matches or doesn’t - it’s kind of like the ‘find’ function but with superpowers. You may be familiar with the concept of a wildcard character. For example, in Microsoft Word, the ? character allows you to search for ‘any character’. So, searching for c?t would find each of the following results: cat, cot, cut, c t.

Regular expressions are similar to this, but it’s a much more expanded and more powerful system. Regular expressions are also more or less standardized across programs (with some dialectal variations depending on context), whereas wildcards are usually program specific and therefore less useful.

Why are regular expressions useful?

Being able to match patterns in text allows you to be precise with what you’re finding and it also allows you to manipulate text in a more advanced and specific way than using something like the ‘find’ function. So, in our example above, with regular expressions you can isolate just the digits of the phone numbers (and format them all the same if you wanted, too!). Regular expressions can also be used in programming languages like Python or R, making them even more powerful. For example, you could write a Python script that uses regular expressions to extract specific information from a text file and then manipulate that information for a particular purpose. After learning regular expressions, it’s likely you’ll find yourself in many situations where they can be used to save time, where you might not have expected it!

Research use cases

Regular expressions become especially meaningful and useful when they are translated to your research context. Your knowledge of the structure and regular patterns of the data that you work with makes regular expressions a powerful means of finding the data that helps answer your research questions. The following are examples of varying granularity from different fields and different phases of research. The examples consist of processing OCR documents, anonymizing data, and information extraction. Processing OCR documents and anonymizing data are important early phases of data processing whereas information extraction can be closer to addressing a research question.

Removing unwanted characters from OCR’d documents

Processing OCR’d documents is an obvious use case, but should not be overlooked because of the variety of ways regular expressions are used. OCR’d documents contain unwanted data that may or may not be part of the original document.

1. Handwritten notes in the margin that result in undecipherable character sequences
2. Specks or marks from the physical document or the scanning process that result in punctuation marks or odd characters
3. Headers, footers, and page numbers that interfere with the flow of the text

These issues can be addressed with various regular expressions such as

• (?i)\b[^aeiou\s]{5,}\b to find words without vowels
• \s[.,;:!?—]\s to find specks or sequences of symbols
• (?i)page\s*\d+ to find page numbers

Anonymizing participant data

Regular expressions are used when research data contains structured information that needs to be anonymized, such as phone numbers or email addresses. The following regular expressions can be used to search and replace.

• Replace phone numbers (\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}) with (XXX) XXX-XXXX
• Replace email addresses (?i)\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b with XXXX@XXXX.XXX

Information extraction

In medical research, regular expressions have been found effective in targeted searches of medical records such as physician’s notes and radiological reports. An older study (Dennstadt et al, 2006) found regular expressions to be “an effective method for focused information extraction tasks related to high-priority disease areas such as hypertension”. A recent study (Turchin et al., 2026) reported that its authors “did not detect a statistically significant difference in the accuracy in extracting BI-RADS scores between Regex and an LLM-based method”.

Testing regular expressions

Before we continue, there is a resource you should pull up on your computer browser to follow along as we go.

regex101 is a free online tool for learning, building, and testing regular expressions. You can write your regular expression and write or paste in test strings or longer text to see the matches that you will get. There are many tools like this out there, but this is the one we usually use at Digital Scholarship at the Library. We’ll be linking to examples on this website a couple of times throughout this workshop so this is your introduction to it!