Personal Backup Version 5.2
	© 2012, J. Rathlev, IEAP, Kiel University

Syntax of Regular Expressions

You can use regular expressions in the program for filtering folder and file names as well as for extensions. Consider the different default settings:

• Folder and file names: filtering is case sensitive.
• Extensions: filtering is case insensitive

Using the modifier /i lets you change this behavior.

The syntax of regular expressions is described by the author of the libraries used (Andrey V. Sorokin) at RegExp Studio. Below is an offline adaption of this description corrected for proofreading errors:

Introduction

Regular expressions are a widely-used method of specifying patterns of text for which to search. Special metacharacters allow you to specify, for instance, that a particular string you are looking for occurs at the beginning or end of a line, or contains n recurrences of a certain character.

Regular expressions look ugly for novices, but in reality they are a very simple (well, relatively simple), handy and powerful tool.

Simple matches

Any single character matches itself, unless it is a metacharacter with a special meaning as described below.

A series of characters matches that series of characters, so the pattern bluh would match bluh in the target string. Quite simple!

You can cause characters that normally function as metacharacters or escape sequences to be interpreted literally by preceding them with the backslash \ escape character. For example, the metacharacter ^ matches the beginning of a string, but \^ matches the character ^ and similarly \\ matches \ and so on.

Examples:

foobar      matches the string foobar

\^FooBarPtr matches the string ^FooBarPtr

Escape sequences

Characters may be specified using an escape sequence syntax, much like that used in C and Perl, where \n matches a new line, \t a tab, etc. More generally, \xnn, where nn is a string of hexadecimal digits, matches the character whose ASCII value is nn.
If you need wide (Unicode) character code, you can use \x{nnnn}, where nnnn is a string of one or more hexadecimal digits.

Examples:

foo\x20bar matches the string foo bar (note the space in the center)

\tfoobar   matches the string foobar predefined by tab

Other characters that need the escape character \ are brackets () and [] and, as already mentioned, the backslash itself \

Character classes

You can specify a character class by enclosing a list of characters in [], which will match any one character from the list.

If the first character after the [ is ^, the class matches any character not in the list.

Examples:

foob[aeiou]r  finds the strings foobar, foober, etc. but not foobbr, foobcr, etc.

foob[^aeiou]r finds the strings foobbr, foobcr, etc. but not foobar, foober, etc.

Within a list, the - character is used to specify a range, so that a-z represents all characters between a and z, inclusive.

If you want the - character itself to be a member of a class, put it at the start or end of the list, or precede it with the \ escape character. If you want the ] character, you may place it at the start of the list or precede it with the \ escape character.

Examples:

[-az]     matches a, z and -

[az-]     matches a, z and -

[a\-z]    matches a, z and -

[a-z]     matches all twenty six lower case characters from a to z

[\n-\x0D] matches any of #10, #11, #12, #13

[\d-t]    matches any digit, - or t

[]-a]     matches any char from ] to a

Metacharacters

Metacharacters are special characters which are the essence of regular expressions. There are different types of metacharacters, as described below.

Metacharacters - line separators

Examples:

^foobar   matches the string foobar only if it occurs at the beginning of the line

foobar$  matches the string foobar only if it occurs at the end of the line

^foobar$ matches the string foobar only if it is the only string in the line

foob.r   matches strings such as foobar, foobbr, foob1r and so on

By default, the ^ metacharacter is only guaranteed to match at the beginning of the input string or text, the $ metacharacter only at the end. Embedded line separators will not be matched by ^ or $.

You may, however, wish to treat a string as a multi-line buffer, such that the ^ metacharacter will match after any line separator within the string, and $ will match before any line separator. You can do this by switching on the modifier /m.

The \A and \Z are just like ^ and $, except that they will not match multiple times when the modifier /m is in use, whereas ^ and $ will match at every internal line separator.

By default, the . metacharacter matches any character, but if you switch off the modifier /s, then the . metacharacter will not match embedded line separators.

TRegExpr works with line separators as recommended at www.unicode.org:

^ is at the beginning of an input string: if the modifier /m is on, the input string will be found when following an occurrence of \x0D\x0A or \x0A or \x0D.
If you are using the Unicode version of TRegExpr, then also the occurrences of \x2028 or \x2029 or \x0B or \x0C or \x85 will match.
Note that there is no empty line within the sequence \x0D\x0A.

$ is at the end of an input string: if the modifier /m is on, the input string will be found when preceding an occurrence of \x0D\x0A or \x0A or \x0D.
If you are using the Unicode version of TRegExpr, then also the occurrences of \x2028 or \x2029 or \x0B or \x0C or \x85 will match.
Note that there is no empty line within the sequence \x0D\x0A.

. matches any character: if the modifier /s if switched off, then . does not match \x0D\x0A and \x0A and \x0D.
If you are using the Unicode version of TRegExpr, then also the occurrences of \x2028 and \x2029 and \x0B and \x0C and \x85 will not match.

Note that ^.*$ (an empty line pattern) does not match the empty string within the sequence \x0D\x0A, but matches the empty string within the sequence \x0A\x0D.

Multiline processing can be easily tuned to suit your own purposes with help of the TRegExpr properties LineSeparators and LinePairedSeparator. You can use Unix style separators \n only, or DOS/Windows style \r\n only, or mix them together (as described above and used by default) or you can define your own line separators!

Metacharacters - predefined classes

You may use \w, \d and \s within custom character classes.

Examples:

foob\dr     matches strings such as foob1r, foob6r and so on, but not foobar, foobbr and so on

foob[\w\s]r matches strings such as foobar, foob r, foobbr and so on, but not foob1r, foob=r and so on

TRegExpr uses the properties SpaceChars and WordChars to define the character classes \w, \W, \s and \S, so you can easily redefine them.

Metacharacters - word boundaries

A word boundary \b is a spot between two characters that has a \w on one side of it and a \W on the other side of it (in either order), counting the imaginary characters of the beginning and end of the string as matching a \W.

Metacharacters - iterators

Any item of a regular expression may be followed by another type of metacharacters, the iterator. Using these metacharacters you can specify a number of occurrences of a previous character, of a metacharacter or of a subexpression.

Hence digits in curly brackets in the form {n,m} specify the minimum number of times the item n and the maximum m are to be matched. The form {n} is equivalent to {n,n} and matches exactly n times. The form {n,} matches n or more times. There is no limit to the size of n or m, but large numbers will consume more memory and slow down the execution of regular expressions.

If a curly bracket occurs in any other context, it is treated as a regular character.

Examples:

foob.*r     matches strings such as foobar, foobalkjdflkj9r and foobr

foob.+r     matches strings such as foobar, foobalkjdflkj9r but not foobr

foob.?r     matches strings such as foobar, foobbr and foobr but not foobalkj9r

fooba{2}r   matches the string foobaar

fooba{2,}r  matches strings such as foobaar, foobaaar, foobaaaar etc.

fooba{2,3}r matches strings such as foobaar, or foobaaar but not foobaaaar

A little explanation about "greediness": "greedy" takes as many as possible, "non-greedy" takes as few as possible.
For example, when applied to the string abbbbc:
b+ and b* return bbbb
b+? returns b
b*? returns an empty string
b{2,3}? returns bb
b{2,3} returns bbb

You can switch all iterators into "non-greedy" mode using the modifier /g.

Metacharacters - alternatives

You can specify a series of alternatives for a pattern using | to separate them, so that fee|fie|foe will match any fee, fie, or foe in the target string, as would f(e|i|o)e.

The first alternative includes everything from the last pattern delimiter (, [, or the beginning of the pattern up to the first |, while the last alternative contains everything from the last | to the next pattern delimiter. For this reason, it is common practice to include alternatives in parentheses, to minimize confusion about where they start and end.

Alternatives are tried from left to right, so the first alternative found for which the entire expression matches, is the one that is chosen. This means that alternatives are not necessarily greedy. For example: when matching (foo|foot) against barefoot, only the foo part will match, as that is the first alternative tried and it successfully matches the target string. This might not seem important, but becomes important when you are capturing matched text using parentheses.

Also remember that | is interpreted as a literal within square brackets, so if you write [fee|fie|foe] you are really only matching [feio|].

Examples:

foo(bar|foo) matches the strings foobar or foofoo

Metacharacters - subexpressions

The bracketing construct ( ... ) may also be used to define regular expressions subexpressions. After parsing you can find subexpression positions, lengths and actual values in MatchPos, MatchLen and Match properties of TRegExpr, and substitute these in template strings using TRegExpr.Substitute.

Subexpressions are numbered based on the left to right order of their opening parenthesis.

The first subexpression has the number 1 (the whole regular expressions match has the number 0 - you can substitute this by using the Replace directive <R> by $0 or $&).

Examples:

(foobar){8,10}  matches strings which contain 8, 9 or 10 instances of the foobar

foob([0-9]|a+)r matches foob0r, foob1r, foobar, foobaar, foobaar etc.

Metacharacters - backreferences

Metacharacters \1 through \9 are interpreted as backreferences; \<n> matches the previously-matched subexpression #<n>.

Examples:

(.)\1+          matches aaaa and cc

(.+)\1+         also matches abab and 123123

(['"]?)(\d+)\1 matches "13" (in double quotes), '4' (in single quotes), or 77 (without quotes), etc.

Modifiers

Modifiers are for changing the behavior of TRegExpr.

There are many ways to set up modifiers. Any modifier may be embedded within the regular expression itself using the (?...) construct.

Also, you can assign them to appropriate TRegExpr properties (a modifier for example to change /x, or ModifierStr to change all modifiers together). The default values for new instances of TRegExpr objects are defined in global variables, for example the global variable RegExprModifierX defines the property value ModifierX of a new TRegExpr instance.

Perform case-insensitive pattern matching (for use when installed in your system locale settings). See also InvertCase.

Treat a string as multiple lines. That is, change ^ and $ from matching only at the very start or the end of the string to matching at the start or the end of any line anywhere within the string. See also Line Separators.

Treat a string as a single line. That is, change . to match any character whatsoever, even a line separator (see also Line Separators), which it would normally not match.

A non-standard modifier. By default this modifier is on. Switching it off will switch all subsequent operators into non-greedy mode. In this case, + works as +?, * as *? and so on

Extend the legibility of your pattern by permitting whitespaces and comments (see explanation below).

Non-standard modifier. Its behavior depends on whether TRegExpr supports Unicode or not.
Personal Backup uses the Unicode version therefore if this modifier is set, the range а-я includes ё, А-Я includes Ё and А-я all Cyrillic characters used in Russian.

Note: The plain text version uses the Windows-1251 code table (not ISO-8859-5), hence the the range à-ÿ (= 0xE0..0xFF) includes ¸ (= 0xB8), À-ß (= 0xC0..0xDF) includes ¨ (= 0xA8) and à-ß (= 0xC0..0xFF) includes all Russian symbols.

In contrast to the original version of TRegExpr, in Personal Backup this modificator is not set by default.

The modifier (?x) itself needs a little more explanation. It tells TRegExpr to ignore a whitespace character that is neither backslashed nor within a character class. You can use this to break up your regular expression into (slightly) more readable parts. The # character is also treated as a metacharacter introducing a comment, for example:

This also means that, if you wish to use real whitespace or # characters in the pattern (outside a character class, where they are unaffected by /x), then you must either use an escape character or encode them using octal or hex escape characters. Taken together, these features go a long way towards making regular expressions text more readable.

Examples:

(?i)Saint-Petersburg       matches Saint-petersburg and Saint-Petersburg

(?i)Saint-(?-i)Petersburg  matches Saint-Petersburg, but not Saint-petersburg

(?i)(Saint-)?Petersburg    matches Saint-petersburg and saint-petersburg

((?i)Saint-)?Petersburg    matches saint-Petersburg, but not saint-petersburg

Miscellaneous

A comment, the text is ignored. Note that TRegExpr closes the comment as soon as it sees a the character ), so there is no way to put a literal character ) into the comment.