diff --git a/Documentation/Evolution/CharacterClasses.md b/Documentation/Evolution/CharacterClasses.md
deleted file mode 100644
index c9ffcbc95..000000000
--- a/Documentation/Evolution/CharacterClasses.md
+++ /dev/null
@@ -1,503 +0,0 @@
-# Character Classes for String Processing
-
-- **Authors:** [Nate Cook](https://github.com/natecook1000), [Michael Ilseman](https://github.com/milseman)
-- **Status:** Draft pitch
-
-## Introduction
-
-[Declarative String Processing Overview][overview] presents regex-powered matching broadly, without details concerning syntax and semantics, leaving clarification to subsequent pitches. [Regular Expression Literals][literals] presents more details on regex _syntax_ such as delimiters and PCRE-syntax innards, but explicitly excludes discussion of regex _semantics_. This pitch and discussion aims to address a targeted subset of regex semantics: definitions of character classes. We propose a comprehensive treatment of regex character class semantics in the context of existing and newly proposed API directly on `Character` and `Unicode.Scalar`.
-
-Character classes in regular expressions include metacharacters like `\d` to match a digit, `\s` to match whitespace, and `.` to match any character. Individual literal characters can also be thought of as character classes, as they at least match themselves, and, in case-insensitive matching, their case-toggled counterpart. For the purpose of this work, then, we consider a *character class* to be any part of a regular expression literal that can match an actual component of a string.
-
-## Motivation
-
-Operating over classes of characters is a vital component of string processing. Swift's `String` provides, by default, a view of `Character`s or [extended grapheme clusters][graphemes] whose comparison honors [Unicode canonical equivalence][canoneq].
-
-```swift
-let str = "Cafe\u{301}" // "Café"
-str == "Café" // true
-str.dropLast() // "Caf"
-str.last == "é" // true (precomposed e with acute accent)
-str.last == "e\u{301}" // true (e followed by composing acute accent)
-```
-
-Unicode leaves all interpretation of grapheme clusters up to implementations, which means that Swift needs to define any semantics for its own usage. Since other regex engines operate, at most, at the semantics level of Unicode scalar values, there is little to no prior art to consult.
-
-Other engines
-
-Character classes in other languages match at either the Unicode scalar value level, or even the code unit level, instead of recognizing grapheme clusters as characters. When matching the `.` character class, other languages will only match the first part of an `"e\u{301}"` grapheme cluster. Some languages, like Perl, Ruby, and Java, support an additional `\X` metacharacter, which explicitly represents a single grapheme cluster.
-
-| Matching `"Cafe\u{301}"` | Pattern: `^Caf.` | Remaining | Pattern: `^Caf\X` | Remaining |
-|---|---|---|---|---|
-| C#, Rust, Go | `"Cafe"` | `"´"` | n/a | n/a |
-| NSString, Java, Ruby, Perl | `"Cafe"` | `"´"` | `"Café"` | `""` |
-
-Other than Java's `CANON_EQ` option, the vast majority of other languages and engines are not capable of comparing with canonical equivalence.
-
-Broad language/engine survey
-
-For these definitions, we cross-referenced Unicode's [UTS\#18][uts18] with a broad survey of existing languages and engines. We found that while these all support a subset of UTS\#18, each language or framework implements a slightly different subset. The following table shows some of the variations:
-
-| Language/Framework | Dot (`.`) matches | Supports `\X` | Canonical Equivalence | `\d` matches FULL WIDTH digit |
-|------------------------------|----------------------------------------------------|---------------|---------------------------|-------------------------------|
-| [ECMAScript][ecmascript] | UTF16 code unit (Unicode scalar in Unicode mode) | no | no | no |
-| [Perl][perl] / [PCRE][pcre] | UTF16 code unit, (Unicode scalar in Unicode mode) | yes | no | no |
-| [Python3][python] | Unicode scalar | no | no | yes |
-| [Raku][raku] | Grapheme cluster | n/a | strings always normalized | yes |
-| [Ruby][ruby] | Unicode scalar | yes | no | no |
-| [Rust][rust] | Unicode scalar | no | no | no |
-| [C#][csharp] | UTF16 code unit | no | no | yes |
-| [Java][java] | Unicode scalar | yes | Only in CANON_EQ mode | no |
-| [Go][go] | Unicode scalar | no | no | no |
-| [`NSRegularExpression`][icu] | Unicode scalar | yes | no | yes |
-
-We are still in the process of evaluating [C++][cplusplus], [RE2][re2], and [Oniguruma][oniguruma].
-
-Rationale
_
-
-Unicode's recommended definition for `\d` is its [numeric type][numerictype] of "Decimal" in contrast to "Digit". It is specifically restricted to sets of ascending contiguously-encoded scalars in a decimal radix positional numeral system. Thus, it excludes "digits" such as superscript numerals from its [definition][derivednumeric] and is a proper subset of `Character.isWholeNumber`.
-
-We interpret Unicode's definition of the set of scalars, especially its requirement that scalars be encoded in ascending chains, to imply that this class is restricted to scalars which meaningfully encode base-10 digits. Thus, we choose to make this Character property _restrictive_, similar to `isHexDigit` and `isWholeNumber` and provide a way to access this value.
-
-It's possible we might add future properties to differentiate Unicode's non-decimal digits, but that is outside the scope of this pitch.
-
-Rationale
_
-
-Word characters include more than letters, and we went with Unicode's recommended scalar semantics. We extend to grapheme clusters similarly to `Character.isLetter`, that is, subsequent (combining) scalars do not change the word-character-ness of the grapheme cluster.
-
-Rationale
_
-
-Note that "whitespace" is a term-of-art and is not correlated with visibility, which is a completely separate concept.
-
-We use Unicode's recommended scalar semantics for horizontal whitespace and extend that to grapheme semantics similarly to `Character.isWhitespace`.
-
-We use ICU's definition for vertical whitespace, similarly extended to grapheme clusters.
-
-Rationale
_
-
-This approach simplifies the use of some common control characters, while making the rest available through a method call.
-
-Grammar Notation
@@ -392,7 +411,7 @@ For non-Unicode properties, only a value is required. These include:
- The special PCRE2 properties `Xan`, `Xps`, `Xsp`, `Xuc`, `Xwd`.
- The special Java properties `javaLowerCase`, `javaUpperCase`, `javaWhitespace`, `javaMirrored`.
-Note that the internal `PropertyContents` syntax is shared by both the `\p{...}` and POSIX-style `[:...:]` syntax, allowing e.g `[:script=Latin:]` as well as `\p{alnum}`.
+Note that the internal `PropertyContents` syntax is shared by both the `\p{...}` and POSIX-style `[:...:]` syntax, allowing e.g `[:script=Latin:]` as well as `\p{alnum}`. Both spellings may be used inside and outside of a custom character class.
#### `\K`
@@ -534,6 +553,7 @@ These operators have a lower precedence than the implicit union of members, e.g
To avoid ambiguity between .NET's subtraction syntax and range syntax, .NET specifies that a subtraction will only be parsed if the right-hand-side is a nested custom character class. We propose following this behavior.
+Note that a custom character class may begin with the `:` character, and only becomes a POSIX character property if a closing `:]` is present. For example, `[:a]` is the character class of `:` and `a`.
### Matching options
@@ -863,7 +883,23 @@ PCRE supports `\N` meaning "not a newline", however there are engines that treat
### Extended character property syntax
-ICU unifies the character property syntax `\p{...}` with the syntax for POSIX character classes `[:...:]`, such that they follow the same internal grammar, which allows referencing any Unicode character property in addition to the POSIX properties. We propose supporting this, though it is a purely additive feature, and therefore should not conflict with regex engines that implement a more limited POSIX syntax.
+ICU unifies the character property syntax `\p{...}` with the syntax for POSIX character classes `[:...:]`. This has two effects:
+
+- They share the same internal grammar, which allows the use of any Unicode character properties in addition to the POSIX properties.
+- The POSIX syntax may be used outside of custom character classes, unlike in PCRE and Oniguruma.
+
+We propose following both of these rules. The former is purely additive, and therefore should not conflict with regex engines that implement a more limited POSIX syntax. The latter does conflict with other engines, but we feel it is much more likely that a user would expect e.g `[:space:]` to be a character property rather than the character class `[:aceps]`. We do however feel that a warning might be warranted in order to avoid confusion.
+
+### POSIX character property disambiguation
+
+PCRE, Oniguruma and ICU allow `[:` to be part of a custom character class if a closing `:]` is not present. For example, `[:a]` is the character class of `:` and `a`. However they each have different rules for detecting the closing `:]`:
+
+- PCRE will scan ahead until it hits either `:]`, `]`, or `[:`.
+- Oniguruma will scan ahead until it hits either `:]`, `]`, or the length exceeds 20 characters.
+- ICU will scan ahead until it hits a known escape sequence (e.g `\a`, `\e`, `\Q`, ...), or `:]`. Note this excludes character class escapes e.g `\d`. It also excludes `]`, meaning that even `[:a][:]` is parsed as a POSIX character property.
+
+We propose unifying these behaviors by scanning ahead until we hit either `[`, `]`, `:]`, or `\`. Additionally, we will stop on encountering `}` or a second occurrence of `=`. These fall out the fact that they would be invalid contents of the alternative `\p{...}` syntax.
+
### Script properties
diff --git a/Documentation/Evolution/RegexTypeOverview.md b/Documentation/Evolution/RegexTypeOverview.md
index bce336551..68dd6ccc7 100644
--- a/Documentation/Evolution/RegexTypeOverview.md
+++ b/Documentation/Evolution/RegexTypeOverview.md
@@ -1,6 +1,11 @@
# Regex Type and Overview
-- Authors: [Michael Ilseman](https://github.com/milseman)
+* Proposal: [SE-0350](0350-regex-type-overview.md)
+* Authors: [Michael Ilseman](https://github.com/milseman)
+* Review Manager: [Ben Cohen](https://github.com/airspeedswift)
+* Status: **Active Review (4 - 28 April 2022)**
+* Implementation: https://github.com/apple/swift-experimental-string-processing
+ * Available in nightly toolchain snapshots with `import _StringProcessing`
## Introduction
@@ -134,11 +139,11 @@ Regexes can be created at run time from a string containing familiar regex synta
```swift
let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"#
-let regex = try! Regex(compiling: pattern)
+let regex = try! Regex(pattern)
// regex: Regex
let regex: Regex<(Substring, Substring, Substring, Substring, Substring)> =
- try! Regex(compiling: pattern)
+ try! Regex(pattern)
```
*Note*: The syntax accepted and further details on run-time compilation, including `AnyRegexOutput` and extended syntaxes, are discussed in [Run-time Regex Construction][pitches].
@@ -207,7 +212,7 @@ func processEntry(_ line: String) -> Transaction? {
// amount: Substring
// )>
- guard let match = regex.matchWhole(line),
+ guard let match = regex.wholeMatch(line),
let kind = Transaction.Kind(match.kind),
let date = try? Date(String(match.date), strategy: dateParser),
let amount = try? Decimal(String(match.amount), format: decimalParser)
@@ -226,7 +231,7 @@ The result builder allows for inline failable value construction, which particip
Swift regexes describe an unambiguous algorithm, where choice is ordered and effects can be reliably observed. For example, a `print()` statement inside the `TryCapture`'s transform function will run whenever the overall algorithm naturally dictates an attempt should be made. Optimizations can only elide such calls if they can prove it is behavior-preserving (e.g. "pure").
-`CustomMatchingRegexComponent`, discussed in [String Processing Algorithms][pitches], allows industrial-strength parsers to be used a regex components. This allows us to drop the overly-permissive pre-parsing step:
+`CustomPrefixMatchRegexComponent`, discussed in [String Processing Algorithms][pitches], allows industrial-strength parsers to be used a regex components. This allows us to drop the overly-permissive pre-parsing step:
```swift
func processEntry(_ line: String) -> Transaction? {
@@ -300,7 +305,7 @@ Regex targets [UTS\#18 Level 2](https://www.unicode.org/reports/tr18/#Extended_U
```swift
/// A regex represents a string processing algorithm.
///
-/// let regex = try Regex(compiling: "a(.*)b")
+/// let regex = try Regex("a(.*)b")
/// let match = "cbaxb".firstMatch(of: regex)
/// print(match.0) // "axb"
/// print(match.1) // "x"
@@ -384,21 +389,25 @@ extension Regex.Match {
// Run-time compilation interfaces
extension Regex {
/// Parse and compile `pattern`, resulting in a strongly-typed capture list.
- public init(compiling pattern: String, as: Output.Type = Output.self) throws
+ public init(_ pattern: String, as: Output.Type = Output.self) throws
}
extension Regex where Output == AnyRegexOutput {
/// Parse and compile `pattern`, resulting in an existentially-typed capture list.
- public init(compiling pattern: String) throws
+ public init(_ pattern: String) throws
}
```
+### Cancellation
+
+Regex is somewhat different from existing standard library operations in that regex processing can be a long-running task.
+For this reason regex algorithms may check if the parent task has been cancelled and end execution.
+
### On severability and related proposals
The proposal split presented is meant to aid focused discussion, while acknowledging that each is interconnected. The boundaries between them are not completely cut-and-dry and could be refined as they enter proposal phase.
Accepting this proposal in no way implies that all related proposals must be accepted. They are severable and each should stand on their own merit.
-
## Source compatibility
Everything in this proposal is additive. Regex delimiters may have their own source compatibility impact, which is discussed in that proposal.
@@ -422,7 +431,7 @@ Regular expressions have a deservedly mixed reputation, owing to their historica
* "Regular expressions are bad because you should use a real parser"
- In other systems, you're either in or you're out, leading to a gravitational pull to stay in when... you should get out
- - Our remedy is interoperability with real parsers via `CustomMatchingRegexComponent`
+ - Our remedy is interoperability with real parsers via `CustomPrefixMatchRegexComponent`
- Literals with refactoring actions provide an incremental off-ramp from regex syntax to result builders and real parsers
* "Regular expressions are bad because ugly unmaintainable syntax"
- We propose literals with source tools support, allowing for better syntax highlighting and analysis
@@ -488,6 +497,16 @@ The generic parameter `Output` is proposed to contain both the whole match (the
The biggest issue with this alternative design is that the numbering of `Captures` elements misaligns with the numbering of captures in textual regexes, where backreference `\0` refers to the entire match and captures start at `\1`. This design would sacrifice familarity and have the pitfall of introducing off-by-one errors.
+### Encoding `Regex`es into the type system
+
+During the initial review period the following comment was made:
+
+> I think the goal should be that, at least for regex literals (and hopefully for the DSL to some extent), one day we might not even need a bytecode or interpreter. I think the ideal case is if each literal was its own function or type that gets generated and optimised as if you wrote it in Swift.
+
+This is an approach that has been tried a few times in a few different languages (including by a few members of the Swift Standard Library and Core teams), and while it can produce attractive microbenchmarks, it has almost always proved to be a bad idea at the macro scale. In particular, even if we set aside witness tables and other associated swift generics overhead, optimizing a fixed pipeline for each pattern you want to match causes significant codesize expansion when there are multiple patterns in use, as compared to a more flexible byte code interpreter. A bytecode interpreter makes better use of instruction caches and memory, and can also benefit from micro architectural resources that are shared across different patterns. There is a tradeoff w.r.t. branch prediction resources, where separately compiled patterns may have more decisive branch history data, but a shared bytecode engine has much more data to use; this tradeoff tends to fall on the side of a bytecode engine, but it does not always do so.
+
+It should also be noted that nothing prevents AOT or JIT compiling of the bytecode if we believe it will be advantageous, but compiling or interpreting arbitrary Swift code at runtime is rather more unattractive, since both the type system and language are undecidable. Even absent this rationale, we would probably not encode regex programs directly into the type system simply because it is unnecessarily complex.
+
### Future work: static optimization and compilation
Swift's support for static compilation is still developing, and future work here is leveraging that to compile regex when profitable. Many regex describe simple [DFAs](https://en.wikipedia.org/wiki/Deterministic_finite_automaton) and can be statically compiled into very efficient programs. Full static compilation needs to be balanced with code size concerns, as a matching-specific bytecode is typically far smaller than a corresponding program (especially since the bytecode interpreter is shared).
@@ -497,7 +516,7 @@ Regex are compiled into an intermediary representation and fairly simple analysi
### Future work: parser combinators
-What we propose here is an incremental step towards better parsing support in Swift using parser-combinator style libraries. The underlying execution engine supports recursive function calls and mechanisms for library extensibility. `CustomMatchingRegexComponent`'s protocol requirement is effectively a [monadic parser](https://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf), meaning `Regex` provides a regex-flavored combinator-like system.
+What we propose here is an incremental step towards better parsing support in Swift using parser-combinator style libraries. The underlying execution engine supports recursive function calls and mechanisms for library extensibility. `CustomPrefixMatchRegexComponent`'s protocol requirement is effectively a [monadic parser](https://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf), meaning `Regex` provides a regex-flavored combinator-like system.
An issues with traditional parser combinator libraries are the compilation barriers between call-site and definition, resulting in excessive and overly-cautious backtracking traffic. These can be eliminated through better [compilation techniques](https://core.ac.uk/download/pdf/148008325.pdf). As mentioned above, Swift's support for custom static compilation is still under development.
@@ -546,7 +565,7 @@ Regexes are often used for tokenization and tokens can be represented with Swift
### Future work: baked-in localized processing
-- `CustomMatchingRegexComponent` gives an entry point for localized processors
+- `CustomPrefixMatchRegexComponent` gives an entry point for localized processors
- Future work includes (sub?)protocols to communicate localization intent
-->
diff --git a/Documentation/Evolution/StringProcessingAlgorithms.md b/Documentation/Evolution/StringProcessingAlgorithms.md
index b976c562e..edefbd19b 100644
--- a/Documentation/Evolution/StringProcessingAlgorithms.md
+++ b/Documentation/Evolution/StringProcessingAlgorithms.md
@@ -8,9 +8,9 @@ We propose:
1. New regex-powered algorithms over strings, bringing the standard library up to parity with scripting languages
2. Generic `Collection` equivalents of these algorithms in terms of subsequences
-3. `protocol CustomMatchingRegexComponent`, which allows 3rd party libraries to provide their industrial-strength parsers as intermixable components of regexes
+3. `protocol CustomPrefixMatchRegexComponent`, which allows 3rd party libraries to provide their industrial-strength parsers as intermixable components of regexes
-This proposal is part of a larger [regex-powered string processing initiative](https://forums.swift.org/t/declarative-string-processing-overview/52459). Throughout the document, we will reference the still-in-progress [`RegexProtocol`, `Regex`](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md), and result builder DSL, but these are in flux and not formally part of this proposal. Further discussion of regex specifics is out of scope of this proposal and better discussed in another thread (see [Pitch and Proposal Status](https://github.com/apple/swift-experimental-string-processing/issues/107) for links to relevant threads).
+This proposal is part of a larger [regex-powered string processing initiative](https://github.com/apple/swift-evolution/blob/main/proposals/0350-regex-type-overview.md), the status of each proposal is tracked [here](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md). Further discussion of regex specifics is out of scope of this proposal and better discussed in their relevant reviews.
## Motivation
@@ -91,18 +91,18 @@ Note: Only a subset of Python's string processing API are included in this table
### Complex string processing
-Even with the API additions, more complex string processing quickly becomes unwieldy. Up-coming support for authoring regexes in Swift help alleviate this somewhat, but string processing in the modern world involves dealing with localization, standards-conforming validation, and other concerns for which a dedicated parser is required.
+Even with the API additions, more complex string processing quickly becomes unwieldy. String processing in the modern world involves dealing with localization, standards-conforming validation, and other concerns for which a dedicated parser is required.
Consider parsing the date field `"Date: Wed, 16 Feb 2022 23:53:19 GMT"` in an HTTP header as a `Date` type. The naive approach is to search for a substring that looks like a date string (`16 Feb 2022`), and attempt to post-process it as a `Date` with a date parser:
```swift
let regex = Regex {
- capture {
- oneOrMore(.digit)
+ Capture {
+ OneOrMore(.digit)
" "
- oneOrMore(.word)
+ OneOrMore(.word)
" "
- oneOrMore(.digit)
+ OneOrMore(.digit)
}
}
@@ -128,21 +128,21 @@ DEBIT 03/24/2020 IRX tax payment ($52,249.98)
Parsing a currency string such as `$3,020.85` with regex is also tricky, as it can contain localized and currency symbols in addition to accounting conventions. This is why Foundation provides industrial-strength parsers for localized strings.
-## Proposed solution
+## Proposed solution
### Complex string processing
-We propose a `CustomMatchingRegexComponent` protocol which allows types from outside the standard library participate in regex builders and `RegexComponent` algorithms. This allows types, such as `Date.ParseStrategy` and `FloatingPointFormatStyle.Currency`, to be used directly within a regex:
+We propose a `CustomPrefixMatchRegexComponent` protocol which allows types from outside the standard library participate in regex builders and `RegexComponent` algorithms. This allows types, such as `Date.ParseStrategy` and `FloatingPointFormatStyle.Currency`, to be used directly within a regex:
```swift
let dateRegex = Regex {
- capture(dateParser)
+ Capture(dateParser)
}
let date: Date = header.firstMatch(of: dateRegex).map(\.result.1)
let currencyRegex = Regex {
- capture(.localizedCurrency(code: "USD").sign(strategy: .accounting))
+ Capture(.localizedCurrency(code: "USD").sign(strategy: .accounting))
}
let amount: [Decimal] = statement.matches(of: currencyRegex).map(\.result.1)
@@ -162,28 +162,30 @@ We also propose the following regex-powered algorithms as well as their generic
|`replace(:with:subrange:maxReplacements)`| Replaces all occurrences of the sequence matching the given `RegexComponent` or sequence with a given collection |
|`split(by:)`| Returns the longest possible subsequences of the collection around elements equal to the given separator |
|`firstMatch(of:)`| Returns the first match of the specified `RegexComponent` within the collection |
+|`wholeMatch(of:)`| Matches the specified `RegexComponent` in the collection as a whole |
+|`prefixMatch(of:)`| Matches the specified `RegexComponent` against the collection at the beginning |
|`matches(of:)`| Returns a collection containing all matches of the specified `RegexComponent` |
+## Detailed design
-## Detailed design
+### `CustomPrefixMatchRegexComponent`
-### `CustomMatchingRegexComponent`
-
-`CustomMatchingRegexComponent` inherits from `RegexComponent` and satisfies its sole requirement; Conformers can be used with all of the string algorithms generic over `RegexComponent`.
+`CustomPrefixMatchRegexComponent` inherits from `RegexComponent` and satisfies its sole requirement. Conformers can be used with all of the string algorithms generic over `RegexComponent`.
```swift
-/// A protocol for custom match functionality.
-public protocol CustomMatchingRegexComponent : RegexComponent {
- /// Match the input string within the specified bounds, beginning at the given index, and return
- /// the end position (upper bound) of the match and the matched instance.
+/// A protocol allowing custom types to function as regex components by
+/// providing the raw functionality backing `prefixMatch`.
+public protocol CustomPrefixMatchRegexComponent: RegexComponent {
+ /// Process the input string within the specified bounds, beginning at the given index, and return
+ /// the end position (upper bound) of the match and the produced output.
/// - Parameters:
/// - input: The string in which the match is performed.
/// - index: An index of `input` at which to begin matching.
/// - bounds: The bounds in `input` in which the match is performed.
/// - Returns: The upper bound where the match terminates and a matched instance, or `nil` if
/// there isn't a match.
- func match(
+ func consuming(
_ input: String,
startingAt index: String.Index,
in bounds: Range
@@ -197,8 +199,8 @@ public protocol CustomMatchingRegexComponent : RegexComponent {
We use Foundation `FloatingPointFormatStyle.Currency` as an example for protocol conformance. It would implement the `match` function with `Match` being a `Decimal`. It could also add a static function `.localizedCurrency(code:)` as a member of `RegexComponent`, so it can be referred as `.localizedCurrency(code:)` in the `Regex` result builder:
```swift
-extension FloatingPointFormatStyle.Currency : CustomMatchingRegexComponent {
- public func match(
+extension FloatingPointFormatStyle.Currency : CustomPrefixMatchRegexComponent {
+ public func consuming(
_ input: String,
startingAt index: String.Index,
in bounds: Range
@@ -389,7 +391,7 @@ extension BidirectionalCollection where SubSequence == Substring {
}
```
-#### First match
+#### Match
```swift
extension BidirectionalCollection where SubSequence == Substring {
@@ -398,6 +400,16 @@ extension BidirectionalCollection where SubSequence == Substring {
/// - Returns: The first match of `regex` in the collection, or `nil` if
/// there isn't a match.
public func firstMatch(of regex: R) -> RegexMatch?
+
+ /// Match a regex in its entirety.
+ /// - Parameter r: The regex to match against.
+ /// - Returns: The match if there is one, or `nil` if none.
+ public func wholeMatch(of r: R) -> Regex.Match?
+
+ /// Match part of the regex, starting at the beginning.
+ /// - Parameter r: The regex to match against.
+ /// - Returns: The match if there is one, or `nil` if none.
+ public func prefixMatch(of r: R) -> Regex.Match?
}
```
@@ -473,7 +485,7 @@ extension RangeReplaceableCollection where SubSequence == Substring {
/// - Returns: A new collection in which all occurrences of subsequence
/// matching `regex` in `subrange` are replaced by `replacement`.
public func replacing(
- _ regex: R,
+ _ r: R,
with replacement: Replacement,
subrange: Range,
maxReplacements: Int = .max
@@ -489,7 +501,7 @@ extension RangeReplaceableCollection where SubSequence == Substring {
/// - Returns: A new collection in which all occurrences of subsequence
/// matching `regex` are replaced by `replacement`.
public func replacing(
- _ regex: R,
+ _ r: R,
with replacement: Replacement,
maxReplacements: Int = .max
) -> Self where Replacement.Element == Element
@@ -502,7 +514,7 @@ extension RangeReplaceableCollection where SubSequence == Substring {
/// - maxReplacements: A number specifying how many occurrences of the
/// sequence matching `regex` to replace. Default is `Int.max`.
public mutating func replace(
- _ regex: R,
+ _ r: R,
with replacement: Replacement,
maxReplacements: Int = .max
) where Replacement.Element == Element
@@ -511,48 +523,48 @@ extension RangeReplaceableCollection where SubSequence == Substring {
/// the given regex are replaced by another regex match.
/// - Parameters:
/// - regex: A regex describing the sequence to replace.
- /// - replacement: A closure that receives the full match information,
- /// including captures, and returns a replacement collection.
/// - subrange: The range in the collection in which to search for `regex`.
/// - maxReplacements: A number specifying how many occurrences of the
/// sequence matching `regex` to replace. Default is `Int.max`.
+ /// - replacement: A closure that receives the full match information,
+ /// including captures, and returns a replacement collection.
/// - Returns: A new collection in which all occurrences of subsequence
/// matching `regex` are replaced by `replacement`.
public func replacing(
_ regex: R,
- with replacement: (RegexMatch) throws -> Replacement,
subrange: Range,
- maxReplacements: Int = .max
+ maxReplacements: Int = .max,
+ with replacement: (RegexMatch) throws -> Replacement
) rethrows -> Self where Replacement.Element == Element
/// Returns a new collection in which all occurrences of a sequence matching
/// the given regex are replaced by another collection.
/// - Parameters:
/// - regex: A regex describing the sequence to replace.
- /// - replacement: A closure that receives the full match information,
- /// including captures, and returns a replacement collection.
/// - maxReplacements: A number specifying how many occurrences of the
/// sequence matching `regex` to replace. Default is `Int.max`.
+ /// - replacement: A closure that receives the full match information,
+ /// including captures, and returns a replacement collection.
/// - Returns: A new collection in which all occurrences of subsequence
/// matching `regex` are replaced by `replacement`.
public func replacing(
_ regex: R,
- with replacement: (RegexMatch) throws -> Replacement,
- maxReplacements: Int = .max
+ maxReplacements: Int = .max,
+ with replacement: (RegexMatch) throws -> Replacement
) rethrows -> Self where Replacement.Element == Element
/// Replaces all occurrences of the sequence matching the given regex with
/// a given collection.
/// - Parameters:
/// - regex: A regex describing the sequence to replace.
- /// - replacement: A closure that receives the full match information,
- /// including captures, and returns a replacement collection.
/// - maxReplacements: A number specifying how many occurrences of the
/// sequence matching `regex` to replace. Default is `Int.max`.
+ /// - replacement: A closure that receives the full match information,
+ /// including captures, and returns a replacement collection.
public mutating func replace(
_ regex: R,
- with replacement: (RegexMatch) throws -> Replacement,
- maxReplacements: Int = .max
+ maxReplacements: Int = .max,
+ with replacement: (RegexMatch) throws -> Replacement
) rethrows where Replacement.Element == Element
}
```
@@ -609,4 +621,4 @@ Trimming a string from both sides shares a similar story. For example, `"ababa".
### Future API
-Some Python functions are not currently included in this proposal, such as trimming the suffix from a string/collection. This pitch aims to establish a pattern for using `RegexComponent` with string processing algorithms, so that further enhancement can to be introduced to the standard library easily in the future, and eventually close the gap between Swift and other popular scripting languages.
+Some common string processing functions are not currently included in this proposal, such as trimming the suffix from a string/collection, and finding overlapping ranges of matched substrings. This pitch aims to establish a pattern for using `RegexComponent` with string processing algorithms, so that further enhancement can to be introduced to the standard library easily in the future, and eventually close the gap between Swift and other popular scripting languages.
diff --git a/Documentation/Evolution/UnicodeForStringProcessing.md b/Documentation/Evolution/UnicodeForStringProcessing.md
new file mode 100644
index 000000000..828d8f53c
--- /dev/null
+++ b/Documentation/Evolution/UnicodeForStringProcessing.md
@@ -0,0 +1,872 @@
+# Unicode for String Processing
+
+Proposal: [SE-NNNN](NNNN-filename.md)
+Authors: [Nate Cook](https://github.com/natecook1000), [Alejandro Alonso](https://github.com/Azoy)
+Review Manager: TBD
+Implementation: [apple/swift-experimental-string-processing][repo]
+Status: **Draft**
+
+
+## Introduction
+
+This proposal describes `Regex`'s rich Unicode support during regex matching, along with the character classes and options that define that behavior.
+
+## Motivation
+
+Swift's `String` type provides, by default, a view of `Character`s or [extended grapheme clusters][graphemes] whose comparison honors [Unicode canonical equivalence][canoneq]. Each character in a string can be composed of one or more Unicode scalar values, while still being treated as a single unit, equivalent to other ways of formulating the equivalent character:
+
+```swift
+let str = "Cafe\u{301}" // "Café"
+str == "Café" // true
+str.dropLast() // "Caf"
+str.last == "é" // true (precomposed e with acute accent)
+str.last == "e\u{301}" // true (e followed by composing acute accent)
+```
+
+This default view is fairly novel. Most languages that support Unicode strings generally operate at the Unicode scalar level, and don't provide the same affordance for operating on a string as a collection of grapheme clusters. In Python, for example, Unicode strings report their length as the number of scalar values, and don't use canonical equivalence in comparisons:
+
+```python
+cafe = u"Cafe\u0301"
+len(cafe) # 5
+cafe == u"Café" # False
+```
+
+Existing regex engines follow this same model of operating at the Unicode scalar level. To match canonically equivalent characters, or have equivalent behavior between equivalent strings, you must normalize your string and regex to the same canonical format.
+
+```python
+# Matches a four-element string
+re.match(u"^.{4}$", cafe) # None
+# Matches a string ending with 'é'
+re.match(u".+é$", cafe) # None
+
+cafeComp = unicodedata.normalize("NFC", cafe)
+re.match(u"^.{4}$", cafeComp) #
+re.match(u".+é$", cafeComp) #
+```
+
+With Swift's string model, this behavior would surprising and undesirable — Swift's default regex semantics must match the semantics of a `String`.
+
+Other engines
+
+Other regex engines match character classes (such as `\w` or `.`) at the Unicode scalar value level, or even the code unit level, instead of recognizing grapheme clusters as characters. When matching the `.` character class, other languages will only match the first part of an `"e\u{301}"` grapheme cluster. Some languages, like Perl, Ruby, and Java, support an additional `\X` metacharacter, which explicitly represents a single grapheme cluster.
+
+| Matching `"Cafe\u{301}"` | Pattern: `^Caf.` | Remaining | Pattern: `^Caf\X` | Remaining |
+|---|---|---|---|---|
+| C#, Rust, Go, Python | `"Cafe"` | `"´"` | n/a | n/a |
+| NSString, Java, Ruby, Perl | `"Cafe"` | `"´"` | `"Café"` | `""` |
+
+Other than Java's `CANON_EQ` option, the vast majority of other languages and engines are not capable of comparing with canonical equivalence.
+
+