diff --git a/Documentation/Evolution/StringProcessingAlgorithms.md b/Documentation/Evolution/StringProcessingAlgorithms.md new file mode 100644 index 000000000..9454396ce --- /dev/null +++ b/Documentation/Evolution/StringProcessingAlgorithms.md @@ -0,0 +1,612 @@ +# String processing algorithms + +## Introduction + +The Swift standard library's string processing algorithms are underpowered compared to other popular programming and scripting languages. Some of these omissions can be found in `NSString`, but these fundamental algorithms should have a place in the standard library. + +We propose: + +1. New regex-powered algorithms over strings, bringing the standard library up to parity with scripting languages +2. Generic `Collection` equivalents of these algorithms in terms of subsequences +3. `protocol CustomMatchingRegexComponent`, which allows 3rd party libraries to provide their industrial-strength parsers as intermixable components of regexes + +This proposal is part of a larger [regex-powered string processing initiative](https://forums.swift.org/t/declarative-string-processing-overview/52459). Throughout the document, we will reference the still-in-progress [`RegexProtocol`, `Regex`](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md), and result builder DSL, but these are in flux and not formally part of this proposal. Further discussion of regex specifics is out of scope of this proposal and better discussed in another thread (see [Pitch and Proposal Status](https://github.com/apple/swift-experimental-string-processing/issues/107) for links to relevant threads). + +## Motivation + +A number of common string processing APIs are missing from the Swift standard library. While most of the desired functionalities can be accomplished through a series of API calls, every gap adds a burden to developers doing frequent or complex string processing. For example, here's one approach to find the number of occurrences a substring ("banana") within a string: + +```swift +let str = "A banana a day keeps the doctor away. I love bananas; banana are my favorite fruit." + +var idx = str.startIndex +var ranges = [Range]() +while let r = str.range(of: "banana", options: [], range: idx.. + Comparison of how Swift's APIs stack up with Python's. + +Note: Only a subset of Python's string processing API are included in this table for the following reasons: + +- Functions to query if all characters in the string are of a specified category, such as `isalnum()` and `isalpha()`, are omitted. These are achievable in Swift by passing in the corresponding character set to `allSatisfy(_:)`, so they're omitted in this table for simplicity. +- String formatting functions such as `center(length, character)` and `ljust(width, fillchar)` are also excluded here as this proposal focuses on matching and searching functionalities. + +##### Search and replace + +|Python |Swift | +|--- |--- | +| `count(sub, start, end)` | | +| `find(sub, start, end)`, `index(sub, start, end)` | `firstIndex(where:)` | +| `rfind(sub, start, end)`, `rindex(sub, start, end)` | `lastIndex(where:)` | +| `expandtabs(tabsize)`, `replace(old, new, count)` | `Foundation.replacingOccurrences(of:with:)` | +| `maketrans(x, y, z)` + `translate(table)` | + +##### Prefix and suffix matching + +|Python |Swift | +|--- |--- | +| `startswith(prefix, start, end)` | `starts(with:)` or `hasPrefix(:)`| +| `endswith(suffix, start, end)` | `hasSuffix(:)` | +| `removeprefix(prefix)` | Test if string has prefix with `hasPrefix(:)`, then drop the prefix with `dropFirst(:)`| +| `removesuffix(suffix)` | Test if string has suffix with `hasSuffix(:)`, then drop the suffix with `dropLast(:)` | + +##### Strip / trim + +|Python |Swift | +|--- |--- | +| `strip([chars])`| `Foundation.trimmingCharacters(in:)` | +| `lstrip([chars])` | `drop(while:)` | +| `rstrip([chars])` | Test character equality, then `dropLast()` iteratively | + +##### Split + +|Python |Swift | +|--- |--- | +| `partition(sep)` | `Foundation.components(separatedBy:)` | +| `rpartition(sep)` | | +| `split(sep, maxsplit)` | `split(separator:maxSplits:...)` | +| `splitlines(keepends)` | `split(separator:maxSplits:...)` | +| `rsplit(sep, maxsplit)` | | + + + + + +### Complex string processing + +Even with the API additions, more complex string processing quickly becomes unwieldy. Up-coming support for authoring regexes in Swift help alleviate this somewhat, but string processing in the modern world involves dealing with localization, standards-conforming validation, and other concerns for which a dedicated parser is required. + +Consider parsing the date field `"Date: Wed, 16 Feb 2022 23:53:19 GMT"` in an HTTP header as a `Date` type. The naive approach is to search for a substring that looks like a date string (`16 Feb 2022`), and attempt to post-process it as a `Date` with a date parser: + +```swift +let regex = Regex { + capture { + oneOrMore(.digit) + " " + oneOrMore(.word) + " " + oneOrMore(.digit) + } +} + +let dateParser = Date.ParseStrategy(format: "\(day: .twoDigits) \(month: .abbreviated) \(year: .padded(4))" +if let dateMatch = header.firstMatch(of: regex)?.0 { + let date = try? Date(dateMatch, strategy: dateParser) +} +``` + +This requires writing a simplistic pre-parser before invoking the real parser. The pre-parser will suffer from being out-of-sync and less featureful than what the real parser can do. + +Or consider parsing a bank statement to record all the monetary values in the last column: + +```swift +let statement = """ +CREDIT 04/06/2020 Paypal transfer $4.99 +CREDIT 04/03/2020 Payroll $69.73 +DEBIT 04/02/2020 ACH transfer ($38.25) +DEBIT 03/24/2020 IRX tax payment ($52,249.98) +""" +``` + +Parsing a currency string such as `$3,020.85` with regex is also tricky, as it can contain localized and currency symbols in addition to accounting conventions. This is why Foundation provides industrial-strength parsers for localized strings. + + +## Proposed solution + +### Complex string processing + +We propose a `CustomMatchingRegexComponent` protocol which allows types from outside the standard library participate in regex builders and `RegexComponent` algorithms. This allows types, such as `Date.ParseStrategy` and `FloatingPointFormatStyle.Currency`, to be used directly within a regex: + +```swift +let dateRegex = Regex { + capture(dateParser) +} + +let date: Date = header.firstMatch(of: dateRegex).map(\.result.1) + +let currencyRegex = Regex { + capture(.localizedCurrency(code: "USD").sign(strategy: .accounting)) +} + +let amount: [Decimal] = statement.matches(of: currencyRegex).map(\.result.1) +``` + +### String algorithm additions + +We also propose the following regex-powered algorithms as well as their generic `Collection` equivalents. See the Detailed design section for a complete list of variation and overloads . + +|Function | Description | +|--- |--- | +|`contains(_:) -> Bool` | Returns whether the collection contains the given sequence or `RegexComponent` | +|`starts(with:) -> Bool` | Returns whether the collection contains the same prefix as the specified `RegexComponent` | +|`trimPrefix(_:)`| Removes the prefix if it matches the given `RegexComponent` or collection | +|`firstRange(of:) -> Range?` | Finds the range of the first occurrence of a given sequence or `RegexComponent`| +|`ranges(of:) -> some Collection` | Finds the ranges of the all occurrences of a given sequence or `RegexComponent` within the collection | +|`replace(:with:subrange:maxReplacements)`| Replaces all occurrences of the sequence matching the given `RegexComponent` or sequence with a given collection | +|`split(by:)`| Returns the longest possible subsequences of the collection around elements equal to the given separator | +|`firstMatch(of:)`| Returns the first match of the specified `RegexComponent` within the collection | +|`matches(of:)`| Returns a collection containing all matches of the specified `RegexComponent` | + + + +## Detailed design + +### `CustomMatchingRegexComponent` + +`CustomMatchingRegexComponent` inherits from `RegexComponent` and satisfies its sole requirement; Conformers can be used with all of the string algorithms generic over `RegexComponent`. + +```swift +/// A protocol for custom match functionality. +public protocol CustomMatchingRegexComponent : RegexComponent { + /// Match the input string within the specified bounds, beginning at the given index, and return + /// the end position (upper bound) of the match and the matched instance. + /// - Parameters: + /// - input: The string in which the match is performed. + /// - index: An index of `input` at which to begin matching. + /// - bounds: The bounds in `input` in which the match is performed. + /// - Returns: The upper bound where the match terminates and a matched instance, or `nil` if + /// there isn't a match. + func match( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) -> (upperBound: String.Index, match: Match)? +} +``` + +
+Example for protocol conformance + +We use Foundation `FloatingPointFormatStyle.Currency` as an example for protocol conformance. It would implement the `match` function with `Match` being a `Decimal`. It could also add a static function `.localizedCurrency(code:)` as a member of `RegexComponent`, so it can be referred as `.localizedCurrency(code:)` in the `Regex` result builder: + +```swift +extension FloatingPointFormatStyle.Currency : CustomMatchingRegexComponent { + public func match( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) -> (upperBound: String.Index, match: Decimal)? +} + +extension RegexComponent where Self == FloatingPointFormatStyle.Currency { + public static func localizedCurrency(code: Locale.Currency) -> Self +} +``` + +Matching and extracting a localized currency amount, such as `"$3,020.85"`, can be done directly within a regex: + +```swift +let regex = Regex { + capture(.localizedCurreny(code: "USD")) +} +``` + +
+ + +### String algorithm additions + +#### Contains + +```swift +extension Collection where Element: Equatable { + /// Returns a Boolean value indicating whether the collection contains the + /// given sequence. + /// - Parameter other: A sequence to search for within this collection. + /// - Returns: `true` if the collection contains the specified sequence, + /// otherwise `false`. + public func contains(_ other: S) -> Bool + where S.Element == Element +} + +extension BidirectionalCollection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the collection contains the + /// given regex. + /// - Parameter regex: A regex to search for within this collection. + /// - Returns: `true` if the regex was found in the collection, otherwise + /// `false`. + public func contains(_ regex: R) -> Bool +} +``` + +#### Starts with + +```swift +extension BidirectionalCollection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the initial elements of the + /// sequence are the same as the elements in the specified regex. + /// - Parameter regex: A regex to compare to this sequence. + /// - Returns: `true` if the initial elements of the sequence matches the + /// beginning of `regex`; otherwise, `false`. + public func starts(with regex: R) -> Bool +} +``` + +#### Trim prefix + +```swift +extension Collection { + /// Returns a new collection of the same type by removing initial elements + /// that satisfy the given predicate from the start. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + /// - Returns: A collection containing the elements of the collection that are + /// not removed by `predicate`. + public func trimmingPrefix(while predicate: (Element) throws -> Bool) rethrows -> SubSequence +} + +extension Collection where SubSequence == Self { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) +} + +extension RangeReplaceableCollection { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) +} + +extension Collection where Element: Equatable { + /// Returns a new collection of the same type by removing `prefix` from the + /// start. + /// - Parameter prefix: The collection to remove from this collection. + /// - Returns: A collection containing the elements that does not match + /// `prefix` from the start. + public func trimmingPrefix(_ prefix: Prefix) -> SubSequence + where Prefix.Element == Element +} + +extension Collection where SubSequence == Self, Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} + +extension RangeReplaceableCollection where Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} + +extension BidirectionalCollection where SubSequence == Substring { + /// Returns a new subsequence by removing the initial elements that matches + /// the given regex. + /// - Parameter regex: The regex to remove from this collection. + /// - Returns: A new subsequence containing the elements of the collection + /// that does not match `prefix` from the start. + public func trimmingPrefix(_ regex: R) -> SubSequence +} + +extension RangeReplaceableCollection + where Self: BidirectionalCollection, SubSequence == Substring +{ + /// Removes the initial elements that matches the given regex. + /// - Parameter regex: The regex to remove from this collection. + public mutating func trimPrefix(_ regex: R) +} +``` + +#### First range + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter sequence: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns nil if `sequence` is not found. + public func firstRange(of sequence: S) -> Range? + where S.Element == Element +} + +extension BidirectionalCollection where Element: Comparable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns `nil` if `sequence` is not found. + public func firstRange(of other: S) -> Range? + where S.Element == Element +} + +extension BidirectionalCollection where SubSequence == Substring { + /// Finds and returns the range of the first occurrence of a given regex + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A range in the collection of the first occurrence of `regex`. + /// Returns `nil` if `regex` is not found. + public func firstRange(of regex: R) -> Range? +} +``` + +#### Ranges + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A collection of ranges of all occurrences of `other`. Returns + /// an empty collection if `other` is not found. + public func ranges(of other: S) -> some Collection> + where S.Element == Element +} + +extension BidirectionalCollection where SubSequence == Substring { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection or ranges in the receiver of all occurrences of + /// `regex`. Returns an empty collection if `regex` is not found. + public func ranges(of regex: R) -> some Collection> +} +``` + +#### First match + +```swift +extension BidirectionalCollection where SubSequence == Substring { + /// Returns the first match of the specified regex within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: The first match of `regex` in the collection, or `nil` if + /// there isn't a match. + public func firstMatch(of regex: R) -> RegexMatch? +} +``` + +#### Matches + +```swift +extension BidirectionalCollection where SubSequence == Substring { + /// Returns a collection containing all matches of the specified regex. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection of matches of `regex`. + public func matches(of regex: R) -> some Collection> +} +``` + +#### Replace + +```swift +extension RangeReplaceableCollection where Element: Equatable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `other`. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: S, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where S.Element == Element, Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: S, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where S.Element == Element, Replacement.Element == Element + + /// Replaces all occurrences of a target sequence with a given collection + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + public mutating func replace( + _ other: S, + with replacement: Replacement, + maxReplacements: Int = .max + ) where S.Element == Element, Replacement.Element == Element +} + +extension RangeReplaceableCollection where SubSequence == Substring { + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` in `subrange` are replaced by `replacement`. + public func replacing( + _ regex: R, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + public mutating func replace( + _ regex: R, + with replacement: Replacement, + maxReplacements: Int = .max + ) where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another regex match. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + with replacement: (RegexMatch) throws -> Replacement, + subrange: Range, + maxReplacements: Int = .max + ) rethrows -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + with replacement: (RegexMatch) throws -> Replacement, + maxReplacements: Int = .max + ) rethrows -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + public mutating func replace( + _ regex: R, + with replacement: (RegexMatch) throws -> Replacement, + maxReplacements: Int = .max + ) rethrows where Replacement.Element == Element +} +``` + +#### Split + +```swift +extension Collection where Element: Equatable { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator. + /// - Parameter separator: The element to be split upon. + /// - Returns: A collection of subsequences, split from this collection's + /// elements. + public func split(by separator: S) -> some Collection + where S.Element == Element +} + +extension BidirectionalCollection where SubSequence == Substring { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator. + /// - Parameter separator: A regex describing elements to be split upon. + /// - Returns: A collection of substrings, split from this collection's + /// elements. + public func split(by separator: R) -> some Collection +} +``` + + + + + +## Alternatives considered + +### Extend `Sequence` instead of `Collection` + +Most of the proposed algorithms are necessarily on `Collection` due to the use of indices or mutation. `Sequence` does not support multi-pass iteration, so even `trimPrefix` would problematic on `Sequence` because it needs to look 1 `Element` ahead to know when to stop trimming. + +## Future directions + +### Backward algorithms + +It would be useful to have algorithms that operate from the back of a collection, including ability to find the last non-overlapping range of a pattern in a string, and/or that to find the first range of a pattern when searching from the back, and trimming a string from both sides. They are deferred from this proposal as the API that could clarify the nuances of backward algorithms are still being explored. + +
+ Nuances of backward algorithms + +There is a subtle difference between finding the last non-overlapping range of a pattern in a string, and finding the first range of this pattern when searching from the back. + +The currently proposed algorithm that finds a pattern from the front, e.g. `"aaaaa".ranges(of: "aa")`, produces two non-overlapping ranges, splitting the string in the chunks `aa|aa|a`. It would not be completely unreasonable to expect to introduce a counterpart, such as `"aaaaa".lastRange(of: "aa")`, to return the range that contains the third and fourth characters of the string. This would be a shorthand for `"aaaaa".ranges(of: "aa").last`. Yet, it would also be reasonable to expect the function to return the first range of `"aa"` when searching from the back of the string, i.e. the range that contains the fourth and fifth characters. + +Trimming a string from both sides shares a similar story. For example, `"ababa".trimming("aba")` can return either `"ba"` or `"ab"`, depending on whether the prefix or the suffix was trimmed first. +
+ + +### Future API + +Some Python functions are not currently included in this proposal, such as trimming the suffix from a string/collection. This pitch aims to establish a pattern for using `RegexComponent` with string processing algorithms, so that further enhancement can to be introduced to the standard library easily in the future, and eventually close the gap between Swift and other popular scripting languages. diff --git a/Package.swift b/Package.swift index 72e17e362..71641ae28 100644 --- a/Package.swift +++ b/Package.swift @@ -42,13 +42,22 @@ let package = Package( .target( name: "_StringProcessing", dependencies: ["_MatchingEngine", "_CUnicode"], + swiftSettings: [ + .unsafeFlags(["-enable-library-evolution"]), + ]), + .target( + name: "RegexBuilder", + dependencies: ["_StringProcessing", "_MatchingEngine"], swiftSettings: [ .unsafeFlags(["-enable-library-evolution"]), .unsafeFlags(["-Xfrontend", "-enable-experimental-pairwise-build-block"]) ]), .testTarget( name: "RegexTests", - dependencies: ["_StringProcessing"], + dependencies: ["_StringProcessing"]), + .testTarget( + name: "RegexBuilderTests", + dependencies: ["_StringProcessing", "RegexBuilder"], swiftSettings: [ .unsafeFlags(["-Xfrontend", "-enable-experimental-pairwise-build-block"]) ]), @@ -73,7 +82,7 @@ let package = Package( // MARK: Exercises .target( name: "Exercises", - dependencies: ["_MatchingEngine", "Prototypes", "_StringProcessing"], + dependencies: ["_MatchingEngine", "Prototypes", "_StringProcessing", "RegexBuilder"], swiftSettings: [ .unsafeFlags(["-Xfrontend", "-enable-experimental-pairwise-build-block"]) ]), diff --git a/Sources/Exercises/Participants/RegexParticipant.swift b/Sources/Exercises/Participants/RegexParticipant.swift index bae3aed42..731b9b6f6 100644 --- a/Sources/Exercises/Participants/RegexParticipant.swift +++ b/Sources/Exercises/Participants/RegexParticipant.swift @@ -10,6 +10,7 @@ //===----------------------------------------------------------------------===// import _StringProcessing +import RegexBuilder /* diff --git a/Sources/_StringProcessing/RegexDSL/Anchor.swift b/Sources/RegexBuilder/Anchor.swift similarity index 98% rename from Sources/_StringProcessing/RegexDSL/Anchor.swift rename to Sources/RegexBuilder/Anchor.swift index 57d8f2ffa..ea2dde382 100644 --- a/Sources/_StringProcessing/RegexDSL/Anchor.swift +++ b/Sources/RegexBuilder/Anchor.swift @@ -10,6 +10,7 @@ //===----------------------------------------------------------------------===// import _MatchingEngine +@_spi(RegexBuilder) import _StringProcessing public struct Anchor { internal enum Kind { diff --git a/Sources/_StringProcessing/RegexDSL/Builder.swift b/Sources/RegexBuilder/Builder.swift similarity index 95% rename from Sources/_StringProcessing/RegexDSL/Builder.swift rename to Sources/RegexBuilder/Builder.swift index 78c122828..8921c8f25 100644 --- a/Sources/_StringProcessing/RegexDSL/Builder.swift +++ b/Sources/RegexBuilder/Builder.swift @@ -9,6 +9,8 @@ // //===----------------------------------------------------------------------===// +@_spi(RegexBuilder) import _StringProcessing + @resultBuilder public enum RegexComponentBuilder { public static func buildBlock() -> Regex { diff --git a/Sources/_StringProcessing/RegexDSL/DSL.swift b/Sources/RegexBuilder/DSL.swift similarity index 75% rename from Sources/_StringProcessing/RegexDSL/DSL.swift rename to Sources/RegexBuilder/DSL.swift index 4c3c382cc..816668b67 100644 --- a/Sources/_StringProcessing/RegexDSL/DSL.swift +++ b/Sources/RegexBuilder/DSL.swift @@ -10,6 +10,15 @@ //===----------------------------------------------------------------------===// import _MatchingEngine +@_spi(RegexBuilder) import _StringProcessing + +extension Regex { + public init( + @RegexComponentBuilder _ content: () -> Content + ) where Content.Output == Output { + self.init(content()) + } +} // A convenience protocol for builtin regex components that are initialized with // a `DSLTree` node. @@ -23,51 +32,6 @@ extension _BuiltinRegexComponent { } } -// MARK: - Primitives - -extension String: RegexComponent { - public typealias Output = Substring - - public var regex: Regex { - .init(node: .quotedLiteral(self)) - } -} - -extension Substring: RegexComponent { - public typealias Output = Substring - - public var regex: Regex { - .init(node: .quotedLiteral(String(self))) - } -} - -extension Character: RegexComponent { - public typealias Output = Substring - - public var regex: Regex { - .init(node: .atom(.char(self))) - } -} - -extension UnicodeScalar: RegexComponent { - public typealias Output = Substring - - public var regex: Regex { - .init(node: .atom(.scalar(self))) - } -} - -extension CharacterClass: RegexComponent { - public typealias Output = Substring - - public var regex: Regex { - guard let ast = self.makeAST() else { - fatalError("FIXME: extended AST?") - } - return Regex(ast: ast) - } -} - // MARK: - Combinators // MARK: Concatenation @@ -96,9 +60,9 @@ public struct QuantificationBehavior { case reluctantly case possessively } - + var kind: Kind - + internal var astKind: AST.Quantification.Kind { switch kind { case .eagerly: return .eager @@ -108,19 +72,49 @@ public struct QuantificationBehavior { } } +extension DSLTree.Node { + /// Generates a DSLTree node for a repeated range of the given DSLTree node. + /// Individual public API functions are in the generated Variadics.swift file. + static func repeating( + _ range: Range, + _ behavior: QuantificationBehavior, + _ node: DSLTree.Node + ) -> DSLTree.Node { + // TODO: Throw these as errors + assert(range.lowerBound >= 0, "Cannot specify a negative lower bound") + assert(!range.isEmpty, "Cannot specify an empty range") + + switch (range.lowerBound, range.upperBound) { + case (0, Int.max): // 0... + return .quantification(.zeroOrMore, behavior.astKind, node) + case (1, Int.max): // 1... + return .quantification(.oneOrMore, behavior.astKind, node) + case _ where range.count == 1: // ..<1 or ...0 or any range with count == 1 + // Note: `behavior` is ignored in this case + return .quantification(.exactly(.init(faking: range.lowerBound)), .eager, node) + case (0, _): // 0..: _BuiltinRegexComponent { // MARK: - Backreference -struct ReferenceID: Hashable, Equatable { - private static var counter: Int = 0 - var base: Int - - init() { - base = Self.counter - Self.counter += 1 - } -} - public struct Reference: RegexComponent { let id = ReferenceID() - + public init(_ captureType: Capture.Type = Capture.self) {} public var regex: Regex { .init(node: .atom(.symbolicReference(id))) } } + +extension Regex.Match { + public subscript(_ reference: Reference) -> Capture { + self[reference.id] + } +} diff --git a/Sources/RegexBuilder/Match.swift b/Sources/RegexBuilder/Match.swift new file mode 100644 index 000000000..3f86f9498 --- /dev/null +++ b/Sources/RegexBuilder/Match.swift @@ -0,0 +1,28 @@ +//===----------------------------------------------------------------------===// +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2021-2022 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// +//===----------------------------------------------------------------------===// + +import _StringProcessing + +extension String { + public func match( + @RegexComponentBuilder _ content: () -> R + ) -> Regex.Match? { + match(content()) + } +} + +extension Substring { + public func match( + @RegexComponentBuilder _ content: () -> R + ) -> Regex.Match? { + match(content()) + } +} diff --git a/Sources/_StringProcessing/RegexDSL/Variadics.swift b/Sources/RegexBuilder/Variadics.swift similarity index 99% rename from Sources/_StringProcessing/RegexDSL/Variadics.swift rename to Sources/RegexBuilder/Variadics.swift index c81f8b555..60292252a 100644 --- a/Sources/_StringProcessing/RegexDSL/Variadics.swift +++ b/Sources/RegexBuilder/Variadics.swift @@ -12,6 +12,7 @@ // BEGIN AUTO-GENERATED CONTENT import _MatchingEngine +@_spi(RegexBuilder) import _StringProcessing extension RegexComponentBuilder { public static func buildPartialBlock( diff --git a/Sources/VariadicsGenerator/VariadicsGenerator.swift b/Sources/VariadicsGenerator/VariadicsGenerator.swift index 683d45e6e..1f41e68d6 100644 --- a/Sources/VariadicsGenerator/VariadicsGenerator.swift +++ b/Sources/VariadicsGenerator/VariadicsGenerator.swift @@ -9,7 +9,7 @@ // //===----------------------------------------------------------------------===// -// swift run VariadicsGenerator --max-arity 10 > Sources/_StringProcessing/RegexDSL/Variadics.swift +// swift run VariadicsGenerator --max-arity 10 > Sources/RegexBuilder/Variadics.swift import ArgumentParser #if os(macOS) @@ -121,6 +121,7 @@ struct VariadicsGenerator: ParsableCommand { // BEGIN AUTO-GENERATED CONTENT import _MatchingEngine + @_spi(RegexBuilder) import _StringProcessing """) diff --git a/Sources/_StringProcessing/CharacterClass.swift b/Sources/_StringProcessing/CharacterClass.swift index d72ecf06c..7989c0943 100644 --- a/Sources/_StringProcessing/CharacterClass.swift +++ b/Sources/_StringProcessing/CharacterClass.swift @@ -178,6 +178,17 @@ public struct CharacterClass: Hashable { } } +extension CharacterClass: RegexComponent { + public typealias Output = Substring + + public var regex: Regex { + guard let ast = self.makeAST() else { + fatalError("FIXME: extended AST?") + } + return Regex(ast: ast) + } +} + extension RegexComponent where Self == CharacterClass { public static var any: CharacterClass { .init(cc: .any, matchLevel: .graphemeCluster) diff --git a/Sources/_StringProcessing/RegexDSL/ASTConversion.swift b/Sources/_StringProcessing/Regex/ASTConversion.swift similarity index 100% rename from Sources/_StringProcessing/RegexDSL/ASTConversion.swift rename to Sources/_StringProcessing/Regex/ASTConversion.swift diff --git a/Sources/_StringProcessing/RegexDSL/AnyRegexOutput.swift b/Sources/_StringProcessing/Regex/AnyRegexOutput.swift similarity index 100% rename from Sources/_StringProcessing/RegexDSL/AnyRegexOutput.swift rename to Sources/_StringProcessing/Regex/AnyRegexOutput.swift diff --git a/Sources/_StringProcessing/RegexDSL/Core.swift b/Sources/_StringProcessing/Regex/Core.swift similarity index 78% rename from Sources/_StringProcessing/RegexDSL/Core.swift rename to Sources/_StringProcessing/Regex/Core.swift index 236888c77..c6433ef3e 100644 --- a/Sources/_StringProcessing/RegexDSL/Core.swift +++ b/Sources/_StringProcessing/Regex/Core.swift @@ -36,6 +36,7 @@ public struct Regex: RegexComponent { init(ast: AST) { self.tree = ast.dslTree } + init(tree: DSLTree) { self.tree = tree } @@ -44,7 +45,8 @@ public struct Regex: RegexComponent { let program: Program // var ast: AST { program.ast } - var root: DSLTree.Node { + @_spi(RegexBuilder) + public var root: DSLTree.Node { program.tree.root } @@ -59,7 +61,8 @@ public struct Regex: RegexComponent { self.program = Program(ast: .init(ast, globalOptions: nil)) } - init(node: DSLTree.Node) { + @_spi(RegexBuilder) + public init(node: DSLTree.Node) { self.program = Program(tree: .init(node, options: nil)) } @@ -84,17 +87,46 @@ public struct Regex: RegexComponent { self = content.regex } - public init( - @RegexComponentBuilder _ content: () -> Content - ) where Content.Output == Output { - self.init(content()) + public var regex: Regex { + self } +} + +// MARK: - Primitive regex components + +extension String: RegexComponent { + public typealias Output = Substring public var regex: Regex { - self + .init(node: .quotedLiteral(self)) + } +} + +extension Substring: RegexComponent { + public typealias Output = Substring + + public var regex: Regex { + .init(node: .quotedLiteral(String(self))) + } +} + +extension Character: RegexComponent { + public typealias Output = Substring + + public var regex: Regex { + .init(node: .atom(.char(self))) + } +} + +extension UnicodeScalar: RegexComponent { + public typealias Output = Substring + + public var regex: Regex { + .init(node: .atom(.scalar(self))) } } +// MARK: - Testing public struct MockRegexLiteral: RegexComponent { public typealias MatchValue = Substring diff --git a/Sources/_StringProcessing/RegexDSL/DSLConsumers.swift b/Sources/_StringProcessing/Regex/DSLConsumers.swift similarity index 100% rename from Sources/_StringProcessing/RegexDSL/DSLConsumers.swift rename to Sources/_StringProcessing/Regex/DSLConsumers.swift diff --git a/Sources/_StringProcessing/RegexDSL/DSLTree.swift b/Sources/_StringProcessing/Regex/DSLTree.swift similarity index 81% rename from Sources/_StringProcessing/RegexDSL/DSLTree.swift rename to Sources/_StringProcessing/Regex/DSLTree.swift index 25a5943c0..e579828d2 100644 --- a/Sources/_StringProcessing/RegexDSL/DSLTree.swift +++ b/Sources/_StringProcessing/Regex/DSLTree.swift @@ -11,7 +11,8 @@ import _MatchingEngine -struct DSLTree { +@_spi(RegexBuilder) +public struct DSLTree { var root: Node var options: Options? @@ -22,7 +23,8 @@ struct DSLTree { } extension DSLTree { - indirect enum Node: _TreeNode { + @_spi(RegexBuilder) + public indirect enum Node: _TreeNode { /// Try to match each node in order /// /// ... | ... | ... @@ -101,7 +103,8 @@ extension DSLTree { } extension DSLTree { - struct CustomCharacterClass { + @_spi(RegexBuilder) + public struct CustomCharacterClass { var members: [Member] var isInverted: Bool @@ -120,7 +123,8 @@ extension DSLTree { } } - enum Atom { + @_spi(RegexBuilder) + public enum Atom { case char(Character) case scalar(Unicode.Scalar) case any @@ -134,18 +138,21 @@ extension DSLTree { } // CollectionConsumer -typealias _ConsumerInterface = ( +@_spi(RegexBuilder) +public typealias _ConsumerInterface = ( String, Range ) -> String.Index? // Type producing consume // TODO: better name -typealias _MatcherInterface = ( +@_spi(RegexBuilder) +public typealias _MatcherInterface = ( String, String.Index, Range ) -> (String.Index, Any)? // Character-set (post grapheme segmentation) -typealias _CharacterPredicateInterface = ( +@_spi(RegexBuilder) +public typealias _CharacterPredicateInterface = ( (Character) -> Bool ) @@ -161,7 +168,8 @@ typealias _CharacterPredicateInterface = ( */ extension DSLTree.Node { - var children: [DSLTree.Node]? { + @_spi(RegexBuilder) + public var children: [DSLTree.Node]? { switch self { case let .orderedChoice(v): return v @@ -256,7 +264,8 @@ extension DSLTree { } } extension DSLTree.Node { - func _captureStructure( + @_spi(RegexBuilder) + public func _captureStructure( _ constructor: inout CaptureStructure.Constructor ) -> CaptureStructure { switch self { @@ -323,14 +332,18 @@ extension DSLTree.Node { } extension DSLTree.Node { - func appending(_ newNode: DSLTree.Node) -> DSLTree.Node { + @_spi(RegexBuilder) + public func appending(_ newNode: DSLTree.Node) -> DSLTree.Node { if case .concatenation(let components) = self { return .concatenation(components + [newNode]) } return .concatenation([self, newNode]) } - func appendingAlternationCase(_ newNode: DSLTree.Node) -> DSLTree.Node { + @_spi(RegexBuilder) + public func appendingAlternationCase( + _ newNode: DSLTree.Node + ) -> DSLTree.Node { if case .orderedChoice(let components) = self { return .orderedChoice(components + [newNode]) } @@ -338,32 +351,13 @@ extension DSLTree.Node { } } -extension DSLTree.Node { - /// Generates a DSLTree node for a repeated range of the given DSLTree node. - /// Individual public API functions are in the generated Variadics.swift file. - static func repeating( - _ range: Range, - _ behavior: QuantificationBehavior, - _ node: DSLTree.Node - ) -> DSLTree.Node { - // TODO: Throw these as errors - assert(range.lowerBound >= 0, "Cannot specify a negative lower bound") - assert(!range.isEmpty, "Cannot specify an empty range") - - switch (range.lowerBound, range.upperBound) { - case (0, Int.max): // 0... - return .quantification(.zeroOrMore, behavior.astKind, node) - case (1, Int.max): // 1... - return .quantification(.oneOrMore, behavior.astKind, node) - case _ where range.count == 1: // ..<1 or ...0 or any range with count == 1 - // Note: `behavior` is ignored in this case - return .quantification(.exactly(.init(faking: range.lowerBound)), .eager, node) - case (0, _): // 0..(_ reference: Reference) -> Capture { - guard let offset = referencedCaptureOffsets[reference.id] else { + @_spi(RegexBuilder) + public subscript(_ id: ReferenceID) -> Capture { + guard let offset = referencedCaptureOffsets[id] else { preconditionFailure( "Reference did not capture any match in the regex") } @@ -98,21 +99,9 @@ extension String { public func match(_ regex: R) -> Regex.Match? { regex.match(in: self) } - - public func match( - @RegexComponentBuilder _ content: () -> R - ) -> Regex.Match? { - match(content()) - } } extension Substring { public func match(_ regex: R) -> Regex.Match? { regex.match(in: self) } - - public func match( - @RegexComponentBuilder _ content: () -> R - ) -> Regex.Match? { - match(content()) - } } diff --git a/Sources/_StringProcessing/RegexDSL/Options.swift b/Sources/_StringProcessing/Regex/Options.swift similarity index 100% rename from Sources/_StringProcessing/RegexDSL/Options.swift rename to Sources/_StringProcessing/Regex/Options.swift diff --git a/Tests/RegexBuilderTests/AlgorithmsTests.swift b/Tests/RegexBuilderTests/AlgorithmsTests.swift new file mode 100644 index 000000000..183d247a7 --- /dev/null +++ b/Tests/RegexBuilderTests/AlgorithmsTests.swift @@ -0,0 +1,62 @@ +//===----------------------------------------------------------------------===// +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2021-2022 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// +//===----------------------------------------------------------------------===// + +import XCTest +import _StringProcessing +@testable import RegexBuilder + +class RegexConsumerTests: XCTestCase { + func testMatches() { + let regex = Capture(OneOrMore(.digit)) { 2 * Int($0)! } + let str = "foo 160 bar 99 baz" + XCTAssertEqual(str.matches(of: regex).map(\.result.1), [320, 198]) + } + + func testMatchReplace() { + func replaceTest( + _ regex: R, + input: String, + result: String, + _ replace: (_MatchResult>) -> String, + file: StaticString = #file, + line: UInt = #line + ) { + XCTAssertEqual(input.replacing(regex, with: replace), result) + } + + let int = Capture(OneOrMore(.digit)) { Int($0)! } + + replaceTest( + int, + input: "foo 160 bar 99 baz", + result: "foo 240 bar 143 baz", + { match in String(match.result.1, radix: 8) }) + + replaceTest( + Regex { int; "+"; int }, + input: "9+16, 0+3, 5+5, 99+1", + result: "25, 3, 10, 100", + { match in "\(match.result.1 + match.result.2)" }) + + // TODO: Need to support capture history + // replaceTest( + // OneOrMore { int; "," }, + // input: "3,5,8,0, 1,0,2,-5,x8,8,", + // result: "16 3-5x16", + // { match in "\(match.result.1.reduce(0, +))" }) + + replaceTest( + Regex { int; "x"; int; Optionally { "x"; int } }, + input: "2x3 5x4x3 6x0 1x2x3x4", + result: "6 60 0 6x4", + { match in "\(match.result.1 * match.result.2 * (match.result.3 ?? 1))" }) + } +} diff --git a/Tests/RegexTests/CustomTests.swift b/Tests/RegexBuilderTests/CustomTests.swift similarity index 84% rename from Tests/RegexTests/CustomTests.swift rename to Tests/RegexBuilderTests/CustomTests.swift index 12d4ad6cd..b405a5399 100644 --- a/Tests/RegexTests/CustomTests.swift +++ b/Tests/RegexBuilderTests/CustomTests.swift @@ -1,5 +1,17 @@ -import _StringProcessing +//===----------------------------------------------------------------------===// +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2021-2022 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// +//===----------------------------------------------------------------------===// + import XCTest +import _StringProcessing +@testable import RegexBuilder // A nibbler processes a single character from a string private protocol Nibbler: CustomRegexComponent { @@ -58,8 +70,7 @@ func customTest( } } -extension RegexTests { - +class CustomRegexComponentTests: XCTestCase { // TODO: Refactor below into more exhaustive, declarative // tests. func testCustomRegexComponents() { @@ -127,5 +138,4 @@ extension RegexTests { XCTAssertEqual(res4.result.0, "123") XCTAssertEqual(res4.result.1, 3) } - } diff --git a/Tests/RegexTests/RegexDSLTests.swift b/Tests/RegexBuilderTests/RegexDSLTests.swift similarity index 99% rename from Tests/RegexTests/RegexDSLTests.swift rename to Tests/RegexBuilderTests/RegexDSLTests.swift index 4ca446b32..a7d307b9b 100644 --- a/Tests/RegexTests/RegexDSLTests.swift +++ b/Tests/RegexBuilderTests/RegexDSLTests.swift @@ -10,7 +10,8 @@ //===----------------------------------------------------------------------===// import XCTest -@testable import _StringProcessing +import _StringProcessing +@testable import RegexBuilder class RegexDSLTests: XCTestCase { func _testDSLCaptures( diff --git a/Tests/RegexTests/AlgorithmsTests.swift b/Tests/RegexTests/AlgorithmsTests.swift index 6a7bf646b..b51f12100 100644 --- a/Tests/RegexTests/AlgorithmsTests.swift +++ b/Tests/RegexTests/AlgorithmsTests.swift @@ -114,52 +114,6 @@ class RegexConsumerTests: XCTestCase { expectReplace("aab", "a+", "X", "Xb") expectReplace("aab", "a*", "X", "XXbX") } - - func testMatches() { - let regex = Capture(OneOrMore(.digit)) { 2 * Int($0)! } - let str = "foo 160 bar 99 baz" - XCTAssertEqual(str.matches(of: regex).map(\.result.1), [320, 198]) - } - - func testMatchReplace() { - func replaceTest( - _ regex: R, - input: String, - result: String, - _ replace: (_MatchResult>) -> String, - file: StaticString = #file, - line: UInt = #line - ) { - XCTAssertEqual(input.replacing(regex, with: replace), result) - } - - let int = Capture(OneOrMore(.digit)) { Int($0)! } - - replaceTest( - int, - input: "foo 160 bar 99 baz", - result: "foo 240 bar 143 baz", - { match in String(match.result.1, radix: 8) }) - - replaceTest( - Regex { int; "+"; int }, - input: "9+16, 0+3, 5+5, 99+1", - result: "25, 3, 10, 100", - { match in "\(match.result.1 + match.result.2)" }) - - // TODO: Need to support capture history - // replaceTest( - // OneOrMore { int; "," }, - // input: "3,5,8,0, 1,0,2,-5,x8,8,", - // result: "16 3-5x16", - // { match in "\(match.result.1.reduce(0, +))" }) - - replaceTest( - Regex { int; "x"; int; Optionally { "x"; int } }, - input: "2x3 5x4x3 6x0 1x2x3x4", - result: "6 60 0 6x4", - { match in "\(match.result.1 * match.result.2 * (match.result.3 ?? 1))" }) - } func testAdHoc() { let r = try! Regex("a|b+") diff --git a/Tests/RegexTests/CaptureTests.swift b/Tests/RegexTests/CaptureTests.swift index cc3568c1d..258aea86d 100644 --- a/Tests/RegexTests/CaptureTests.swift +++ b/Tests/RegexTests/CaptureTests.swift @@ -1,6 +1,16 @@ +//===----------------------------------------------------------------------===// +// +// This source file is part of the Swift.org open source project +// +// Copyright (c) 2021-2022 Apple Inc. and the Swift project authors +// Licensed under Apache License v2.0 with Runtime Library Exception +// +// See https://swift.org/LICENSE.txt for license information +// +//===----------------------------------------------------------------------===// import XCTest -@testable import _StringProcessing +@testable @_spi(RegexBuilder) import _StringProcessing import _MatchingEngine extension StructuredCapture {