Skip to content

Update local proposal copies #317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 22, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions Documentation/Evolution/RegexSyntaxRunTimeConstruction.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@

# Regex Syntax and Run-time Construction

- Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman)
* Proposal: [SE-NNNN](NNNN-filename.md)
* Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman)
* Review Manager: [Ben Cohen](https://github.com/airspeedswift)
* Status: **Awaiting review**
* Implementation: https://github.com/apple/swift-experimental-string-processing
* Available in nightly toolchain snapshots with `import _StringProcessing`

## Introduction

Expand Down Expand Up @@ -81,11 +86,11 @@ We propose initializers to declare and compile a regex from syntax. Upon failure
```swift
extension Regex {
/// Parse and compile `pattern`, resulting in a strongly-typed capture list.
public init(compiling pattern: String, as: Output.Type = Output.self) throws
public init(_ pattern: String, as: Output.Type = Output.self) throws
}
extension Regex where Output == AnyRegexOutput {
/// Parse and compile `pattern`, resulting in an existentially-typed capture list.
public init(compiling pattern: String) throws
public init(_ pattern: String) throws
}
```

Expand Down Expand Up @@ -156,6 +161,20 @@ extension Regex.Match where Output == AnyRegexOutput {
}
```

We propose adding API to query and access captures by name in an existentially typed regex match:

```swift
extension Regex.Match where Output == AnyRegexOutput {
/// If a named-capture with `name` is present, returns its value. Otherwise `nil`.
public subscript(_ name: String) -> AnyRegexOutput.Element? { get }
}

extension AnyRegexOutput {
/// If a named-capture with `name` is present, returns its value. Otherwise `nil`.
public subscript(_ name: String) -> AnyRegexOutput.Element? { get }
}
```

The rest of this proposal will be a detailed and exhaustive definition of our proposed regex syntax.

<details><summary>Grammar Notation</summary>
Expand Down
31 changes: 25 additions & 6 deletions Documentation/Evolution/RegexTypeOverview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# Regex Type and Overview

- Authors: [Michael Ilseman](https://github.com/milseman)
* Proposal: [SE-0350](0350-regex-type-overview.md)
* Authors: [Michael Ilseman](https://github.com/milseman)
* Review Manager: [Ben Cohen](https://github.com/airspeedswift)
* Status: **Active Review (4 - 28 April 2022)**
* Implementation: https://github.com/apple/swift-experimental-string-processing
* Available in nightly toolchain snapshots with `import _StringProcessing`

## Introduction

Expand Down Expand Up @@ -207,7 +212,7 @@ func processEntry(_ line: String) -> Transaction? {
// amount: Substring
// )>

guard let match = regex.matchWhole(line),
guard let match = regex.wholeMatch(line),
let kind = Transaction.Kind(match.kind),
let date = try? Date(String(match.date), strategy: dateParser),
let amount = try? Decimal(String(match.amount), format: decimalParser)
Expand Down Expand Up @@ -384,21 +389,25 @@ extension Regex.Match {
// Run-time compilation interfaces
extension Regex {
/// Parse and compile `pattern`, resulting in a strongly-typed capture list.
public init(compiling pattern: String, as: Output.Type = Output.self) throws
public init(_ pattern: String, as: Output.Type = Output.self) throws
}
extension Regex where Output == AnyRegexOutput {
/// Parse and compile `pattern`, resulting in an existentially-typed capture list.
public init(compiling pattern: String) throws
public init(_ pattern: String) throws
}
```

### Cancellation

Regex is somewhat different from existing standard library operations in that regex processing can be a long-running task.
For this reason regex algorithms may check if the parent task has been cancelled and end execution.

### On severability and related proposals

The proposal split presented is meant to aid focused discussion, while acknowledging that each is interconnected. The boundaries between them are not completely cut-and-dry and could be refined as they enter proposal phase.

Accepting this proposal in no way implies that all related proposals must be accepted. They are severable and each should stand on their own merit.


## Source compatibility

Everything in this proposal is additive. Regex delimiters may have their own source compatibility impact, which is discussed in that proposal.
Expand Down Expand Up @@ -488,6 +497,16 @@ The generic parameter `Output` is proposed to contain both the whole match (the

The biggest issue with this alternative design is that the numbering of `Captures` elements misaligns with the numbering of captures in textual regexes, where backreference `\0` refers to the entire match and captures start at `\1`. This design would sacrifice familarity and have the pitfall of introducing off-by-one errors.

### Encoding `Regex`es into the type system

During the initial review period the following comment was made:

> I think the goal should be that, at least for regex literals (and hopefully for the DSL to some extent), one day we might not even need a bytecode or interpreter. I think the ideal case is if each literal was its own function or type that gets generated and optimised as if you wrote it in Swift.

This is an approach that has been tried a few times in a few different languages (including by a few members of the Swift Standard Library and Core teams), and while it can produce attractive microbenchmarks, it has almost always proved to be a bad idea at the macro scale. In particular, even if we set aside witness tables and other associated swift generics overhead, optimizing a fixed pipeline for each pattern you want to match causes significant codesize expansion when there are multiple patterns in use, as compared to a more flexible byte code interpreter. A bytecode interpreter makes better use of instruction caches and memory, and can also benefit from micro architectural resources that are shared across different patterns. There is a tradeoff w.r.t. branch prediction resources, where separately compiled patterns may have more decisive branch history data, but a shared bytecode engine has much more data to use; this tradeoff tends to fall on the side of a bytecode engine, but it does not always do so.

It should also be noted that nothing prevents AOT or JIT compiling of the bytecode if we believe it will be advantageous, but compiling or interpreting arbitrary Swift code at runtime is rather more unattractive, since both the type system and language are undecidable. Even absent this rationale, we would probably not encode regex programs directly into the type system simply because it is unnecessarily complex.

### Future work: static optimization and compilation

Swift's support for static compilation is still developing, and future work here is leveraging that to compile regex when profitable. Many regex describe simple [DFAs](https://en.wikipedia.org/wiki/Deterministic_finite_automaton) and can be statically compiled into very efficient programs. Full static compilation needs to be balanced with code size concerns, as a matching-specific bytecode is typically far smaller than a corresponding program (especially since the bytecode interpreter is shared).
Expand Down Expand Up @@ -551,4 +570,4 @@ Regexes are often used for tokenization and tokens can be represented with Swift

-->

[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md
[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md