Add cut operator (^) to grammar

[traviscross]

The cut operator (^) is a backtracking fence. Once the expression to its left succeeds, we become committed to the alternative; the remainder of the expression must parse successfully or parsing will fail. See Packrat Parsers Can Handle Practical Grammars in Mostly Constant Space, Mizushima et al., https://kmizu.github.io/papers/paste513-mizushima.pdf.

This operator solves a problem for us with C string literals. These literals cannot contain a null escape. But if we simply fail to lex the literal (e.g. c"\0"), we may instead lex it successfully as two separate tokens (`c "\0"), and that would be incorrect.

As long as we only use cut to express constraints that can be expressed in a regular language and we keep our alternations disjoint, the grammar can still be mechanically converted to a CFG.

Let's add the cut operator to our grammar and use it for C string literals and some similar constructs.

In the railroad diagrams, we'll render the cut as a "no backtracking" box around the expression or sequence of expressions after the cut. The idea is that once you enter the box the only way out is forward.

(H/t to @ehuss for suggesting the cut operator to solve this problem.)

cc @ehuss

---

This is stacked on #2097 and should merge after it.

[mattheww]

When you're defining a cut operator, it's important to specify the scope over
which it cancels re-attempts.

For the lexer's purposes it would be fine to make that scope unlimited, saying that if the right-hand side of a production containing a cut fails after reaching the cut then the entire lexing process fails.

But in this PR the nearest thing to a definition of the cut operator is the reference to https://kmizu.github.io/papers/paste513-mizushima.pdf .

That paper defines a cut operator with a narrower scope: it allows a cut only on the left-hand side of an ordered choice expression, and cancels only the re-attempt of the right-hand side of that expression.

That definition doesn't work for the positions in which this PR is placing cuts.

[mattheww]

If you're still planning to use prioritised choice more widely in the Reference, then (given that the Reference already has the notion of reserved forms) perhaps the simplest way to define cut is to say that:

FOO ← a ^ b

is a shorthand for

FOO ← a b
RESERVED_PREFIX_OF_FOO ← a

with RESERVED_PREFIX_OF_FOO appearing immediately after FOO in the lexical grammar's top-level ordered choice for tokens.

That characterisation also illustrates why adding the notion of a cut doesn't buy very much.

In particular if you use prioritised choice for the token rules then (for the lexing dialect used in Rust 2021 and later) you can simplify the existing reserved token rules, getting rid of the "except b or c or r or br or cr" business, and end up with something like this:

C_STRING_LITERAL ← c " (more stuff)
RESERVED_TOKEN_DOUBLE_QUOTE ← IDENTIFIER_OR_KEYWORD "
IDENTIFIER_OR_KEYWORD ← (same as at present)

This way c"\0" is rejected as a reserved form, and there's no need to bother the Reference's readers with a discussion of cuts.

[traviscross]

Yes, @ehuss and I had earlier discussed these same matters, point for point. Thanks for elaborating them here; good to have these written out on the PR.

With regard to using RESERVED_ and ordered choice, that was my first thought too for solving this problem. But it's less theoretically satisfying than cut (with global escape) -- to me anyway -- since the input is still parsed or lexed successfully as far as the grammar is concerned. The idea that RESERVED_ rules matching are failures is something that has to be overlaid.

[traviscross]

With regard to the locality (or lack thereof) of cut, it's interesting that Python's grammar has a rule with cut that would be a no-op under the Mizushima et al. interpretation:

assignment_expression:
    | NAME ':=' ~ expression 

But their description of cut does not clearly suggest a global escape:

~ (“cut”): commit to the current alternative and fail the rule even if this fails to parse

[traviscross]

One library that takes the local interpretation is pegase. As they describe:

Used outside an ordered choice expression, it's simply a no-op.

[mattheww]

With regard to the locality (or lack thereof) of cut, it's interesting that Python's grammar has a rule with cut that would be a no-op under the Mizushima et al. interpretation:

AIUI the cuts in the Python grammar are there for performance (and debuggability) reasons, not to change the accepted language.

The parser can discard some state as soon as it gets to the cut (as described in the paper you linked), so it's useful even when there's no choice operator following.

[mattheww]

While you're looking at the quoted literals, you might consider helping the cut operator earn its keep by changing

SUFFIX → IDENTIFIER_OR_KEYWORD_{except _}

to something equivalent to

SUFFIX → XID_Start XID_Continue^* | _ ^ XID_Continue⁺

That would fix another bug in this family, by preventing something like "xxx"_ being analysed as two tokens.

[traviscross]

@ehuss has a forthcoming PR, likely to supersede this one, that resolves a large number of grammar issues. On that branch, looks like he went with:

SUFFIX ->
      `_` ^ XID_Continue+
    | XID_Start XID_Continue*

[ehuss]

Here's a commit that changes this to be an unary operator: ehuss@21ea969

[traviscross]

Thanks; cherry-picked.