In libsyntax, parser.bump_and_get actually does "get and then bump" (function name confusing)

[allan-simon]

The name really get me confused because i was doing something like that

(just a little context here, in this piece of code , I use a custom lexer which pop token::EOF also for a custom token ( %> for a <% xxxx %> templating system ), EOF was chosen because

1. i needed to have a value in the token::Token enum to be able to leverage the other parser functions
2. it needed to be something that would not be parsed as part of an expression
3. something for which i will still have a way to make the difference between the actual token and my custom one (for EOF i still have parser.reader.is_eof() for me )

    match (
        parser.parse_ident(),
        parser.parse_fn_decl(true),
        parser.bump_and_get()
    ) {
        (
            functioname,
            ref function_decl,
            token::EOF,

        ) if parser.token == token::EOF && !parser.reader.is_eof()  => {

and it was not working because parser.token == token::EOF was actually testing the NEXT token , because bump_and_get was actually first returning the last token (the one used in the match) and AFTER moving the parser by one more token, (the one stored in parser.token)

anyway I've already been to fix my problem, so the problem is more about naming

my proposition

• replace bump_and_get by get_and_bump
• maybe permit one more value in the Token enum a token::USER_DEFINED(uint), so in order to help people writing syntax extension that superset rust, to still be able to use the normal parser, and just have to implement their own custom lexer::Reader trait.

[allan-simon]

or maybe we want both bump_and_get and get_and_bump ?

[steveklabnik]

/cc @rust-lang/compiler, is this a cleanup we want to do?

[nrc]

replace bump_and_get by get_and_bump

I'd take a PR for this, I don't think it's high priority or anything though.

AN extra token should probably have some discussion on internals or as an RFC.

[steveklabnik]

#40266 fixed the first part of this, I would suggest opening an internals discussion if you're still interested in the other part!