Fixes importing inline comments #13139

[Achilles-96]

Signed-off-by: Raghuram Vadapalli [email protected]

[nijel]

From the test failure, it seems that it breaks parsing comments terminated by \n.

[Achilles-96]

I am not much familiar with parser project. Is it possible to run these tests on local machine.

[nijel]

Indeed the documentation on this was missing, I've just added it: https://github.com/phpmyadmin/sql-parser/blob/master/CONTRIBUTING.md

[nijel]

I've also added script bin/tokenize-query which you can use for testing parser as well.

With your changes the following query is tokenized wrongly:

#comment
SELECT 1

See:

$ ./bin/tokenize-query --query '#comment
SELECT 1'
[TOKEN 0]
Type = 4
Flags = 1
Value = '#comment
SELECT 1'
Token = '#comment
SELECT 1'

[TOKEN 1]
Type = 9
Flags = 0
Value = NULL
Token = NULL

While the correct (and current) tokenization is:

[TOKEN 0]
Type = 4
Flags = 1
Value = '#comment'
Token = '#comment'

[TOKEN 1]
Type = 1
Flags = 3
Value = 'SELECT'
Token = 'SELECT'

[TOKEN 2]
Type = 3
Flags = 0
Value = ' '
Token = ' '

[TOKEN 3]
Type = 6
Flags = 0
Value = 1
Token = '1'

[TOKEN 4]
Type = 9
Flags = 0
Value = NULL
Token = NULL

[nijel]

I've come with different fix, which does not break tokenization, see #148.

[nijel]

Closing this for now as it's breaking tokenization. Either we will use #148 or different patch is needed.