Query with missing semi-colon gets wrongfully parsed

[proddata]

The following 4 statements

select 1;
select 1 I can put anything here
select 3

... get parsed into 2 seemly valid statements ...

[
  Statement {
    query: 'select 1',
    originalQuery: 'select 1;\nselect 1 I can put anything here\nselect 3;',
    tree: '(statement (query (queryNoWith (queryTerm (querySpec select (selectItem (expr (booleanExpression (predicated (valueExpression (primaryExpression (parameterOrLiteral (parameterOrSimpleLiteral (numericLiteral (integerLiteral 1)))))))))))))))',
    type: 'select',
    ctx: DefaultContext {
      parentCtx: [StatementsContext],
      invokingState: 250,
      children: [Array],
      start: [Pt],
      stop: [Pt],
      exception: null,
      parser: [_SqlBaseParser],
      ruleIndex: 3
    },
    exception: null,
    metadata: Metadata {
      tables: [],
      parameterizedProperties: {},
      withProperties: {}
    }
  },
  Statement {
    query: 'select 1 I',
    originalQuery: 'select 1;\nselect 1 I can put anything here\nselect 3;',
    tree: '(statement (query (queryNoWith (queryTerm (querySpec select (selectItem (expr (booleanExpression (predicated (valueExpression (primaryExpression (parameterOrLiteral (parameterOrSimpleLiteral (numericLiteral (integerLiteral 1))))))))) (ident (unquotedIdent I))))))))',
    type: 'select',
    ctx: DefaultContext {
      parentCtx: [StatementsContext],
      invokingState: 252,
      children: [Array],
      start: [Pt],
      stop: [Pt],
      exception: null,
      parser: [_SqlBaseParser],
      ruleIndex: 3
    },
    exception: null,
    metadata: Metadata {
      tables: [],
      parameterizedProperties: {},
      withProperties: {}
    }
  }
]

... ignoring the missing semicolon ; and dropping the select 2; query.

[surister]

The way I see it is that for example in:

select 1;
select 2
select 3;

As we are missing an end of line ";" in line 2, therefore this query has only two statements:
st1 [select 1;]
st2 [select 2 select 3;]

st2 should generate an exception, like

t = sqlparse("""
select 1
select 2
""")
print(t[0].exception)
# InputMismatchException[line 3:0 mismatched input 'select' expecting {<EOF>, ';'}]

but in fact, it doesn't

t = sqlparse("""
select 1;
select 2
select 3;
""")
print(t[1].exception)
# 

[surister]

It's one of those situation edge cases where antlr4 doesn't give us context of the error, so we have to make up for that and use some heuristics. It's where we differ from the original CrateDB implementation, they only parse single statements while we do multiple statements, issues arises that never appeared to the core team hence the novelty.

These are very edge cases and we might find more in the future but at least they are very self contained (I know exactly where they happen) and at some point the heuristics will be good enough to cover most if not all. Growing pain unfortunately.