[Tokenizer] Segfault during tokenization

[akshanshbhatt]

While working on #509, I noticed that the tokenizer failed to tokenize this code -

def main():
    if foo:
        pass
#this is a comment
        pass
        if bar:
            pass
        pass
    pass

The first time I tried this, I was prompted with a segfault traceback. For subsequent tries, my terminal froze and I had to manually interrupt the process -

❯ lpy --show-tokens examples/expr2.py
Traceback (most recent call last):
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 821
    return emit_tokens(arg_file, true, compiler_options);
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 93
    auto res = LFortran::tokens(al, input, diagnostics, &stypes, &locations);
  File "tokenizer.re", line 612
  File "tokenizer.re", line 181
  Binary file "/usr/lib/system/libsystem_platform.dylib", local address: 0x1803434a3
Segfault: Signal SIGSEGV (segmentation fault) received
❯ lpy --show-tokens examples/expr2.py
^C
❯ lpy --show-tokens examples/expr2.py
^C

CPython's tokenizer returns the tokens as expected.

[certik]

Thanks for the report. It's probably some stuff with the indentation tracking, it needs to be debugged and fixed.