Implement a reliable way to test the syntax definition

[daviwil]

We need to find a way to test the syntax definition to ensure that any new changes don't break the grammar. It would be ideal if this could be done with the fewest number of dependencies possible so that we could run our tests in AppVeyor when PRs are sent.

I believe @vors might have some initial idea for how we could do that.

[vors]

A while ago @Jaykul proposed use https://github-lightshow.herokuapp.com/ for testing. I think it's the right way to go about it. It's fairly editor agnostic.

[daviwil]

I just found this GitHub repo: https://github.com/microsoft/vscode-textmate

Looks like we can use VS Code's TextMate parser as an API, might be helpful for testing the results of using our TextMate grammar against test files!

[daviwil]

The VS Code team uses some custom code to test the syntax definitions they ship (including the one for PowerShell). They use a well-known example code file for each language and then use their syntax tokenizer to output a JSON file which is then compared against a JSON file from a "known good" tokenization pass. Simple approach but it seems to work for them. We might be able to use this as a starting point for our own CI tests.

Here are some links to the relevant files:

• Core testing code
• PowerShell example file
• PowerShell expected token results

[daviwil]

Looks like the shared syntax definition repo for TypeScript also has a similar solution and it's also using the vscode-textmate library:

https://github.com/Microsoft/TypeScript-TmLanguage/tree/master/tests

[vors]

As a heads-up, I'm planning to spend some time this and next week working on this tests automation. Please, feel free to use https://gitter.im/PowerShell/EditorSyntax if you have some ideas to share about it.

[vors]

TypeScript is a good example!

Plan

How to evaluate tests

I don't particularly like an idea of storing serialized regions. I would probably prefer run current grammar vs previous release (tag or just commit sha1) grammar and if it gives the same results on the tests included in the previous release, then CI passes. That way

1. We don't need to store serialized intermediate tests in the repo
2. We are confident that there are no regressions on the previous release test subset.
3. There could be regressions (or fixes) on the new (compare to the baseline) tests, but maybe just leave them as a warnings.

How to store tests

I like the way how https://github.com/jgm/CommonMark/blob/master/spec.txt works.
It's the single source for http://spec.commonmark.org/0.25/ and for tests https://github.com/jgm/CommonMark/blob/master/test/spec_tests.py
All in one single file. Very convinient imo!

In SublimeText/PowerShell we have a single file, but it's a .ps1 file.
https://github.com/SublimeText/PowerShell/blob/dev/tests/samples/test-file.ps1

It forces everybody to put comments as powershell comments, which interfere with the highlighting.
For example, if something is broken in the middle of document and the rest of the doc is treated as a string, then test just show this fact and loose all other information.
It's kind of old compilers that reports only one error at a time. Not convenient.

If we take a similar to CommonMark approach, we can use a spec document with some loose structure, that allows to identify test-case regions. For simplicity and popularity I would say a markdown document.

Test document

It could look like that

## Basics

declarations should be consistent for functions

```powershell
function foo.bar() {}
Function foo() {}
```

And classes

```powershell
class A {}
Class Foo-Bar {}
```

And workflows

```powershell
workflow w1 {}
Workflow work {}
```

And configurations

```powershell
configuration c {}
Configuration c {}
```

## Highlight types
    Some explanation about test.

```powershell
[int[]][char[]]"hello world"
[string]$someVariable = [char[]](104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100)
[Collections.Generic.List``2[char]]$x = [char[]](104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100)
[Collections.Generic.List[char]]$x = [char[]](104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100)
```

Every

```powershell
   foobar
```

block would be treated as a separate test case.
This way, it's easy to read test cases subsequently (because they are listed subsequently) and get the context and intentions of them from the comments above and below.

Feedback is welcome! Please let me know if you see any problems with this approach

[daviwil]

So for the CommonMark approach, do they just have a "last known good" HTML file that they're comparing the Markdown formatting against? If so, that sounds fine to me. I like having a document serve a dual purpose to be both an example and a test artifact.

[vors]

No, they explicitly say what rendered view should look like, i.e.
https://github.com/jgm/CommonMark/blob/master/spec.txt#L531

```````````````````````````````` example
***
---
___
.
<hr />
<hr />
<hr />
````````````````````````````````

[daviwil]

Ahhh! Clever, I missed that.

[vors]

I started a node.js - based prototype for test harness that uses https://github.com/Microsoft/vscode-textmate

Opened a couple of issues:
microsoft/vscode-textmate#20
microsoft/vscode-textmate#21