Support for syntax test output highlighting

[deathaxe]

Problem description

Reading the syntax test output can sometimes be a bit tedious. A syntax highlighting of the output could propaply help improving the readability of the test output.

Preferred solution

To do so, the run_syntax_tests.py needs to be patched to add support for the syntax key in sublime-build files.

The proposal also contains a syntax definition. But I don't have a final idea about how to scope the different parts of the build output in detail. Maybe the scope names need to be tweaked a little bit.

1. Add support for the `syntax` parameter of `sublime-build` files to the `run_syntax_tests` command.

--- C:\Apps\Sublime Text 3\Data\Packages\Default\run_syntax_tests.py    Thu Oct 17 18:22:14 2019
+++ C:\Apps\Sublime Text 3\Data\Packages\Default\run_syntax_tests.py    Thu Oct 17 18:22:34 2019
@@ -10,7 +10,11 @@
 
 
 class RunSyntaxTestsCommand(sublime_plugin.WindowCommand):
-    def run(self, find_all=False, **kwargs):
+
+    def run(self,
+            find_all=False,
+            syntax="Syntax Test.sublime-syntax",
+            **kwargs):
 
         if not hasattr(self, 'output_view'):
             # Try not to call get_output_panel until the regexes are assigned
@@ -23,6 +27,7 @@
         settings.set('line_numbers', False)
         settings.set('gutter', False)
         settings.set('scroll_past_end', False)
+        settings.set('syntax', syntax)
 
         # Call create_output_panel a second time after assigning the above
         # settings, so that it'll be picked up as a result buffer

2. Add a Syntax Test.sublime-syntax file

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: Sublime Syntax Tests
hidden: true
scope: text.syntax-tests
variables:
  scope_segment: '\w+(?:[\w-]*\+*)' # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    - match: ^(\[)Finished(\])
      scope: markup.info.syntax-tests
      captures:
        1: punctuation.definition.markup.begin.syntax-tests
        2: punctuation.definition.markup.end.syntax-tests
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      captures:
        1: markup.error.failed.syntax-tests
        2: punctuation.separator.syntax-tests
        3: markup.error.num-failed.syntax-tests
        4: markup.info.num-asserts.syntax-tests
        5: markup.info.num-files.syntax-tests
    - match: ^
      push:
        - match: '[/.]'
          scope: punctuation.separator.path.syntax-tests
        - match: (:)(\d+)(:)(\d+)(:)
          captures:
            1: punctuation.separator.sequence.syntax-tests
            2: constant.numeric.integer.decimal.syntax-tests
            3: punctuation.separator.sequence.syntax-tests
            4: constant.numeric.integer.decimal.syntax-tests
            5: punctuation.separator.sequence.syntax-tests
          set:
            - match: \n
              pop: true
            - match: \[
              scope: punctuation.definition.selector.begin.syntax-tests
              push:
                - match: \]
                  scope: punctuation.definition.selector.end.syntax-tests
                  pop: true
                - match: '\)'
                  scope: invalid.illegal.stray-bracket.scope-selector
                - include: scope-selector

###[ PackageDev's Scope Selector ]#############################################

  scope-selector:
    - match: (?:^\s*)?({{scope_segment}})
      captures:
        1: string.unquoted.scope-segment.scope-selector
      push:
        - match: (\.)(\.+)?
          captures:
            1: punctuation.separator.scope-segments.scope-selector
            2: invalid.illegal.empty-scope-segment.scope-selector
          pop: true
        - match: \s+\(
          scope: invalid.illegal.operator-required-between-scope-and-group.scope-selector
        - match: \s+(?=\w)
          scope: keyword.operator.right.scope-selector
          pop: true
        - match: ''
          pop: true
    - match: '-'
      scope: keyword.operator.without.scope-selector
    - match: '&'
      scope: keyword.operator.with.scope-selector
    - match: '[,|]'
      scope: keyword.operator.or.scope-selector
    - match: \(
      scope: punctuation.section.group.begin.scope-selector
      push:
        - match: \)
          scope: punctuation.section.group.end.scope-selector
          set:
            - match: (\s*{{scope_segment}}|\.)+
              scope: invalid.illegal.operator-required-after-group.scope-selector
            - match: \s*
              pop: true
        - include: scope-selector

Alternatives

If the syntax definition seems too much, please, at least implement step 1 with Plain Text.tmLanguage as default syntax to enable custom sublime-build files with a custom syntax definition.

Maybe a 3rd party package like PackageDev could implement the highlighting then, without the need of patching the run_syntax_tests.py implementation or hack around to make it work.

[FichteFoll]

Even in the case 1. was implemented, the existing build system wouldn't know to use it and PD would have to provide a separate one leading to duplicate entries in the build system selection.

I think the build output of syntax tests is in general not optimized for human readability.

[deathaxe]

My current implementation looks like ...

Packages/Default/Syntax Tests.sublime-syntax

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: Sublime Syntax Tests
hidden: true
scope: text.syntax-tests
variables:
  scope_segment: \w+(?:[\w-]*\+*) # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    # summaries
    - match: ^(\[)Finished(\])
      scope: meta.summary.finished.syntax-tests markup.info.finished.syntax-tests
      captures:
        1: punctuation.definition.markup.begin.syntax-tests
        2: punctuation.definition.markup.end.syntax-tests
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      scope: meta.summary.failure.syntax-tests
      captures:
        1: markup.error.failed.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests
        5: constant.numeric.syntax-tests
    - match: ^(Success)(:) (\d+) assertions in (\d+) files passed
      scope: meta.summary.success.syntax-tests
      captures:
        1: markup.info.success.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests

    # locations
    - match: ^Packages/(..[^:]*)(:)(\d+)(:)(\d+)
      scope: meta.path.syntax-tests entity.name.section
      captures:
        2: punctuation.separator.sequence.syntax-tests
        3: constant.numeric.integer.decimal.syntax-tests
        4: punctuation.separator.sequence.syntax-tests
        5: constant.numeric.integer.decimal.syntax-tests

    # assertions
    - match: ^\s*(expect|found)(:)
      captures:
        1: keyword.other.syntax-tests
        2: punctuation.separator.syntax-tests
      push: assertion-body

  assertion-body:
    - match: \n
      pop: 1
    - match: '{{scope_segment}}'
      scope: string.unquoted.scope-segment.scope-selector
    - match: \.
      scope: punctuation.separator.scope-segments.scope-selector

Packages/Default/run_syntax_tests.py

--- run_syntax_tests.py	Sat Feb 10 16:02:06 2024
+++ run_syntax_tests.py	Sat Feb 10 16:02:36 2024
@@ -23,6 +23,7 @@
         settings.set('line_numbers', False)
         settings.set('gutter', False)
         settings.set('scroll_past_end', False)
+        settings.set("syntax", kwargs.get("syntax", "Packages/Default/Syntax Tests.sublime-syntax"))
 
         # Call create_output_panel a second time after assigning the above
         # settings, so that it'll be picked up as a result buffer
@@ -65,13 +66,15 @@
         total_assertions = 0
         failed_assertions = 0
 
+        pattern = re.compile(r'(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]')
+        repl = r'\1:\2:\3\n  expect: \4\n   found: \5\n'
+
         for t in tests:
             assertions, test_output_lines = sublime_api.run_syntax_test(t)
             total_assertions += assertions
             if len(test_output_lines) > 0:
                 failed_assertions += len(test_output_lines)
-                for line in test_output_lines:
-                    append(self.output_view, line + '\n')
+                append(self.output_view, ''.join(pattern.sub(repl, line) for line in test_output_lines))
 
         if failed_assertions > 0:
             message = 'FAILED: {} of {} assertions in {} files failed\n'

Resulting syntax test output looks like:

[BenjaminSchaaf]

Build 4170 adds the syntax parameter as requested. I have not added the sublime-syntax for this, it's not clear to me whether that would be best as an internal syntax definition or as part of the default packages.

[deathaxe]

Maybe treat it as regular expressions syntaxes, which are assigned to find inputs in Default but are provided via separate Regular Expressions package.

Note, the syntax as provided in my former comment however works only together with reformatted syntax test output, which is IMHO as important as syntax highlighting for 2 reasons:

1. long selectors cause output to soft-wrap on normal screens, which tanks readability and results in messed up wall of text, if many assertions fail.
2. expected and actual selector printed aligned on separate lines helps comparing them.

IMHO, reformatted display is even more important than syntax highlighting.

Maybe use symbols to reduce repetition of expect: and found: key words.

Long story short: We could add a Syntax Tests package (or any suitable alternative name) to sublimehq/Packages with a syntax definition for current test output.

I'd still keep going with my reformatted one as it eases up debugging though.

[deathaxe]

I slightly tweaked reformatted version to use symbols instead of text and update scopes.

It is part of https://github.com/deathaxe/sublime-commands

run_syntax_tests.py

--- Packages\Default\run_syntax_tests.py	Sat Feb 24 16:28:59 2024
+++ Packages\Default\run_syntax_tests.py	Sat Feb 24 16:29:05 2024
@@ -75,18 +68,15 @@
         total_assertions = 0
         failed_assertions = 0
 
-        pattern = re.compile(r"(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]")
-        repl = r"\1:\2:\3\n  ✓ \4\n  ! \5\n"
+        pattern = re.compile(r'(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]')
+        repl = r'\1:\2:\3\n  ✓ \4\n  ! \5\n'
 
         for t in tests:
             assertions, test_output_lines = sublime_api.run_syntax_test(t)
             total_assertions += assertions
             if len(test_output_lines) > 0:
                 failed_assertions += len(test_output_lines)
-                append(
-                    self.output_view,
-                    "".join(pattern.sub(repl, line) for line in test_output_lines),
-                )
+                append(self.output_view, ''.join(pattern.sub(repl, line) for line in test_output_lines))
 
         if failed_assertions > 0:
             message = "FAILED: {} of {} assertions in {} files failed\n"

Sublime Syntax Test Results.sublime-syntax

%YAML 1.2
---
# https://www.sublimetext.com/docs/syntax.html
name: Sublime Syntax Test Results
hidden: true
scope: text.sublime.syntax-test-results
variables:
  scope_segment: \w+(?:[\w-]*\+*) # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    # comments
    - match: ^\#.+\n?
      scope: comment.line.double-dash.syntax-test-results
    # summaries
    - match: ^(\[)Finished(\])
      scope: meta.summary.finished.sublime.syntax-test-results markup.info.finished.sublime.syntax-test-results
      captures:
        1: punctuation.definition.markup.begin.sublime.syntax-test-results
        2: punctuation.definition.markup.end.sublime.syntax-test-results
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      scope: meta.summary.failure.sublime.syntax-test-results
      captures:
        1: markup.error.failed.sublime.syntax-test-results
        2: punctuation.separator.other.sublime.syntax-test-results
        3: meta.assertions.failed.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        4: meta.assertions.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        5: meta.test-files.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
    - match: ^(Success)(:) (\d+) assertions in (\d+) files passed
      scope: meta.summary.success.sublime.syntax-test-results
      captures:
        1: markup.info.success.sublime.syntax-test-results
        2: punctuation.separator.other.sublime.syntax-test-results
        3: meta.assertions.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        4: meta.test-files.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
    # locations
    - match: ^(?=Packages/)
      push: location
    - match: ^\s*(✓)\s
      captures:
        1: markup.info.sublime.syntax-test-results
      push: expected-scope
    - match: ^\s*(!)\s
      captures:
        1: markup.error.sublime.syntax-test-results
      push: found-scope

  location:
    - meta_content_scope: meta.path.filename.sublime.syntax-test-results entity.name.filename.sublime.syntax-test-results
    - match: (:)(\d+)(:)(\d+)
      captures:
        1: meta.path.row.sublime.syntax-test-results punctuation.separator.row.sublime.syntax-test-results
        2: meta.path.row.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        3: meta.path.column.sublime.syntax-test-results punctuation.separator.column.sublime.syntax-test-results
        4: meta.path.column.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
      pop: 1
    - match: /
      scope: punctuation.separator.path.sublime.syntax-test-results
    - match: $
      pop: 1

  expected-scope:
    - meta_content_scope: meta.scope.expected.sublime.syntax-test-results
    - match: \n
      pop: 1
    - include: scope-selectors

  found-scope:
    - meta_content_scope: meta.scope.found.sublime.syntax-test-results
    - match: \n
      pop: 1
    - include: scope-selectors

  scope-selectors:
    - meta_scope: meta.group.scope-selector
    - match: \(
      scope: punctuation.section.group.begin.scope-selector
      push: scope-selectors
    - match: \)
      scope: punctuation.section.group.end.scope-selector
      pop: 1
    - match: '{{scope_segment}}'
      scope: string.unquoted.scope-segment.scope-selector
    - match: \.
      scope: punctuation.separator.scope-segments.scope-selector
    - match: '-'
      scope: keyword.operator.without.scope-selector
    - match: '&'
      scope: keyword.operator.with.scope-selector
    - match: '[,|]'
      scope: keyword.operator.or.scope-selector

Mariana displays it as

With some additional color scheme rules added to my custom Brackets Dark color scheme it looks like

[michaelblyons]

If you're going to change the output formatting, IMO why not also roll up into a single error where expected/actual are also the same for consecutive characters on the same line?

Packages/C#/tests/syntax_test_C#11.cs:26:5-12

Edit: Maybe that's harder. Sorry.

[deathaxe]

Not sure if next_result/prev_result would pick those resulting regions up.