[mlir] Testing infra improvements ideas

[banach-space]

Meta-ticket to collect ideas for improving the testing infrastructure in MLIR.

At the moment, this is a collection of suggestions extracted from:

• https://discourse.llvm.org/t/rfc-should-we-aim-for-more-consistency-in-tests/, and
• Extend TestingGuide (adds notes on formatting) mlir-www#216.

The order of these items is random. I will try to add more data/notes over time.

Please, if you decide to work on any of these, leave a comment so that we avoid duplicating efforts.

IDEA 1

Separate examples from ODS definitions into a new field: let examples = {}. Build infra so that ninja check-mlir would extra those examples and sanity-check them with mlir-opt (to avoid the examples becoming out-dated).

This could be used to demonstrate both "valid" and invalid usage.

IDEA 2

Generate, somehow, examples from the ODS assemblyFormat (even if we will likely want to update them manually).

IDEA 3

Revisit the concept of round-trip tests in "ops.mlir" files. They shouldn't be strictly necessary for ops with let assemblyFormat MLIR thoroughly tests the printer/parser generator.

IDEA 4

Improve "split" locations when encountering errors. Currently these are OK for humans to read - make them easier for tools to parse (and to jump to locations).

IDEA 5

Add more metadata to “splits” so that one could do something like:

---- Test: Verify the folding behavior with single dynamic rank.

This would then be printed alongside the error/failure message. This should be integrated with the existing metadata like here.

IDEA 6

Make generate-test-checks.py generate better names. For example:

// Automatically generated - possible near-future version (mild improvement)
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[BITCAST_1:.*]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[BITCAST_2.:*]] = vector.bitcast %[[BITCAST_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[BITCAST_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}

instead of:

// Automatically generated - today’s version
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[VAL_0:[0-9]+|[a-zA-Z$._-][a-zA-Z0-9$._-]*]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[VAL_1:.*]] = vector.bitcast %[[VAL_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[VAL_2:.*]] = vector.bitcast %[[VAL_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[VAL_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}

[llvmbot]

@llvm/issue-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

Meta-ticket to collect ideas for improving the testing infrastructure in MLIR.

At the moment, this is a collection of suggestions extracted from:

• https://discourse.llvm.org/t/rfc-should-we-aim-for-more-consistency-in-tests/, and
• Extend TestingGuide (adds notes on formatting) mlir-www#216.

The order of these items is random. I will try to add more data/notes over time.

Please, if you decide to work on any of these, leave a comment so that we avoid duplicating efforts.

IDEA 1

Separate examples from ODS definitions into a new field: let examples = {}. Build infra so that ninja check-mlir would extra those examples and sanity-check them with mlir-opt (to avoid the examples becoming out-dated).

This could be used to demonstrate both "valid" and invalid usage.

IDEA 2

Generate, somehow, examples from the ODS assemblyFormat (even if we will likely want to update them manually).

IDEA 3

Revisit the concept of round-trip tests in "ops.mlir" files. They shouldn't be strictly necessary for ops with let assemblyFormat MLIR thoroughly tests the printer/parser generator.

IDEA 4

Improve "split" locations when encountering errors. Currently these are OK for humans to read - make them easier for tools to parse (and to jump to locations).

IDEA 5

Add more metadata to “splits” so that one could do something like:

---- Test: Verify the folding behavior with single dynamic rank.

This would then be printed alongside the error/failure message. This should be integrated with the existing metadata like here.

IDEA 6

Make generate-test-checks.py generate better names. For example:

// Automatically generated - possible near-future version (mild improvement)
// CHECK-LABEL:   func.func @<!-- -->bitcast_2d(
// CHECK-SAME:                          %[[ARG_0]]: vector&lt;2x4xi32&gt;) -&gt; vector&lt;2x16xi8&gt; {
// CHECK:           %[[BITCAST_1:.*]] = vector.bitcast %[[ARG_0]] : vector&lt;2x4xi32&gt; to vector&lt;2x2xi64&gt;
// CHECK:           %[[BITCAST_2.:*]] = vector.bitcast %[[BITCAST_1]] : vector&lt;2x2xi64&gt; to vector&lt;2x16xi8&gt;
// CHECK:           return %[[BITCAST_2]] : vector&lt;2x16xi8&gt;
func.func @<!-- -->bitcast_2d(%arg0: vector&lt;2x4xi32&gt;) -&gt; vector&lt;2x16xi8&gt; {
  %0 = vector.bitcast %arg0 : vector&lt;2x4xi32&gt; to vector&lt;2x2xi64&gt;
  %1 = vector.bitcast %0 : vector&lt;2x2xi64&gt; to vector&lt;2x16xi8&gt;
  return %1 : vector&lt;2x16xi8&gt;
}

instead of:

// Automatically generated - today’s version
// CHECK-LABEL:   func.func @<!-- -->bitcast_2d(
// CHECK-SAME:                          %[[VAL_0:[0-9]+|[a-zA-Z$._-][a-zA-Z0-9$._-]*]]: vector&lt;2x4xi32&gt;) -&gt; vector&lt;2x16xi8&gt; {
// CHECK:           %[[VAL_1:.*]] = vector.bitcast %[[VAL_0]] : vector&lt;2x4xi32&gt; to vector&lt;2x2xi64&gt;
// CHECK:           %[[VAL_2:.*]] = vector.bitcast %[[VAL_1]] : vector&lt;2x2xi64&gt; to vector&lt;2x16xi8&gt;
// CHECK:           return %[[VAL_2]] : vector&lt;2x16xi8&gt;
func.func @<!-- -->bitcast_2d(%arg0: vector&lt;2x4xi32&gt;) -&gt; vector&lt;2x16xi8&gt; {
  %0 = vector.bitcast %arg0 : vector&lt;2x4xi32&gt; to vector&lt;2x2xi64&gt;
  %1 = vector.bitcast %0 : vector&lt;2x2xi64&gt; to vector&lt;2x16xi8&gt;
  return %1 : vector&lt;2x16xi8&gt;
}