[Benchmark] geglu example and test

[Sibylau]

[Benchmark] add geglu example and test

• Add geglu Helion example
• Register Helion geglu in Tritonbench to compare against torch geglu in llama, geglu in liger_kernels, and torch_compile.
• Add a test to check the generated Triton code

For the Triton kernel benchmarking issue #234.

[oulgen]

if we now need transformers, you're gonna need to update the CI to handle it.

[oulgen]

@Sibylau not needing transformers would be better if you wanna write baselines

[yf225]

Wonder does python benchmarks/run.py --op geglu --metrics accuracy pass (i.e. showing accuracy check = 1 for all backends)?

[Sibylau]

Yes, it passes. Maybe it's good to post accuracy pass info in each PR, and document the performance.

[Sibylau]

But torch.compile for the geglu kernel seems to have accuracy issues

[yf225]

@Sibylau I just merged #596 to allow passing the TB operator instance as the first argument to the Helion integration wrapper geglu_tritonbench - now we should be able to access the TB baseline's model weights in helion tritonbench wrapper and copy the weights into the helion MLP.

It would be great to run the tritonbench accuracy check again to confirm it passes, thanks!

[Sibylau]

Thank you! The new commit copies the weights, and the accuracy matches

[yf225]

We shouldn't be commenting out these fp32 precision lines - could you share the errors you were seeing?

[Sibylau]

Sorry I should've uncommented before pushing the code. But I ran into this error when using these fp32 precision:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/main.py", line 101, in __init__
    self.parseArgs(argv)
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/main.py", line 150, in parseArgs
    self.createTests()
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/main.py", line 161, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/loader.py", line 232, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/loader.py", line 232, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/unittest/loader.py", line 162, in loadTestsFromName
    module = __import__(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jieeliu/workspace/helion/test/test_examples.py", line 17, in <module>
    torch.backends.cuda.matmul.fp32_precision = "tf32"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jieeliu/envs/conda/envs/helion/lib/python3.11/site-packages/torch/backends/cuda/__init__.py", line 149, in __setattr__
    raise AttributeError("Unknown attribute " + name)
AttributeError: Unknown attribute fp32_precision

[yf225]

hmm wonder are you using the latest PyTorch nightly?

[Sibylau]

My torch version is

__all__ = ['__version__', 'debug', 'cuda', 'git_version', 'hip', 'xpu']
__version__ = '2.8.0+cu128'
debug = False
cuda: Optional[str] = '12.8'
git_version = 'a1cb3cc05d46d198467bebbb6e8fba50a325d4e7'
hip: Optional[str] = None
xpu: Optional[str] = None

Yeah, I remember it was PyTorch 2.8.0 nightly. I can try different versions too.

[Sibylau]

The BaselineModel and TritonBenchOperator classes are type annotations to pass pyright type checking. If there are better ways to write, please suggest

[yf225]

To keep it simple, maybe we can just use object as type for tb_op (similar to

helion/examples/jagged_mean.py

Line 141 in 5190605

tb_op: object, x: torch.Tensor, B: int, M: int, seqlen: int, sparsity: float

), and add type ignore there as necessary, so that we don't need these classes.

[yf225]

To keep it simple, maybe we can just use object as type for tb_op (similar to

helion/examples/jagged_mean.py

Line 141 in 5190605

tb_op: object, x: torch.Tensor, B: int, M: int, seqlen: int, sparsity: float

), and add type ignore there as necessary, so that we don't need these classes.

[yf225]

thanks @Sibylau !