[Example] Layer Norm Forward #170
Conversation
[ghstack-poisoned]
facebook-github-bot
added
the
CLA Signed
| for n in nodes: | ||
| if "output_index" in n.meta: | ||
| output_nodes[n.meta["output_index"]] = n.name | ||
| import pdb; pdb.set_trace() |
There was a problem hiding this comment.
| import pdb; pdb.set_trace() | |
| breakpoint() |
same thing but shorter.
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
There was a problem hiding this comment.
btw probably need to add a unit test to test_examples.py similar to other examples
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
jansel
left a comment
jansel
left a comment
There was a problem hiding this comment.
ghstack doesn't work in this repo, https://github.com/modular/stack-pr does though
| return out | ||
|
|
||
|
|
||
| def helion_layer_norm_wrapper( |
There was a problem hiding this comment.
PyTorch's layer_norm https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.layer_norm.html takes in normalized_shape as second arg, a list[int]. This allows us to work around it by not having to pass this into the kernel
There was a problem hiding this comment.
We should be able to pass it into the kernel, I think we can remove this.
| import helion.language as hl | ||
|
|
||
|
|
||
| # TODO(PaulZhang12): Support autotuning, setting reduction_loops currently errors |
There was a problem hiding this comment.
What is the error you are getting? Do you need help fixing this one? For benchmarking we need to run the autotuner.
There was a problem hiding this comment.
I filed an issue #345, haven't had time to look into it but can address it before merging
Stack from ghstack (oldest at bottom):