Wider support for gelu. Remove unused activation. Use same torch layer for silu and swish

[hypnopump]

Adds support for GeLU in modules where only silu/swish and mish were supported. Uses the same layer for silu and swish instead of a lambda definition for swish and the layer for silu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

I think we should create a simple activations.py file as is done here: https://github.com/huggingface/transformers/blob/main/src/transformers/activations.py and import from there

[hypnopump]

Probably @patrickvonplaten but that would be more of a design choice, whereas this is a path to fix a current bug.

[hypnopump]

Specifically, for a UNet1D model, it only works (forward+backward pass) if an activation is specified, but no failure is raised upon module creation, only at runtime. Also, there is an inconsistency between the timestep embedding module and the rest, where the timestep embedding does not accept gelu but other modules do.

As you say, probably it would be better for the future to have all activations grouped on a single place. That would help with maintainability and avoid code duplications.

[patrickvonplaten]

cc @williamberman let's move all those activation functions into their own file

[patrickvonplaten]

@williamberman could you try to help move all the activation functions to their own filesL

[williamberman]

hey @hypnopump sorry for the delay! I put up a PR here just since this branch is a bit behind tip and has some failing ci. Happy to merge this PR as well if you want to rebase and cherry pick the commit from my branch or what not. Up to you! :)

#3656

[hypnopump]

makes sense. im closing this then as functionality is implemented in your newer PR (left a review btw :)).