Transformers v4.12.0 compatible #107

JulianneKnott · 2021-11-22T18:00:41Z

Updating Fastseq for compatibility with huggingface transformers v4.12.0.
Benchmarks (samples/sec):

Bart       13.0 (2.9x)
DistilBart 19.1 (3.5x)
T5         29.2 (2.0x)
GPT2       20.3 (5.1x)
ProphetNet TBD

feihugis

@JulianneKnott Thanks for updating HF models! I did a pass and left some minor comments. Did you check if all the changed code have been used during the inference?

benchmarks/models/hf_bart.sh

benchmarks/models/hf_t5.sh

examples/bart/README.md

fastseq/ops/ngram_repeat_block.py

fastseq/optimizer/transformers/beam_search_optimizer.py

fastseq/optimizer/transformers/modeling_bart_optimizer.py

tests/optimizer/transformers/data/expected_prophetnet_output_no_cache.hypo

azure-pipelines.yml

JiushengChen · 2021-11-30T19:15:14Z

benchmarks/hf.sh

-    #https://github.com/huggingface/transformers.git \
+    export BASELINE_REPO=$CACHE_DIR/transformers_v4.12.0
    git_clone_if_not_in_cache \
-	https://github.com/JiushengChen/transformers.git \


Some context about this forked repo, it is to add param "no_repeat_ngram_size", see
JiushengChen/Transformers@db97043

To clarify, this is no longer needed since fastseq uses it's own run_eval_hf.py for the baseline?

README.md

azure-pipelines.yml

benchmarks/models/hf_mbart.sh

docker/Dockerfile

examples/bart/README.md

examples/gpt2/README.md

examples/prophetnet/README.md

fastseq/config.py

tests/run_fastseq_tests.sh

fastseq/optimizer/transformers/beam_search_optimizer.py

feihugis · 2021-12-13T18:20:27Z

fastseq/optimizer/transformers/beam_search_optimizer.py

+        is_greedy_gen_mode = (num_beams == 1) and (num_beam_groups == 1) and do_sample is False
+        is_sample_gen_mode = (num_beams == 1) and (num_beam_groups == 1) and do_sample is True
+        is_beam_gen_mode = (num_beams > 1) and (num_beam_groups == 1) and do_sample is False
+        is_beam_sample_gen_mode = (num_beams > 1) and (num_beam_groups == 1) and do_sample is True
+        is_group_beam_gen_mode = (num_beams > 1) and (num_beam_groups > 1)


Are our changes compatible with all these generation modes?

The beam search updates are only applied for is_beam_gen_mode. The model-specific updates (ie attention) are applied in all cases.

Does it also work with is_group_beam_gen_mode?

no is_group_beam_gen_mode does not use the updates. Should that be added?

If the optimizations can also work for group_beam_gen_mode, we can add it. We can support it in the another PR later if it will take some time to support it.

I think it will take some time to make it work for both, so I think it's best if group_beam_gen_mode goes in another pr

Sounds good.

fastseq_cli/transformers_generate.py

feihugis

Thanks @JulianneKnott for the revisions! The PR looks good to me!

examples/prophetnet/README.md

JulianneKnott added 15 commits October 28, 2021 17:42

update transformers version number

12b9985

update transformer version number (4.12.0)

e2b7a1d

bart and gpt2 updates (no el attn, old cli)

2566867

t5 updates (no el attn, old cli)

434185c

debug

2a8f779

updated (existing fastseq optimization)

67bf684

bart, gpt2, t5 benchmarks updated

6f8e451

add prophetnet

c3fc882

add distilbart benchmark

c6b98b6

import prophetnet optimizer

f00c752

simplify prophetnet test

9c32b81

update benchmarks checks

940a78d

prophetnet benchmarks

39b33ba

cleanup

f4ce9c2

cleanup

2ac5317

feihugis reviewed Nov 22, 2021

View reviewed changes

JulianneKnott added 7 commits November 23, 2021 17:57

cleanup

08cc31a

azure pipeline updates

bd58d06

sentencepiece update (docker + azure pipeline)

d9466e4

pipelines update torch=1.10.0

cc4494c

fix prophetnet fs expected output + cleanup

c3c98cb

benchmarks, examples, + readmes

e4354c3

fixed t5 ngram repeat block

9fb69a7

JiushengChen reviewed Nov 30, 2021

View reviewed changes

JulianneKnott added 6 commits December 2, 2021 01:03

fix gpt2, t5, prophetnet unit hf tests

298ce3a

fixed tests broken by fastseq update

b6e899e

unit test cleanup (w/ hf docker)

6faf3fb

update pipeline image

e549380

update pipeline

e1e76d5

pipeline debug

8fb5575

JulianneKnott added 17 commits December 9, 2021 02:33

debug

f9e8c16

debug

c464dee

debug

addd2a9

debug

c61bedb

pipeline debug

645887b

debug pipeline

cc34fc7

debug pipeline

8490d64

debug pipeline

1e43d3f

fastseq tests working, run fairseq tests

816cab9

debug venv

124f768

debug

7aab0d5

debug

f28672d

fairseq tests working, run with fastseq tests

7d83023

try running all tests

1633ef3

typo fix

49beb3d

fixed last 4 failing tests

980efa6

cleanup

0ad879e

feihugis reviewed Dec 13, 2021

View reviewed changes

JulianneKnott added 2 commits December 13, 2021 18:39

compatibility up to 4.12.5

943f198

compatibility

5233d86

feihugis reviewed Dec 13, 2021

View reviewed changes

JulianneKnott added 4 commits December 13, 2021 18:51

cleanup

31a49ff

fix bart benchmarks

8ad78b3

cleanup

d501760

cleanup

61fb12a

feihugis approved these changes Dec 13, 2021

View reviewed changes

prophetnet benchmarks

c66ec06

feihugis reviewed Dec 14, 2021

View reviewed changes

examples/prophetnet/README.md Outdated Show resolved Hide resolved

typo fixed

3513f0f

JulianneKnott merged commit df98314 into main Dec 14, 2021

Transformers v4.12.0 compatible #107

Transformers v4.12.0 compatible #107

Uh oh!

Conversation

JulianneKnott commented Nov 22, 2021

Uh oh!

feihugis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

feihugis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants