[finetune_trainer] --evaluate_during_training is no more

[stas00]

In examples/seq2seq/builtin_trainer/ all scripts reference --evaluate_during_training but it doesn't exist in pt trainer, but does exist in tf trainer:

grep -Ir evaluate_during
builtin_trainer/finetune.sh:    --do_train --do_eval --do_predict --evaluate_during_training \
builtin_trainer/train_distil_marian_enro.sh:    --do_train --do_eval --do_predict --evaluate_during_training\
builtin_trainer/finetune_tpu.sh:    --do_train --do_eval --evaluate_during_training \
builtin_trainer/train_distilbart_cnn.sh:    --do_train --do_eval --do_predict --evaluate_during_training \
builtin_trainer/train_distil_marian_enro_tpu.sh:    --do_train --do_eval --evaluate_during_training \
builtin_trainer/train_mbart_cc25_enro.sh:    --do_train --do_eval --do_predict --evaluate_during_training \

Traceback (most recent call last):
  File "finetune_trainer.py", line 310, in <module>
    main()
  File "finetune_trainer.py", line 118, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/transformers/hf_argparser.py", line 144, in parse_args_into_dataclasses
    raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")
ValueError: Some specified arguments are not used by the HfArgumentParser: ['--evaluate_during_training']

Is this meant to be replaced by: --evaluation_strategy - this is the closest I found in training_args.py

If so which one? steps or epoch?

Also the help output is borked:

$ python finetune_trainer.py -h
...
 [--evaluation_strategy {EvaluationStrategy.NO,EvaluationStrategy.STEPS,EvaluationStrategy.EPOCH}]

probably this is not what what's intended, but

 [--evaluation_strategy {no, steps, epochs}

But perhaps it's a bigger issue - I see trainer.args.evaluate_during_training:

src/transformers/integrations.py:        ) and (not trainer.args.do_eval or not trainer.args.evaluate_during_training):

and also --evaluate_during_training in many other files under examples/.

Thank you.

@sgugger, @patrickvonplaten

[stas00]

Found the source of breakage: #8604 - I guess that PR needs more work