TransformerXL: StopIteration: Caught StopIteration in replica 0 on device 0

[davidliujiafeng]

Environment info

• transformers version: 3.4.0
• Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-debian-stretch-sid
• Python version: 3.6.9
• PyTorch version (GPU?): 1.6.0 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Yes

@TevenLeScao

Error I get

Traceback (most recent call last):
  File "/ai/fzc/minGPT/transformerXLtest.py", line 163, in <module>
    input_ids=inputs["input_ids"].to(device),
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/opt/conda/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_transfo_xl.py", line 866, in forward
    mems = self.init_mems(bsz)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_transfo_xl.py", line 800, in init_mems
    param = next(self.parameters())
StopIteration

To reproduce the problem

Run Code below:

import torch
from torch.nn import DataParallel
from transformers import TransfoXLTokenizer, TransfoXLModel

device = "cuda:0"

# Get model
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103', return_dict=True)
model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

# Run forward
inputs = tokenizer(["This is an example"], return_tensors="pt")

outputs = model(
    input_ids=inputs["input_ids"].to(device),
)

print(f"outputs: {outputs}")
print("Success.")

[davidliujiafeng]

The same code I tested on GPT-2, works fine for me. Guess something wrong with transformer-xl

GPT-2 code below:

import torch
from torch.nn import DataParallel
from transformers import GPT2Tokenizer, GPT2LMHeadModel

device = "cuda:0"

# Get model
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

# Run forward
inputs = tokenizer(["This is an example"], return_tensors="pt")
outputs = model(
    input_ids=inputs["input_ids"].to(device),
    attention_mask=inputs["attention_mask"].to(device),
    labels=inputs["input_ids"].to(device),
)

print(f"outputs: {outputs}")
print("Success.")

[TevenLeScao]

Seems like this was overlooked in #4300 ! I'll update TransfoXL in the same way.

[TevenLeScao]

As of now, Pytorch doesn't support calling self.parameters() within DataParallel, which causes the current issue. Even after fixing that, which was straightforward, Pytorch also doesn't support calling self.ParameterList and self.ParameterDict, which are also used in TransfoXL, which will cause another issue. As Pytorch is moving people away from DataParallel, they are unlikely to fix this anytime soon on their end. On our end, this is going to be much harder to fix in a non-BC way, as changing the way the model is organized means previous checkpoints cannot be loaded. In the meantime, you could use DistributedDataParallel instead.

[yashbonde]

I used torch.nn.parallel.DistributedDataParallel to run the model in forward pass with the script below:

import os
import sys
import tempfile
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from transformers import TransfoXLTokenizer, TransfoXLLMHeadModel

def setup(rank, world_size):
     os.environ['MASTER_ADDR'] = 'localhost'
     os.environ['MASTER_PORT'] = '12355'

     # initialize the process group
     dist.init_process_group("gloo", rank=rank, world_size=world_size)

def demo_model_parallel(rank, world_size):
    print(f"Running DDP with model parallel example on rank {rank}.")
    setup(rank, world_size)

    # transfoXL model
    tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
    mp_model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103', return_dict=True)
    ddp_mp_model = DDP(mp_model, find_unused_parameters=True)
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_mp_model.parameters(), lr=0.001)
    
    for i in range(10):
        optimizer.zero_grad()
        # check to see if the model returns different losses
        if rank == 0:
            inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
        else:
            inputs = tokenizer("Borat and the republic of Kazakhistan!", return_tensors = "pt")
        outputs = ddp_mp_model(input_ids = inputs["input_ids"], labels=inputs["input_ids"], return_dict = True)
        _l = outputs.losses.mean() # documentation is incorrect there is no `loss` but `losses`
        print(_l)
        _l.backward()
        optimizer.step()

def run_demo(demo_fn, world_size):
    mp.spawn(demo_fn,
             args=(world_size,),
             nprocs=world_size,
             join=True)
    
    
if __name__ == "__main__":
    run_demo(demo_model_parallel, 2)

However during backward pass I get this error:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that
your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1)
passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making
sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the
distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward`
function. Please include the loss function and the structure of the return value of `forward` of your module when reporting
this issue (e.g. list, dict, iterable).

The code is modified from tutorial

[ismymajia]

@TevenLeScao
I have the same error when trainning TransformerXL.

File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
output = module(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 1056, in forward
return_dict=return_dict,
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 866, in forward
mems = self.init_mems(bsz)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 800, in init_mems
param = next(self.parameters())
StopIteration

how to solve it quikly?

[TevenLeScao]

@yashbonde this seems to be an unrelated issue! I'll take a look tomorrow.
@ismymajia see my message above - this is a Pytorch issue that we cannot fix without breaking backwards compatibility of checkpoints, as they're slowly stopping support for DataParallel.

[ismymajia]

Now the problem is that huggingface tansformer-xl model cannot be trained. huggingface tansformer-xl model will not be supported? Do you plan to update the tansformer-xl code?
@TevenLeScao

[TevenLeScao]

As I said in my previous post, you can just use single-GPU or distributed training instead. Of course transformer-xl is supported ; but we cannot update its code to bypass the Pytorch issues with DataParallel without breaking backwards compatibility with previous checkpoints.

[ismymajia]

when i train tansformer-xl as below:

python -m torch.distributed.launch --nproc_per_node 4 run_language_modeling.py --output_dir ${model_dir}
--tokenizer_name $data_dir/wordpiece-custom.json
--config_name $data_dir/$config_file
--train_data_files "$data_dir/train*.txt"
--eval_data_file $data_dir/valid.txt
--block_size=128
--do_train
--do_eval
--per_device_train_batch_size 4
--gradient_accumulation_steps 1
--learning_rate 6e-4
--weight_decay 0.01
--adam_epsilon 1e-6
--adam_beta1 0.9
--adam_beta2 0.98
--max_steps 500_000
--warmup_steps 24_000
--fp16
--logging_dir ${model_dir}/tensorboard
--save_steps 5000
--save_total_limit 20
--seed 108
--max_steps -1
--num_train_epochs 20
--dataloader_num_workers 0
--overwrite_output_dir

occur error:

[INFO|language_modeling.py:324] 2020-11-11 13:50:49,520 >> Loading features from cached file /opt/ml/input/data/training/mm/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train3.txt [took 93.739 s]
[INFO|language_modeling.py:324] 2020-11-11 13:52:30,959 >> Loading features from cached file /opt/ml/input/data/training/mm/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train2.txt [took 101.436 s]
Traceback (most recent call last):
File "run_language_modeling.py", line 350, in
main()
File "run_language_modeling.py", line 313, in main
trainer.train(model_path=model_path)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 657, in train
else True
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 333, in init
self.broadcast_bucket_size)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled cuda error, NCCL version 2.4.8
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 261, in
main()
File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 257, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-u', 'run_language_modeling.py', '--local_rank=3', '--output_dir', '/opt/ml/input/data/training/mm/huggingface/data/20201107/checkpoints/transfo-xl_1L_dembed1024_dhead64_dInner4096_dmodel1024_heads16_1', '--tokenizer_name', '/opt/ml/input/data/training/mm/huggingface/data/20201107/wordpiece-custom.json', '--config_name', '/opt/ml/input/data/training/mm/huggingface/data/20201107/config-transfo-xl.json', '--train_data_files', '/opt/ml/input/data/training/mm/huggingface/data/20201107/train*.txt', '--eval_data_file', '/opt/ml/input/data/training/mm/huggingface/data/20201107/valid.txt', '--block_size=128', '--do_train', '--do_eval', '--per_device_train_batch_size', '16', '--gradient_accumulation_steps', '1', '--learning_rate', '6e-4', '--weight_decay', '0.01', '--adam_epsilon', '1e-6', '--adam_beta1', '0.9', '--adam_beta2', '0.98', '--max_steps', '500_000', '--warmup_steps', '24_000', '--fp16', '--logging_dir', '/opt/ml/input/data/training/mm/huggingface/data/20201107/checkpoints/transfo-xl_1L_dembed1024_dhead64_dInner4096_dmodel1024_heads16_1/tensorboard', '--save_steps', '5000', '--save_total_limit', '20', '--seed', '108', '--max_steps', '-1', '--num_train_epochs', '20', '--overwrite_output_dir']' died with <Signals.SIGKILL: 9>.

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

my env is below:
pytorch:1.6+cu101
transformer 3.4
tokenizer 0.9.3

@TevenLeScao How to solve it ?

[TevenLeScao]

Hey, looking at the error message (the SIGKILL) this looks more like the machine killing the process than like a bug. What's your setup? This happens if the machine runs out of RAM for example.

[ismymajia]

I am trainning the transformer-xl on one machine with multi-gpus by ddp.
I don't know if this is a problem.
@TevenLeScao

[TevenLeScao]

Hey, usually when you get a mysterious CUDA error like this ("RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled cuda error, NCCL version 2.4.8") it's because of GPU memory. I'll close the issue now as this is unrelated, and does not particularly look like a library bug. You should probably post on the forums at https://discuss.huggingface.co/ to see if you can get help with debugging!

[ismymajia]

[INFO|language_modeling.py:242] 2020-11-11 11:54:46,363 >> Loading features from cached file /opt/ml/input/data/training/kyzhan/huggingface/data/train40G/cached_lm_PreTrainedTokenizerFast_126_train3.txt [took 116.431 s]
/ th_index_copy
main()
File "run_hf_train_lm_ti.py", line 338, in main
trainer.train(model_path=model_path)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 758, in train
tr_loss += self.training_step(model, inputs)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1056, in training_step
loss = self.compute_loss(model, inputs)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1082, in compute_loss
outputs = model(**inputs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 511, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 1056, in forward
return_dict=return_dict,
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 888, in forward
word_emb = self.word_emb(input_ids)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_transfo_xl.py", line 448, in forward
emb_flat.index_copy(0, indices_i, emb_i)
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #4 'source' in call to th_index_copy

Now encounter this problem. @TevenLeScao

[TevenLeScao]

I'm closing this issue as it concerns an unrelated problem that we cannot solve. Can you open a new one with a complete description ?