Which part of the code should I modify when I have less than 8 GPUs ?

[yaoxingcheng]

After preprocessing, I got files train.{lang}.[0..7].pth. But when I try to pretrain a model with MLM, I got an error saying 'no file called train.{lang}.pth exists'.
I think that might because I have to regroup my dataset into a single file, but which part of the code should I modify or extra parameters should I put in to achieve that?

[baptisteroziere]

If you have only one GPU, the model will load the file named train.{lang}.pth at training time. If you have several, each GPU will load the file corresponding to its ID train.{lang}.{gpu_node_id}.pth. If you have less than 8 GPUs, your model will not be trained on the whole dataset but it should still work. If you have only one GPU, it fails because train.{lang}.pth doesn't exist.

If you're only running it on one GPU to test your pipeline and you don't care about loading only 1/8 of the data, you can just create a symlink from train.{lang}.pth to train.{lang}.0.pth for every lang. Otherwise you need to regroup every train.{lang}.[0..7].pth file into one called train.{lang}.pth. The part of the code to modify is preprocessing/preprocess.py but you may find it easier (and faster) to just load every .pth file and concatenate the 'positions', 'sentences' and 'unk_words' to create your new .pth file.

[orazheng]

After preprocessing, I got files train.{lang}.[0..7].pth. But when I try to pretrain a model with MLM, I got an error saying 'no file called train.{lang}.pth exists'.
I think that might because I have to regroup my dataset into a single file, but which part of the code should I modify or extra parameters should I put in to achieve that?

Hi @yaoxingcheng, I saw you successfully passed the preprocessing part and got files. May I ask how you run the preprocessing code? I got the error FileNotFoundError: [Errno 2] No such file or directory: 'data/test_dataset/cpp/all.with_comments.tok'
Thank you very much for your help!