How to correctly run transformer?

[sfraczek]

Hi,

I have encountered a number of problems with fluid/neural_machine_translation/transformer model. Am I doing something wrong? How to correctly run it?

Steps I have taken

Following instructions in https://github.com/PaddlePaddle/models/blob/develop/fluid/neural_machine_translation/transformer/README_cn.md I have downloaded WMT'16 EN-DE from https://github.com/google/seq2seq/blob/master/docs/data.md by clicking download.

Next I extracted it to wmt16_en_de directory.

Next I did paste -d ' \ t ' train.tok.clean.bpe.32000.en train.tok.clean.bpe.32000.de > train.tok.clean.bpe.32000.en-de

Then I did sed -i '1i\<s>\n<e>\n<unk>' vocab.bpe.32000

in config.py I changed use_gpu = True to False.
In train.py I added import multiprocessing and changed dev_count = fluid.core.get_cuda_device_count() to dev_count = fluid.core.get_cuda_device_count() if TrainTaskConfig.use_gpu else multiprocessing.cpu_count().

Training

I launched training by python -u train.py --src_vocab_fpath wmt16_en_de/vocab.bpe.32000 --trg_vocab_fpath wmt16_en_de/vocab.bpe.32000 --special_token '<s>' '<e>' '<unk>' --train_file_pattern wmt16_en_de/train.tok.clean.bpe.32000.en-de --use_token_batch True --batch_size 3200 --sort_type pool --pool_size 200000

but I got

E0719 14:26:29.439303 55138 graph.cc:43] softmax_with_cross_entropy_grad input var not in all_var list: softmax_with_cross_entropy_0.tmp_0@GRAD
epoch: 0, consumed 0.000161s
Traceback (most recent call last):
  File "train.py", line 428, in <module>
    train(args)
  File "train.py", line 419, in train
    "pass_" + str(pass_id) + ".checkpoint"))
  File "/home/sfraczek/Paddle/build/python/paddle/fluid/io.py", line 288, in save_persistables
    filename=filename)
  File "/home/sfraczek/Paddle/build/python/paddle/fluid/io.py", line 166, in save_vars
    filename=filename)
  File "/home/sfraczek/Paddle/build/python/paddle/fluid/io.py", line 197, in save_vars
    executor.run(save_program)
  File "/home/sfraczek/Paddle/build/python/paddle/fluid/executor.py", line 449, in run
    self.executor.run(program.desc, scope, 0, True, True)
paddle.fluid.core.EnforceNotMet: holder_ should not be null
Tensor not initialized yet when Tensor::type() is called. at [/home/sfraczek/Paddle/paddle/fluid/framework/tensor.h:139]
PaddlePaddle Call Stacks:
0       0x7f060e948f1cp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
1       0x7f060e94b901p paddle::framework::Tensor::type() const + 209
2       0x7f060f617bf6p paddle::operators::SaveOp::SaveLodTensor(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_,
boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::va
riant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> con
st&, paddle::framework::Variable*) const + 614
3       0x7f060f618472p paddle::operators::SaveOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boos
t::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::varian
t::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::
detail::variant::void_> const&) const + 210

So I have commented out

#fluid.io.save_persistables(
#    exe,
#    os.path.join(TrainTaskConfig.ckpt_dir,
#                 "pass_" + str(pass_id) + ".checkpoint"))

and it worked.

Inference

So next I have tried to run inference.
I have found that the file wmt16_en_de/newstest2013.tok.bpe.32000.en-de doesn't exist but based on the README I guessed that I should run
paste -d ' \ t ' newstest2013.tok.bpe.32000.en newstest2013.tok.bpe.32000.de > newstest2013.tok.bpe.32000.en-de is this correct?

python -u infer.py --src_vocab_fpath wmt16_en_de/vocab.bpe.32000 --trg_vocab_fpath wmt16_en_de/vocab.bpe.32000 --special_token '<s>' '<e>' '<unk>' --test_file_pattern wmt16_en_de/newstest2013.tok.bpe.32000.en-de --batch_size 4 model_path trained_models/pass_20.infer.model beam_size 5 but there was no ouptut from the script. It ended without error too.

I tried giving other files but it doesn't output anything either.

I added profiling by adding import paddle.fluid.profiler as profiler and

+    parser.add_argument(
+        "--profile",
+        type=bool,
+        default=False,
+        help="Enables/disables profiling.")

and

+    if args.profile:
+        with profiler.profiler("CPU", sorted_key='total') as cpuprof:
+            infer(args)
+    else:
+        infer(args)

But there is no output from the profile.

Please help.

[sfraczek]

update on debugging

I found that there was a problem with paste command. It didn't insert the delimiter between the English and German sentences, so the files were not parsed correctly. Are you not experiencing this problem?

I have replaced the call to paste with a python script:

from itertools import izip

def concat_files(f1, f2, f3):
    with open(f1) as textfile1, open(f2) as textfile2, open(f3, "w") as output_file:
        for x, y in izip(textfile1, textfile2):
            x = x.strip()
            y = y.strip()
            output_file.write("{0} \t {1}\n".format(x, y))

def main():
    f1="newstest2013.tok.bpe.32000.en"
    f2="newstest2013.tok.bpe.32000.de"
    f3="newstest2013.tok.bpe.32000.en-de"
    concat_files(f1,f2,f3)

    f1="train.tok.clean.bpe.32000.en"
    f2="train.tok.clean.bpe.32000.de"
    f3="train.tok.clean.bpe.32000.en-de"
    concat_files(f1,f2,f3)

if __name__ == "__main__":
    main()

now I'm repeating the training and inference and I will keep you updated.

[kuke]

Are you sure you didn't miss \t when using paste cmd? It works fine in our linux systems.

[sfraczek]

I have run into

Traceback (most recent call last):
  File "train.py", line 428, in <module>
    train(args)
  File "train.py", line 401, in train
    feed=feed_list)
  File "/home/sfraczek/Paddle/build/python/paddle/fluid/parallel_executor.py", line 269, in run
    self.executor.run(fetch_list, fetch_var_name)
paddle.fluid.core.EnforceNotMet: enforce posix_memalign(&p, 4096ul, size) == 0 failed, 12 != 0
Alloc 5242880000 error! at [/home/sfraczek/Paddle/paddle/fluid/memory/detail/system_allocator.cc:52]
PaddlePaddle Call Stacks:
0       0x7f7950c082e9p paddle::memory::detail::CPUAllocator::Alloc(unsigned long*, unsigned long) + 5209
1       0x7f7950c044ecp paddle::memory::detail::BuddyAllocator::RefillPool() + 92
2       0x7f7950c04d68p paddle::memory::detail::BuddyAllocator::Alloc(unsigned long) + 760
3       0x7f7950b2a1f0p void* paddle::memory::Alloc<paddle::platform::CPUPlace>(paddle::platform::CPUPlace, unsigned long) + 192
4       0x7f7950b1d7fap paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::type_index) + 410
5       0x7f794fa7d571p float* paddle::framework::Tensor::mutable_data<float>(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>) + 97
6       0x7f7950592f3dp paddle::operators::LayerNormKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 781
7       0x7f79505932cfp std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::LayerNormKernel<paddle::platform::CPUDeviceContext, float>, paddle::operators::LayerNormKernel<paddle::platform::CPUDeviceContext, double> >::operator()(char const*, char const*) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 47
8       0x7f7950a6114bp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 747
9       0x7f7950a5ca0dp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 221
10      0x7f7950935246p
11      0x7f795096a4c1p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 65
12      0x7f7950934d63p paddle::framework::details::ComputationOpHandle::RunImpl() + 83
13      0x7f795096e1e9p paddle::framework::details::OpHandleBase::Run(bool) + 393
14      0x7f7950961208p
15      0x7f7950962210p
16      0x7f79507a152ep std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) + 46
17      0x7f79fea9ba99p
18      0x7f795096033dp
19      0x7f7950966304p std::thread::_Impl<std::_Bind_simple<ThreadPool::ThreadPool(unsigned long)::{lambda()#1} ()> >::_M_run() + 340
20      0x7f79f8582c80p
21      0x7f79fea946bap
22      0x7f79fe7ca41dp clone + 109

[sfraczek]

I need to update the transformer code to a new version and start again because there were a lot of changes.

[sfraczek]

I have been trying to run the transformer on the most recent develop branch but I keep running out of memory even with --batch_size 1 --pool_size 2 (I don't know what pool_size does exactly though). The process gets killed by the system displaying just Killed. I have roughly 180GB RAM. I don't see any iteration before it gets killed and it takes a lot of time before it gets killed. Also I didn't notice any obvious memory usage increasing steadily so it probably happens quickly.
Can you help me? Have you successfully ran training on CPU?

The command I have been using

python -u train.py   --src_vocab_fpath wmt16_en_de/vocab.bpe.32000   --trg_vocab_fpath wmt16_en_de/vocab.bpe.32000   --special_token '<s>' '<e>' '<unk>'   --train_file_pattern wmt16_en_de/train.tok.clean.bpe.32000.en-de   --use_token_batch True   --batch_size 1   --sort_type pool_size 2 --device CPU

I think it is caused by out of memory because of this command dmesg -T| grep -E -i -B100 'killed process' returning this:

[wto lip 24 14:58:27 2018] Out of memory: Kill process 128566 (python) score 878 or sacrifice child
[wto lip 24 14:58:27 2018] Killed process 128566 (python) total-vm:393885000kB, anon-rss:176935556kB, file-rss:70104kB, shmem-rss:4096kB

[sfraczek]

I tried running transformer with valgrind's tool named massif but the tool didn't work so I couldn't track the issue.
This is a blocker for us. Please fix the CPU path so we can work on optimizing it.
We must suspend the effort until it starts working.