Hanging with more than 64 hosts

[ceandrade]

Problem

I can not run jobs in parallel in a cluster with more than 64 machines.

---

System

Using openmpi-3.0.0.tar.bz2, Sep 12, 2017, compiled from source/distribution tarball.

Linux version 2.6.32-573.3.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Aug 13 22:55:16 UTC 2015

---

Details of the problem

I'm trying to use OpenMPI in a cluster with 200 machines. Unfortunately, mpirun looks like be frozen when I submit the jobs:

$ mpirun -np 1 -hostfile hosts.txt -pernode hostname

where hosts.txt has 200 machines.

If the number of machines is at most 64, I can get the answer pretty fast:

$ mpirun -np 64 -hostfile hosts_64.txt -pernode hostname
machine1
machine2
....
machine64

This will be blocked forever:

$ mpirun -np 65 -hostfile hosts_65.txt -pernode hostname
<no response for 10min...>

[rhc54]

If you add "-mca routed_radix 300" to your command line, does it make a difference?

[ceandrade]

It got better but didn't scale to more than 128 nodes, even "-mca routed_radix 600". I also tried "routed_debruijn" and "routed_binomial", but no success.

[ggouaillardet]

@ceandrade can you please confirm the node on which you run mpirun is not part of your hosts_64.txt hostfile ?

[ceandrade]

@ggouaillardet I tried both actually. I launched from a machine outside the cluster, and from machines within the cluster. This is the line:

$ mpirun -v -display-devel-map -display-allocation -mca routed_radix 600 --bind-to none -np 1 -hostfile hosts_266.txt -pernode hostname

but no answer. Note that I have 266 identical machines working in my cluster now. Is there a way to create some more detailed logging info that I can share with you guys?

[ggouaillardet]

I can reproduce the issue with 65 tasks and with the default radix, but only when mpirun is invoked from a node not in the hostfile. I will resume investigations tomorrow. Stay tuned !

[rhc54]

Another thing to try: add "-mca plm_rsh_num_concurrent 300" to your cmd line and see if that helps.

[rhc54]

Alternatively, you can add "-mca plm_rsh_no_tree_spawn 1" to the cmd line - affect in your case will be the same.

[ceandrade]

@rhc54 It worked with the options "-mca plm_rsh_num_concurrent 300 -mca routed_radix 600" and the option "-mca plm_rsh_no_tree_spawn 1" alone. The latter is much faster than the former options.

I don't understand very well MPI (I just use it as another tool), but apparently, the tree-based launch has some problem with more than 128 nodes.

So, this solution looks be enough to me. Therefore, I think it is up to you guys to leave this ticket open for further investigation.

Thank @rhc54 and @ggouaillardet for your help and prompt answers!

[rhc54]

ok, thanks for the report, and sorry for the problem. Since @ggouaillardet can reproduce it, I will defer to him for the eventual solution now that we have isolated the problem to the tree spawn code.

[ggouaillardet]

@ceandrade this issue will be fixed in upcoming 3.0.1.
Meanwhile, you can manually download and apply the patch available at https://github.com/ggouaillardet/ompi/commit/e445d47c3f8ff908d19eb18cbd6d953e1272f7ca.patch

Thanks for the report !

[ceandrade]

Thank you very much @ggouaillardet. I am amazed how fast you guys are!