--report-bindings on whole machine, plus numa markers

I combined a few related features here:
1. --report-bindings on whole machine
2. mark unallowed (eg cgroup) parts of the whole machine with ~
3. numa markers
4. allow hwloc tree to not have sockets/cores

1. whole machine:

The incoming topology to the pretty-print functions probably didn't use
the WHOLE_SYSTEM flag. In general we don't want WHOLE_SYSTEM, but for
pretty printing I think it makes more sense. So I'm caching a WHOLE_SYSTEM
load of the current machine and using it if the incoming topology also
identifies as the current machine.

Examples of what pretty-printing looks like with/without whole system:

Suppose the machine is
    [..../..../..../....][..../..../..../....]
     0    4    8    12    16   20   24   28
and we run with
    cgset -r cpuset.cpus=24,25,28,29 mycgroup1
    cgset -r cpuset.cpus=26,27,30,31 mycgroup2
to leave only these hardware threads active:
    mycgroup1: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/..~~/..~~]
    mycgroup2: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~../~~..]

Without whole-system the printout (for both of the above) would be
(-np 2)
    MCW rank 0 bound to socket 1[core 0[hwt 0-1]]: [][BB/..]
    MCW rank 1 bound to socket 1[core 1[hwt 0-1]]: [][../BB]

With whole-system the output is this, which I think is more informative
mycgroup1 (-np 2):
    MCW rank 0 bound to socket 1[core 6[hwt 0-1]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/BB~~/..~~]
    MCW rank 1 bound to socket 1[core 7[hwt 0-1]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/..~~/BB~~]
mycgroup2 (-np 2):
    MCW rank 0 bound to socket 1[core 6[hwt 2-3]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~BB/~~..]
    MCW rank 1 bound to socket 1[core 7[hwt 2-3]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~../~~BB]

2. mark unallowed (~)

When using the whole-machine option there's a bitmask available to identify
the allowed PUs, eg omitting PUs not in our cgroup.  To distinguish those
PUs I'm using "~"

3. numa markers (<>)

I like having numa markers as well as the existing separators between
sockets and cores.  They're a little harder since the numas are more fluid,
eg sockets always contain cores not vice versa, so you can hard code a loop
over sockets follwed by a loop over cores. But numas might be be above or
below sockets in the tree.

This code identifies which level should be considered the child of the
numas, and has each of the hard coded loops capable of adding numa markers.

Currently I don't have any tunable to turn off the numa markers. A lot of
machines have fairly simple numa output where each socket contains one numa,
and that ends up looking like this:
    [<..../..../..../....>][<..../..../..../....>]
If others feel that's too cluttered I'm okay with having some tunable so
people have to ask for numa markers.

4. allow hwloc tree to not have sockets/cores

I may be behind the times on hwloc development, but as far as I know
hwloc trees aren't guaranteed to have sockets and cores, just a MACHINE
at the top and PU at the bottom. So I added a little code to the loops
so it would still print the PUs on a hypothetical machine that lacked any
structuring of the PUs into cores/sockets.

Signed-off-by: Mark Allen <[email protected]>

Author: markalle

opal/mca/hwloc/base/hwloc_base_util.c

@@ -1704,6 +1704,8 @@ static int build_map(int *num_sockets_arg, int *num_cores_arg,
     return OPAL_SUCCESS;
 }
 
+static hwloc_topology_t whole_system = NULL;
+
 /*
  * Make a prettyprint string for a hwloc_cpuset_t
  */
@@ -1720,6 +1722,15 @@ int opal_hwloc_base_cset2str(char *str, int len,
     hwloc_obj_t root;
     opal_hwloc_topo_data_t *sum;
 
+    if (!whole_system) {
+        hwloc_topology_init(&whole_system);
+        hwloc_topology_set_flags(whole_system, HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM);
+        hwloc_topology_load(whole_system);
+    }
+    if (hwloc_topology_is_thissystem(topo) && whole_system) {
+        topo = whole_system;
+    }
+
     str[0] = tmp[stmp] = '\0';
 
     /* if the cpuset is all zero, then not bound */
@@ -1769,13 +1780,107 @@ int opal_hwloc_base_cset2str(char *str, int len,
     return OPAL_SUCCESS;
 }
 
+// given an input obj somewhere in the hwloc tree, look for a
+// numa object that contains it
+static hwloc_obj_t
+find_my_numa(hwloc_obj_t obj)
+{
+    hwloc_obj_t p, numa;
+    int i;
+
+    p = obj;
+    while (p && p->memory_arity == 0) {
+        p = p->parent;
+    }   
+    // p should have either found a level that contains numas or reached NULL
+    if (p == NULL) { return NULL; }
+    for (i=0; i<p->memory_arity; ++i) {
+        numa = &(p->memory_first_child[i]);
+
+        if (hwloc_bitmap_isincluded(obj->cpuset, numa->cpuset)) {
+            return numa;
+        }   
+    }
+    return NULL;
+}
+
+// I added numa-markers to the --report_bindings output but I'm
+// considering the possibility it might look cluttered for the
+// trivial cases.  Eg
+//    [<../../../..>][<../../../..>]
+// So I'm adding a check we could enable for whether the numa
+// markers are interesting enough to print.
+//
+// Using multiple numa nodes under a direct parent is a possible
+// criteria.  For now I'm always returning 1.
+static int
+is_numa_output_worth_printing(hwloc_topology_t topo)
+{
+    hwloc_obj_t p;
+
+    // keeping all the numa output for now:
+    return 1;
+
+    p = hwloc_get_obj_by_type(topo, HWLOC_OBJ_PU, 0);
+    while (p && p->memory_arity == 0) {
+        p = p->parent;
+    }
+    if (p && p->memory_arity > 1) {
+        return 1;
+    }
+
+    return 0;
+}
+
+// which level from the set {socket, core, pu} has
+// the first descendent underneath the lowest numa level.
+// returns MACHINE if there is no numa level
+//
+// Eg if an hwloc tree had numas containing sockets like this
+//   <[../..][../..]><[../..][../..]>
+// the tree would be
+//   mach               +memory_children: n n
+//   s   s   s   s
+//   c c c c c c c c
+//   pppppppppppppppp
+// so this should return SOCKET
+static hwloc_obj_type_t
+first_type_under_a_numa(hwloc_topology_t topo)
+{
+    hwloc_obj_t p;
+    hwloc_obj_type_t type;
+
+    p = hwloc_get_obj_by_type(topo, HWLOC_OBJ_PU, 0);
+    while (p && p->memory_arity == 0) {
+        if (p->type == HWLOC_OBJ_PU ||
+            p->type == HWLOC_OBJ_CORE ||
+            p->type == HWLOC_OBJ_SOCKET)
+        {
+            type = p->type;
+        }
+        p = p->parent;
+    }
+    if (p && p->memory_arity > 0) {
+        return type;
+    }
+
+    return HWLOC_OBJ_MACHINE;
+}
+
 /*
  * Make a prettyprint string for a cset in a map format.
  * Example: [B./..]
  * Key:  [] - signifies socket
+ *       <> - signifies numa
  *        / - divider between cores
  *        . - signifies PU a process not bound to
  *        B - signifies PU a process is bound to
+ *        ~ - signifies PU that is disallowed, eg not in our cgroup:
+ *
+ * The incoming topo is expected to be from a topology_load without
+ * WHOLE_SYSTEM. For the purpose of printing though we want a whole-system
+ * version. So we'll cache one here for the current system and use that
+ * if the incoming topo is also the current system.
  */
 int opal_hwloc_base_cset2mapstr(char *str, int len,
                                 hwloc_topology_t topo,
@@ -1787,6 +1892,27 @@ int opal_hwloc_base_cset2mapstr(char *str, int len,
     hwloc_obj_t socket, core, pu;
     hwloc_obj_t root;
     opal_hwloc_topo_data_t *sum;
+    hwloc_cpuset_t allowed;
+    hwloc_obj_t prev_numa = NULL;
+    hwloc_obj_t cur_numa = NULL;
+    hwloc_obj_type_t type_under_numa;
+    int a_numa_marker_is_open = 0;
+    int print_numa_markers = 1;
+
+    if (!whole_system) {
+        hwloc_topology_init(&whole_system);
+        hwloc_topology_set_flags(whole_system, HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM);
+        hwloc_topology_load(whole_system);
+    }
+    if (hwloc_topology_is_thissystem(topo) && whole_system) {
+        topo = whole_system;
+    }
+
+    allowed = hwloc_topology_get_allowed_cpuset(topo);
+    print_numa_markers = is_numa_output_worth_printing(topo);
+    if (print_numa_markers) {
+        type_under_numa = first_type_under_a_numa(topo);
+    }
 
     str[0] = tmp[stmp] = '\0';
 
@@ -1807,44 +1933,137 @@ int opal_hwloc_base_cset2mapstr(char *str, int len,
         }
     }
 
+// As far as I know hwloc trees aren't required to have sockets and cores,
+// just a MACHINE at the top and PU at the bottom. The 'fake_*' vars make
+// the loops always iterate at least once, even if the initial socket = ...
+// etc lookup is NULL.
+
+    int fake_on_first_socket;
+    int fake_on_first_core;
+    hwloc_cpuset_t cpuset_for_socket; // can be fake, eg the machine's
+                                      // cpuset if there are no sockets
+    hwloc_cpuset_t cpuset_for_core;
+
     /* Iterate over all existing sockets */
+    fake_on_first_socket = 1;
     for (socket = hwloc_get_obj_by_type(topo, HWLOC_OBJ_SOCKET, 0);
-         NULL != socket;
+         NULL != socket || fake_on_first_socket;
          socket = socket->next_cousin) {
-        strncat(str, "[", len - strlen(str) - 1);
+        fake_on_first_socket = 0;
+
+// if numas contain sockets, example output <[../..][../..]><[../..][../..]>
+        if (print_numa_markers && type_under_numa == HWLOC_OBJ_SOCKET) {
+            prev_numa = cur_numa;
+            cur_numa = find_my_numa(socket);
+            if (cur_numa && cur_numa != prev_numa) {
+                if (a_numa_marker_is_open) {
+                    strncat(str, ">", len - strlen(str) - 1);
+                }
+                strncat(str, "<", len - strlen(str) - 1);
+                a_numa_marker_is_open = 1;
+            }
+        }
+
+        if (socket != NULL) { strncat(str, "[", len - strlen(str) - 1); }
+
+        if (socket != NULL) {
+            cpuset_for_socket = socket->cpuset;
+        } else {
+            cpuset_for_socket = root->cpuset;
+        }
 
         /* Iterate over all existing cores in this socket */
+        fake_on_first_core = 1;
         core_index = 0;
         for (core = hwloc_get_obj_inside_cpuset_by_type(topo,
-                                                        socket->cpuset,
+                                                        cpuset_for_socket,
                                                         HWLOC_OBJ_CORE, core_index);
-             NULL != core;
+             NULL != core || fake_on_first_core;
              core = hwloc_get_obj_inside_cpuset_by_type(topo,
-                                                        socket->cpuset,
+                                                        cpuset_for_socket,
                                                         HWLOC_OBJ_CORE, ++core_index)) {
+            fake_on_first_core = 0;
+
+// if numas contain cores and are contained by sockets,
+// example output [<../..><../..>][<../../../..>]
+            if (print_numa_markers && type_under_numa == HWLOC_OBJ_CORE) {
+                prev_numa = cur_numa;
+                cur_numa = find_my_numa(core);
+                if (cur_numa && cur_numa != prev_numa) {
+                    if (a_numa_marker_is_open) {
+                        strncat(str, ">", len - strlen(str) - 1);
+                    }
+                    strncat(str, "<", len - strlen(str) - 1);
+                    a_numa_marker_is_open = 1;
+                }
+            }
+
+
             if (core_index > 0) {
                 strncat(str, "/", len - strlen(str) - 1);
             }
 
+            if (core != NULL) {
+                cpuset_for_core = core->cpuset;
+            } else {
+                cpuset_for_core = cpuset_for_socket;
+            }
+
             /* Iterate over all existing PUs in this core */
             pu_index = 0;
             for (pu = hwloc_get_obj_inside_cpuset_by_type(topo,
-                                                          core->cpuset,
+                                                          cpuset_for_core,
                                                           HWLOC_OBJ_PU, pu_index);
                  NULL != pu;
                  pu = hwloc_get_obj_inside_cpuset_by_type(topo,
-                                                          core->cpuset,
+                                                          cpuset_for_core,
                                                           HWLOC_OBJ_PU, ++pu_index)) {
 
+// if numas contain PU and are contained by cores (seems unlikely)
+// example output [<..../....>/<..../....>/<..../....>/<..../....>]
+                if (print_numa_markers && type_under_numa == HWLOC_OBJ_PU) {
+                    prev_numa = cur_numa;
+                    cur_numa = find_my_numa(pu);
+                    if (cur_numa && cur_numa != prev_numa) {
+                        if (a_numa_marker_is_open) {
+                            strncat(str, ">", len - strlen(str) - 1);
+                        }
+                        strncat(str, "<", len - strlen(str) - 1);
+                        a_numa_marker_is_open = 1;
+                    }
+                }
+
                 /* Is this PU in the cpuset? */
                 if (hwloc_bitmap_isset(cpuset, pu->os_index)) {
                     strncat(str, "B", len - strlen(str) - 1);
                 } else {
-                    strncat(str, ".", len - strlen(str) - 1);
+                    if (hwloc_bitmap_isset(allowed, pu->os_index)) {
+                        strncat(str, ".", len - strlen(str) - 1);
+                    } else {
+                        strncat(str, "~", len - strlen(str) - 1);
+                    }
+                }
+            }
+            if (print_numa_markers && type_under_numa == HWLOC_OBJ_PU) {
+                if (a_numa_marker_is_open) {
+                    strncat(str, ">", len - strlen(str) - 1);
+                    a_numa_marker_is_open = 0;
                 }
             }
         }
-        strncat(str, "]", len - strlen(str) - 1);
+        if (print_numa_markers && type_under_numa == HWLOC_OBJ_CORE) {
+            if (a_numa_marker_is_open) {
+                strncat(str, ">", len - strlen(str) - 1);
+                a_numa_marker_is_open = 0;
+            }
+        }
+        if (socket != NULL) { strncat(str, "]", len - strlen(str) - 1); }
+    }
+    if (print_numa_markers && type_under_numa == HWLOC_OBJ_SOCKET) {
+        if (a_numa_marker_is_open) {
+            strncat(str, ">", len - strlen(str) - 1);
+            a_numa_marker_is_open = 0;
+        }
     }
 
     return OPAL_SUCCESS;