README.md: make the heterogeneous support more clear

Remove ambiguous warning language (it's not clear if the "THIS
FUNCTIONALITY..." warning applies to the option above or below the
warning) and make it clear exactly what the heterogeneous option
supports and does not support.

This is a port of 6674ba8 from master; it's not a direct
cherry pick because master has README.md and the v4.0.x branch has
README.

Signed-off-by: Jeff Squyres <[email protected]>

Author: jsquyres

README

@@ -1524,13 +1524,29 @@ MISCELLANEOUS FUNCTIONALITY
   Enable the PERUSE MPI data analysis interface.
 
 --enable-heterogeneous
-  Enable support for running on heterogeneous clusters (e.g., machines
-  with different endian representations).  Heterogeneous support is
-  disabled by default because it imposes a minor performance penalty.
-
-  *** THE HETEROGENEOUS FUNCTIONALITY IS CURRENTLY BROKEN - DO NOT USE ***
-
- --enable-spc
+  Enable support for running on heterogeneous clusters where data
+  types are equivalent sizes across nodes, but may have differing
+  endian representations.  Heterogeneous support is disabled by
+  default because it imposes a minor performance penalty.
+
+  Note that the MPI standard does not guarantee that all
+  heterogeneous communication will function properly, especially
+  when the conversion between the different representations leads to
+  loss of accuracy or range.  For example, if a message with a
+  16-bit integer datatype is sent with value 0x10000 to a receiver
+  where the same integer datatype is only 8 bits, the value will be
+  truncated at the receiver.  Similarly, problems can occur if a
+  floating point datatype in one MPI process uses X1 bits for its
+  mantissa and Y1 bits for its exponent, but the same floating point
+  datatype in another MPI process uses X2 and Y2 bits, respectively
+  (where X1 != X2 and/or Y1 != Y2).  Type size differences like this
+  can lead to unexpected behavior.
+
+  Open MPI's heterogeneous support correctly handles endian
+  differences between datatype representations that are otherwise
+  compatible.
+
+--enable-spc
   Enable software-based performance counters capability.
 
 --with-wrapper-cflags=<cflags>