DOCS-538 new step to verify connectivity before deploy rs

Author: Bob Grabar

source/administration/replica-sets.txt

@@ -656,6 +656,49 @@ Possible causes of replication lag include:
   - The :ref:`replica-set-procedure-change-oplog-size` topic this document.
   - The :doc:`/tutorial/change-oplog-size` tutorial.
 
+.. _replica-set-troubleshooting-check-connection:
+
+Test the Connection Between Each Member
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There must be connectivity from every :term:`replica set` member to
+every other member in order for replication to work. Problems with
+network or firewall rules can prevent this connectivity and prevent
+replication from working. To test the connection from every member to
+every other member, in both directions, consider the following example:
+
+.. example:: Given a replica set with three members running on three separate
+   hosts:
+
+   - ``m1.example.net``
+   - ``m2.example.net``
+   - ``m3.example.net``
+
+   1. Test the connection from ``m1.example.net`` to the other hosts by running
+      the following operations from ``m1.example.net``:
+
+      .. code-block:: sh
+
+         mongo --host m2.example.net --port 27017"
+
+         mongo --host m3.example.net --port 27017"
+
+   #. Test the connection from ``m2.example.net`` to the other two
+      hosts by running similar appropriate operations from ``m2.example.net``.
+
+      This means you have now tested the connection between
+      ``m2.example.net`` and ``m1.example.net`` twice, but each time
+      from a different direction. This is important to verifying
+      connectivity. Network topologies and firewalls might allow a
+      connection in one direction but not the other. Therefore you must
+      make sure to verify that the connection works in both directions.
+
+   #. Test the connection from ``m3.example.net`` to the other two
+      hosts by running the operations from ``m3.example.net``.
+
+   If a connection in any direction fails, there's a networking or
+   firewall issue that needs to be diagnosed separately.
+
 .. index:: pair: replica set; failover
 .. _replica-set-failover-administration:
 .. _failover:

source/tutorial/deploy-replica-set.txt

@@ -49,7 +49,11 @@ Deploy a Development or Testing Replica Set
 
 The examples in this procedure create a new replica set named ``rs0``.
 
-1. Start three instances of :program:`mongod` as members of a replica
+1. Before creating your replica set, verify there is full connectivity
+   between all the hosts to be in the replica set, as described in
+   :ref:`replica-set-troubleshooting-check-connection`.
+
+#. Start three instances of :program:`mongod` as members of a replica
    set named ``rs0``, as described in this step. For ephemeral tests and
    the purposes of this guide, you may run the :program:`mongod`
    instances in separate windows of GNU Screen. OS X and most Linux
@@ -186,7 +190,11 @@ deployment described above, with the following differences:
 
 To deploy a production replica set:
 
-1. On each system start the :program:`mongod` process by issuing a
+1. Before creating your replica set, verify there is full connectivity
+   between all the hosts to be in the replica set, as described in
+   :ref:`replica-set-troubleshooting-check-connection`.
+
+#. On each system start the :program:`mongod` process by issuing a
    command similar to following:
 
    .. code-block:: sh