diff --git a/source/administration/sharding.txt b/source/administration/sharding.txt index bf30cb187c6..172213ce909 100644 --- a/source/administration/sharding.txt +++ b/source/administration/sharding.txt @@ -7,85 +7,81 @@ Shard Cluster Administration .. default-domain:: mongodb -This document provides a collection of basic operations and procedures -for administering :term:`shard clusters `. - +This document describes common administrative tasks. For a full introduction to sharding in MongoDB see -":doc:`/core/sharding`," and for a complete overview of all sharding -documentation in the MongoDB Manual, see ":doc:`/sharding`." The -":doc:`/administration/sharding-architectures`" document provides an +:doc:`/core/sharding`, and for a complete overview of all sharding +documentation in the MongoDB Manual, see :doc:`/sharding`. The +:doc:`/administration/sharding-architectures` document provides an overview of deployment possibilities to help deploy a shard -cluster. Finally, the ":doc:`/core/sharding-internals`" document +cluster. Finally, the :doc:`/core/sharding-internals` document provides a more detailed introduction to sharding when troubleshooting issues or understanding your cluster's behavior. + .. contents:: Sharding Procedures: :backlinks: none :local: .. _sharding-procedure-setup: -Setup ------ +Set up a Shard Cluster +---------------------- -If you have an existing replica set, you can use the -":doc:`/tutorial/convert-replica-set-to-replicated-shard-cluster`" -tutorial as a guide. If you're deploying a :term:`shard cluster` from -scratch, see the ":doc:`/tutorial/deploy-shard-cluster`" tutorial for -more detail or use the following procedure as a quick starting point: +Before deploying a shard cluster, see the requirements listed in +:ref:`Requirements for Shard Clusters `. -#. Provision the required hardware. +For testing purposes, you can run all the required shard processes on a +single server. For production, use the configurations described in +:doc:`/administration/replication-architectures`. - The ":ref:`sharding-requirements`" section describes what you'll - need to get started. +.. include:: /includes/warning-sharding-hostnames.rst - .. warning:: Sharding and "localhost" Addresses +.. TODO Keep this or delete it: + If you have an existing replica set, you can use the + :doc:`/tutorial/convert-replica-set-to-replicated-shard-cluster` + tutorial as a guide. If you're deploying a :term:`shard cluster` from + scratch, see the :doc:`/tutorial/deploy-shard-cluster` tutorial for + more detail or use the following procedure as a quick starting point: - If you use either "localhost" or "``127.0.0.1``" as the host - identifier, then you must use "localhost" or "``127.0.0.1``" for - *all* host settings for any MongoDB instances in the cluster. If - you mix localhost addresses with remote host address, MongoDB will - produce errors. +1. Create data directories for each of the three (3) config server instances. -#. On all three (3) config server instances, issue the following - command to start the :program:`mongod` process: +#. Start the three config server instances. For example, to start a + config server instance running on TCP port ``27018`` with the data + stored in ``/data/configdb``, type the following: .. code-block:: sh - mongod --configsvr + mongod --configsvr --dbpath /data/configdb --port 27018 - This starts a :program:`mongod` instance running on TCP port - ``27018``, with the data stored in the ``/data/configdb`` path. All other - :doc:`command line ` and :doc:`configuration - file ` options are available for config - server instances. + For additional command options, see :doc:`/reference/mongod` + and :doc:`/reference/configuration-options`. .. include:: /includes/note-config-server-startup.rst -#. Start a :program:`mongos` instance issuing the following command: - - .. code-block:: sh +#. Start the three :program:`mongos` instances. For example, to start a + :program:`mongos` instance running on the following hosts: - mongos --configdb config0.mongodb.example.net,config1.mongodb.example.net,config2.mongodb.example.net --port 27017 + - ``mongos0.example.net`` + - ``mongos1.example.net`` + - ``mongos2.example.net`` -#. Connect to the :program:`mongos` instance using the :program:`mongo` - shell. + You would issue the following command: .. code-block:: sh - mongo mongos.mongodb.example.net - -#. Add shards to the cluster. + mongos --configdb mongos0.example.net,mongos1.example.net,mongos2.example.net - Run the following commands while connected to a :program:`mongos` to - initialize the cluster. +#. Connect to one of the :program:`mongos` instances. For example, if + a :program:`mongos` is accessible at ``mongos0.example.net`` on + port ``27017``, issue the following command: - First, you need to tell the cluster where to find the individual - shards. You can do this using the :dbcommand:`addShard` command. + .. code-block:: sh - .. code-block:: javascript + mongo mongos0.example.net - sh.addShard( "[hostname]:[port]" ) +#. Add shards to the cluster. From the :program:`mongo` shell connected + to the :program:`mongos` instance, call the :method:`sh.addShard()` + method for each shard to add to the cluster. For example: @@ -93,8 +89,8 @@ more detail or use the following procedure as a quick starting point: sh.addShard( "mongodb0.example.net:27027" ) - MongoDB will discover all other members of the replica set, if - ``mongodb0.example.net:27027`` is a member of a replica set. + If ``mongodb0.example.net:27027`` is a member of a replica + set, MongoDB will discover all other members of the replica set. .. note:: In production deployments, all shards should be replica sets. @@ -102,21 +98,8 @@ more detail or use the following procedure as a quick starting point: .. optional:: - You may specify a "name" as an argument to the - :dbcommand:`addShard`, as follows: - - .. code-block:: javascript - - db.runCommand( { addShard: mongodb0.example.net, name: "mongodb0" } ) - - Or: - - .. code-block:: javascript - - sh.addShard( mongodb0.example.net, name: "mongodb0" ) - - If you do not specify a shard name, then MongoDB will assign a - name upon creation. + You can specify a name for the shard and a maximum size. See + :dbcommand:`addShard`. .. note:: @@ -129,32 +112,33 @@ more detail or use the following procedure as a quick starting point: replicaSetName/,, - For example, if the name of the replica set is "``repl0``", then - your :method:`sh.addShard` command would be: + For example, if the name of the replica set is ``repl0``, then + your :method:`sh.addShard()` command would be: .. code-block:: javascript sh.addShard( "repl0/mongodb0.example.net:27027,mongodb1.example.net:27017,mongodb2.example.net:27017" ) -#. Enable sharding for any database that you want to shard. +#. Enable sharding for each database you want to shard. + While sharding operates on a per-collection basis, you must enable + sharding for each database that holds collections you want to shard. + This step is a meta-data change and will not redistribute your data. - MongoDB enables sharding on a per-database basis. This is only a - meta-data change and will not redistribute your data. To enable - sharding for a given database, use the :dbcommand:`enableSharding` - command or the :method:`sh.enableSharding()` shell function. + To enable sharding for a given database, use the + :dbcommand:`enableSharding` command or the + :method:`sh.enableSharding()` shell function, as shown below. Replace + ```` with the name of the database on which to enable + sharding. .. code-block:: javascript - db.runCommand( { enableSharding: [database] } ) + db.runCommand( { enableSharding: } ) Or: .. code-block:: javascript - sh.enableSharding([database]) - - Replace ``[database]`` with the name of the database you wish to - enable sharding on. + sh.enableSharding() .. note:: @@ -166,7 +150,7 @@ more detail or use the following procedure as a quick starting point: #. Enable sharding on a per-collection basis. - Finally, you must explicitly specify collections to shard. The + Finally, you must explicitly specify collections to shard. The collections must belong to a database for which you have enabled sharding. When you shard a collection, you also choose the shard key. To shard a collection, run the :dbcommand:`shardCollection` @@ -174,13 +158,13 @@ more detail or use the following procedure as a quick starting point: .. code-block:: javascript - db.runCommand( { shardCollection: "[database].[collection]", key: "[shard-key]" } ) + db.runCommand( { shardCollection: ".", key: "" } ) Or: .. code-block:: javascript - sh.shardCollection("[database].[collection]", "key") + sh.shardCollection(".", "key") For example: @@ -199,9 +183,9 @@ more detail or use the following procedure as a quick starting point: the distribution of data. Furthermore, you cannot change a collection's shard key once it has been set. - See the ":ref:`Shard Key Overview `" and the - more in depth documentation of ":ref:`Shard Key Qualities - `" to help you select better shard + See the :ref:`Shard Key Overview ` and the + more in depth documentation of :ref:`Shard Key Qualities + ` to help you select better shard keys. If you do not specify a shard key, MongoDB will shard the @@ -215,6 +199,17 @@ This section describes common maintenance procedure, including: how to add and remove nodes, how to manually split chunks, and how to disable the balancer for backups. +List the Set of Configured Shards +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To list the current set of configured shards and verify that all shards +have been committed to the system, run the :dbcommand:`listShards` +command: + +.. code-block:: javascript + + db.runCommand( { listshards : 1 } ) + .. _sharding-procedure-add-shard: Adding a Shard to a Cluster @@ -232,9 +227,9 @@ procedure: .. code-block:: javascript - sh.addShard( "[hostname]:[port]" ) + sh.addShard( ":" ) - Replace ``[hostname]`` and ``[port]`` with the hostname and TCP + Replace ```` and ```` with the hostname and TCP port number of where the shard is accessible. For example: @@ -271,7 +266,7 @@ procedure: following form: the replica set name, followed by a forward slash, followed by a comma-separated list of seeds for the replica set. For example, if the name of the replica set is - "myapp1", then your :method:`sh.addShard` command might resemble: + "myapp1", then your :method:`sh.addShard()` command might resemble: .. code-block:: javascript @@ -282,9 +277,9 @@ procedure: It may take some time for :term:`chunks ` to migrate to the new shard. - See the ":ref:`Balancing and Distribution `" - section for an overview of the balancing operation and the ":ref:`Balancing Internals - `" section for additional information. + See the :ref:`Balancing and Distribution ` + section for an overview of the balancing operation and the :ref:`Balancing Internals + ` section for additional information. .. _sharding-procedure-remove-shard: @@ -695,7 +690,7 @@ Balancer Operations This section provides an overview of common administrative procedures related to balancing and the balancing process. -.. seealso:: ":ref:`sharding-balancing`" and the +.. seealso:: :ref:`sharding-balancing` and the :dbcommand:`moveChunk` that provides manual :term:`chunk` migrations. @@ -947,7 +942,7 @@ name. #. Shut down the config server that you're moving. This will render all config data for your cluster :ref:`read only - `." + `. #. Change the DNS entry that points to the system that provided the old config server, so that the *same* hostname points to the new @@ -1072,7 +1067,7 @@ of the cluster metadata from the config database is straight forward: .. [#read-only] While one of the three config servers unavailable, no the cluster cannot split any chunks nor can it migrate chunks between shards. Your application will be able to write data to the - cluster. The ":ref:`sharding-config-server`" section of the + cluster. The :ref:`sharding-config-server` section of the documentation provides more information on this topic. .. index:: troubleshooting; sharding diff --git a/source/core/sharding-internals.txt b/source/core/sharding-internals.txt index a884bde8140..b528ba5dbc4 100644 --- a/source/core/sharding-internals.txt +++ b/source/core/sharding-internals.txt @@ -495,3 +495,34 @@ All chunk migrations use the following procedure: When the ``_secondaryThrottle`` is ``true`` for :dbcommand:`moveChunk` or the :term:`balancer`, MongoDB ensure that *one* :term:`secondary` member has replicated changes before allowing new chunk migrations. + +Config Database +--------------- + +The ``config`` database contains sharding configuration information. To +start the ``config`` database from the :program:`mongo` shell, issue the +following command: + +.. code-block:: javascript + + use config + +The config database includes the following collections used by sharding: + +.. data:: chunks + +.. data:: databases + +.. data:: lockpings + +.. data:: locks + +.. data:: mongos + +.. data:: settings + +.. data:: shards + +.. data:: system.indexes + +.. data:: version \ No newline at end of file diff --git a/source/core/sharding.txt b/source/core/sharding.txt index d495f9c8c29..a4dd5c30c19 100644 --- a/source/core/sharding.txt +++ b/source/core/sharding.txt @@ -154,9 +154,10 @@ A :term:`shard cluster` has the following components: configuration server process, but always use exactly three config servers for redundancy and safety in production. -- Two or more :program:`mongod` instances, to hold data. +- Two or more shards. Each shard consists of one or more :program:`mongod` + instances, which store the data for that shard. - These are "normal," :program:`mongod` instances that hold all of the + These "normal" :program:`mongod` instances hold all of the actual data for the cluster. Typically each shard is a :term:`replica sets `. Each diff --git a/source/includes/warning-sharding-hostnames.rst b/source/includes/warning-sharding-hostnames.rst new file mode 100644 index 00000000000..5ae3718e072 --- /dev/null +++ b/source/includes/warning-sharding-hostnames.rst @@ -0,0 +1,9 @@ +.. warning:: Sharding and "localhost" Addresses + + If you use either "localhost" or ``127.0.0.1`` as the hostname + portion of any host identifier, for example as the ``host`` argument + to :dbcommand:`addShard` or the value to the :option:`mongos + --configdb` run time option, then you must use "localhost" or + ``127.0.0.1`` for *all* host settings for any MongoDB instances in + the cluster. If you mix localhost addresses and remote host address, + MongoDB will error. \ No newline at end of file diff --git a/source/reference/method/sh.addShard.txt b/source/reference/method/sh.addShard.txt index 7e72854a0ce..047eee4ab78 100644 --- a/source/reference/method/sh.addShard.txt +++ b/source/reference/method/sh.addShard.txt @@ -8,10 +8,6 @@ sh.addShard() :param host: Specify the hostname of a new shard server. - :param optional name: Specify the name of the new shard. If you do - not specify a name for this shard, MongoDB - will assign one by default. - Use this to add shard instances to the present :term:`shard cluster`. The ``host`` parameter can be in any of the following forms: :: diff --git a/source/tutorial/deploy-shard-cluster.txt b/source/tutorial/deploy-shard-cluster.txt index 3e43fa07e10..6433454b249 100644 --- a/source/tutorial/deploy-shard-cluster.txt +++ b/source/tutorial/deploy-shard-cluster.txt @@ -4,120 +4,115 @@ Deploy a Shard Cluster .. default-domain:: mongodb -Synopsis --------- - This document describes how to deploy a :term:`shard cluster` for a -standalone :program:`mongod` instance. - -To deploy a shard cluster for an existing replica set, see +standalone :program:`mongod` instance. To deploy a shard cluster for an +existing replica set, see :doc:`/tutorial/convert-replica-set-to-replicated-shard-cluster`. +Procedure +--------- + Before deploying a shard cluster, see the requirements listed in :ref:`Requirements for Shard Clusters `. -.. warning:: Sharding and "localhost" Addresses +.. include:: /includes/warning-sharding-hostnames.rst - If you use either "localhost" or ``127.0.0.1`` as the hostname - portion of any host identifier, for example as the ``host`` argument to - :dbcommand:`addShard` or the value to the :option:`mongos --configdb` - run time option, then you must use "localhost" or ``127.0.0.1`` - for *all* host settings. If you mix localhost addresses and remote - host address, MongoDB will error. +Start the Config Server Database Instances +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Procedure ---------- - -Start Config Server Database Instances -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Begin by configuring three config server database processes. These are small +The config server database processes are small :program:`mongod` instances that store the cluster's metadata. You must -have exactly *three* instances in production deployments. Each has a complete copy -of the cluster's metadata for redundancy. These instances should run on different servers +have exactly *three* instances in production deployments. Each stores a complete copy +of the cluster's metadata, for redundancy. These instances should run on different servers to assure good uptime and data safety. Since config database :program:`mongod` instances receive relatively little traffic and demand only a small portion of system resources, you can run the instances on systems that run other cluster components. -To start a config database, type the following command at a system prompt: - -.. code-block:: sh - - mongod --configsvr - -By default a :option:`--configsrv ` process stores its data files +By default a :program:`mongod` :option:`--configsrv ` process stores its data files in the `/data/configdb` directory. You can specify a different -location using the :setting:`dbpath` run-time option option. The config :program:`mongod` instance +location using the :setting:`dbpath` run-time option. The config :program:`mongod` instance is accessible via port ``27019``. In addition to :setting:`configsvr`, use other :program:`mongod` :doc:`runtime options ` as needed. -Repeat this process for all three config server instances. +To create a data directory for each config server, issue a command +similar to the following for each: -Start ``mongos`` Instances -~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. code-block:: sh -All client operations against a shard cluster go through :program:`mongos` -instances. The :program:`mongos` instance routes queries and operations + mkdir /data/db/config + +To start each config server, issue a command similar to the following +for each: + +.. code-block:: sh + + mongod --configsvr --dbpath --port + +Start the ``mongos`` Instances +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :program:`mongos` instance routes queries and operations to the appropriate shards and interacts with the config server instances. +All client operations against a shard cluster go through :program:`mongos` +instances. -:program:`mongos` instances are lightweight, and a shard cluster typically -has several instances. For example, you run one :program:`mongos` +:program:`mongos` instances are lightweight and do not require data directories. +A shard cluster typically +has several instances. For example, you might run one :program:`mongos` instance on each of your application servers, or you might run a :program:`mongos` instance on each of the servers running a :program:`mongod` process. You must the specify resolvable hostnames [#names]_ for the *3* config servers -when starting the :program:`mongos` instance, either in the -configuration file, or as command line parameters. +when starting the :program:`mongos` instance. You specify the hostnames either in the +configuration file or as command line parameters. The :program:`mongos` instance runs on the default MongoDB TCP port: ``27017``. -Use the following command at a system prompt to start a -:program:`mongos` instance: +To start a :program:`mongos` instance running on the following hosts: -.. code-block:: sh +- ``mongos0.example.net`` +- ``mongos1.example.net`` +- ``mongos2.example.net`` - mongos --configdb config0.mongodb.example.net,config1.mongodb.example.net,config2.mongodb.example.net --port 27017 +You would issue the following command: -The above example assumes that you have config instances running on the following hosts: +.. code-block:: sh -- ``config0.mongodb.example.net`` -- ``config1.mongodb.example.net`` -- ``config2.mongodb.example.net`` + mongos --configdb mongos0.example.net,mongos1.example.net,mongos2.example.net .. [#names] Use DNS names for the config servers rather than explicit - IP addresses for operational flexibility. You cannot change the - config server names, or IP address if you're not using resolvable - hostname, without a restarting *ever* :program:`mongos` and + IP addresses for operational flexibility. If you're not using resolvable + hostame, + you cannot change the config server names or IP addresses + without a restarting *every* :program:`mongos` and :program:`mongod` instance. - Add Shards to the Cluster ~~~~~~~~~~~~~~~~~~~~~~~~~ -In a production shard cluster, each shard is itself a :term:`replica -set`. You must deploy at least one replica set or one shard to -begin. You may add additional shards later to a running cluster. For -instructions on deploying replica sets, see :doc:`/tutorial/deploy-replica-set`. +You must deploy at least one :term:`shard` or one :term:`replica set` to +begin. In a production shard cluster, each shard is a replica set. You +may add additional shards to a running cluster later. For instructions +on deploying replica sets, see :doc:`/tutorial/deploy-replica-set`. -When you have two active and initiated replica sets, perform the -following procedure, which describes the process for adding the first -two shards to the cluster. +This procedure assumes you have two active and initiated replica sets +and describes how to add the first two shards to the cluster. -Using the :program:`mongo` shell, connect to a :program:`mongos`. For example, -if the :program:`mongos` is accessible at -``mongos0.mongodb.example.net`` on port ``27017`` you would issue the -following command: +First, connect to one of the :program:`mongos` instances. For example, +if a :program:`mongos` is accessible at ``mongos0.example.net`` on +port ``27017``, issue the following command: .. code-block:: sh - mongo mongos0.mongodb.example.net + mongo mongos0.example.net -For each shard that you want to add to the cluster, call the :method:`sh.addShard()` -method in the :program:`mongo` shell, as in the following example: +Then, from the :program:`mongo` shell connected to the :program:`mongos` +instance, call the :method:`sh.addShard()` method for each shard that +you want to add to the cluster: .. code-block:: javascript @@ -132,7 +127,7 @@ These operations add two shards, provided by: - the replica set name and ``s1``, that includes the ``sfo40.example.net`` host. -.. admonition:: All shards should be replica sets. +.. admonition:: All shards should be replica sets .. versionchanged:: 2.0.3 @@ -158,9 +153,9 @@ Enable Sharding for Databases ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ While sharding operates on a per-collection basis, you must enable -sharding for each database that holds a database that you would like -to shard. A single cluster can have many databases, with each database -housing collections. +sharding for each database that holds collections you want to shard. A +single shard cluster can have many databases, with each database housing +collections. Use the following operation in a :program:`mongo` shell session connected to a :program:`mongos` instance in your cluster: @@ -178,12 +173,10 @@ Enable Sharding for Collections ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can enable sharding on a per-collection basis. Because -MongoDB uses "range based sharding," you must specify a :term:`shard -key` MongoDB can use to distribute your documents among the -shards. For more information, see the sections of this manual that give -an :ref:`overview of shard keys ` and that -give an in-depth exploration of the -:ref:`features of good shard keys `. +MongoDB uses "range based sharding," you must specify the :term:`shard +key` MongoDB uses to distribute your documents among the +shards. For more information, see the +:ref:`overview of shard keys `. To enable sharding for a collection, use the :method:`sh.shardCollection()` helper in the :program:`mongo` shell.