Skip to content

DOCS-298 DOCS-299 migrate deploy shards #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 11, 2012
Merged

DOCS-298 DOCS-299 migrate deploy shards #300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1adb540
DOCS-298 DOCS-299 initial edits
Oct 9, 2012
9f7d3b3
DOCS-298 DOCS-299 migrate deploy shards
Oct 10, 2012
64fc4ae
DOCS-298 DOCS-299 edit to internals document
Oct 10, 2012
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 85 additions & 90 deletions source/administration/sharding.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,116 +7,99 @@ Shard Cluster Administration

.. default-domain:: mongodb

This document provides a collection of basic operations and procedures
for administering :term:`shard clusters <shard cluster>`.

This document describes common administrative tasks.
For a full introduction to sharding in MongoDB see
":doc:`/core/sharding`," and for a complete overview of all sharding
documentation in the MongoDB Manual, see ":doc:`/sharding`." The
":doc:`/administration/sharding-architectures`" document provides an
:doc:`/core/sharding`, and for a complete overview of all sharding
documentation in the MongoDB Manual, see :doc:`/sharding`. The
:doc:`/administration/sharding-architectures` document provides an
overview of deployment possibilities to help deploy a shard
cluster. Finally, the ":doc:`/core/sharding-internals`" document
cluster. Finally, the :doc:`/core/sharding-internals` document
provides a more detailed introduction to sharding when troubleshooting
issues or understanding your cluster's behavior.


.. contents:: Sharding Procedures:
:backlinks: none
:local:

.. _sharding-procedure-setup:

Setup
-----
Set up a Shard Cluster
----------------------

If you have an existing replica set, you can use the
":doc:`/tutorial/convert-replica-set-to-replicated-shard-cluster`"
tutorial as a guide. If you're deploying a :term:`shard cluster` from
scratch, see the ":doc:`/tutorial/deploy-shard-cluster`" tutorial for
more detail or use the following procedure as a quick starting point:
Before deploying a shard cluster, see the requirements listed in
:ref:`Requirements for Shard Clusters <sharding-requirements>`.

#. Provision the required hardware.
For testing purposes, you can run all the required shard processes on a
single server. For production, use the configurations described in
:doc:`/administration/replication-architectures`.

The ":ref:`sharding-requirements`" section describes what you'll
need to get started.
.. include:: /includes/warning-sharding-hostnames.rst

.. warning:: Sharding and "localhost" Addresses
.. TODO Keep this or delete it:
If you have an existing replica set, you can use the
:doc:`/tutorial/convert-replica-set-to-replicated-shard-cluster`
tutorial as a guide. If you're deploying a :term:`shard cluster` from
scratch, see the :doc:`/tutorial/deploy-shard-cluster` tutorial for
more detail or use the following procedure as a quick starting point:

If you use either "localhost" or "``127.0.0.1``" as the host
identifier, then you must use "localhost" or "``127.0.0.1``" for
*all* host settings for any MongoDB instances in the cluster. If
you mix localhost addresses with remote host address, MongoDB will
produce errors.
1. Create data directories for each of the three (3) config server instances.

#. On all three (3) config server instances, issue the following
command to start the :program:`mongod` process:
#. Start the three config server instances. For example, to start a
config server instance running on TCP port ``27018`` with the data
stored in ``/data/configdb``, type the following:

.. code-block:: sh

mongod --configsvr
mongod --configsvr --dbpath /data/configdb --port 27018

This starts a :program:`mongod` instance running on TCP port
``27018``, with the data stored in the ``/data/configdb`` path. All other
:doc:`command line </reference/mongod>` and :doc:`configuration
file </reference/configuration-options>` options are available for config
server instances.
For additional command options, see :doc:`/reference/mongod`
and :doc:`/reference/configuration-options`.

.. include:: /includes/note-config-server-startup.rst

#. Start a :program:`mongos` instance issuing the following command:

.. code-block:: sh
#. Start the three :program:`mongos` instances. For example, to start a
:program:`mongos` instance running on the following hosts:

mongos --configdb config0.mongodb.example.net,config1.mongodb.example.net,config2.mongodb.example.net --port 27017
- ``mongos0.example.net``
- ``mongos1.example.net``
- ``mongos2.example.net``

#. Connect to the :program:`mongos` instance using the :program:`mongo`
shell.
You would issue the following command:

.. code-block:: sh

mongo mongos.mongodb.example.net

#. Add shards to the cluster.
mongos --configdb mongos0.example.net,mongos1.example.net,mongos2.example.net

Run the following commands while connected to a :program:`mongos` to
initialize the cluster.
#. Connect to one of the :program:`mongos` instances. For example, if
a :program:`mongos` is accessible at ``mongos0.example.net`` on
port ``27017``, issue the following command:

First, you need to tell the cluster where to find the individual
shards. You can do this using the :dbcommand:`addShard` command.
.. code-block:: sh

.. code-block:: javascript
mongo mongos0.example.net

sh.addShard( "[hostname]:[port]" )
#. Add shards to the cluster. From the :program:`mongo` shell connected
to the :program:`mongos` instance, call the :method:`sh.addShard()`
method for each shard to add to the cluster.

For example:

.. code-block:: javascript

sh.addShard( "mongodb0.example.net:27027" )

MongoDB will discover all other members of the replica set, if
``mongodb0.example.net:27027`` is a member of a replica set.
If ``mongodb0.example.net:27027`` is a member of a replica
set, MongoDB will discover all other members of the replica set.

.. note:: In production deployments, all shards should be replica sets.

Repeat this step for each new shard in your cluster.

.. optional::

You may specify a "name" as an argument to the
:dbcommand:`addShard`, as follows:

.. code-block:: javascript

db.runCommand( { addShard: mongodb0.example.net, name: "mongodb0" } )

Or:

.. code-block:: javascript

sh.addShard( mongodb0.example.net, name: "mongodb0" )

If you do not specify a shard name, then MongoDB will assign a
name upon creation.
You can specify a name for the shard and a maximum size. See
:dbcommand:`addShard`.

.. note::

Expand All @@ -129,32 +112,33 @@ more detail or use the following procedure as a quick starting point:

replicaSetName/<seed1>,<seed2>,<seed3>

For example, if the name of the replica set is "``repl0``", then
your :method:`sh.addShard` command would be:
For example, if the name of the replica set is ``repl0``, then
your :method:`sh.addShard()` command would be:

.. code-block:: javascript

sh.addShard( "repl0/mongodb0.example.net:27027,mongodb1.example.net:27017,mongodb2.example.net:27017" )

#. Enable sharding for any database that you want to shard.
#. Enable sharding for each database you want to shard.
While sharding operates on a per-collection basis, you must enable
sharding for each database that holds collections you want to shard.
This step is a meta-data change and will not redistribute your data.

MongoDB enables sharding on a per-database basis. This is only a
meta-data change and will not redistribute your data. To enable
sharding for a given database, use the :dbcommand:`enableSharding`
command or the :method:`sh.enableSharding()` shell function.
To enable sharding for a given database, use the
:dbcommand:`enableSharding` command or the
:method:`sh.enableSharding()` shell function, as shown below. Replace
``<database>`` with the name of the database on which to enable
sharding.

.. code-block:: javascript

db.runCommand( { enableSharding: [database] } )
db.runCommand( { enableSharding: <database> } )

Or:

.. code-block:: javascript

sh.enableSharding([database])

Replace ``[database]`` with the name of the database you wish to
enable sharding on.
sh.enableSharding(<database>)

.. note::

Expand All @@ -166,21 +150,21 @@ more detail or use the following procedure as a quick starting point:

#. Enable sharding on a per-collection basis.

Finally, you must explicitly specify collections to shard. The
Finally, you must explicitly specify collections to shard. The
collections must belong to a database for which you have enabled
sharding. When you shard a collection, you also choose the shard
key. To shard a collection, run the :dbcommand:`shardCollection`
command or the :method:`sh.shardCollection()` shell helper.

.. code-block:: javascript

db.runCommand( { shardCollection: "[database].[collection]", key: "[shard-key]" } )
db.runCommand( { shardCollection: "<database>.<collection>", key: "<shard-key>" } )

Or:

.. code-block:: javascript

sh.shardCollection("[database].[collection]", "key")
sh.shardCollection("<database>.<collection>", "key")

For example:

Expand All @@ -199,9 +183,9 @@ more detail or use the following procedure as a quick starting point:
the distribution of data. Furthermore, you cannot change a
collection's shard key once it has been set.

See the ":ref:`Shard Key Overview <sharding-shard-key>`" and the
more in depth documentation of ":ref:`Shard Key Qualities
<sharding-internals-shard-keys>`" to help you select better shard
See the :ref:`Shard Key Overview <sharding-shard-key>` and the
more in depth documentation of :ref:`Shard Key Qualities
<sharding-internals-shard-keys>` to help you select better shard
keys.

If you do not specify a shard key, MongoDB will shard the
Expand All @@ -215,6 +199,17 @@ This section describes common maintenance procedure, including: how to
add and remove nodes, how to manually split chunks, and how to disable
the balancer for backups.

List the Set of Configured Shards
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To list the current set of configured shards and verify that all shards
have been committed to the system, run the :dbcommand:`listShards`
command:

.. code-block:: javascript

db.runCommand( { listshards : 1 } )

.. _sharding-procedure-add-shard:

Adding a Shard to a Cluster
Expand All @@ -232,9 +227,9 @@ procedure:

.. code-block:: javascript

sh.addShard( "[hostname]:[port]" )
sh.addShard( "<hostname>:<port>" )

Replace ``[hostname]`` and ``[port]`` with the hostname and TCP
Replace ``<hostname>`` and ``<port>`` with the hostname and TCP
port number of where the shard is accessible.

For example:
Expand Down Expand Up @@ -271,7 +266,7 @@ procedure:
following form: the replica set name, followed by a forward
slash, followed by a comma-separated list of seeds for the
replica set. For example, if the name of the replica set is
"myapp1", then your :method:`sh.addShard` command might resemble:
"myapp1", then your :method:`sh.addShard()` command might resemble:

.. code-block:: javascript

Expand All @@ -282,9 +277,9 @@ procedure:
It may take some time for :term:`chunks <chunk>` to migrate to the
new shard.

See the ":ref:`Balancing and Distribution <sharding-balancing>`"
section for an overview of the balancing operation and the ":ref:`Balancing Internals
<sharding-balancing-internals>`" section for additional information.
See the :ref:`Balancing and Distribution <sharding-balancing>`
section for an overview of the balancing operation and the :ref:`Balancing Internals
<sharding-balancing-internals>` section for additional information.

.. _sharding-procedure-remove-shard:

Expand Down Expand Up @@ -695,7 +690,7 @@ Balancer Operations
This section provides an overview of common administrative procedures
related to balancing and the balancing process.

.. seealso:: ":ref:`sharding-balancing`" and the
.. seealso:: :ref:`sharding-balancing` and the
:dbcommand:`moveChunk` that provides manual :term:`chunk`
migrations.

Expand Down Expand Up @@ -947,7 +942,7 @@ name.
#. Shut down the config server that you're moving.

This will render all config data for your cluster :ref:`read only
<sharding-config-server>`."
<sharding-config-server>`.

#. Change the DNS entry that points to the system that provided the old
config server, so that the *same* hostname points to the new
Expand Down Expand Up @@ -1072,7 +1067,7 @@ of the cluster metadata from the config database is straight forward:
.. [#read-only] While one of the three config servers unavailable, no
the cluster cannot split any chunks nor can it migrate chunks
between shards. Your application will be able to write data to the
cluster. The ":ref:`sharding-config-server`" section of the
cluster. The :ref:`sharding-config-server` section of the
documentation provides more information on this topic.

.. index:: troubleshooting; sharding
Expand Down
31 changes: 31 additions & 0 deletions source/core/sharding-internals.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,3 +495,34 @@ All chunk migrations use the following procedure:
When the ``_secondaryThrottle`` is ``true`` for :dbcommand:`moveChunk`
or the :term:`balancer`, MongoDB ensure that *one* :term:`secondary`
member has replicated changes before allowing new chunk migrations.

Config Database
---------------

The ``config`` database contains sharding configuration information. To
start the ``config`` database from the :program:`mongo` shell, issue the
following command:

.. code-block:: javascript

use config

The config database includes the following collections used by sharding:

.. data:: chunks

.. data:: databases

.. data:: lockpings

.. data:: locks

.. data:: mongos

.. data:: settings

.. data:: shards

.. data:: system.indexes

.. data:: version
5 changes: 3 additions & 2 deletions source/core/sharding.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,9 +154,10 @@ A :term:`shard cluster` has the following components:
configuration server process, but always use exactly three config
servers for redundancy and safety in production.

- Two or more :program:`mongod` instances, to hold data.
- Two or more shards. Each shard consists of one or more :program:`mongod`
instances, which store the data for that shard.

These are "normal," :program:`mongod` instances that hold all of the
These "normal" :program:`mongod` instances hold all of the
actual data for the cluster.

Typically each shard is a :term:`replica sets <replica set>`. Each
Expand Down
9 changes: 9 additions & 0 deletions source/includes/warning-sharding-hostnames.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.. warning:: Sharding and "localhost" Addresses

If you use either "localhost" or ``127.0.0.1`` as the hostname
portion of any host identifier, for example as the ``host`` argument
to :dbcommand:`addShard` or the value to the :option:`mongos
--configdb` run time option, then you must use "localhost" or
``127.0.0.1`` for *all* host settings for any MongoDB instances in
the cluster. If you mix localhost addresses and remote host address,
MongoDB will error.
4 changes: 0 additions & 4 deletions source/reference/method/sh.addShard.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@ sh.addShard()

:param host: Specify the hostname of a new shard server.

:param optional name: Specify the name of the new shard. If you do
not specify a name for this shard, MongoDB
will assign one by default.

Use this to add shard instances to the present :term:`shard
cluster`. The ``host`` parameter can be in any of the following
forms: ::
Expand Down
Loading