From e8a0cc7cdf1943a0663e3be10906c089ff654481 Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 17 Dec 2012 13:10:19 -0500 Subject: [PATCH 1/4] DOCS-685 migrate padding factor wiki page --- source/core/write-operations.txt | 80 ++++++++++++++++++++++++++++++++ source/faq/developers.txt | 44 ++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/source/core/write-operations.txt b/source/core/write-operations.txt index 42d642df1dc..0b1833584ce 100644 --- a/source/core/write-operations.txt +++ b/source/core/write-operations.txt @@ -341,6 +341,86 @@ write operation that affects multiple documents using the To isolate a sequence of write operations from other read and write operations, see :doc:`/tutorial/perform-two-phase-commits`. +.. _write-operations-padding-factor: + +Padding Factor +-------------- + +When updating a document, if an update operation does not cause the +document to increase in size, the update occurs in-place. If, however, +the update causes the document to increase in size, for instance if you +use the :operator:`$push` operator, **and** exceeds its allocated space, +then the document is relocated on disk to a new location with enough +contiguous space to fit the larger document. Since the move will +require an update of all the indexes for the document, if the +collection has many indexes, the move will impact write performance. + +To minimize document movements, Mongo employs padding. Mongo adaptively +learns if documents in a collection tend to grow, and if they do, adds +a :stats:`paddingFactor` so that the document has room to grow on +subsequent writes. The :stats:`paddingFactor` indicates the padding for +new inserts and moves. + +.. versionadded:: 2.2 + You can use the :dbcommand:`collMod` command with the + ``usePowerOf2Sizes`` flag to ensure that MongoDB allocates space for + documents in sizes that are powers of 2. + +To check the current :stats:`paddingFactor` on a collection, you can +run the :dbcommand:`db.collection.stats()` command in the +:program:`mongo` shell, as in the following example: + +.. code-block:: javascript + + db.myCollection.stats() + +Since each document is written at a different point in time, the +padding for each document will not be the same. You can calculate the +padding size by subtracting 1 from the paddingFactor: + +.. code-block:: none + + padding size = (paddingFactor - 1) * . + +For example, a ``paddingFactor`` of ``1.0`` specifies a padding size of ``0`` +whereas a paddingFactor of ``1.5`` specifies a padding size of ``0.5`` or ``50 +percent (50%)`` of the document size. + +Because the :stats:`paddingFactor` is relative to the size of each +document, you cannot calculate the exact amount of padding for a +collection based on the average document size and padding factor. + +If an update operation causes the document to *decrease* in size, for +instance if you perform an :operator:`$unset` or :operator:`$pop` +updates, the document remains in place and effectively has more +padding. If the document remains this size, the space is never +reclaimed unless you perform a :dbcommand:`compact` or a +:dbcommand:`repairDatabase` operation. + +.. note:: + + Padding is removed when you perform :dbcommand:`compact`, + :dbcommand:`repairDatabase`, or initial replica sync operations. + Additionally, if you :program:`export ` from a + collection with padding and :program:`import ` into a + new collection, or into the same collection but first drop the + collection, the new collection is created without padding. + Therefore, after these operations, you may see slower update + performance, but the size required for storage will be lower. You + can, however, run the :dbcommand:`compact` command with a + ``paddingFactor`` or a ``paddingBytes`` parameter. + +.. COMMENT -- not sure if we really need this manual padding info or if + it belongs here, but ... + +.. seealso:: + + - :ref:`faq-developers-manual-padding` + + - `Fast Updates with MongoDB (update-in-place ) + `_ + Architecture ------------ diff --git a/source/faq/developers.txt b/source/faq/developers.txt index 24585fc82b3..f57706766ae 100644 --- a/source/faq/developers.txt +++ b/source/faq/developers.txt @@ -660,3 +660,47 @@ information on indexes. .. Commenting out.. If your small documents are approximately the page cache unit size, there is no benefit for ram cache efficiency, although embedding will provide some benefit regarding random disk i/o. +======= +.. _faq-developers-manual-padding: + +Can I manually pad documents to prevent moves during updates? +------------------------------------------------------------- + +An update can cause a document to move on disk if the document grows in +size. To minimize document movements, MongoDB uses +:ref:`padding `. + +You should not have to pad manually because MongoDB handles +:ref:`padding automatically `. MongoDB +adaptively learns if documents in a collection tend to grow, and if +they do, adds a :stats:`paddingFactor` so that the document has room to +grow on subsequent writes. You can change the default +:stats:`paddingFactor` calculation by using the the +:dbcommand:`collMod` command with the ``usePowerOf2Sizes`` flag. +The ``usePowerOf2Sizes`` flag ensures that MongoDB allocates document space +in sizes that are powers of 2. + +However, in those exceptions where you must pad manually, you can use +the strategy of first adding a temporary field to a document and then +:operator:`$unsetting <$unset>` the field, as in the following example: + +.. code-block:: javascript + + var myTempPadding = [ "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]; + + db.myCollection.insert( { _id: 5, paddingField: myTempPadding } ); + + db.myCollection.update( { _id: 5 }, + { $unset: { paddingField: "" } } + ) + + db.myCollection.update( { _id: 5 }, + { $set: { realField: "Some text that I might have needed padding for" } } + ) + +.. seealso:: + + :ref:`write-operations-padding-factor` From 79b0c5d65f35bbcc8fd9ea614866dd505de473ee Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 7 Jan 2013 12:31:49 -0500 Subject: [PATCH 2/4] DOCS-685 incorporate ed and dan's comments --- source/core/write-operations.txt | 40 +++++++++++++----------- source/faq/developers.txt | 11 ++----- themes/mongodb/static/mongodb-docs.css_t | 1 - 3 files changed, 24 insertions(+), 28 deletions(-) diff --git a/source/core/write-operations.txt b/source/core/write-operations.txt index 0b1833584ce..899734da65a 100644 --- a/source/core/write-operations.txt +++ b/source/core/write-operations.txt @@ -355,7 +355,7 @@ contiguous space to fit the larger document. Since the move will require an update of all the indexes for the document, if the collection has many indexes, the move will impact write performance. -To minimize document movements, Mongo employs padding. Mongo adaptively +To minimize document movements, MongoDB employs padding. MongoDB adaptively learns if documents in a collection tend to grow, and if they do, adds a :stats:`paddingFactor` so that the document has room to grow on subsequent writes. The :stats:`paddingFactor` indicates the padding for @@ -363,8 +363,9 @@ new inserts and moves. .. versionadded:: 2.2 You can use the :dbcommand:`collMod` command with the - ``usePowerOf2Sizes`` flag to ensure that MongoDB allocates space for - documents in sizes that are powers of 2. + ``usePowerOf2Sizes`` flag so that MongoDB allocates document space + in sizes that are powers of 2. As with all padding, using powers of + 2 minimizes, but does not eliminate, document movements. To check the current :stats:`paddingFactor` on a collection, you can run the :dbcommand:`db.collection.stats()` command in the @@ -382,33 +383,36 @@ padding size by subtracting 1 from the paddingFactor: padding size = (paddingFactor - 1) * . -For example, a ``paddingFactor`` of ``1.0`` specifies a padding size of ``0`` -whereas a paddingFactor of ``1.5`` specifies a padding size of ``0.5`` or ``50 -percent (50%)`` of the document size. +For example, a ``paddingFactor`` of ``1.0`` specifies no padding +whereas a paddingFactor of ``1.5`` specifies a padding size of ``0.5`` or 50 +percent (50%) of the document size. Because the :stats:`paddingFactor` is relative to the size of each document, you cannot calculate the exact amount of padding for a collection based on the average document size and padding factor. If an update operation causes the document to *decrease* in size, for -instance if you perform an :operator:`$unset` or :operator:`$pop` -updates, the document remains in place and effectively has more -padding. If the document remains this size, the space is never -reclaimed unless you perform a :dbcommand:`compact` or a +instance if you perform an :operator:`$unset` or a :operator:`$pop` +update, the document remains in place and effectively has more +padding. If the document remains this size, the space is not reclaimed +until you perform a :dbcommand:`compact` or a :dbcommand:`repairDatabase` operation. .. note:: Padding is removed when you perform :dbcommand:`compact`, :dbcommand:`repairDatabase`, or initial replica sync operations. - Additionally, if you :program:`export ` from a - collection with padding and :program:`import ` into a - new collection, or into the same collection but first drop the - collection, the new collection is created without padding. - Therefore, after these operations, you may see slower update - performance, but the size required for storage will be lower. You - can, however, run the :dbcommand:`compact` command with a - ``paddingFactor`` or a ``paddingBytes`` parameter. + However, with the :dbcommand:`compact` command, you can run the + command with a ``paddingFactor`` or a ``paddingBytes`` parameter. + + Padding is also removed if you :program:`export ` from + a collection. If you :program:`import ` into a new + collection, the new collection is created without padding. If you + :program:`import ` into an existing collection with + padding, the existing padding is maintained. + + If padding is removed during these operations, updates may be slower + although the size requirements for storage are lower. .. COMMENT -- not sure if we really need this manual padding info or if it belongs here, but ... diff --git a/source/faq/developers.txt b/source/faq/developers.txt index f57706766ae..1aa8092ef9b 100644 --- a/source/faq/developers.txt +++ b/source/faq/developers.txt @@ -667,18 +667,11 @@ Can I manually pad documents to prevent moves during updates? ------------------------------------------------------------- An update can cause a document to move on disk if the document grows in -size. To minimize document movements, MongoDB uses +size. To *minimize* document movements, MongoDB uses :ref:`padding `. You should not have to pad manually because MongoDB handles -:ref:`padding automatically `. MongoDB -adaptively learns if documents in a collection tend to grow, and if -they do, adds a :stats:`paddingFactor` so that the document has room to -grow on subsequent writes. You can change the default -:stats:`paddingFactor` calculation by using the the -:dbcommand:`collMod` command with the ``usePowerOf2Sizes`` flag. -The ``usePowerOf2Sizes`` flag ensures that MongoDB allocates document space -in sizes that are powers of 2. +:ref:`padding automatically `. However, in those exceptions where you must pad manually, you can use the strategy of first adding a temporary field to a document and then diff --git a/themes/mongodb/static/mongodb-docs.css_t b/themes/mongodb/static/mongodb-docs.css_t index 29e0fdb00bf..dc4bc101714 100644 --- a/themes/mongodb/static/mongodb-docs.css_t +++ b/themes/mongodb/static/mongodb-docs.css_t @@ -727,7 +727,6 @@ div.section > h2, div.section > h3,div.section > h4 { div.admonition p { line-height:1.5em; - padding: 1em 0; } dd > div.admonition { margin-left: 0; } From 8d621ab088f111860908ae932c917bcff174170e Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 7 Jan 2013 17:32:21 -0500 Subject: [PATCH 3/4] DOCS-685 power of 2 in faq padding --- source/core/write-operations.txt | 16 ++++++++-------- source/faq/developers.txt | 11 +++++++++-- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/source/core/write-operations.txt b/source/core/write-operations.txt index 899734da65a..3c1ed4c01d8 100644 --- a/source/core/write-operations.txt +++ b/source/core/write-operations.txt @@ -355,17 +355,17 @@ contiguous space to fit the larger document. Since the move will require an update of all the indexes for the document, if the collection has many indexes, the move will impact write performance. -To minimize document movements, MongoDB employs padding. MongoDB adaptively -learns if documents in a collection tend to grow, and if they do, adds -a :stats:`paddingFactor` so that the document has room to grow on -subsequent writes. The :stats:`paddingFactor` indicates the padding for -new inserts and moves. +To minimize document movements, MongoDB employs padding. MongoDB +adaptively learns if documents in a collection tend to grow, and if +they do, adds a :stats:`paddingFactor` so that the documents have room +to grow on subsequent writes. The :stats:`paddingFactor` indicates the +padding for new inserts and moves. .. versionadded:: 2.2 You can use the :dbcommand:`collMod` command with the - ``usePowerOf2Sizes`` flag so that MongoDB allocates document space - in sizes that are powers of 2. As with all padding, using powers of - 2 minimizes, but does not eliminate, document movements. + :collflag:`usePowerOf2Sizes` flag so that MongoDB allocates document + space in sizes that are powers of 2. As with all padding, using + powers of 2 minimizes, but does not eliminate, document movements. To check the current :stats:`paddingFactor` on a collection, you can run the :dbcommand:`db.collection.stats()` command in the diff --git a/source/faq/developers.txt b/source/faq/developers.txt index 1aa8092ef9b..dd86f94ea1b 100644 --- a/source/faq/developers.txt +++ b/source/faq/developers.txt @@ -660,7 +660,7 @@ information on indexes. .. Commenting out.. If your small documents are approximately the page cache unit size, there is no benefit for ram cache efficiency, although embedding will provide some benefit regarding random disk i/o. -======= + .. _faq-developers-manual-padding: Can I manually pad documents to prevent moves during updates? @@ -671,7 +671,14 @@ size. To *minimize* document movements, MongoDB uses :ref:`padding `. You should not have to pad manually because MongoDB handles -:ref:`padding automatically `. +:ref:`padding automatically `. MongoDB +adaptively learns if documents in a collection tend to grow, and if +they do, adds a :stats:`paddingFactor` so that documents have room to +grow on subsequent writes. You can change the default +:stats:`paddingFactor` calculation by using the the +:dbcommand:`collMod` command with the :collflag:`usePowerOf2Sizes` +flag. The :collflag:`usePowerOf2Sizes` flag ensures that MongoDB +allocates document space in sizes that are powers of 2. However, in those exceptions where you must pad manually, you can use the strategy of first adding a temporary field to a document and then From e7053bfd94a65446e3ca6edf804fbdb6bc1bbecd Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 7 Jan 2013 19:24:17 -0500 Subject: [PATCH 4/4] DOCS-685 add powersof2 sentence from Dan --- source/core/write-operations.txt | 6 ++++-- source/faq/developers.txt | 4 +++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/source/core/write-operations.txt b/source/core/write-operations.txt index 3c1ed4c01d8..81f1e90cb42 100644 --- a/source/core/write-operations.txt +++ b/source/core/write-operations.txt @@ -364,8 +364,10 @@ padding for new inserts and moves. .. versionadded:: 2.2 You can use the :dbcommand:`collMod` command with the :collflag:`usePowerOf2Sizes` flag so that MongoDB allocates document - space in sizes that are powers of 2. As with all padding, using - powers of 2 minimizes, but does not eliminate, document movements. + space in sizes that are powers of 2. This helps ensure that the + space freed due to either deletions or document relocations is + reused efficiently. As with all padding, using powers of 2 + minimizes, but does not eliminate, document movements. To check the current :stats:`paddingFactor` on a collection, you can run the :dbcommand:`db.collection.stats()` command in the diff --git a/source/faq/developers.txt b/source/faq/developers.txt index dd86f94ea1b..c8225b09501 100644 --- a/source/faq/developers.txt +++ b/source/faq/developers.txt @@ -678,7 +678,9 @@ grow on subsequent writes. You can change the default :stats:`paddingFactor` calculation by using the the :dbcommand:`collMod` command with the :collflag:`usePowerOf2Sizes` flag. The :collflag:`usePowerOf2Sizes` flag ensures that MongoDB -allocates document space in sizes that are powers of 2. +allocates document space in sizes that are powers of 2, which helps +ensure that the space freed due to either deletions or document +relocations is reused efficiently. However, in those exceptions where you must pad manually, you can use the strategy of first adding a temporary field to a document and then