DOCS-1105 sort w limit and other optimization

Author: kay-kim

source/applications/aggregation.txt

@@ -236,6 +236,83 @@ affecting the result. However, at this time place
 :agg:pipeline:`$match` operators at the beginning of the pipeline when
 possible.
 
+Pipeline Sequence Optimization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionchanged:: 2.4
+
+:term:`Aggregation` operations have an optimization phase which
+attempts to re-arrange the pipeline for improved performance. 
+
+``$sort`` + ``$skip`` + ``$limit`` Sequence Optimization
+````````````````````````````````````````````````````````
+
+When you have sequence of :agg:pipeline:`$sort` followed by a
+:agg:pipeline:`$skip` followed by a :agg:pipeline:`$limit`, an
+optimization occurs whereby the :agg:pipeline:`$limit` moves in front
+of the :agg:pipeline:`$skip`. For example, if the pipeline consists of
+the following stages:
+
+.. code-block:: javascript
+
+   { $sort: { age : -1 } },
+   { $skip: 10 },
+   { $limit: 5 }
+
+During the optimization phase, the optimizer transforms the sequence to
+the following:
+
+.. code-block:: javascript
+
+   { $sort: { age : -1 } },
+   { $limit: 15 }
+   { $skip: 10 }
+
+.. note::
+
+   The :agg:pipeline:`$limit` value has increased to the sum of the
+   initial value and the :agg:pipeline:`$skip` value.
+
+``$limit`` + ``$skip`` + ``$limit`` + ``$skip`` Sequence Optimization
+`````````````````````````````````````````````````````````````````````
+
+When you have continuous sequence of :agg:pipeline:`$limit` pipeline
+stage followed by a :agg:pipeline:`$skip` pipeline stage, the
+aggregation will attempt to re-arrange the pipeline stages to combine
+the limits together and the skips together. For example, if the
+pipeline consists of the following stages:
+
+.. code-block:: javascript
+
+   { $limit: 100 },
+   { $skip: 5 },
+   { $limit: 10},
+   { $skip: 2 }
+
+During the intermediate step, the optimizer reverses the position of
+the :agg:pipeline:`$skip` followed by a :agg:pipeline:`$limit` to
+:agg:pipeline:`$limit` followed by the :agg:pipeline:`$skip`.
+
+.. code-block:: javascript
+
+   { $limit: 100 },
+   { $limit: 15},
+   { $skip: 5 },
+   { $skip: 2 }
+
+The :agg:pipeline:`$limit` value has increased to the sum of the
+initial value and the :agg:pipeline:`$skip` value. Then, for the final
+:agg:pipeline:`$limit` value, the optimizer selects the minimum between
+the adjacent :agg:pipeline:`$limit` values. For the final
+:agg:pipeline:`$skip` value, the optimizer adds the adjacent
+:agg:pipeline:`$skip` values, to transform the sequence to the
+following:
+
+.. code-block:: javascript
+
+   { $limit: 15 },
+   { $skip: 7 }
+
 Memory for Cumulative Operators
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -247,6 +324,8 @@ implementation of :agg:pipeline:`$sort` does not go to disk in these
 cases: in order to sort the contents of the pipeline, the entire input
 must fit in memory.
 
+.. include:: /includes/fact-agg-sort-limit.rst
+
 :agg:pipeline:`$group` has similar characteristics: Before any
 :agg:pipeline:`$group` passes its output along the pipeline, it must
 receive the entirety of its input. For the :agg:pipeline:`$group`

source/includes/fact-agg-sort-limit.rst

@@ -0,0 +1,6 @@
+.. versionchanged:: 2.4
+   When a :agg:pipeline:`$sort` immediately precedes a
+   :agg:pipeline:`$limit` in the pipeline, the :agg:pipeline:`$sort`
+   operation only maintains the top n results as it progresses, where n
+   is the specified limit. Before 2.4, :agg:pipeline:`$sort` would sort
+   all the results in memory, and then limit the results to n results.

source/reference/aggregation/limit.txt

@@ -24,3 +24,6 @@ $limit (aggregation)
    by the pipeline. :pipeline:`$limit` has no effect on the content
    of the documents it passes.
 
+   .. note::
+
+      .. include:: /includes/fact-agg-sort-limit.rst

source/reference/aggregation/sort.txt

@@ -43,19 +43,20 @@ $sort (aggregation)
 
       .. todo:: mention the importance of order preserving objects
 
-      - :agg:pipeline:`$skip` 
+      - :pipeline:`$skip`
 
    :pipeline:`$sort` operator can take advantage of an index when
    placed at the **beginning** of the pipeline or placed **before**
    the following aggregation operators:
 
-   - :agg:pipeline:`$project`
-   - :agg:pipeline:`$unwind`
-   - :agg:pipeline:`$group`.
+   - :pipeline:`$project`
+   - :pipeline:`$unwind`
+   - :pipeline:`$group`.
+
+   .. include:: /includes/fact-agg-sort-limit.rst
 
    .. warning:: Unless the :pipeline:`$sort` operator can use an index,
-      in the current release, the sort must fit within memory. This
-      may cause problems when sorting large numbers of documents.
+      the :pipeline:`$sort` operation must fit within memory.
 
       .. todo:: if a sort precedes the first $group in a sharded system,
          all documents must go to the mongos for sorting.

source/release-notes/2.4.txt

@@ -462,3 +462,8 @@ Improvements to the Aggregation Framework
 - :agg:group:`$min` operator only considers non-null and existing field
   values. If all the values for a field are ``null`` or are missing,
   the operator returns ``null`` for the minimum value.
+
+- When a :agg:pipeline:`$sort` immediately precedes a
+  :agg:pipeline:`$limit` in the pipeline, the :agg:pipeline:`$sort`
+  operation only maintains the top n results as it progresses, where n
+  is the specified limit.