Skip to content

Commit 80252e9

Browse files
author
Dave Cuthbert
authored
DOCS-15709 collection level filter (#103)
* DOCS-15709 Collection level filtering * Staging fixes * Staging fixes * Staging fixes * Staging fixes * Review feedback * Staging fixes * Review feedback - not ready to push * Examples * Examples * Examples * Staging fixes * Alans feedback * Review feedback * Staging * Staging * Review comments * Review feedback * Review feedback * Staging fixes
1 parent f70aeeb commit 80252e9

10 files changed

+533
-30
lines changed

‎source/includes/api/facts/includeNamespaces-syntax.rst‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
.. code-block:: javascript
2+
3+
[
4+
{
5+
"database": "databaseOne", // required
6+
"collections": [ // optional
7+
"collectionOne",
8+
"collectionTwo"
9+
]
10+
},
11+
{
12+
"database": "databaseTwo"
13+
}
14+
]
15+

‎source/includes/api/requests/start-filtered.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
curl -X POST "http://localhost:27182/api/v1/start" --data '
2+
{
3+
"source": "cluster0",
4+
"destination": "cluster1",
5+
"includeNamespaces" : [
6+
{
7+
"database" : "sales",
8+
"collections": [ "EMEA", "APAC" ]
9+
},
10+
{
11+
"database" : "marketing"
12+
}
13+
]
14+
} '

‎source/includes/example-filter-collection-with-renaming.rst‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
.. include:: /includes/intro-start-api-example-intro.rst
2+
3+
``cluster0`` contains the ``students``, ``staff``, and ``prospects``
4+
databases.
5+
6+
- The ``students`` database contains the ``undergrad`` and ``graduate``
7+
collections.
8+
- The ``staff`` database contains the ``employees`` and ``contractors``
9+
collections.
10+
11+
The ``includeNamespaces`` array in this example defines a filter on two
12+
of the databases:
13+
14+
.. code-block:: shell
15+
16+
{
17+
"source": "cluster0",
18+
"destination": "cluster1",
19+
"includeNamespaces":
20+
[
21+
{ "database" : "students", "collections": ["undergrad", "graduate", "adjuncts"] },
22+
{ "database" : "staff" }
23+
]
24+
}
25+
26+
With this filter in place, ``mongosync`` syncs:
27+
28+
- The entire ``staff`` database
29+
- The ``undergrad``, ``graduate``, and ``adjuncts`` collections in the
30+
``students`` database
31+
32+
``mongosync`` does not sync any information from the ``prospects``
33+
database.
34+
35+
Adding a Collection
36+
```````````````````
37+
38+
``mongosync`` syncs the entire ``staff`` database. If you add new
39+
collections to the ``staff`` database, ``mongosync`` syncs them too.
40+
41+
``mongosync`` does not sync new collections that are added to
42+
the ``students`` database unless the collection is a part of the filter.
43+
44+
For example, ``mongosync`` does not sync the new collection if you add
45+
the ``postdocs`` collection to the ``students`` database. If you add the
46+
``adjuncts`` collection, ``mongosync`` syncs it since ``adjuncts`` is
47+
part of the filter.
48+
49+
50+
Renaming a Collection
51+
`````````````````````
52+
53+
You can rename any collection in the ``staff`` database.
54+
55+
.. code-block:: shell
56+
57+
// This code works
58+
use admin
59+
db.runCommand( { renameCollection: "staff.employees", to: "staff.salaried" } )
60+
61+
You can only rename a collection within the ``students`` database if the
62+
new and old names are both in the filter. If either of the names is not
63+
in the filter, ``monogsync`` reports an error and exists.
64+
65+
.. code-block:: shell
66+
67+
// This code works
68+
use admin
69+
db.runCommand( { renameCollection: "students.graduate", to: "students.adjuncts" } )
70+
71+
If a collection is specified in the filter, you can drop it, but you
72+
cannot rename it to remove it from the filter.
73+
74+
.. code-block:: shell
75+
:copyable: false
76+
77+
// This code produces an error and mongosync stops syncing
78+
use admin
79+
db.runCommand( { renameCollection: "students.graduate", to: "students.notAFilteredCollection" } )
80+
81+
When the whole target database is included in the filter, you can rename
82+
collections to add them to the filter:
83+
84+
- Source collection is specified in the filter
85+
86+
.. code-block:: shell
87+
88+
use admin
89+
db.runCommand( { renameCollection: "students.adjuncts", to: "staff.adjuncts" } )
90+
91+
- Source collection is not specified in the filter
92+
93+
.. code-block:: shell
94+
95+
use admin
96+
db.runCommand( { renameCollection: "prospects.current", to: "staff.newHires" } )
97+
98+
You can also rename collections in the source database when the whole
99+
target database is in the filter:
100+
101+
.. code-block:: shell
102+
103+
use admin
104+
db.runCommand( { renameCollection: "staff.employees", to: "staff.onPayroll" } )
105+
106+
.. important::
107+
108+
If you anticipate renaming collections, consider adding the entire
109+
database to the filter rather than specifying individual collections.
110+

‎source/includes/example-filter-collection.rst‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
.. include:: /includes/intro-start-api-example-intro.rst
2+
3+
``cluster0`` contains the ``sales``, ``marketing``, and
4+
``engineering`` databases.
5+
6+
The ``sales`` database contains the ``EMEA``, ``APAC``, and ``AMER``
7+
collections.
8+
9+
The ``includeNamespaces`` array in this example defines a filter on two
10+
of the databases, ``sales`` and ``marketing``.
11+
12+
The ``sales`` database also filters on the ``EMEA`` and ``APAC``
13+
collections.
14+
15+
.. code-block:: javascript
16+
17+
"includeNamespaces" : [
18+
{
19+
"database" : "sales",
20+
"collections": [ "EMEA", "APAC" ]
21+
},
22+
{
23+
"database" : "marketing"
24+
}
25+
]
26+
27+
After you call the ``/start`` API with this filter in place,
28+
``mongosync``:
29+
30+
- Syncs all of the collections in the ``marketing`` database
31+
- Filters out the ``engineering`` database
32+
- Syncs the ``EMEA`` and ``APAC`` collections from the ``sales``
33+
database
34+
- Filters out the ``AMER`` collection
35+

‎source/includes/intro-start-api-example-intro.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
The following example starts a sync job between ``cluster0`` and
2+
``cluster1``. The source cluster is ``cluster0`` and the destination
3+
cluster is ``cluster1``.

‎source/includes/limitations-filtering.rst‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
- Filtering is not supported with :ref:`reversible sync
2+
<c2c-api-reverse>`.
3+
- The destination cluster must not contain user data prior to starting.
4+
- The destination cluster must not contain the
5+
``mongosync_reserved_for_internal_use`` system database prior to
6+
starting.
7+
- You cannot modify a filter that is in use. To create a new filter,
8+
see: :ref:`c2c-change-filter`.
9+
- You can only rename collections in certain situations. For more
10+
details see: :ref:`c2c-filter-renaming-collections`.
11+
- If a filter includes a :ref:`view <views-landing-page>` but not the
12+
base collection, only the view is replicated.
13+
- You cannot specify system collections or system databases in a filter.
14+
- Operations that use the :pipeline:`$out` aggregation stage are only
15+
supported if the entire database is specified in the filter. You
16+
cannot limit the filter to a collection within the database. See:
17+
:ref:`c2c-filter-with-out`.
18+

‎source/reference.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Reference
1111
/reference/configuration
1212
/reference/api
1313
/reference/mongosync-states
14+
/reference/collection-level-filtering
1415
/reference/oplog-sizing
1516
/reference/disaster-recovery
1617
/reference/limitations

‎source/reference/api/start.txt‎

Lines changed: 72 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,40 @@ Request Body Parameters
9191
- Required
9292
- Name of the destination cluster.
9393

94+
* - ``enableUserWriteBlocking``
95+
- boolean
96+
- Optional
97+
- If set to ``true``, blocks writes on the destination cluster
98+
while the synchronization is in progress. After the
99+
synchronization is committed to the destination cluster, the
100+
original source cluster blocks writes and the destination cluster
101+
accepts writes.
102+
103+
Default value is ``false``.
104+
105+
* - ``includeNamespaces``
106+
- array
107+
- Optional
108+
- Filters the databases or collections to sync. The filter syntax
109+
is:
110+
111+
.. include:: /includes/api/facts/includeNamespaces-syntax.rst
112+
113+
.. versionadded:: 1.1
114+
115+
If you configure a filter on a source cluster that has multiple
116+
databases, ``mongosync`` only syncs the databases specified in
117+
the filter definition. ``mongosync`` will not sync the other
118+
existing databases.
119+
120+
If you want to modify the filter to add a newly created database,
121+
you have to :ref:`restart the filtered sync <c2c-change-filter>`
122+
from the beginning.
123+
124+
For more details, see: :ref:`c2c-filtered-sync`.
125+
126+
For current limitations, see: :ref:`c2c-filtering-limitations`.
127+
94128
* - ``reversible``
95129
- boolean
96130
- Optional
@@ -118,18 +152,6 @@ Request Body Parameters
118152

119153
.. versionadded:: 1.1
120154

121-
* - ``enableUserWriteBlocking``
122-
- boolean
123-
- Optional
124-
- If set to ``true``, blocks writes on the destination cluster
125-
while the synchronization is in progress. After the
126-
synchronization is committed to the destination cluster, the
127-
original source cluster blocks writes and the destination cluster
128-
accepts writes.
129-
130-
Default value is ``false``.
131-
132-
133155
.. _c2c-api-start-sharding:
134156

135157
Sharding Parameters
@@ -162,8 +184,8 @@ The ``sharding`` option has the following parameters:
162184
during sync.
163185

164186
Collections not included in this array sync to unsharded
165-
collections on the destination cluster. If set with an empty array,
166-
no collections are sharded.
187+
collections on the destination cluster. If set with an empty
188+
array, no collections are sharded.
167189

168190
* - | ``shardingEntries``
169191
| ``.collection``
@@ -190,18 +212,18 @@ The ``sharding`` option has the following parameters:
190212

191213
``mongosync`` throws an error if the ``sharding`` option is not set when
192214
syncing from a replica set to a sharded cluster. ``mongosync`` also
193-
throws an error if the ``sharding`` option is set with any other configuration.
215+
throws an error if the ``sharding`` option is set with any other
216+
configuration.
194217

195218
Response
196219
--------
197220

198221
.. include:: /includes/api/tables/basic-response.rst
199222

200-
Example 1 - Start a Standard Sync Job
201-
-------------------------------------
223+
Example: Start a Sync Job
224+
-------------------------
202225

203-
The following example starts a synchronization job where ``cluster0`` is
204-
the source and ``cluster1`` is the destination.
226+
.. include:: /includes/intro-start-api-example-intro.rst
205227

206228
Request
207229
~~~~~~~
@@ -216,12 +238,14 @@ Response
216238
:language: json
217239
:copyable: false
218240

219-
Example 2 - Start a Reversible Sync Job
220-
---------------------------------------
241+
Example: Start a Reversible Sync Job
242+
------------------------------------
221243

222-
The following example starts a synchronization job where ``cluster0`` is
223-
the source and ``cluster1`` is the destination. The ``reversible`` and
224-
``enableUserWriteBlocking`` fields allow the sync to be reversed.
244+
.. include:: /includes/intro-start-api-example-intro.rst
245+
246+
The ``reversible`` and ``enableUserWriteBlocking`` fields allow the sync
247+
to be reversed. To reverse the sync direction, see: :ref:`reverse
248+
<c2c-api-reverse>`.
225249

226250
Request
227251
~~~~~~~
@@ -236,18 +260,39 @@ Response
236260
:language: json
237261
:copyable: false
238262

239-
Example 3 - Start Sync from Replica Set to Sharded Cluster
240-
----------------------------------------------------------
263+
264+
Example: Start a Filtered Sync Job
265+
----------------------------------
266+
267+
.. include:: /includes/example-filter-collection.rst
268+
269+
The ``includeNamespaces`` option creates a filter. To filter the sync,
270+
see: :ref:`c2c-filtered-sync`
241271

242272
Request
243273
~~~~~~~
244274

245-
.. literalinclude:: /includes/api/requests/start-rs-shard.sh
275+
.. literalinclude:: /includes/api/requests/start-filtered.sh
246276
:language: shell
247277

248278
Response
249279
~~~~~~~~
250280

281+
.. literalinclude:: /includes/api/responses/success.json
282+
:language: json
283+
:copyable: false
284+
285+
Example: Start Sync from Replica Set to Sharded Cluster
286+
-------------------------------------------------------
287+
288+
Request
289+
~~~~~~~
290+
291+
.. literalinclude:: /includes/api/requests/start-rs-shard.sh
292+
293+
Response
294+
~~~~~~~~
295+
251296
.. literalinclude:: /includes/api/responses/success.json
252297
:language: json
253298
:copyable: false

0 commit comments

Comments
 (0)