Skip to content

Commit b69ce54

Browse files
Chris Choschmalliso
authored andcommitted
DOCSP-15810: multiple sources usage example (#165)
* DOCSP-15810: multiple sources usage example
1 parent 159f385 commit b69ce54

File tree

2 files changed

+83
-6
lines changed

2 files changed

+83
-6
lines changed

source/source-connector/configuration-properties/change-stream.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,8 @@ Settings
4444

4545
.. tip:: Additional Examples
4646

47-
- (TODO: link to Custom pipeline example)
48-
- (TODO: link to Multiple source example)
47+
- :ref:`<source-usage-example-custom-pipeline>`
48+
- :ref:`<source-usage-example-multiple-sources>`
4949

5050
| **Default**: ``[]``
5151
| **Accepted Values**: Valid aggregation pipeline stage

source/source-connector/usage-examples/multiple-sources.txt

Lines changed: 81 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,82 @@
1-
================
2-
Multiple Sources
3-
================
1+
.. _source-usage-example-multiple-sources:
2+
3+
======================================
4+
Listen for Changes on Multiple Sources
5+
======================================
6+
7+
.. default-domain:: mongodb
8+
9+
This usage example demonstrates how to configure a MongoDB Kafka source
10+
connector to listen for change events on multiple MongoDB collections, and
11+
publish them to a Kafka topic.
12+
13+
If you need your connector to listen for change events on a more particular
14+
set of databases and collections, you can use a **pipeline**. A pipeline is a
15+
MongoDB aggregation pipeline composed of instructions to the database to
16+
filter or transform data. See the next section for examples of how to
17+
configure your connector ``pipeline`` setting to match multiple database and
18+
collection names using a regular expression.
19+
20+
.. note::
21+
22+
The ``database`` and ``collection`` configuration settings also affect
23+
which databases and collections on which the connector listens for change
24+
events. To learn more about these settings, see the
25+
:ref:`<source-configuration-mongodb-connection>` guide.
26+
27+
Examples
28+
--------
29+
30+
The following examples show you how to use an aggregation pipeline to select
31+
specific database or collection names on which to listen for change events.
32+
33+
Include Change Events from Multiple Databases
34+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
35+
36+
You can define an aggregation pipeline to select only change events on
37+
multiple databases by specifying the following in the ``pipeline``
38+
setting:
39+
40+
- A ``$match`` aggregation operator
41+
- The ``ns.db``, field which identifies the database part of the namespace
42+
- The ``$regex`` operator and a regular expression that matches the database
43+
names
44+
45+
The following sample configuration shows how you can set your source connector
46+
to listen for change events on the ``sandbox`` and ``firewall`` databases:
47+
48+
.. code-block:: ini
49+
50+
pipeline=[{"$match": {"ns.db": {"$regex": "/^(sandbox|firewall)$/"}}}]
51+
52+
Exclude Change Events from Multiple Collections
53+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
54+
55+
You can define an aggregation pipeline to ignore change events on
56+
multiple collections by specifying the following in the ``pipeline``
57+
setting:
58+
59+
- A ``$match`` aggregation operator
60+
- The ``ns.coll`` field, which identifies the collection part of the namespace
61+
- The ``$regex`` operator and a regular expression that matches the
62+
collection names
63+
- The ``$not`` operator which instructs the enclosing ``$regex`` operator to
64+
match everything the regular expression does not match
65+
66+
The following sample configuration shows how you can set your source connector
67+
to filter out change events that originate from all collections named
68+
"hyperspace" in any database:
69+
70+
.. code-block:: ini
71+
72+
pipeline=[{"$match": {"ns.coll": {"$regex": {"$not": "/^hyperspace$/"}}}}]
73+
74+
Additional Information
75+
----------------------
76+
77+
- The :manual:`$match aggregation operator </reference/operator/aggregation/match/>`
78+
- :manual:`MongoDB change events </change-events/>`
79+
- :manual:`MongoDB namespace </reference/limits/#faq-dev-namespace>`
80+
- Regular expression syntax using the `Patterns class <https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html>`__
81+
- :manual:`</reference/operator/query/not/#-not-and-regular-expressions>`
482

5-
TODO:

0 commit comments

Comments
 (0)