|
1 |
| -================ |
2 |
| -Multiple Sources |
3 |
| -================ |
| 1 | +.. _source-usage-example-multiple-sources: |
| 2 | + |
| 3 | +====================================== |
| 4 | +Listen for Changes on Multiple Sources |
| 5 | +====================================== |
| 6 | + |
| 7 | +.. default-domain:: mongodb |
| 8 | + |
| 9 | +This usage example demonstrates how to configure a MongoDB Kafka source |
| 10 | +connector to listen for change events on multiple MongoDB collections, and |
| 11 | +publish them to a Kafka topic. |
| 12 | + |
| 13 | +If you need your connector to listen for change events on a more particular |
| 14 | +set of databases and collections, you can use a **pipeline**. A pipeline is a |
| 15 | +MongoDB aggregation pipeline composed of instructions to the database to |
| 16 | +filter or transform data. See the next section for examples of how to |
| 17 | +configure your connector ``pipeline`` setting to match multiple database and |
| 18 | +collection names using a regular expression. |
| 19 | + |
| 20 | +.. note:: |
| 21 | + |
| 22 | + The ``database`` and ``collection`` configuration settings also affect |
| 23 | + which databases and collections on which the connector listens for change |
| 24 | + events. To learn more about these settings, see the |
| 25 | + :ref:`<source-configuration-mongodb-connection>` guide. |
| 26 | + |
| 27 | +Examples |
| 28 | +-------- |
| 29 | + |
| 30 | +The following examples show you how to use an aggregation pipeline to select |
| 31 | +specific database or collection names on which to listen for change events. |
| 32 | + |
| 33 | +Include Change Events from Multiple Databases |
| 34 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 35 | + |
| 36 | +You can define an aggregation pipeline to select only change events on |
| 37 | +multiple databases by specifying the following in the ``pipeline`` |
| 38 | +setting: |
| 39 | + |
| 40 | +- A ``$match`` aggregation operator |
| 41 | +- The ``ns.db``, field which identifies the database part of the namespace |
| 42 | +- The ``$regex`` operator and a regular expression that matches the database |
| 43 | + names |
| 44 | + |
| 45 | +The following sample configuration shows how you can set your source connector |
| 46 | +to listen for change events on the ``sandbox`` and ``firewall`` databases: |
| 47 | + |
| 48 | +.. code-block:: ini |
| 49 | + |
| 50 | + pipeline=[{"$match": {"ns.db": {"$regex": "/^(sandbox|firewall)$/"}}}] |
| 51 | + |
| 52 | +Exclude Change Events from Multiple Collections |
| 53 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 54 | + |
| 55 | +You can define an aggregation pipeline to ignore change events on |
| 56 | +multiple collections by specifying the following in the ``pipeline`` |
| 57 | +setting: |
| 58 | + |
| 59 | +- A ``$match`` aggregation operator |
| 60 | +- The ``ns.coll`` field, which identifies the collection part of the namespace |
| 61 | +- The ``$regex`` operator and a regular expression that matches the |
| 62 | + collection names |
| 63 | +- The ``$not`` operator which instructs the enclosing ``$regex`` operator to |
| 64 | + match everything the regular expression does not match |
| 65 | + |
| 66 | +The following sample configuration shows how you can set your source connector |
| 67 | +to filter out change events that originate from all collections named |
| 68 | +"hyperspace" in any database: |
| 69 | + |
| 70 | +.. code-block:: ini |
| 71 | + |
| 72 | + pipeline=[{"$match": {"ns.coll": {"$regex": {"$not": "/^hyperspace$/"}}}}] |
| 73 | + |
| 74 | +Additional Information |
| 75 | +---------------------- |
| 76 | + |
| 77 | +- The :manual:`$match aggregation operator </reference/operator/aggregation/match/>` |
| 78 | +- :manual:`MongoDB change events </change-events/>` |
| 79 | +- :manual:`MongoDB namespace </reference/limits/#faq-dev-namespace>` |
| 80 | +- Regular expression syntax using the `Patterns class <https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html>`__ |
| 81 | +- :manual:`</reference/operator/query/not/#-not-and-regular-expressions>` |
4 | 82 |
|
5 |
| -TODO: |
|
0 commit comments