Skip to content

Commit 6c7e009

Browse files
rustagirrustagir
authored
DOCSP-26355: startup mode (#59)
* DOCSP-26355-startup-mode-properties * first pass fixes * add old refs back * VK technical review 1 * fixes * RL PR fixes 1 * small fixes * MW PR fixes 1
1 parent b32c092 commit 6c7e009

File tree

7 files changed

+213
-31
lines changed

7 files changed

+213
-31
lines changed

‎source/includes/usage-examples/copy/source.properties

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@ connector.class=com.mongodb.kafka.connect.MongoSourceConnector
22
connection.uri=<your production MongoDB connection uri>
33
database=shopping
44
collection=customers
5-
copy.existing=true
6-
copy.existing.pipeline=[{ "$match": { "country": "Mexico" } }]
5+
startup.mode=copy_existing
6+
startup.mode.copy.existing.pipeline=[{ "$match": { "country": "Mexico" } }]

‎source/source-connector/configuration-properties.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ See the following categories for a list of related configuration properties:
4747
- Specify the format of the data the connector publishes to your
4848
Kafka topic.
4949

50-
* - :ref:`Copy Existing Properties <source-configuration-copy-existing>`
50+
* - :ref:`Startup Properties <source-configuration-startup>`
5151
- Specify what data the connector should convert to Change Stream
5252
events.
5353

@@ -68,7 +68,7 @@ for more information on these settings.
6868
Kafka Topic </source-connector/configuration-properties/kafka-topic>
6969
Change Stream </source-connector/configuration-properties/change-stream>
7070
Output Format </source-connector/configuration-properties/output-format>
71-
Copy Existing </source-connector/configuration-properties/copy-existing>
71+
Startup </source-connector/configuration-properties/startup>
7272
Error Handling and Resuming from Interruption </source-connector/configuration-properties/error-handling>
7373
All Properties </source-connector/configuration-properties/all-properties>
7474

‎source/source-connector/configuration-properties/all-properties.txt

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -78,19 +78,19 @@ To view only the options related to the format of your output, see the
7878
:start-after: source-configuration-output-format-table-start
7979
:end-before: source-configuration-output-format-table-end
8080

81-
Copy Existing
82-
-------------
81+
Startup
82+
-------
8383

84-
.. include:: /source-connector/configuration-properties/copy-existing.txt
85-
:start-after: source-configuration-copy-existing-description-start
86-
:end-before: source-configuration-copy-existing-description-end
84+
.. include:: /source-connector/configuration-properties/startup.txt
85+
:start-after: source-configuration-startup-description-start
86+
:end-before: source-configuration-startup-description-end
8787

88-
To view only the options related to copying data, see the
89-
:ref:`<source-configuration-copy-existing>` page.
88+
To view only the options related to startup, see the
89+
:ref:`<source-configuration-startup>` page.
9090

91-
.. include:: /source-connector/configuration-properties/copy-existing.txt
92-
:start-after: source-configuration-copy-existing-table-start
93-
:end-before: source-configuration-copy-existing-table-end
91+
.. include:: /source-connector/configuration-properties/startup.txt
92+
:start-after: source-configuration-startup-table-start
93+
:end-before: source-configuration-startup-table-end
9494

9595
Error Handling and Resuming from Interruption
9696
---------------------------------------------

‎source/source-connector/configuration-properties/copy-existing.txt

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,14 @@ Overview
1717

1818
.. _source-configuration-copy-existing-description-start:
1919

20+
.. important:: ``copy.existing*`` Properties are Deprecated
21+
22+
Starting in Version 1.9 of the {+mkc+}, ``copy.existing*`` properties
23+
are deprecated and may be removed in a future release. You should use
24+
``startup.mode*`` properties to configure the copy existing feature.
25+
To learn about ``startup.mode*`` settings, see
26+
:ref:`source-configuration-startup`.
27+
2028
Use the following configuration settings to enable the copy existing
2129
feature which converts MongoDB collections into Change Stream events.
2230

@@ -63,19 +71,18 @@ Settings
6371
| **Description:**
6472
| Regular expression the connector uses to match namespaces from
6573
which to copy data. A namespace describes the MongoDB database name
66-
and collection separated by a period, e.g.
67-
``databaseName.collectionName``.
74+
and collection separated by a period (for example, ``databaseName.collectionName``).
6875

6976
.. example::
7077

71-
In the following example, the regular expression setting matches
72-
collections that start with "page" in the "stats" database.
78+
In the following example, the regular-expression setting matches
79+
collections that start with "page" in the ``stats`` database.
7380

7481
.. code-block:: none
7582

7683
copy.existing.namespace.regex=stats\.page.*
7784

78-
The "\\" character in the example above escapes the "." character
85+
The "\" character in the example above escapes the "." character
7986
that follows it in the regular expression. For more information on
8087
how to build regular expressions, see the Java API documentation on
8188
`Patterns <https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html>`__.
@@ -125,9 +132,8 @@ Settings
125132
- | **Type:** boolean
126133
|
127134
| **Description:**
128-
| When set to ``true``,
129-
the connector sets the copy existing aggregation
130-
to use temporary disk storage.
135+
| When set to ``true``, the connector uses temporary disk storage
136+
for the copy existing aggregation.
131137
| **Default**: ``true``
132138

133-
.. _source-configuration-copy-existing-table-end:
139+
.. _source-configuration-copy-existing-table-end:

‎source/source-connector/configuration-properties/startup.txt

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
.. _source-configuration-startup:
2+
3+
==================
4+
Startup Properties
5+
==================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
Overview
16+
--------
17+
18+
.. _source-configuration-startup-description-start:
19+
20+
Use the following configuration settings to configure startup of the
21+
source connector to convert MongoDB collections into Change Stream
22+
events.
23+
24+
.. _source-configuration-startup-description-end:
25+
26+
.. tip::
27+
28+
For an example using the copy existing feature, see the
29+
:ref:`<source-usage-example-copy-existing-data>` Usage Example.
30+
31+
.. include:: /includes/source-config-link.rst
32+
33+
Settings
34+
--------
35+
36+
.. _source-configuration-startup-table-start:
37+
38+
.. list-table::
39+
:header-rows: 1
40+
:widths: 40 60
41+
42+
* - Name
43+
- Description
44+
45+
* - | **startup.mode**
46+
- | **Type:** string
47+
|
48+
| **Description:**
49+
| Specifies how the connector should start up when there is no
50+
source offset available. Resuming a change stream requires a
51+
resume token, which the connector gets from the source offset.
52+
If no source offset is available, the connector may either
53+
ignore all or some of the existing source data, or may at first
54+
copy all existing source data and then continue with processing
55+
new data.
56+
|
57+
| If ``startup.mode=latest``, the connector ignores all existing
58+
source data.
59+
|
60+
| If ``startup.mode=timestamp``, the connector
61+
actuates ``startup.mode.timestamp.*`` properties. If no
62+
properties are configured, ``timestamp`` is equivalent to
63+
``latest``.
64+
|
65+
| If ``startup.mode=copy_existing``, the connector
66+
copies all existing source data to Change Stream events. This
67+
setting is equivalent to the deprecated setting ``copy.existing=true``.
68+
69+
.. include:: /includes/copy-existing-admonition.rst
70+
71+
| **Default**:``latest``
72+
| **Accepted Values**: ``latest``, ``timestamp``, ``copy_existing``
73+
74+
* - | **startup.mode.timestamp.start.at.operation.time**
75+
- | **Type:** string
76+
|
77+
| **Description:**
78+
| Actuated only if ``startup.mode=timestamp``. Specifies the
79+
starting point for the change stream. To learn more about
80+
Change Stream parameters, see the :rapid:`Server manual entry </reference/operator/aggregation/changeStream/>`.
81+
| **Default**: ``""``
82+
| **Accepted Values**:
83+
| * An integer number of seconds since the
84+
Epoch in decimal format (for example, ``30``)
85+
| * An instant in the ISO-8601 format with one second precision (for example,
86+
``1970-01-01T00:00:30Z``)
87+
| * A BSON Timestamp in the canonical extended JSON (v2) format
88+
(for example, ``{"$timestamp": {"t": 30, "i": 0}}``)
89+
90+
* - | **startup.mode.copy.existing.namespace.regex**
91+
- | **Type:** string
92+
|
93+
| **Description:**
94+
| Regular expression the connector uses to match namespaces from
95+
which to copy data. A namespace describes the MongoDB database name
96+
and collection separated by a period (for example, ``databaseName.collectionName``).
97+
98+
.. example::
99+
100+
In the following example, the regular-expression setting matches
101+
collections that start with "page" in the ``stats`` database.
102+
103+
.. code-block:: none
104+
105+
startup.mode.copy.existing.namespace.regex=stats\.page.*
106+
107+
The "\" character in the example above escapes the "." character
108+
that follows it in the regular expression. For more information on
109+
how to build regular expressions, see the Java API documentation on
110+
`Patterns <https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html>`__.
111+
112+
| **Default**: ``""``
113+
| **Accepted Values**: A valid regular expression
114+
115+
* - | **startup.mode.copy.existing.pipeline**
116+
- | **Type:** string
117+
|
118+
| **Description:**
119+
| An inline array of :manual:`pipeline operations </core/aggregation-pipeline/>`
120+
the connector runs when copying existing data. You can use this
121+
setting to filter the source collection and improve the use of
122+
indexes in the copying process.
123+
124+
.. example::
125+
126+
The following example shows how you can use the :manual:`$match </reference/operator/aggregation/match/>`
127+
aggregation operator to instruct the connector to copy only
128+
documents that contain a ``closed`` field with a value of ``false``.
129+
130+
.. code-block:: none
131+
132+
startup.mode.copy.existing.pipeline=[ { "$match": { "closed": "false" } } ]
133+
134+
| **Default**: ``""``
135+
| **Accepted Values**: Valid aggregation pipeline stages
136+
137+
* - | **startup.mode.copy.existing.max.threads**
138+
- | **Type:** int
139+
|
140+
| **Description:**
141+
| The maximum number of threads the connector can use to copy data.
142+
| **Default**: number of processors available in the environment
143+
| **Accepted Values**: An integer
144+
145+
* - | **startup.mode.copy.existing.queue.size**
146+
- | **Type:** int
147+
|
148+
| **Description:**
149+
| The size of the queue the connector can use when copying data.
150+
| **Default**: ``16000``
151+
| **Accepted Values**: An integer
152+
153+
* - | **startup.mode.copy.existing.allow.disk.use**
154+
- | **Type:** boolean
155+
|
156+
| **Description:**
157+
| When set to ``true``, the connector uses temporary disk storage
158+
for the copy existing aggregation.
159+
| **Default**: ``true``
160+
| **Accepted Values**: ``true`` or ``false``
161+
162+
.. _source-configuration-startup-table-end:

‎source/source-connector/usage-examples/copy-existing-data.txt

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ specifying the following configuration options in your source connector:
6262

6363
database=shopping
6464
collection=customers
65-
copy.existing=true
65+
startup.mode=copy_existing
6666

6767
Your source connector copies your collection by creating change event documents
6868
that describe inserting each document into your collection.
@@ -72,25 +72,25 @@ that describe inserting each document into your collection.
7272
To learn more about change event documents, see the
7373
:ref:`Change Streams <source-connector-fundamentals-change-event>` guide.
7474

75-
To learn more about the ``copy.existing`` option, see
76-
:ref:`<source-configuration-copy-existing>` in the {+mkc+}.
75+
To learn more about the ``startup.mode`` option, see
76+
:ref:`<source-configuration-startup>`.
7777

7878
.. _source-usage-example-copy-existing-data-mask-data:
7979

8080
Filter Data
8181
~~~~~~~~~~~
8282

8383
You can filter data by specifying an aggregation pipeline in the
84-
``copy.existing.pipeline`` option of your source connector configuration. The
84+
``startup.mode.copy.existing.pipeline`` option of your source connector configuration. The
8585
following configuration specifies an aggregation pipeline that matches all
8686
documents with "Mexico" in the ``country`` field:
8787

8888
.. code-block:: properties
8989

90-
copy.existing.pipeline=[{ "$match": { "country": "Mexico" } }]
90+
startup.mode.copy.existing.pipeline=[{ "$match": { "country": "Mexico" } }]
9191

92-
To learn more about the ``copy.existing.pipeline`` option, see
93-
:ref:`<source-configuration-copy-existing>` in the {+mkc+}.
92+
To learn more about the ``startup.mode.copy.existing.pipeline`` option, see
93+
:ref:`<source-configuration-startup>`.
9494

9595
To learn more about aggregation pipelines, see the following resources:
9696

@@ -101,7 +101,7 @@ To learn more about aggregation pipelines, see the following resources:
101101
Specify the Configuration
102102
~~~~~~~~~~~~~~~~~~~~~~~~~
103103

104-
Your source connector configuration to copy the ``customers`` collection should
104+
Your final source connector configuration to copy the ``customers`` collection should
105105
look like this:
106106

107107
.. literalinclude:: /includes/usage-examples/copy/source.properties

‎source/whats-new.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ What's New
1212

1313
Learn what's new by version:
1414

15+
* :ref:`Version 1.9 <kafka-connector-whats-new-1.9>`
1516
* :ref:`Version 1.8.1 <kafka-connector-whats-new-1.8.1>`
1617
* :ref:`Version 1.8 <kafka-connector-whats-new-1.8>`
1718
* :ref:`Version 1.7 <kafka-connector-whats-new-1.7>`
@@ -24,6 +25,19 @@ Learn what's new by version:
2425
* :ref:`Version 1.1 <kafka-connector-whats-new-1.1>`
2526
* :ref:`Version 1.0 <kafka-connector-whats-new-1.0>`
2627

28+
.. _kafka-connector-whats-new-1.9:
29+
30+
What's New in 1.9
31+
-----------------
32+
33+
- Introduced ``startup.mode=timestamp`` setting that allows you to
34+
start a Change Stream at a specific timestamp by setting the new
35+
``startup.mode.timestamp.start.at.operation.time`` property.
36+
- Deprecated the ``copy.existing`` property and all ``copy.existing.*``
37+
properties. You should use ``startup.mode=copy_existing`` and
38+
``startup.mode.copy.existing.*`` properties to configure the copy existing
39+
feature.
40+
2741
.. _kafka-connector-whats-new-1.8.1:
2842

2943
What's New in 1.8.1

0 commit comments

Comments
 (0)