Skip to content

Commit 8098ebc

Browse files
Chris Choschmalliso
authored andcommitted
DOCS-13647: Add dead letter queue configuration information (#19)
1 parent bc1abef commit 8098ebc

File tree

1 file changed

+83
-0
lines changed

1 file changed

+83
-0
lines changed

source/kafka-sink-properties.txt

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,3 +256,86 @@ the following behavior for data consumed from ``topicA``:
256256
up to 100.
257257
- Generate a UUID to be stored in the ``_id`` field for each new document.
258258
- Omit fields ``k2`` and ``k4`` from the value projection using a blacklist.
259+
260+
261+
Dead Letter Queue Configuration Settings
262+
----------------------------------------
263+
264+
Kafka connectors send messages that cannot be processed to the **dead letter
265+
queue**. The connector sends invalid messages to this queue in order to
266+
allow manual inspection, updates, and re-submission for processing. It is
267+
disabled by default and requires restarting the connector for the changes to
268+
the configuration to take effect.
269+
270+
The following is an example configuration for enabling the dead letter queue
271+
topic ``example.deadletterqueue`` that specifies that both the invalid
272+
message as well as the log file should record invalid messages, and that
273+
context headers should be included in the dead letter queue messages.
274+
275+
.. code-block:: properties
276+
277+
errors.tolerance=all
278+
errors.log.enable=true
279+
errors.log.include.messages=true
280+
errors.deadletterqueue.topic.name=example.deadletterqueue
281+
errors.deadletterqueue.context.headers.enable=true
282+
283+
284+
The table below describes the configuration settings relevant to the dead
285+
letter queue.
286+
287+
.. list-table::
288+
:header-rows: 1
289+
:stub-columns: 1
290+
:widths: 2 1 4
291+
292+
293+
* - Name
294+
- Type
295+
- Description
296+
297+
* - errors.tolerance
298+
- string
299+
- | Which errors to allow before the message processing task fails. When the connector encounters an error when tolerance is set to "none", it reports an error and blocks further processing of the rest of the records. When set to "all", the connector silently ignores any bad messages.
300+
|
301+
| **Default:** "none"
302+
| **Accepted Values**: "none" or "all"
303+
304+
* - errors.log.enable
305+
- boolean
306+
- | Whether details of failed operations should be written to the log file. When set to **true**, both errors that are tolerated (determined by the **errors.tolerance** setting) and not tolerated are written. When set to **false**, errors that are tolerated are omitted.
307+
|
308+
| **Default:** false
309+
| **Accepted Values**: ``true`` or ``false``
310+
311+
* - errors.log.include.messages
312+
- boolean
313+
- | Whether to include the invalid message, including record keys, values, and headers, in the error log file.
314+
|
315+
| **Default:** false
316+
| **Accepted Values**: ``true`` or ``false``
317+
318+
* - errors.deadletterqueue.topic.name
319+
- string
320+
- | Name of topic to use as the dead letter queue. If blank, none of the invalid messages are written to the dead letter queue.
321+
|
322+
| **Default:** ""
323+
| **Accepted Values**: A valid Kafka topic name
324+
325+
* - errors.deadletterqueue.context.headers.enable
326+
- boolean
327+
- | Whether to provide the context headers in messages written to the dead letter queue.
328+
|
329+
| **Default:** false
330+
| **Accepted Values**: ``true`` or ``false``
331+
332+
* - errors.deadletterqueue.topic.replication.factor
333+
- integer
334+
- | The number of nodes on which to replicate the dead letter queue topic. If running a single-node Kafka cluster, this must be set to 1.
335+
|
336+
| **Default:** 3
337+
| **Accepted Values**: A valid number
338+
339+
See the `Confluent Sink Connector documentation <https://docs.confluent.io/current/installation/configuration/connect/sink-connect-configs.html>`__
340+
for more information on these settings.
341+

0 commit comments

Comments
 (0)