Skip to content

Commit 7e7f3b0

Browse files
lmkerbey-mdblmkerbey-mdb
authored
(DOCSP-36026) Add Atlas SP mongosh commands. (#7001)
* (DOCSP-36026) Add Atlas SP mongosh commands. * (DOCSP-36026) Formatting fixes. * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) Various corrections to createStreamProcessor * (DOCSP-36026) Various fixes to create and list * (DOCSP-36026) Add ToC Page + various fixes * (DOCSP-36026) ToC fixes. * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) Fixes to sample * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) * (DOCSP-36026) Copy review. * (DOCSP-36026)
1 parent 5996801 commit 7e7f3b0

11 files changed

+1253
-0
lines changed

‎snooty.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,7 @@ toc_landing_pages = [
154154
"/reference/inconsistency-type",
155155
"/reference/method",
156156
"/reference/method/js-atlas-search",
157+
"/reference/method/js-atlas-streams",
157158
"/reference/method/js-bulk",
158159
"/reference/method/js-client-side-field-level-encryption",
159160
"/reference/method/js-collection",

‎source/reference/method.txt

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,69 @@ Atlas Search Index Methods
3434

3535
/reference/method/js-atlas-search
3636

37+
Atlas Stream Processing Methods
38+
------------------------------------------------------
39+
40+
:atlas:`Atlas Stream Processors
41+
</atlas-sp/overview/#mongodb-expression-exp.Stream-Processor>`
42+
let you perform aggregation operations against streams of
43+
continuous data using the same data model and query API that
44+
you use with at-rest data.
45+
46+
Use the following methods to manage Stream Processors:
47+
48+
.. important::
49+
50+
The following methods can only be run on deployments hosted on
51+
:atlas:`MongoDB Atlas </>`.
52+
53+
.. list-table::
54+
:widths: 30 70
55+
:header-rows: 1
56+
57+
* - Name
58+
59+
- Description
60+
61+
* - :method:`sp.createStreamProcessor()`
62+
63+
- Creates a stream processor.
64+
65+
* - :method:`sp.listStreamProcessors()`
66+
67+
- Lists all existing stream processors on the current stream
68+
processing instance.
69+
70+
* - :method:`sp.process()`
71+
72+
- Creates an ephemeral stream processor.
73+
74+
* - :method:`sp.processor.drop()`
75+
76+
- Deletes an existing stream processor.
77+
78+
* - :method:`sp.processor.sample()`
79+
80+
- Returns an array of sampled results from a currently running stream processor.
81+
82+
* - :method:`sp.processor.start()`
83+
84+
- Starts an existing stream processor.
85+
86+
* - :method:`sp.processor.stats()`
87+
88+
- Returns statistics summarizing an existing stream processor.
89+
90+
* - :method:`sp.processor.stop()`
91+
92+
- Stops a currently running stream processor.
93+
94+
.. toctree::
95+
:titlesonly:
96+
:hidden:
97+
98+
/reference/method/js-atlas-streams
99+
37100
Collection
38101
----------
39102

‎source/reference/method/js-atlas-streams.txt

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
.. _doc-stream-methods:
2+
3+
===============================
4+
Atlas Stream Processing Methods
5+
===============================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 1
13+
:class: singlecol
14+
15+
.. note:: ``mongosh`` Methods
16+
17+
.. include:: /includes/fact-mongosh-shell-method-toc.rst
18+
19+
:atlas:`Atlas Stream Processors
20+
</atlas-sp/overview/#mongodb-expression-exp.Stream-Processor>`
21+
let you perform aggregation operations against streams of
22+
continuous data using the same data model and query API that
23+
you use with at-rest data.
24+
25+
Use the following methods to manage Stream Processors
26+
27+
.. important::
28+
29+
The following methods can only be run on deployments hosted on
30+
:atlas:`MongoDB Atlas </>`.
31+
32+
.. include:: /includes/extracts/methods-toc-explanation.rst
33+
34+
.. list-table::
35+
:widths: 30 70
36+
:header-rows: 1
37+
38+
* - Name
39+
40+
- Description
41+
42+
* - :method:`sp.createStreamProcessor()`
43+
44+
- Creates a stream processor.
45+
46+
* - :method:`sp.listStreamProcessors()`
47+
48+
- Lists all existing stream processors on the current stream
49+
processing instance.
50+
51+
* - :method:`sp.process()`
52+
53+
- Creates an ephemeral stream processor.
54+
55+
* - :method:`sp.processor.drop()`
56+
57+
- Deletes an existing stream processor.
58+
59+
* - :method:`sp.processor.sample()`
60+
61+
- Returns an array of sampled results from a currently running stream processor.
62+
63+
* - :method:`sp.processor.start()`
64+
65+
- Starts an existing stream processor.
66+
67+
* - :method:`sp.processor.stats()`
68+
69+
- Returns statistics summarizing an existing stream processor.
70+
71+
* - :method:`sp.processor.stop()`
72+
73+
- Stops a currently running stream processor.
74+
75+
.. toctree::
76+
:titlesonly:
77+
:hidden:
78+
79+
/reference/method/sp.createStreamProcessor
80+
/reference/method/sp.listStreamProcessors
81+
/reference/method/sp.process
82+
/reference/method/sp.processor.drop
83+
/reference/method/sp.processor.sample
84+
/reference/method/sp.processor.start
85+
/reference/method/sp.processor.stats
86+
/reference/method/sp.processor.stop

‎source/reference/method/sp.createStreamProcessor.txt

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
==========================
2+
sp.createStreamProcessor()
3+
==========================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Definition
14+
-----------
15+
16+
.. method:: sp.createStreamProcessor()
17+
18+
.. versionadded:: 7.0
19+
20+
Creates a :atlas:`Stream Processor
21+
</atlas-sp/overview/#mongodb-expression-exp.Stream-Processor>` on
22+
the current :atlas:`Stream Processing Instance
23+
</atlas-sp/overview/#mongodb-expression-exp.Stream-Processing-Instance>`.
24+
25+
Syntax
26+
-----------
27+
28+
The :method:`sp.createStreamProcessor()` method has the following
29+
syntax:
30+
31+
.. code-block:: json
32+
33+
sp.createStreamProcessor(
34+
<name>,
35+
[
36+
<pipeline>
37+
],
38+
{
39+
<options>
40+
}
41+
)
42+
43+
Command Fields
44+
---------------------------
45+
46+
``sp.createStreamProcessor()`` takes these fields:
47+
48+
.. list-table::
49+
:header-rows: 1
50+
:widths: 20 20 20 40
51+
52+
* - Field
53+
- Type
54+
- Necessity
55+
- Description
56+
57+
* - ``name``
58+
- string
59+
- Required
60+
- Logical name for the stream processor. This must be unique
61+
within the stream processing instance.
62+
63+
* - ``pipeline``
64+
- array
65+
- Required
66+
- :ref:`Stream aggregation pipeline <stream-aggregation>` you
67+
want to apply to your streaming data.
68+
69+
* - ``options``
70+
- object
71+
- Optional
72+
- Object defining various optional settings for your stream
73+
processor.
74+
75+
* - ``options.dlq``
76+
- object
77+
- Conditional
78+
- Object assigning a
79+
:term:`dead letter queue` for your stream processing instance.
80+
This field is necessary if you define the ``options`` field.
81+
82+
* - ``options.dlq.connectionName``
83+
- string
84+
- Conditional
85+
- Label that identifies a connection in your
86+
connection registry. This connection must reference an
87+
Atlas cluster. This field is necessary if you define the
88+
``options.dlq`` field.
89+
90+
* - ``options.dlq.db``
91+
- string
92+
- Conditional
93+
- Name of an Atlas database on the cluster specified
94+
in ``options.dlq.connectionName``. This field is necessary if
95+
you define the ``options.dlq`` field.
96+
97+
* - ``options.dlq.coll``
98+
- string
99+
- Conditional
100+
- Name of a collection in the database specified in
101+
``options.dlq.db``. This field is necessary if you
102+
define the ``options.dlq`` field.
103+
104+
105+
Behavior
106+
---------------
107+
108+
``sp.createStreamProcessor()`` creates a persistent, named stream
109+
processor on the current stream processing instance. You can
110+
initialize this stream processor with
111+
:method:`sp.processor.start()`. If you try to create a stream
112+
processor with the same name as an existing stream processor,
113+
``mongosh`` will return an error.
114+
115+
Access Control
116+
------------------------
117+
118+
The user running ``sp.createStreamProcessor()`` must have the
119+
:atlasrole:`atlasAdmin` role.
120+
121+
Example
122+
----------------
123+
124+
The following example creates a stream processor named ``solarDemo``
125+
which ingests data from the ``sample_stream_solar`` connection. The
126+
processor excludes all documents where the value of the ``device_id``
127+
field is ``device_8``, passing the rest to a :atlas:`tumbling window
128+
</atlas-sp/overview/#tumbling-windows>` with a 10-second
129+
duration. Each window groups the documents it receives, then returns
130+
various useful statistics of each group. The stream processor then
131+
merges these records to ``solar_db.solar_coll`` over the ``mongodb1``
132+
connection.
133+
134+
.. code-block:: json
135+
:copyable: true
136+
137+
sp.createStreamProcessor(
138+
'solarDemo',
139+
[
140+
{
141+
$source: {
142+
connectionName: 'sample_stream_solar',
143+
timeField: {
144+
$dateFromString: {
145+
dateString: '$timestamp'
146+
}
147+
}
148+
}
149+
},
150+
{
151+
$match: {
152+
$expr: {
153+
$ne: [
154+
"$device_id",
155+
"device_8"
156+
]
157+
}
158+
}
159+
},
160+
{
161+
$tumblingWindow: {
162+
interval: {
163+
size: NumberInt(10),
164+
unit: "second"
165+
},
166+
"pipeline": [
167+
{
168+
$group: {
169+
"_id": { "device_id": "$device_id" },
170+
"max_temp": { $max: "$obs.temp" },
171+
"max_watts": { $max: "$obs.watts" },
172+
"min_watts": { $min: "$obs.watts" },
173+
"avg_watts": { $avg: "$obs.watts" },
174+
"median_watts": {
175+
$median: {
176+
input: "$obs.watts",
177+
method: "approximate"
178+
}
179+
}
180+
}
181+
}
182+
]
183+
}
184+
},
185+
{
186+
$merge: {
187+
into: {
188+
connectionName: "mongodb1",
189+
db: "solar_db",
190+
coll: "solar_coll"
191+
},
192+
on: ["_id"]
193+
}
194+
}
195+
]
196+
)
197+
198+
Learn More
199+
------------------
200+
201+
- :atlas:`Stream Aggregation <atlas-sp/stream-aggregation>`
202+
- :atlas:`Manage Stream Processors <atlas-sp/manage-stream-processor>`

0 commit comments

Comments
 (0)