Skip to content

Commit 2e47bd5

Browse files
jeff-allen-mongojeff-allen-mongo
committed
(DOCS-15065): Long-running queries page (#434)
* (DOCS-15065): Long-running queries page * Formatting fix * Apply tweaks * Updates per review * add missing word * word change * weight > age * updates per external review * Add example to modify min snapshot window * Updates per copy review * add ref link to 'sessions' * use present tense * Add new example to long running query page * edits * lower case white * tweaks * add glossary links * add snapshotTooOld warnings * typo fix * code block formatting * add example links * fix typos * replace midnight * updates per technical review * active voice * add missing word * Updates per copy review * formatting * add missing text * move value prop to top of page * intro formatting tweak * updates per Jane's feedback * typo fix * update toc depth * wording * add c and python examples * fix indentation for python example * add go and motor tabs * fix indentations * adjust indentations * update indentation * add php examples * add c++ and ruby examples
1 parent 4fc2b44 commit 2e47bd5

File tree

3 files changed

+357
-1
lines changed

3 files changed

+357
-1
lines changed

‎source/release-notes/5.0.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -953,7 +953,8 @@ Extended Support for Read Concern ``"snapshot"``
953953

954954
Starting in MongoDB 5.0, read concern :readconcern:`"snapshot"` is
955955
supported for some read operations outside of multi-document
956-
transactions on primaries and secondaries.
956+
transactions on primaries and secondaries. See
957+
:ref:`tutorial-long-running-queries`.
957958

958959
``minSnapshotHistoryWindowInSeconds`` Server Parameter
959960
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

‎source/tutorial/long-running-queries.txt

Lines changed: 354 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
.. _tutorial-long-running-queries:
2+
3+
=====================================
4+
Perform Long-Running Snapshot Queries
5+
=====================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
Snapshot queries allow you to read data as it appeared at a single point
16+
in time in the recent past.
17+
18+
Starting in MongoDB 5.0, you can use read concern
19+
:readconcern:`"snapshot"` to query data on :term:`secondary` nodes. This
20+
feature increases the versatility and resilience of your application's
21+
reads. You do not need to create a static copy of your data, move it out
22+
into a separate system, and manually isolate these long-running queries
23+
from interfering with your operational workload. Instead, you can
24+
perform long-running queries against a live, transactional database
25+
while reading from a consistent state of the data.
26+
27+
Using read concern :readconcern:`"snapshot"` on secondary nodes does not
28+
impact your application's write workload. Only application reads benefit
29+
from long-running queries being isolated to secondaries.
30+
31+
Use snapshot queries when you want to:
32+
33+
- Perform multiple related queries and ensure that each query reads data
34+
from the same point in time.
35+
36+
- Ensure that you read from a consistent state of the data from some
37+
point in the past.
38+
39+
Comparing Local and Snapshot Read Concerns
40+
------------------------------------------
41+
42+
When MongoDB performs long-running queries using the default
43+
:readconcern:`"local"` read concern, the query results may contain data
44+
from writes that occur at the same time as the query. As a result, the
45+
query may return unexpected or inconsistent results.
46+
47+
To avoid this scenario, create a :ref:`session <sessions>` and specify
48+
read concern :readconcern:`"snapshot"`. With read concern
49+
:readconcern:`"snapshot"`, MongoDB runs your query with snapshot
50+
isolation, meaning that your query reads data as it appeared at a single
51+
point in time in the recent past.
52+
53+
Examples
54+
--------
55+
56+
The examples on this page show how you can use snapshot queries to:
57+
58+
- :ref:`example-snapshot-queries-multiple`
59+
60+
- :ref:`example-snapshot-pit`
61+
62+
.. _example-snapshot-queries-multiple:
63+
64+
Run Related Queries From the Same Point in Time
65+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
66+
67+
Read concern :readconcern:`"snapshot"` lets you run multiple related
68+
queries within a session and ensure that each query reads data from the
69+
same point in time.
70+
71+
An animal shelter has a ``pets`` database that contains collections for
72+
each type of pet. The ``pets`` database has these collections:
73+
74+
- ``cats``
75+
- ``dogs``
76+
77+
Each document in each collection contains an ``adoptable`` field,
78+
indicating whether the pet is available for adoption. For example, a
79+
document in the ``cats`` collection looks like this:
80+
81+
.. code-block:: javascript
82+
83+
{
84+
"name": "Whiskers",
85+
"color": "white",
86+
"age": 10,
87+
"adoptable": true
88+
}
89+
90+
You want to run a query to see the total number of pets available for
91+
adoption across all collections. To provide a consistent view of the
92+
data, you want to ensure that the data returned from each collection is
93+
from a single point in time.
94+
95+
To accomplish this goal, use read concern :readconcern:`"snapshot"`
96+
within a session:
97+
98+
.. tabs-selector:: drivers
99+
100+
.. tabs-drivers::
101+
102+
.. tab::
103+
:tabid: python
104+
105+
.. literalinclude:: /driver-examples/test_examples.py
106+
:language: python
107+
:dedent: 8
108+
:start-after: Start Snapshot Query Example 1
109+
:end-before: End Snapshot Query Example 1
110+
111+
.. tab::
112+
:tabid: c
113+
114+
.. literalinclude:: /driver-examples/test-mongoc-sample-commands.c
115+
:language: c
116+
:dedent: 3
117+
:start-after: Start Snapshot Query Example 1
118+
:end-before: End Snapshot Query Example 1
119+
120+
.. tab::
121+
:tabid: go
122+
123+
.. literalinclude:: /driver-examples/go_examples.go
124+
:language: go
125+
:dedent: 1
126+
:start-after: Start Snapshot Query Example 1
127+
:end-before: End Snapshot Query Example 1
128+
129+
.. tab::
130+
:tabid: motor
131+
132+
.. literalinclude:: /driver-examples/test_examples_motor.py
133+
:language: python
134+
:dedent: 8
135+
:start-after: Start Snapshot Query Example 1
136+
:end-before: End Snapshot Query Example 1
137+
138+
.. tab::
139+
:tabid: php
140+
141+
.. literalinclude:: /driver-examples/DocumentationExamplesTest.php
142+
:language: php
143+
:dedent: 8
144+
:start-after: Start Snapshot Query Example 1
145+
:end-before: End Snapshot Query Example 1
146+
147+
.. tab::
148+
:tabid: ruby
149+
150+
.. literalinclude:: /driver-examples/snapshot_query_examples_spec.rb
151+
:language: ruby
152+
:dedent: 6
153+
:start-after: Start Snapshot Query Example 1
154+
:end-before: End Snapshot Query Example 1
155+
156+
.. tab::
157+
:tabid: cpp
158+
159+
.. literalinclude:: /driver-examples/cpp-documentation-examples.cpp
160+
:language: cpp
161+
:dedent: 4
162+
:start-after: Start Snapshot Query Example 1
163+
:end-before: End Snapshot Query Example 1
164+
165+
The preceding series of commands:
166+
167+
- Uses ``MongoClient()`` to establish a connection to the MongoDB
168+
deployment.
169+
170+
- Switches to the ``pets`` database.
171+
172+
- Establishes a session. The command specifies ``snapshot=True``,
173+
so the session uses read concern :readconcern:`"snapshot"`.
174+
175+
- Performs these actions for each collection in the ``pets`` database:
176+
177+
- Uses :pipeline:`$match` to filter for documents where the
178+
``adoptable`` field is ``True``.
179+
180+
- Uses :pipeline:`$count` to return a count of the filtered documents.
181+
182+
- Increments the ``adoptablePetsCount`` variable with the count from
183+
the database.
184+
185+
- Prints the ``adoptablePetsCount`` variable.
186+
187+
All queries within the session read data as it appeared at the same
188+
point in time. As a result, the final count reflects a consistent
189+
snapshot of the data.
190+
191+
.. note::
192+
193+
If the session lasts longer than the WiredTiger history retention
194+
period (300 seconds, by default), the query errors with a
195+
``SnapshotTooOld`` error. To learn how to configure snapshot
196+
retention and enable longer-running queries, see
197+
:ref:`configure-snapshot-retention`.
198+
199+
.. _example-snapshot-pit:
200+
201+
Read from a Consistent State of the Data from Some Point in the Past
202+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
203+
204+
Read concern :readconcern:`"snapshot"` ensures that your query reads
205+
data as it appeared at some single point in time in the recent past.
206+
207+
An online shoe store has a ``sales`` collection that contains data for
208+
each item sold at the store. For example, a document in the ``sales``
209+
collection looks like this:
210+
211+
.. code-block:: javascript
212+
213+
{
214+
"shoeType": "boot",
215+
"price": 30,
216+
"saleDate": ISODate("2022-02-02T06:01:17.171Z")
217+
}
218+
219+
Each day at midnight, a query runs to see how many pairs of shoes were
220+
sold that day. The daily sales query looks like this:
221+
222+
.. tabs-drivers::
223+
224+
.. tab::
225+
:tabid: python
226+
227+
.. literalinclude:: /driver-examples/test_examples.py
228+
:language: python
229+
:dedent: 8
230+
:start-after: Start Snapshot Query Example 2
231+
:end-before: End Snapshot Query Example 2
232+
233+
.. tab::
234+
:tabid: c
235+
236+
.. literalinclude:: /driver-examples/test-mongoc-sample-commands.c
237+
:language: c
238+
:dedent: 3
239+
:start-after: Start Snapshot Query Example 2
240+
:end-before: End Snapshot Query Example 2
241+
242+
.. tab::
243+
:tabid: go
244+
245+
.. literalinclude:: /driver-examples/go_examples.go
246+
:language: go
247+
:dedent: 1
248+
:start-after: Start Snapshot Query Example 2
249+
:end-before: End Snapshot Query Example 2
250+
251+
.. tab::
252+
:tabid: motor
253+
254+
.. literalinclude:: /driver-examples/test_examples_motor.py
255+
:language: python
256+
:dedent: 8
257+
:start-after: Start Snapshot Query Example 2
258+
:end-before: End Snapshot Query Example 2
259+
260+
.. tab::
261+
:tabid: php
262+
263+
.. literalinclude:: /driver-examples/DocumentationExamplesTest.php
264+
:language: php
265+
:dedent: 8
266+
:start-after: Start Snapshot Query Example 2
267+
:end-before: End Snapshot Query Example 2
268+
269+
The preceding query:
270+
271+
- Uses :pipeline:`$match` with :query:`$expr` to specify a filter on the
272+
``saleDate`` field.
273+
274+
- :query:`$expr` allows the use of :ref:`aggregation expressions
275+
<aggregation-expressions>` (such as :variable:`NOW`) in the
276+
:pipeline:`$match` stage.
277+
278+
- Uses the :query:`$gt` operator and :expression:`$dateSubtract`
279+
expression to return documents where the ``saleDate`` is
280+
greater than one day before the time the query is executed.
281+
282+
- Uses :pipeline:`$count` to return a count of the matching documents.
283+
The count is stored in the ``totalDailySales`` variable.
284+
285+
- Specifies read concern :readconcern:`"snapshot"` to ensure that the
286+
query reads from a single point in time.
287+
288+
The ``sales`` collection is quite large, and as a result this query may
289+
take a few minutes to run. Because the store is online, sales can occur
290+
at any time of day.
291+
292+
For example, consider if:
293+
294+
- The query begins executing at 12:00 AM.
295+
- A customer buys three pairs of shoes at 12:02 AM.
296+
- The query finishes executing at 12:04 AM.
297+
298+
If the query doesn't use read concern :readconcern:`"snapshot"`, sales
299+
that occur between when the query starts and when it finishes can be
300+
included in the query count, despite not occurring on the day the report
301+
is for. This could result in inaccurate reports with some sales being
302+
counted twice.
303+
304+
By specifying read concern :readconcern:`"snapshot"`, the query only
305+
returns data that was present in the database at a point in time shortly
306+
before the query started executing.
307+
308+
.. note::
309+
310+
If the query takes longer than the WiredTiger history retention
311+
period (300 seconds, by default), the query errors with a
312+
``SnapshotTooOld`` error. To learn how to configure snapshot
313+
retention and enable longer-running queries, see
314+
:ref:`configure-snapshot-retention`.
315+
316+
.. _configure-snapshot-retention:
317+
318+
Configure Snapshot Retention
319+
----------------------------
320+
321+
By default, the WiredTiger storage engine retains history for 300
322+
seconds. You can use a session with ``snapshot=true`` for a total of 300
323+
seconds from the time of the first operation in the session to the last.
324+
If you use the session for a longer period of time, the session fails
325+
with a ``SnapshotTooOld`` error. Similarly, if you query data using read
326+
concern :readconcern:`"snapshot"` and your query lasts longer than 300
327+
seconds, the query fails.
328+
329+
If your query or session run for longer than 300 seconds, consider
330+
increasing the snapshot retention period. To increase the retention
331+
period, modify the :parameter:`minSnapshotHistoryWindowInSeconds`
332+
parameter.
333+
334+
For example, this command sets the value of
335+
:parameter:`minSnapshotHistoryWindowInSeconds` to 600 seconds:
336+
337+
.. code-block:: javascript
338+
339+
db.adminCommand( { setParameter: 1, minSnapshotHistoryWindowInSeconds: 600 } )
340+
341+
.. important::
342+
343+
To modify :parameter:`minSnapshotHistoryWindowInSeconds` for a
344+
:atlas:`MongoDB Atlas </>` cluster, you must contact :atlas:`Atlas
345+
Support </support>`.
346+
347+
Disk Space and History
348+
~~~~~~~~~~~~~~~~~~~~~~
349+
350+
Increasing the value of :parameter:`minSnapshotHistoryWindowInSeconds`
351+
increases disk usage because the server must maintain the history of
352+
older modified values within the specified time window. The amount of
353+
disk space used depends on your workload, with higher volume workloads
354+
requiring more disk space.

‎source/tutorial/query-documents.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -626,4 +626,5 @@ To learn how to iterate through documents in a cursor, refer to your
626626
/tutorial/query-array-of-documents
627627
/tutorial/project-fields-from-query-results
628628
/tutorial/query-for-null-fields
629+
/tutorial/long-running-queries
629630
/tutorial/iterate-a-cursor/

0 commit comments

Comments
 (0)