Skip to content

Commit 82e7d97

Browse files
committed
improve description
1 parent 8faf59e commit 82e7d97

File tree

1 file changed

+7
-3
lines changed

1 file changed

+7
-3
lines changed

airbyte_cdk/sources/streams/http/http_client.py

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -145,9 +145,13 @@ def _request_session(self) -> requests.Session:
145145
if cache_dir
146146
else "file::memory:?cache=shared"
147147
)
148-
backend = requests_cache.SQLiteCache(
149-
sqlite_path, fast_save=True, wal=True
150-
) # By using `PRAGMA synchronous=OFF` and `PRAGMA journal_mode=WAL`, we avoid having `database table is locked` errors. Note that those were blindly added at the same time and one or the other might be sufficient to prevent the issues but we have seen good results with both. Feel free to revisit given more information.
148+
# By using `PRAGMA synchronous=OFF` and `PRAGMA journal_mode=WAL`, we reduce the possible occurrences of `database table is locked` errors.
149+
# Note that those were blindly added at the same time and one or the other might be sufficient to prevent the issues but we have seen good results with both. Feel free to revisit given more information.
150+
# There are strong signals that `fast_save` might create problems but if the sync crashes, we start back from the beginning in terms of sqlite anyway so the impact should be minimal. Signals are:
151+
# * https://github.com/requests-cache/requests-cache/commit/7fa89ffda300331c37d8fad7f773348a3b5b0236#diff-f43db4a5edf931647c32dec28ea7557aae4cae8444af4b26c8ecbe88d8c925aaR238
152+
# * https://github.com/requests-cache/requests-cache/commit/7fa89ffda300331c37d8fad7f773348a3b5b0236#diff-2e7f95b7d7be270ff1a8118f817ea3e6663cdad273592e536a116c24e6d23c18R164-R168
153+
# * `If the application running SQLite crashes, the data will be safe, but the database [might become corrupted](https://www.sqlite.org/howtocorrupt.html#cfgerr) if the operating system crashes or the computer loses power before that data has been written to the disk surface.` in [this description](https://www.sqlite.org/pragma.html#pragma_synchronous).
154+
backend = requests_cache.SQLiteCache(sqlite_path, fast_save=True, wal=True)
151155
return CachedLimiterSession(
152156
sqlite_path, backend=backend, api_budget=self._api_budget, match_headers=True
153157
)

0 commit comments

Comments
 (0)