Litestream compatibility?

[Koeng101]

In the documentation:

To work around this limitation, SQLite is patched to always use EXCLUSIVE locking mode for WAL databases.

This seems to mean software like litestream, which uses taking over the WAL file for backups, wouldn't work. I just wanted to confirm I'm correct about that, so I don't use those two pieces of software together!

[ncruces]

The driver should be fully compatible with Litestream on:

• Linux and macOS since Implement shared memory WAL. #71,
• Windows since Windows shared memory. #181, and
• BSD/illumos since Improve Wal locking on BSD #204.

---

Your intuition is correct: as things stand this driver is not compatible with Litestream.
Supporting WAL in a way that enables (an external process like) Litestream to work is particularly hard.

[ncruces]

As an update on this, I'm considering asynchronous replication based on a VFS and the LTX format instead.

If an ecosystem of tools forms around LTX (against which I can target compatibility) that seems a much better bet.

[ncruces]

I haven't started any work on this.

I still think LTX might have a better replication story, but this module should now be compatible with Litestream, at least.

LTX is still good to consider, but given the current state of the ecosystem, I think there are other things to work on first.

[ncruces]

I don't know how that looks like, but I guess, yeah, go for it. We'll find a home for it. It probably needs to have its own go.mod at least, or it blows up the dependencies. I'm also unsure how often I'd run it.

⚠️ One word of warning: Litestream needs to run out of process. It can't be used as a library.

[ncruces]

Yeah, I already use that strategy for the GORM bindings and think that it helps cull what clients need to download.

Litestream needs to be its own process because of file-locking issues. It depends on mattn and multiple SQLite versions in one process is a big red flag.

[ncruces]

Not 100% sure I understand what this means.
Is it that this driver works with marmot doing leaderless replication out-of-process?
That's great if so!

[ncruces]

One thing I'd be interested in knowing, and that may make sense here (for read-only replicas), is if a OPEN_READONLY WAL connection works and gets updates as they come in. Just because it's not something I've tested, and read-only connections have subtly different locking.