FATAL Error: Database invalidated. Failed to create checkpoint as .wal file does not exist

[bineetsingh30]

Java 21, Duckdb 1.2.2.0, OS: Linux Redhat 10

In my Java application, I am opening multiple connections from different threads to Duckdb in persistent mode. In the start of the application, I create a main connection, and now any subsequent connection is created using DuckDBConnection.duplicate.
DuckDB file is only accessed from this process.

My connection is created with property jdbc_stream_results set to true.

One of my update query failed with error:

java.sql.SQLException: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
Original error: "Failed to create checkpoint because of error: Could not remove file "/home/ec2-user/duckdb/./duckdb.db.wal": No such file or directory"

Scenario:

1. DuckDb is being queried by a select query on a different table parallely.
2. Update/INSERT queries are coming. They are processed in different threads. To run a query, I create new connection (using duplicate) and run the query.

My duckdb.db file is around 160 GB.

[staticlibs]

@bineetsingh30

Hi, thanks for the report! The problem sounds similar to one reported in #206, that is unfortunately not easily reproducible (one attempt). I wonder if there are any known exact steps that can help with reproducing this?

[eramitmittal]

hi, I work with @bineetsingh30.

We saw #206 but we believe its a different issue because the error in #206 specifically mentions "The process cannot access the file because it is being used by another process.". Over here the error is "No such file or directory". This indicates that the wal file was merged with main DB (possibly) by a thread while (possibly) another thread still expected it to be there.

We are trying to reproduce the issue and discover exact steps but due to multi-threaded nature of the application it is a bit tricky.

[eramitmittal]

I see @Mytherin making a change 3 months back in this area where RemoveFile has been replaced with TryRemoveFile which checks if WAL file exists or not before trying to remove it. Not sure what the motivation behind this change was and if it is related to this problem. Anyway we will test with latest duckdb version.

[staticlibs]

@eramitmittal

I think you are correct with your analysis, normal way to deal with issues like this is to get a somewhat stable Java reproducer, then convert it to C++ and then run/debug it under sanitizers. If you get a more or less stable reproducer in your env - I can pick it up from there.

[staticlibs]

@bineetsingh30 @eramitmittal

Just FYI, this may be the same problem as duckdb/duckdb#19343. The fix to it is merged recently - duckdb/duckdb#19344.