You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When wrapping a transactional file:// backend in DirFileSystem (e.g. filesystem("dir", …, fs=filesystem("file"))), entering with fs.transaction: on the dir:// wrapper sets only the wrapper’s _intrans flag. Because DirFileSystem never delegates its transaction context down to the wrapped LocalFileSystem, all writes (even via fs.open(..., "wb")) commit immediately rather than being deferred to temp files and renamed on commit.
Steps to Reproduce
Create a transactional file:// filesystem and wrap it in dir://:
Observe that exists_inside is True, even though no commit has occurred yet.
Expected Behavior
Inside the with fs.transaction: block, no file should appear on disk.
After exiting the block without errors, the file should be atomically renamed into place.
On error, no partial files should remain.
Actual Behavior
The file is created on disk immediately, inside the transaction block.
There is no deferral to a temp file, and no atomic rename on commit
Proposed Fix
Override DirFileSystem.transaction to delegate to self.fs.transaction, so that the wrapper and wrapped FS share the same transaction context and _intrans flag.
Minimal Repro Code
importos, tempfile, fsspectmp=tempfile.mkdtemp()
base=fsspec.filesystem("file")
fs=fsspec.filesystem("dir", path=tmp, fs=base)
print("Before transaction:", base._intrans, fs._intrans)
withfs.transaction:
print("Inside transaction:", base._intrans, fs._intrans)
withfs.open("data.txt", "wb") asf:
f.write(b"hello")
# Should be False, but is Trueprint("Exists inside?", os.path.exists(os.path.join(tmp, "data.txt")))
print("Clean up")
os.remove(os.path.join(tmp, "data.txt"))
os.rmdir(tmp)
Output
Before transaction: False False
Inside transaction: False True
Exists inside? True
Clean up
Environment
fsspec version: 2025.3.2
The text was updated successfully, but these errors were encountered:
patrickwolf
added a commit
to patrickwolf/filesystem_spec
that referenced
this issue
Apr 23, 2025
DirFileSystem now delegates its `transaction` and `transaction_type` to the wrapped filesystem, ensuring that `with fs.transaction:` on a `dir://` instance correctly toggles the underlying `file://` transaction. This restores atomic write semantics when using the directory wrapper.
Closesfsspec#1823
This test verifies that the fix for issue fsspec#1823 works correctly by ensuring
that DirFileSystem properly propagates its transaction context to the underlying
filesystem. When a transaction is started on a DirFileSystem, both the wrapper
and the wrapped filesystem should have their _intrans flags set, allowing writes
to be deferred until the transaction is committed.
Test confirms that:
- Both filesystems have _intrans=True inside transaction
- Files do not appear on disk until transaction commits
- Files appear correctly after transaction commit
Uh oh!
There was an error while loading. Please reload this page.
When wrapping a transactional file:// backend in DirFileSystem (e.g. filesystem("dir", …, fs=filesystem("file"))), entering with fs.transaction: on the dir:// wrapper sets only the wrapper’s _intrans flag. Because DirFileSystem never delegates its transaction context down to the wrapped LocalFileSystem, all writes (even via fs.open(..., "wb")) commit immediately rather than being deferred to temp files and renamed on commit.
Steps to Reproduce
Create a transactional file:// filesystem and wrap it in dir://:
Enter a transaction on the dir:// wrapper and write a file:
Observe that exists_inside is True, even though no commit has occurred yet.
Expected Behavior
Actual Behavior
Proposed Fix
Override DirFileSystem.transaction to delegate to self.fs.transaction, so that the wrapper and wrapped FS share the same transaction context and _intrans flag.
Minimal Repro Code
Output
Environment
fsspec version: 2025.3.2
The text was updated successfully, but these errors were encountered: