feat: Add support for deltalake

[AndreasAlbertQC]

Motivation

I would like to be able to serialize data to deltalake, including the dataframely metadata.

Changes

• Refactored the tests from test_read_write_parquet.py such that they can easily be run for each storage backend.
  • Since different backends use different args / kwargs, support slightly different features (e.g. no truly lazy sink in deltalake atm), and need different code for mocking, I implemented a thin wrapper Tester interface that translates the calls needed for the tests into the specifics of the backends.
  • Tests are then simply pytest.mark.parametrized over the differet backend testers
  • If we want to add another backend in the future, you would just implement the tester class and can then fully reuse existing tests.
• Added implementation for reading from and writing to delta lake in Schema, Collection, and FailureInfo
  • data is stored in delta tables
  • meta data is stored in custom commit meta data. I initially considered table metadata, but switched to commits because table metadata is not changeable after an initial write, and may not reflect the current state of the table. With commit metadata, we can make sure to only trust the metadata if the last write came from dataframely.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1211217826559584

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (4fd1b4d) to head (0785c2d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #134    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           45        49     +4     
  Lines         2577      2812   +235     
==========================================
+ Hits          2577      2812   +235     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[AndreasAlbertQC]

@delsner @borchero this turned out bigger than I thought because I had to refactor the way that the storage testing is done. (So the tests are reusable across backends). Let me know if you'd like to chat about this synchronously to make it easier to review.

[borchero]

Thank you, this looks great! The abstraction really shows its worth here 😄

One additional request: can we add deltalake as an optional dependency to pyproject.toml?

[AndreasAlbertQC]

One additional request: can we add deltalake as an optional dependency to pyproject.toml?

Good point, done. I also added sqlalchemy and pyarrow because I don't know why they were not already added.

[AndreasAlbertQC]

Let's figure out the treatment of optional dependency testing in #135

[borchero]

LGTM, just small suggestions! :)