Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

varun-edachali-dbx · 2025-05-26T13:30:00Z

What type of PR is this?

Refactor

Description

Create ResultSet interface and make the existing ResultSet (as ThriftResultSet) implement it. Concrete Thrift Backend layer should return ThriftResultSet from execute commands.
Generalise the return types of some functions in the backend interface to prevent alignment with Thrift (create new Status enum to avoid using Thrift's Status type).

How is this tested?

Unit tests
- some of the existing unit tests were slightly altered to account for the introduction of the interface. No unit tests were removed or introduced.
E2E Tests
Manually
N/A

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing
https://databricks.atlassian.net/browse/PECOBLR-445?atlOrigin=eyJpIjoiMjA3ZTljYTUzNDgxNDY4MmFhMThjYjI5MWY0Yjc0MTIiLCJwIjoiaiJ9

Signed-off-by: varun-edachali-dbx <[email protected]>

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

…t to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: varun-edachali-dbx <[email protected]>

* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

* Modified the gitignore file to not have .idea file * [PECO-1803] Splitting the PySql connector into the core and the non core part (#417) * Implemented ColumnQueue to test the fetchall without pyarrow Removed token removed token * order of fields in row corrected * Changed the folder structure and tested the basic setup to work * Refractored the code to make connector to work * Basic Setup of connector, core and sqlalchemy is working * Basic integration of core, connect and sqlalchemy is working * Setup working dynamic change from ColumnQueue to ArrowQueue * Refractored the test code and moved to respective folders * Added the unit test for column_queue Fixed __version__ Fix * venv_main added to git ignore * Added code for merging columnar table * Merging code for columnar * Fixed the retry_close sesssion test issue with logging * Fixed the databricks_sqlalchemy tests and introduced pytest.ini for the sqla_testing * Added pyarrow_test mark on pytest * Fixed databricks.sqlalchemy to databricks_sqlalchemy imports * Added poetry.lock * Added dist folder * Changed the pyproject.toml * Minor Fix * Added the pyarrow skip tag on unit tests and tested their working * Fixed the Decimal and timestamp conversion issue in non arrow pipeline * Removed not required files and reformatted * Fixed test_retry error * Changed the folder structure to src / databricks * Removed the columnar non arrow flow to another PR * Moved the README to the root * removed columnQueue instance * Revmoved databricks_sqlalchemy dependency in core * Changed the pysql_supports_arrow predicate, introduced changes in the pyproject.toml * Ran the black formatter with the original version * Extra .py removed from all the __init__.py files names * Undo formatting check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * BIG UPDATE * Refeactor code * Refractor * Fixed versioning * Minor refractoring * Minor refractoring * Changed the folder structure such that sqlalchemy has not reference here * Fixed README.md and CONTRIBUTING.md * Added manual publish * On push trigger added * Manually setting the publish step * Changed versioning in pyproject.toml * Bumped up the version to 4.0.0.b3 and also changed the structure to have pyarrow as optional * Removed the sqlalchemy tests from integration.yml file * [PECO-1803] Print warning message if pyarrow is not installed (#468) Print warning message if pyarrow is not installed Signed-off-by: Jacky Hu <[email protected]> * [PECO-1803] Remove sqlalchemy and update README.md (#469) Remove sqlalchemy and update README.md Signed-off-by: Jacky Hu <[email protected]> * Removed all sqlalchemy related stuff * generated the lock file * Fixed failing tests * removed poetry.lock * Updated the lock file * Fixed poetry numpy 2.2.2 issue * Workflow fixes --------- Signed-off-by: Jacky Hu <[email protected]> Co-authored-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Removed python3.8 support * Minor fix Signed-off-by: varun-edachali-dbx <[email protected]>

Support for Py till 3.12 Signed-off-by: varun-edachali-dbx <[email protected]>

* Increased the number of retry attempts allowed (#486) Updated the number of attempts allowed * bump version to 3.7.1 (#487) bumped up version * Refractore * Minor change Signed-off-by: varun-edachali-dbx <[email protected]>

bumped up the version Signed-off-by: varun-edachali-dbx <[email protected]>

* Remove upper caps on dependencies (#452) * Remove upper caps on numpy and pyarrow versions Signed-off-by: David Black <[email protected]> * Added CI CD upto python 3.13 Signed-off-by: David Black <[email protected]> * Specify pandas 2.2.3 as the lower bound for python 3.13 Signed-off-by: David Black <[email protected]> * Specify pyarrow 18.0.0 as the lower bound for python 3.13 Signed-off-by: David Black <[email protected]> * Move `numpy` to dev dependencies Signed-off-by: Dave Hirschfeld <[email protected]> * Updated lockfile Signed-off-by: Dave Hirschfeld <[email protected]> --------- Signed-off-by: David Black <[email protected]> Signed-off-by: Dave Hirschfeld <[email protected]> Co-authored-by: David Black <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Improve debugging + add PR review template * case sensitivity of PR template Signed-off-by: varun-edachali-dbx <[email protected]>

* Base changes * Black formatter * Cache version fix * Added the changed test_retry.py file * retry_test_mixins changes Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T01:40:00Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-03T04:42:15Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jayantsing-db

thanks. a minor comment for a separate PR.

jayantsing-db · 2025-06-03T05:08:17Z

src/databricks/sql/backend/thrift_backend.py

+        return ThriftResultSet(
+            connection=cursor.connection,
+            execute_response=execute_response,
+            thrift_client=self,


we already pass connection which has session which in turn has the backend? can we look into normalising this init params (in a separate PR)

…574) * ensure backend client returns a ResultSet type in backend tests Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * newline for cleanliness Signed-off-by: varun-edachali-dbx <[email protected]> * fix circular import Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * to_hex_id -> get_hex_id Signed-off-by: varun-edachali-dbx <[email protected]> * better comment on protocol version getter Signed-off-by: varun-edachali-dbx <[email protected]> * formatting (black) Signed-off-by: varun-edachali-dbx <[email protected]> * stricter typing for cursor Signed-off-by: varun-edachali-dbx <[email protected]> * correct typing Signed-off-by: varun-edachali-dbx <[email protected]> * correct tests and merge artifacts Signed-off-by: varun-edachali-dbx <[email protected]> * remove accidentally modified workflow files remnants of old merge Signed-off-by: varun-edachali-dbx <[email protected]> * chore: remove accidentally modified workflow files Signed-off-by: varun-edachali-dbx <[email protected]> * add back accidentally removed docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * clean up docstrings Signed-off-by: varun-edachali-dbx <[email protected]> * log hex Signed-off-by: varun-edachali-dbx <[email protected]> * remove unnecessary _replace call Signed-off-by: varun-edachali-dbx <[email protected]> * add __str__ for CommandId Signed-off-by: varun-edachali-dbx <[email protected]> * take TOpenSessionResp in get_protocol_version to maintain existing interface Signed-off-by: varun-edachali-dbx <[email protected]> * active_op_handle -> active_mmand_id Signed-off-by: varun-edachali-dbx <[email protected]> * ensure None returned for close_command Signed-off-by: varun-edachali-dbx <[email protected]> * account for ResultSet return in new pydocs Signed-off-by: varun-edachali-dbx <[email protected]> * pydoc for types Signed-off-by: varun-edachali-dbx <[email protected]> * move common state to ResultSet aprent Signed-off-by: varun-edachali-dbx <[email protected]> * stronger typing in resultSet behaviour Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundant patch in test Signed-off-by: varun-edachali-dbx <[email protected]> * add has_been_closed_server_side assertion Signed-off-by: varun-edachali-dbx <[email protected]> * remove redundancies in tests Signed-off-by: varun-edachali-dbx <[email protected]> * more robust close check Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised state in e2e test Signed-off-by: varun-edachali-dbx <[email protected]> * simplify corrected test Signed-off-by: varun-edachali-dbx <[email protected]> * add line gaps after multi-line pydocs for consistency Signed-off-by: varun-edachali-dbx <[email protected]> * use normalised CommandState type in ExecuteResponse Signed-off-by: varun-edachali-dbx <[email protected]> --------- Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 13:30 — with GitHub Actions Failure

varun-edachali-dbx temporarily deployed to azure-prod May 26, 2025 15:01 — with GitHub Actions Inactive

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 15:01 — with GitHub Actions Failure

varun-edachali-dbx force-pushed the fetch-interface branch from 25158cc to 9b265c1 Compare May 26, 2025 15:02

varun-edachali-dbx had a problem deploying to azure-prod May 26, 2025 15:02 — with GitHub Actions Failure

varun-edachali-dbx temporarily deployed to azure-prod May 26, 2025 15:44 — with GitHub Actions Inactive

varun-edachali-dbx had a problem deploying to azure-prod May 28, 2025 03:31 — with GitHub Actions Failure

varun-edachali-dbx had a problem deploying to azure-prod May 28, 2025 03:40 — with GitHub Actions Failure

varun-edachali-dbx and others added 20 commits May 28, 2025 03:40

ensure backend client returns a ResultSet type in backend tests

8ea5cf4

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

3eba92a

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: correct client tests

6735fd5

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

12388e8

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess comments (nit)

853e6c3

Signed-off-by: varun-edachali-dbx <[email protected]>

Update CODEOWNERS (#562)

4f8b54c

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

Update github actions run conditions (#569)

2c2984e

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend->backend, ResultSet -> ThriftResultSet

4979422

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend -> backend in e2e test_driver

6361e85

Signed-off-by: varun-edachali-dbx <[email protected]>

Removed CI CD for python3.8 (#490)

e5ac8c6

* Removed python3.8 support * Minor fix Signed-off-by: varun-edachali-dbx <[email protected]>

Added CI CD upto python 3.12 (#491)

e06fd2f

Support for Py till 3.12 Signed-off-by: varun-edachali-dbx <[email protected]>

Merging changes from v3.7.1 release (#488)

7126f97

* Increased the number of retry attempts allowed (#486) Updated the number of attempts allowed * bump version to 3.7.1 (#487) bumped up version * Refractore * Minor change Signed-off-by: varun-edachali-dbx <[email protected]>

Bumped up to version 4.0.0 (#493)

b1245da

bumped up the version Signed-off-by: varun-edachali-dbx <[email protected]>

Improve debugging + fix PR review template (#514)

e1d7f71

* Improve debugging + add PR review template * case sensitivity of PR template Signed-off-by: varun-edachali-dbx <[email protected]>

Forward porting all changes into 4.x.x. uptil v3.7.3 (#529)

1c721e0

* Base changes * Black formatter * Cache version fix * Added the changed test_retry.py file * retry_test_mixins changes Signed-off-by: varun-edachali-dbx <[email protected]>

Updated the actions/cache version (#532)

f1fa67a

Signed-off-by: varun-edachali-dbx <[email protected]>

remove redundancies in tests

1f0c81f

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 10:09 — with GitHub Actions Failure

more robust close check

aa7207e

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 2, 2025 11:30 — with GitHub Actions Failure

use normalised state in e2e test

fde4634

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 3, 2025 01:39 — with GitHub Actions Failure

simplify corrected test

54f2b93

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 3, 2025 01:53 — with GitHub Actions Inactive

add line gaps after multi-line pydocs for consistency

db73ecf

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod June 3, 2025 04:16 — with GitHub Actions Failure

varun-edachali-dbx requested a review from jayantsing-db June 3, 2025 04:28

use normalised CommandState type in ExecuteResponse

cf8a629

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx temporarily deployed to azure-prod June 3, 2025 04:42 — with GitHub Actions Inactive

jayantsing-db approved these changes Jun 3, 2025

View reviewed changes

varun-edachali-dbx merged commit 3c78ed7 into sea-migration Jun 3, 2025
22 of 23 checks passed

databricks deleted a comment from github-actions bot Jun 7, 2025

varun-edachali-dbx mentioned this pull request Jun 18, 2025

Generalise Backend Layer #604

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Uh oh!

varun-edachali-dbx commented May 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

jayantsing-db left a comment

Uh oh!

jayantsing-db Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Implement ResultSet Abstraction (backend interfaces for fetch phase) #574

Uh oh!

Conversation

varun-edachali-dbx commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Description

How is this tested?

Related Tickets & Documents

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

jayantsing-db left a comment

Choose a reason for hiding this comment

Uh oh!

jayantsing-db Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

varun-edachali-dbx commented May 26, 2025 •

edited

Loading