FEAT: streaming support in fetchone for varcharmax data type

[gargsaumya]

Work Item / Issue Reference

AB#38110
AB#34162

GitHub Issue: #<ISSUE_NUMBER>

---

Summary

This pull request significantly improves the handling of large object (LOB) data types (such as large strings and binary data) in the MSSQL Python driver, especially for fetching and streaming variable-length data. The changes introduce robust streaming logic for LOB columns, prevent data truncation, and ensure correct type handling for both single-row and batch fetches. Additionally, the code now detects LOB columns and automatically switches to per-row streaming when necessary, improving reliability and correctness for large datasets.

LOB Streaming and Fetching Improvements:

• Introduced the FetchLobColumnData function in ddbc_bindings.cpp to stream LOB data (CHAR, WCHAR, and BINARY types) in chunks, correctly handling nulls, null-terminators, and platform-specific encoding. This prevents truncation and errors when fetching large columns.
• Updated SQLGetData_wrap to use streaming for LOB columns or when data length is unknown/too large, for both narrow and wide character types, as well as binary data. This ensures correct retrieval of all data regardless of size. [1] [2] [3]

Batch Fetch Logic Enhancements:

• Modified FetchBatchData to detect LOB columns and use streaming fetch for those columns, avoiding exceptions and ensuring all data is retrieved for large columns in batch operations. [1] [2] [3] [4] [5]
• Updated FetchMany_wrap to pre-scan columns for LOB types and, if any are found, fall back to row-by-row streaming fetch for those rows; otherwise, it proceeds with standard batch fetching.

Type Mapping and Constants:

• Adjusted _map_sql_type in cursor.py to map long string types to SQL_WVARCHAR/SQL_VARCHAR with length 0 for streaming, aligning with the new LOB streaming logic.
• Defined SQL_MAX_LOB_SIZE (8000) as the threshold for LOB streaming, centralizing the logic for when to treat columns as LOBs.

These changes collectively make LOB handling more robust, reduce the risk of data truncation, and improve compatibility across platforms.

[Copilot]

Pull Request Overview

This PR adds comprehensive streaming support for VARCHAR(MAX) data types by introducing a new LOB (Large Object) streaming mechanism in the C++ bindings and updating the Python cursor layer to handle long strings more efficiently.

Key changes:

• Implements streaming-based data retrieval for large VARCHAR(MAX) columns to handle values that exceed buffer limits
• Refactors SQL type mapping to use zero column size for long strings, triggering proper LOB handling
• Adds comprehensive test coverage for VARCHAR(MAX) scenarios including boundary conditions, large values, and edge cases

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
mssql_python/pybind/ddbc_bindings.cpp	Adds FetchLobColumnData function for streaming large column data and updates SQLGetData_wrap to use streaming for VARCHAR(MAX)
mssql_python/cursor.py	Updates _map_sql_type to use SQL_VARCHAR/SQL_WVARCHAR with zero column size for long strings
tests/test_004_cursor.py	Adds comprehensive test suite for VARCHAR(MAX) covering various data sizes, edge cases, and transaction scenarios

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[sumitmsft]

Left a few comments. Please resolve

[bewithgaurav]

need a re-review post solving conflicts

[gargsaumya]

need a re-review post solving conflicts

The conflicts are now resolved. You can go ahead and re-review.