Allow ingesting in-memory file-like objects

[dhirschfeld]

Writing large amounts of data to disk, only for databricks-sql-connector to then read it back in from disk, is incredibly inefficient.

It would be much more efficient to be able to provide a file-like object to use instead of a filepath. In that way a user could write the data to an in-memory io.BytesIO object instead of writing the data to disk.

[dhirschfeld]

i.e. allow passing through fh rather than creating it internally by opening a file from the filesystem:

databricks-sql-python/src/databricks/sql/client.py

Lines 656 to 668 in d31063c

	def _handle_staging_put(
	self, presigned_url: str, local_file: str, headers: Optional[dict] = None
	):
	"""Make an HTTP PUT request
	Raise an exception if request fails. Returns no data.
	"""
	if local_file is None:
	raise Error("Cannot perform PUT without specifying a local_file")
	with open(local_file, "rb") as fh:
	r = requests.put(url=presigned_url, data=fh, headers=headers)

[kravets-levko]

Hi @dhirschfeld! This indeed sounds like an intersting feature, thank you for sharing it! I have to talk with the rest of team first. Databricks SQL GET and PUT commands should have local file path specified, but I don't know if we ever considered using streams instead of real files. If we agree that there are no risks with this approach - we would have to implement it across all drivers eventually

[susodapop]

Some added context, @dhirschfeld's idea is exactly how the e2e tests for this feature behave (since we ran them in github actions where we don't have a real file system to write to). Should be a straightforward modification.

[gopalldb]

This is an interesting ask. We have implemented very similar solution in other SQL driver, should be able to port that solution here.

[jeffreyh1003]

+1 to OP's ask. This would be a huge win.