Skip to content

Add 'date_dtype', 'datetime_dtype', 'time_dtype' and 'timetamp_dtype' to the 'to_dataframe' APIs #1546

@chelsea-lin

Description

@chelsea-lin

Is your feature request related to a problem? Please describe.
We want to add `date_dtype, datetime_dtype, time_dtype, and timestamp_dtype to the to_dataframe API, similar to bool_dtype, int_dtype, float_dtype, and string_dtype (#1529).

Pyarrow's conversion of data values to nanosecond precision can result in an out-of-bounds error for datetime values like 9999-12-31T23:59:59.999999. Therefore, the default datetime_dtype mapping will be ignored, and all datetime columns will be of object type. Once pandas supports pyarrow dtype backend (link), we can use more flexible custom time dtypes, such as pandas.ArrowDtype(pyarrow.timestamp("us", tz="UTC")).

Describe the solution you'd like
Add custom dtype mapping for these time dtypes conversion.

Describe alternatives you've considered
We can always cast the object dtypes to these custom dtype but would cause performance loss.

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the googleapis/python-bigquery API.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions