Uploading RECORD with ARRAY type using load_table_from_dataframe yields intermediate list.item structure

[jonyvp]

I'm currently attempting to upload a pandas DF containing several columns which either contain a value, or a list of json objects, which result in a nested RECORD field on the bigquery side.

i.e. a column version might contain:
[{"type": "specific_api_name_1", "value": ["0.0.1"]}, {"type": "specific_api_name_2", "value": ["0.0.1"]}]
Uploading this results in the desired format: a nested RECORD in bigquery for a specific row.

Alas, this also yields the schema:

version                              | RECORD   | NULLABLE |   
version.list                         | RECORD   | REPEATED |   
version.list.item                    | RECORD   | NULLABLE |   
version.list.item.type               | STRING   | NULLABLE |   
version.list.item.value              | RECORD   | NULLABLE |   
version.list.item.value.list         | RECORD   | REPEATED |  
version.list.item.value.list.item    | STRING   | NULLABLE |

I would expect (or want) this to generate the schema:

version                              | RECORD   | NULLABLE |  
version.type                         | STRING   | NULLABLE |  
version.value                        | STRING   | REPEATED |  

How could this be achieved?

[tswast]

Closing as duplicate of #19

Due to the way pyarrow serializes parquet files and/or the backend parses these files, array columns are not supported in load from DataFrame. As a workaround, you can use insert_rows_from_dataframe on an existing table.