json_normalize Support for Generators

[msteijaert]

The data argument in json_normalize requires be an dictionary or list of dictionaries. If an iterator is passed, unexpected behavior can occur. In particular, this will result in the loss of the first row. See https://stackoverflow.com/questions/56362810/missing-first-document-when-loading-multi-document-yaml-file-in-pandas-dataframe.

I think this should be catched, either by throwing an error (as proposed in #26646) or by ensuring that the output for the iterator is the same as for the list created by "materializing" the iterator.

[WillAyd]

I think the issue can be expressed more succinctly. Basically this works:

In [22]: data = [{'id': 1, 'name': {'first': 'foo'}}, {'id': 2, 'name': {'first': 'bar'}}]
In [23]: json_normalize(data)
Out[23]:
   id name.first
0   1        foo
1   2        bar

But this doesn't

In [23]: json_normalize(x for x in data)
Out[24]:
   id name.first
0   2        bar

I think this could be supported

[WillAyd]

If you'd like to try a PR to support this I think would be OK

[msteijaert]

If you'd like to try a PR to support this I think would be OK

What would be preferable? Raising an error or turning data into a list if data is an iterator? The latter is pretty straightforward. My (naive) solution would be the following:

if data is iter(data):
    data = [x for x in data]

[WillAyd]

I don't think using is is a good approach. If you can check the source to see where the first entry is getting dropped would be preferable to refactor without inference

[zifeo]

@WillAyd The issue is here (pulling at least one element out of the iterator), not sure this can be so easily refactored. Do you see any trouble data = list(data) could cause?

[WillAyd]

closed by #33585

[kylekeppler]

Note #33585 does not add support for generators, but clarifies the docstrings and raises an exception if not list (which is great, this is not intend as a complaint), Has the team decided it is undesirable to allow generators?

Would something like the following be a reasonable approach?

if isinstance(obj, typing.Generator):
    obj = RepeatsFirstItemGenerator(obj)

Where RepeatsFirstItemGenerator is a class that wraps a generator, caches, and repeats the first item?