'Connection.lookup' can return 'missing' and 'deferred' results

[tseaver]

Do we need to support those fields?

[tseaver]

Options:

• Have the user pass optional missing and / or deferred lists into lookup(), and populate them IFF they are not None.
• Change lookup() to always return a three-tuple, (found, missing, deferred), where each item is a list (of entities, key-only entities, and keys, respectively).

[dhermes]

I prefer the first option or just a different verb like fetch() versus fetch_page() with ndb.

[silvolu]

IRT "do we need to support", they come quite cheap in the end, so it'd be good to have them.
@pcostell, out of curiosity, how likely is that deferred contains results?
I don't have any strong opinions about your proposals.

[pcostell]

A small percentage of responses have deferred results. The field will only be populated if there were too many keys requested or the result size was too large. I would consider reissuing a lookup to get any deferred keys under the hood so that the user just gets all the results they requested.

[silvolu]

I like @pcostell proposal.

[dhermes]

I do too, with the caveat that we should let people pass in a boolean to disable the "more RPCs without telling the user" behavior and under that flag we could return the keys that were deferred?

[tseaver]

Can we have the value passed in to disable extra under-the-covers RPCs be a list, into which the contents of the deferred field from the response would be copied? We could handle missing similarly. E.g.:

deferred = []
missing = []
entities = con.lookup(dataset_id, key_pbs, deferred=deferred, missing=missing)

If deferred isn't a list, then we do the "try again" dance as suggested by @pcostell. If missing isn't a list, then we just ignore it (the status quo).

[silvolu]

SGTM. @dhermes ?

[dhermes]

Yes I like this approach. Simple and convenient.