Upgrade to google-cloud-datastore 2.x?

[snarfed]

Hi all! I know NDB is still on google-cloud-datastore 1.x since 2.0's switch to the new proto-plus generator is backward incompatible. Background in #568. Totally understandable! Those kinds of migrations can be tedious and thankless, so I can imagine why it hasn't been a priority.

There's nothing in the google-cloud-datastore 2.x changes that I'm dying to get, but being on 1.x puts us on older major versions of many other Google packages, eg google-api-core, google-auth, and google-cloud-core, and pip has been spending longer and longer recently on backtracking to find and install compatible versions of those and other packages.

Not a big deal, just filing this to track the migration to google-cloud-datastore 2.x, if and when you all are able to prioritize it. Thanks again for maintaining NDB!

[snarfed]

In case it helps, here's a trimmed log from a recent pip install on Cloud Build:

Step #2 - "build": Running "python3 --version"
Step #2 - "build": Python 3.9.7
Step #2 - "build": Done "python3 --version" (28.682635ms)
Step #2 - "build": --------------------------------------------------------------------------------
Step #2 - "build": Running "python3 -m pip install --requirement requirements.txt --upgrade --upgrade-strategy only-if-needed --no-warn-script-location --no-warn-conflicts --force-reinstall --no-compile --user (PIP_CACHE_DIR=/layers/google.python.pip/pipcache PIP_DISABLE_PIP_VERSION_CHECK=1)"
...
Step #2 - "build": Collecting google-cloud-ndb~=1.10
Step #2 - "build":   Downloading google_cloud_ndb-1.10.4-py3-none-any.whl (176 kB)
...
Step #2 - "build": Collecting google-cloud-datastore<2.0.0dev,>=1.7.0
Step #2 - "build":   Downloading google_cloud_datastore-1.15.3-py2.py3-none-any.whl (134 kB)
Step #2 - "build": Collecting google-cloud-datastore<2.0.0dev,>=1.7.0
Step #2 - "build":   Downloading google_cloud_datastore-1.15.2-py2.py3-none-any.whl (134 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.15.1-py2.py3-none-any.whl (133 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.15.0-py2.py3-none-any.whl (132 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.14.0-py2.py3-none-any.whl (131 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.13.2-py2.py3-none-any.whl (131 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.13.1-py2.py3-none-any.whl (131 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.13.0-py2.py3-none-any.whl (105 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.12.0-py2.py3-none-any.whl (97 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.11.0-py2.py3-none-any.whl (97 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.10.0-py2.py3-none-any.whl (97 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.9.0-py2.py3-none-any.whl (97 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.8.0-py2.py3-none-any.whl (96 kB)
Step #2 - "build": Collecting google-cloud-core<3.0.0dev,>=1.4.1
Step #2 - "build":   Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)
Step #2 - "build": Collecting google-cloud-datastore<2.0.0dev,>=1.7.0
Step #2 - "build":   Downloading google_cloud_datastore-1.7.4-py2.py3-none-any.whl (82 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.7.3-py2.py3-none-any.whl (80 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.7.2-py2.py3-none-any.whl (80 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.7.1-py2.py3-none-any.whl (79 kB)
Step #2 - "build":   Downloading google_cloud_datastore-1.7.0-py2.py3-none-any.whl (74 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-core to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": Collecting google-cloud-core<3.0.0dev,>=1.4.1
Step #2 - "build":   Downloading google_cloud_core-2.0.0-py2.py3-none-any.whl (27 kB)
Step #2 - "build":   Downloading google_cloud_core-1.7.1-py2.py3-none-any.whl (28 kB)
Step #2 - "build":   Downloading google_cloud_core-1.7.0-py2.py3-none-any.whl (28 kB)
Step #2 - "build":   Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB)
Step #2 - "build":   Downloading google_cloud_core-1.5.0-py2.py3-none-any.whl (27 kB)
Step #2 - "build":   Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)
Step #2 - "build":   Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)
Step #2 - "build":   Downloading google_cloud_core-1.4.2-py2.py3-none-any.whl (26 kB)
Step #2 - "build":   Downloading google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-audit-log to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": Collecting google-cloud-audit-log<1.0.0dev,>=0.1.0
Step #2 - "build":   Downloading google_cloud_audit_log-0.1.0-py2.py3-none-any.whl (14 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-appengine-logging to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": Collecting google-cloud-appengine-logging<1.0.0dev,>=0.1.0
Step #2 - "build":   Downloading google_cloud_appengine_logging-0.1.4-py2.py3-none-any.whl (14 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-core to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build":   Downloading google_cloud_appengine_logging-0.1.3-py2.py3-none-any.whl (14 kB)
Step #2 - "build":   Downloading google_cloud_appengine_logging-0.1.1-py2.py3-none-any.whl (14 kB)
Step #2 - "build":   Downloading google_cloud_appengine_logging-0.1.0-py2.py3-none-any.whl (25 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": Collecting google-auth<3.0dev,>=1.25.0
Step #2 - "build":   Downloading google_auth-2.2.1-py2.py3-none-any.whl (153 kB)
Step #2 - "build": INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Step #2 - "build": INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-audit-log to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build":   Downloading google_auth-2.2.0-py2.py3-none-any.whl (153 kB)
Step #2 - "build": INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Step #2 - "build": INFO: pip is looking at multiple versions of google-cloud-appengine-logging to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build":   Downloading google_auth-2.1.0-py2.py3-none-any.whl (153 kB)
Step #2 - "build":   Downloading google_auth-2.0.2-py2.py3-none-any.whl (152 kB)
Step #2 - "build": INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Step #2 - "build":   Downloading google_auth-2.0.1-py2.py3-none-any.whl (152 kB)
Step #2 - "build":   Downloading google_auth-2.0.0-py2.py3-none-any.whl (152 kB)
Step #2 - "build":   Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build":   Downloading google_auth-1.34.0-py2.py3-none-any.whl (152 kB)
Step #2 - "build":   Downloading google_auth-1.33.1-py2.py3-none-any.whl (152 kB)
Step #2 - "build":   Downloading google_auth-1.33.0-py2.py3-none-any.whl (151 kB)
Step #2 - "build":   Downloading google_auth-1.32.1-py2.py3-none-any.whl (147 kB)
Step #2 - "build":   Downloading google_auth-1.32.0-py2.py3-none-any.whl (147 kB)
Step #2 - "build": INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Step #2 - "build":   Downloading google_auth-1.31.0-py2.py3-none-any.whl (147 kB)
Step #2 - "build":   Downloading google_auth-1.30.2-py2.py3-none-any.whl (146 kB)
Step #2 - "build":   Downloading google_auth-1.30.1-py2.py3-none-any.whl (146 kB)
Step #2 - "build":   Downloading google_auth-1.30.0-py2.py3-none-any.whl (146 kB)
Step #2 - "build":   Downloading google_auth-1.29.0-py2.py3-none-any.whl (142 kB)
Step #2 - "build":   Downloading google_auth-1.28.1-py2.py3-none-any.whl (136 kB)
Step #2 - "build":   Downloading google_auth-1.28.0-py2.py3-none-any.whl (136 kB)
Step #2 - "build":   Downloading google_auth-1.27.1-py2.py3-none-any.whl (136 kB)
Step #2 - "build":   Downloading google_auth-1.27.0-py2.py3-none-any.whl (135 kB)
Step #2 - "build":   Downloading google_auth-1.26.1-py2.py3-none-any.whl (116 kB)
Step #2 - "build":   Downloading google_auth-1.26.0-py2.py3-none-any.whl (135 kB)
Step #2 - "build":   Downloading google_auth-1.25.0-py2.py3-none-any.whl (116 kB)
Step #2 - "build": INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Step #2 - "build": Collecting google-api-core[grpc]<3.0.0dev,>=1.26.0
Step #2 - "build":   Downloading google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
Step #2 - "build":   Downloading google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Step #2 - "build":   Downloading google_api_core-1.31.3-py2.py3-none-any.whl (93 kB)
...
Step #2 - "build": Successfully installed ... gdata-python3-3.0.1 google-api-core-1.31.3 google-auth-1.35.0 google-cloud-appengine-logging-0.1.5 google-cloud-audit-log-0.1.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-error-reporting-1.2.3 google-cloud-logging-2.6.0 google-cloud-ndb-1.10.4 google-cloud-tasks-2.6.0 googleapis-common-protos-1.53.0 ...

[tseaver]

@snarfed Thanks for the report. Because of google-cloud-ndb's role as a forward-compatibility bridge for classic GAE users, we have a current mandate to keep the main branch compatible with Python 2.7, which is the driver for pinning google-cloud-datastore < 2.0dev (and, transitively, the Python 2.7-compatible versions of google-api-core, google-cloud-core, google-auth, etc.)

I've argued elsewhere that we could keep a v1 branch around, make from the current tip of main, and then have main drop support for Python 2.7, but my argument was, alas, not persuasive enough.

/cc @andrewsg

[snarfed]

Thanks @tseaver et al! Definitely understood. We may need to keep looking at options here, though: I just noticed that the transitive pin to google-auth 1.35.0 depends on rsa<4.6, and there's a year old CVE for rsa that's only fixed in 4.7 and up:

sybrenstuvel/python-rsa#165
https://snyk.io/vuln/SNYK-PYTHON-RSA-1038401
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-25658

Looks like there isn't perfect consensus on severity, some people are calling it high, others only medium...but still, I'd rather not leave my services and users vulnerable for longer than I have to. Let me know if I can help at all here!

[snarfed]

Friendly nudge? I'm all for backward compatibility and being a bridge, but hopefully that doesn't mean we have to leave security vulnerabilities unpatched. Any thoughts on how to reconcile the two?

[tseaver]

@snarfed a couple of things:

• Within google-auth-library-python, the rsa dependency is a fallback for users who cannot install cryptography.
• Even when falling back to rsa, google-auth uses the library only to sign bytes and verify signatures, which does not (AFAICT) have the same vulnerability (signatures are fixed-length, the verify method does not search for b'\x00).
• The pin you point to is only valid for Python 2.7, for which the rsa package dropped support in their 4.6 release. E.g.:

$ git remote -v
origin	git@github.com:googleapis/python-ndb (fetch)
origin	git@github.com:googleapis/python-ndb (push)
$ ls -d .nox/unit-2-7/lib/python2.7/site-packages/rsa*
.nox/unit-2-7/lib/python2.7/site-packages/rsa
.nox/unit-2-7/lib/python2.7/site-packages/rsa-4.5.dist-info
$ ls -d .nox/unit-3-6/lib/python3.6/site-packages/rsa*
.nox/unit-3-6/lib/python3.6/site-packages/rsa
.nox/unit-3-6/lib/python3.6/site-packages/rsa-4.7.2.dist-info

So, the real exposure is for people who:

• Still run 2.7 nearly two years after its final drop-dead date;
• For whatever reason cannot install crpytography (which made its last 2.7-compatible release in February 2021);
• Are otherwise using the rsa module outside of the scope of the use made of it by google-auth.

One could make the case that google-auth-library-python should be pushing people more strongly to install cryptography, but that isn't an issue for python-ndb to address. I suspect that "Classic" GAE (not GAE Flex might be the one place where Python 2.7 without cryptography might actually obtain.

[snarfed]

Thank you for the deep dive! That all makes sense; I'll happily agree that this vulnerability doesn't seem to affect me or most ndb users. Definitely reassuring.

I'm still a bit curious about the broader question of reconciling backward compatibility, notably google-cloud-datastore < 2.0dev, with security patches. Do you know if google-cloud-datastore still patches and releases a 1.x branch? If not, any idea how you all will handle the next vulnerability there, or in any other of your frozen deps, that affect more of your users?

[tseaver]

@snarfed

I'm still a bit curious about the broader question of reconciling backward compatibility, notably google-cloud-datastore < 2.0dev, with security patches. Do you know if google-cloud-datastore still patches and releases a 1.x branch? If not, any idea how you all will handle the next vulnerability there, or in any other of your frozen deps, that affect more of your users?

We haven't kept a v1 branch active. If a sufficiently urgent vulnerability appeared, we could start that branch from the last 1.x release tag, and then handle releasing from it: we do this already for python-api-core (e.g. googleapis/python-api-core#305 ).

[snarfed]

Glad to hear you're ready to do this if/when the need arises. Thanks again for the detailed explanations. And for all your work maintaining ndb and the rest of these libraries!