Skip to content

PYTHON-3460 Implement OIDC SASL mechanism #1138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 138 commits into from
May 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
2381380
PYTHON-3460 Implement OIDC SASL mechanism
blink1073 Dec 2, 2022
5048aea
fix log cat
blink1073 Jan 11, 2023
17972eb
print tokens
blink1073 Jan 11, 2023
0380ded
try again
blink1073 Jan 11, 2023
950df79
undo comments
blink1073 Jan 17, 2023
ab10e68
test multiple principals
blink1073 Jan 19, 2023
e9a8992
fix handling of principal name
blink1073 Jan 19, 2023
883d427
try different token
blink1073 Jan 19, 2023
ccf5a3f
test with auth_oidc scripts
blink1073 Jan 20, 2023
be21e22
use bash shell
blink1073 Jan 20, 2023
1595f31
fix handling of python bin
blink1073 Jan 20, 2023
8bd23c6
do not cache aws device creds
blink1073 Jan 23, 2023
0e93762
remove skipping server step 1 and add todo
blink1073 Jan 24, 2023
eef2712
update handling of cache key
blink1073 Jan 24, 2023
8e5638e
fix clientId name
blink1073 Jan 24, 2023
905b89d
handle when there is no principal name
blink1073 Jan 24, 2023
43d49ad
handle device_name
blink1073 Jan 24, 2023
e93c4c0
add lock
blink1073 Jan 24, 2023
324437e
fix locking logic
blink1073 Jan 24, 2023
0ddeaf3
update for running both servers on the same replicaset
blink1073 Jan 25, 2023
b9c0d47
update oidc bootstrap
blink1073 Jan 25, 2023
50b72e1
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 Jan 26, 2023
0313d6c
fix lint and typings
blink1073 Jan 26, 2023
b02a969
test 27017
blink1073 Jan 26, 2023
8d6d503
update for spec compliance
blink1073 Jan 26, 2023
1a12de9
add mechanism field
blink1073 Jan 26, 2023
f408386
clean up caching behavior
blink1073 Jan 27, 2023
b7f161a
typing fixes
blink1073 Jan 27, 2023
2826753
clean up caching behavior
blink1073 Jan 30, 2023
57046f8
cleanup
blink1073 Jan 30, 2023
9f4fb49
add handling of auth connection string tests
blink1073 Jan 30, 2023
62d78dc
add initial handling of reauth required
blink1073 Feb 1, 2023
8bff0ff
finish reauthentication
blink1073 Feb 1, 2023
35182f3
add timeout handling
blink1073 Feb 2, 2023
7ac2fed
fix connection string tests
blink1073 Feb 2, 2023
8ce95bc
add callback return validation
blink1073 Feb 2, 2023
0ecfba4
add more callback validation
blink1073 Feb 2, 2023
b42cc7d
fix function validation
blink1073 Feb 2, 2023
834c130
implement some prose tests
blink1073 Feb 2, 2023
818911d
add unified tests for reconnect
blink1073 Feb 3, 2023
3ccd850
update tests
blink1073 Feb 7, 2023
a59ba53
finish prose tests
blink1073 Feb 8, 2023
5f59c31
update docs and prose test on reauthentication
blink1073 Feb 8, 2023
2cc0a62
debug
blink1073 Feb 8, 2023
7479a24
debug
blink1073 Feb 8, 2023
caabfb6
try clearing auth_ctx
blink1073 Feb 8, 2023
b236f2a
try another way to re-auth
blink1073 Feb 8, 2023
eab4d83
try another way to re-auth
blink1073 Feb 8, 2023
933be5f
cleanup
blink1073 Feb 8, 2023
20aecb7
try again
blink1073 Feb 8, 2023
7ed72fe
try again
blink1073 Feb 8, 2023
c4c64bc
use username as principal name
blink1073 Feb 9, 2023
3acf277
add read and write reauth tests using failcommand
blink1073 Feb 9, 2023
0da3037
lint
blink1073 Feb 9, 2023
370ef94
add event listeners for reauth tests
blink1073 Feb 10, 2023
10e9f20
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 Feb 10, 2023
0abc433
add changelog and example
blink1073 Feb 10, 2023
93a4884
clean up example
blink1073 Feb 10, 2023
b0d1e3b
clean up example
blink1073 Feb 10, 2023
256915a
clean up reauth handling
blink1073 Feb 10, 2023
51c6639
clean up server response
blink1073 Feb 10, 2023
b78e404
clean up config
blink1073 Feb 10, 2023
ce250de
typing
blink1073 Feb 10, 2023
3339837
clarify the supported auth types
blink1073 Feb 13, 2023
2c33731
use mongosh
blink1073 Feb 13, 2023
529cf63
add test for multiple principals and no username
blink1073 Feb 14, 2023
5f0fde0
add principal name to callbacks, and include callbacks in cache key
blink1073 Feb 17, 2023
765dde6
fix test
blink1073 Feb 17, 2023
6186ed9
update cache exp on access
blink1073 Feb 21, 2023
c2e91e2
switch to OIDC_TOKEN_DIR
blink1073 Feb 22, 2023
ca6dd7d
switch to activate-authoidcvenv.sh
blink1073 Feb 22, 2023
5f730ec
upgrade pip and setuptools
blink1073 Feb 22, 2023
f1998d6
remove reauth write prose test
blink1073 Feb 22, 2023
ba2253b
remove temp file
blink1073 Feb 23, 2023
b023041
use main drivers-evergreen-tools
blink1073 Feb 24, 2023
3b6288b
install xml and coverage
blink1073 Feb 27, 2023
a574237
better reauth support
blink1073 Feb 27, 2023
3607f72
Revert "install xml and coverage"
blink1073 Feb 27, 2023
75a707b
implement speculativeAuth for OIDC
blink1073 Feb 28, 2023
35fc668
fix errors
blink1073 Feb 28, 2023
3ceda8a
clarify reauth behavior
blink1073 Feb 28, 2023
6976ef8
fix speculativeAuth and add prose tests
blink1073 Mar 1, 2023
43a5269
update docstring
blink1073 Mar 2, 2023
40ed893
rename DEVICE_NAME to PRINCIPAL_NAME
blink1073 Mar 7, 2023
a46ba28
fix test runner
blink1073 Mar 7, 2023
c7df390
better reauth support
blink1073 Mar 9, 2023
02ce52d
add test numbering and new prose test
blink1073 Mar 13, 2023
28a804b
lint
blink1073 Mar 13, 2023
19dda74
wip clean up tests
blink1073 Mar 14, 2023
9c33a02
wip refractor with test headings
blink1073 Mar 14, 2023
c21771f
refactor prose tests
blink1073 Mar 14, 2023
bad4527
remove speculative auth error test
blink1073 Mar 14, 2023
7beac7a
Refactor to address review
blink1073 Mar 20, 2023
8c8088a
update for prose test clarifications
blink1073 Mar 20, 2023
6c45619
lint
blink1073 Mar 20, 2023
500c98f
allow for extra keys
blink1073 Mar 20, 2023
11261b9
validate callback results
blink1073 Mar 23, 2023
f4ebaf6
updates for security mitigations
blink1073 Mar 30, 2023
f584885
make allowed_hosts a mechanism property
blink1073 Mar 31, 2023
beb2b24
fix auth spec test
blink1073 Mar 31, 2023
6d3ebe2
Merge principal name and timeout into client info object
blink1073 Apr 3, 2023
e46047d
fix validation
blink1073 Apr 3, 2023
3454bab
fix auth spec test
blink1073 Apr 3, 2023
8e8912d
strip sasl commands
blink1073 Apr 4, 2023
2133125
clean up handling of allowed hosts
blink1073 Apr 5, 2023
e9024ec
add another allowed_host and update docs
blink1073 Apr 6, 2023
f522d0d
refactor and adjust allowed_hosts handling
blink1073 Apr 17, 2023
b0c0919
update callback parameters
blink1073 Apr 17, 2023
5828220
clear all info on reauth if no refresh callback
blink1073 Apr 17, 2023
a78ad30
add reauth guard
blink1073 Apr 17, 2023
ef991f9
wip better handling of reauth and locks
blink1073 Apr 18, 2023
4cf5110
add prose tests for reauth and lock guards
blink1073 Apr 19, 2023
a354ce6
use gen_id instead of datetimes
blink1073 Apr 20, 2023
66f2276
update callback parameters
blink1073 Apr 25, 2023
710add4
remove principal name from context
blink1073 Apr 26, 2023
b98aa5c
update for spec changes
blink1073 Apr 28, 2023
5fe805f
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 Apr 28, 2023
eb3ced0
use unittest asserts
blink1073 Apr 28, 2023
c7d33b2
address review
blink1073 May 2, 2023
d23ef6b
address review
blink1073 May 3, 2023
b431cdc
address review
blink1073 May 3, 2023
504d2b1
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 May 3, 2023
a089dc0
address review
blink1073 May 3, 2023
7c56888
try more bytes
blink1073 May 3, 2023
7bb678f
Revert "try more bytes"
blink1073 May 3, 2023
bd53fdc
wip better handling of reauth
blink1073 May 3, 2023
b13427e
Revert "wip better handling of reauth"
blink1073 May 4, 2023
8d24dd9
clean up reauth handling
blink1073 May 5, 2023
c73be94
lint
blink1073 May 5, 2023
1924f3e
lint
blink1073 May 5, 2023
770e4f6
always use for source
blink1073 May 8, 2023
828e22a
undo change to uri
blink1073 May 8, 2023
c719618
Revert "undo change to uri"
blink1073 May 8, 2023
1739968
address review
blink1073 May 10, 2023
f4d6f24
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 May 10, 2023
cbadc70
fix uri
blink1073 May 10, 2023
109a204
fix handling of no_mongos
blink1073 May 11, 2023
7baf5a7
update schema version
blink1073 May 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .evergreen/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -749,6 +749,68 @@ functions:
fi
PYTHON_BINARY=${PYTHON_BINARY} ASSERT_NO_URI_CREDS=true .evergreen/run-mongodb-aws-test.sh

"bootstrap oidc":
- command: ec2.assume_role
params:
role_arn: ${aws_test_secrets_role}
- command: shell.exec
type: test
params:
working_dir: "src"
shell: bash
script: |
${PREPARE_SHELL}
if [ "${skip_EC2_auth_test}" = "true" ]; then
echo "This platform does not support the oidc auth test, skipping..."
exit 0
fi

cd ${DRIVERS_TOOLS}/.evergreen/auth_oidc
export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
export AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}
export OIDC_TOKEN_DIR=/tmp/tokens

. ./activate-authoidcvenv.sh
python oidc_write_orchestration.py
python oidc_get_tokens.py

"run oidc auth test with aws credentials":
- command: shell.exec
type: test
params:
working_dir: "src"
shell: bash
script: |
${PREPARE_SHELL}
if [ "${skip_EC2_auth_test}" = "true" ]; then
echo "This platform does not support the oidc auth test, skipping..."
exit 0
fi
cd ${DRIVERS_TOOLS}/.evergreen/auth_oidc
mongosh setup_oidc.js
- command: shell.exec
type: test
params:
working_dir: "src"
silent: true
script: |
# DO NOT ECHO WITH XTRACE (which PREPARE_SHELL does)
cat <<'EOF' > "${PROJECT_DIRECTORY}/prepare_mongodb_oidc.sh"
export OIDC_TOKEN_DIR=/tmp/tokens
EOF
- command: shell.exec
type: test
params:
working_dir: "src"
script: |
${PREPARE_SHELL}
if [ "${skip_web_identity_auth_test}" = "true" ]; then
echo "This platform does not support the oidc auth test, skipping..."
exit 0
fi
PYTHON_BINARY=${PYTHON_BINARY} ASSERT_NO_URI_CREDS=true .evergreen/run-mongodb-oidc-test.sh

"run aws auth test with aws credentials as environment variables":
- command: shell.exec
type: test
Expand Down Expand Up @@ -2034,6 +2096,19 @@ tasks:
- func: "run aws auth test with aws web identity credentials"
- func: "run aws ECS auth test"

- name: "oidc-auth-test-latest"
commands:
- func: "bootstrap oidc"
- func: "bootstrap mongo-orchestration"
vars:
AUTH: "auth"
ORCHESTRATION_FILE: "auth-oidc.json"
TOPOLOGY: "replica_set"
VERSION: "latest"
- func: "run oidc auth test with aws credentials"
vars:
AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/tokens/test1

- name: load-balancer-test
commands:
- func: "bootstrap mongo-orchestration"
Expand Down Expand Up @@ -3103,6 +3178,14 @@ buildvariants:
# macOS MongoDB servers do not staple OCSP responses and only support RSA.
- name: ".ocsp-rsa !.ocsp-staple"

- matrix_name: "oidc-auth-test"
matrix_spec:
platform: [ ubuntu-20.04 ]
python-version: ["3.9"]
display_name: "MONGODB-OIDC Auth ${platform} ${python-version}"
tasks:
- name: "oidc-auth-test-latest"

- matrix_name: "aws-auth-test"
matrix_spec:
platform: [ubuntu-20.04]
Expand Down
3 changes: 3 additions & 0 deletions .evergreen/resync-specs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ for spec in "$@"
do
# Match the spec dir name, the python test dir name, and/or common abbreviations.
case "$spec" in
auth)
cpjson auth/tests/ auth
;;
atlas-data-lake-testing|data_lake)
cpjson atlas-data-lake-testing/tests/ data_lake
;;
Expand Down
85 changes: 85 additions & 0 deletions .evergreen/run-mongodb-oidc-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
#!/bin/bash

set -o xtrace
set -o errexit # Exit the script with error if any of the commands fail

############################################
# Main Program #
############################################

# Supported/used environment variables:
# MONGODB_URI Set the URI, including an optional username/password to use
# to connect to the server via MONGODB-OIDC authentication
# mechanism.
# PYTHON_BINARY The Python version to use.

echo "Running MONGODB-OIDC authentication tests"
# ensure no secrets are printed in log files
set +x

# load the script
shopt -s expand_aliases # needed for `urlencode` alias
[ -s "${PROJECT_DIRECTORY}/prepare_mongodb_oidc.sh" ] && source "${PROJECT_DIRECTORY}/prepare_mongodb_oidc.sh"

MONGODB_URI=${MONGODB_URI:-"mongodb://localhost"}
MONGODB_URI_SINGLE="${MONGODB_URI}/?authMechanism=MONGODB-OIDC"
MONGODB_URI_MULTIPLE="${MONGODB_URI}:27018/?authMechanism=MONGODB-OIDC&directConnection=true"

if [ -z "${OIDC_TOKEN_DIR}" ]; then
echo "Must specify OIDC_TOKEN_DIR"
exit 1
fi

export MONGODB_URI_SINGLE="$MONGODB_URI_SINGLE"
export MONGODB_URI_MULTIPLE="$MONGODB_URI_MULTIPLE"
export MONGODB_URI="$MONGODB_URI"

echo $MONGODB_URI_SINGLE
echo $MONGODB_URI_MULTIPLE
echo $MONGODB_URI

if [ "$ASSERT_NO_URI_CREDS" = "true" ]; then
if echo "$MONGODB_URI" | grep -q "@"; then
echo "MONGODB_URI unexpectedly contains user credentials!";
exit 1
fi
fi

# show test output
set -x

# Workaround macOS python 3.9 incompatibility with system virtualenv.
if [ "$(uname -s)" = "Darwin" ]; then
VIRTUALENV="/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 -m virtualenv"
else
VIRTUALENV=$(command -v virtualenv)
fi

authtest () {
if [ "Windows_NT" = "$OS" ]; then
PYTHON=$(cygpath -m $PYTHON)
fi

echo "Running MONGODB-OIDC authentication tests with $PYTHON"
$PYTHON --version

$VIRTUALENV -p $PYTHON --never-download venvoidc
if [ "Windows_NT" = "$OS" ]; then
. venvoidc/Scripts/activate
else
. venvoidc/bin/activate
fi
python -m pip install -U pip setuptools
python -m pip install '.[aws]'
python test/auth_aws/test_auth_oidc.py -v
deactivate
rm -rf venvoidc
}

PYTHON=${PYTHON_BINARY:-}
if [ -z "$PYTHON" ]; then
echo "Cannot test without specifying PYTHON_BINARY"
exit 1
fi

authtest
62 changes: 53 additions & 9 deletions pymongo/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
from bson.binary import Binary
from bson.son import SON
from pymongo.auth_aws import _authenticate_aws
from pymongo.auth_oidc import _authenticate_oidc, _get_authenticator, _OIDCProperties
from pymongo.errors import ConfigurationError, OperationFailure
from pymongo.saslprep import saslprep

Expand All @@ -48,6 +49,7 @@
[
"GSSAPI",
"MONGODB-CR",
"MONGODB-OIDC",
"MONGODB-X509",
"MONGODB-AWS",
"PLAIN",
Expand Down Expand Up @@ -101,7 +103,7 @@ def __hash__(self):

def _build_credentials_tuple(mech, source, user, passwd, extra, database):
"""Build and return a mechanism specific credentials tuple."""
if mech not in ("MONGODB-X509", "MONGODB-AWS") and user is None:
if mech not in ("MONGODB-X509", "MONGODB-AWS", "MONGODB-OIDC") and user is None:
raise ConfigurationError("%s requires a username." % (mech,))
if mech == "GSSAPI":
if source is not None and source != "$external":
Expand Down Expand Up @@ -137,6 +139,32 @@ def _build_credentials_tuple(mech, source, user, passwd, extra, database):
aws_props = _AWSProperties(aws_session_token=aws_session_token)
# user can be None for temporary link-local EC2 credentials.
return MongoCredential(mech, "$external", user, passwd, aws_props, None)
elif mech == "MONGODB-OIDC":
properties = extra.get("authmechanismproperties", {})
request_token_callback = properties.get("request_token_callback")
refresh_token_callback = properties.get("refresh_token_callback", None)
provider_name = properties.get("PROVIDER_NAME", "")
default_allowed = [
"*.mongodb.net",
"*.mongodb-dev.net",
"*.mongodbgov.net",
"localhost",
"127.0.0.1",
"::1",
]
allowed_hosts = properties.get("allowed_hosts", default_allowed)
if not request_token_callback and provider_name != "aws":
raise ConfigurationError(
"authentication with MONGODB-OIDC requires providing an request_token_callback or a provider_name of 'aws'"
)
oidc_props = _OIDCProperties(
request_token_callback=request_token_callback,
refresh_token_callback=refresh_token_callback,
provider_name=provider_name,
allowed_hosts=allowed_hosts,
)
return MongoCredential(mech, "$external", user, passwd, oidc_props, None)

elif mech == "PLAIN":
source_database = source or database or "$external"
return MongoCredential(mech, source_database, user, passwd, None, None)
Expand Down Expand Up @@ -439,7 +467,7 @@ def _authenticate_x509(credentials, sock_info):
# MONGODB-X509 is done after the speculative auth step.
return

cmd = _X509Context(credentials).speculate_command()
cmd = _X509Context(credentials, sock_info.address).speculate_command()
sock_info.command("$external", cmd)


Expand Down Expand Up @@ -482,6 +510,7 @@ def _authenticate_default(credentials, sock_info):
"MONGODB-CR": _authenticate_mongo_cr,
"MONGODB-X509": _authenticate_x509,
"MONGODB-AWS": _authenticate_aws,
"MONGODB-OIDC": _authenticate_oidc,
"PLAIN": _authenticate_plain,
"SCRAM-SHA-1": functools.partial(_authenticate_scram, mechanism="SCRAM-SHA-1"),
"SCRAM-SHA-256": functools.partial(_authenticate_scram, mechanism="SCRAM-SHA-256"),
Expand All @@ -490,15 +519,16 @@ def _authenticate_default(credentials, sock_info):


class _AuthContext(object):
def __init__(self, credentials):
def __init__(self, credentials, address):
self.credentials = credentials
self.speculative_authenticate = None
self.address = address

@staticmethod
def from_credentials(creds):
def from_credentials(creds, address):
spec_cls = _SPECULATIVE_AUTH_MAP.get(creds.mechanism)
if spec_cls:
return spec_cls(creds)
return spec_cls(creds, address)
return None

def speculate_command(self):
Expand All @@ -512,8 +542,8 @@ def speculate_succeeded(self):


class _ScramContext(_AuthContext):
def __init__(self, credentials, mechanism):
super(_ScramContext, self).__init__(credentials)
def __init__(self, credentials, address, mechanism):
super(_ScramContext, self).__init__(credentials, address)
self.scram_data = None
self.mechanism = mechanism

Expand All @@ -534,16 +564,30 @@ def speculate_command(self):
return cmd


class _OIDCContext(_AuthContext):
def speculate_command(self):
authenticator = _get_authenticator(self.credentials, self.address)
cmd = authenticator.auth_start_cmd(False)
if cmd is None:
return
cmd["db"] = self.credentials.source
return cmd


_SPECULATIVE_AUTH_MAP: Mapping[str, Callable] = {
"MONGODB-X509": _X509Context,
"SCRAM-SHA-1": functools.partial(_ScramContext, mechanism="SCRAM-SHA-1"),
"SCRAM-SHA-256": functools.partial(_ScramContext, mechanism="SCRAM-SHA-256"),
"MONGODB-OIDC": _OIDCContext,
"DEFAULT": functools.partial(_ScramContext, mechanism="SCRAM-SHA-256"),
}


def authenticate(credentials, sock_info):
def authenticate(credentials, sock_info, reauthenticate=False):
"""Authenticate sock_info."""
mechanism = credentials.mechanism
auth_func = _AUTH_MAP[mechanism]
auth_func(credentials, sock_info)
if mechanism == "MONGODB-OIDC":
_authenticate_oidc(credentials, sock_info, reauthenticate)
else:
auth_func(credentials, sock_info)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes it seem like non-ODIC mechs also support reauthentication. Is that intended?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we are using SCRAM in the unified spec tests.

Loading