Skip to content

Commit 2fa07b2

Browse files
feat: Implement support for postgresql (#49)
* Implement postgresql support
1 parent 669e8d8 commit 2fa07b2

File tree

9 files changed

+459
-63
lines changed

9 files changed

+459
-63
lines changed

README.md

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,13 @@
44
This project is a self-deployed service that provides support for managing [Cloud SQL IAM Database Authentication](https://cloud.google.com/sql/docs/mysql/authentication) for groups. This service leverages [Cloud Run](https://cloud.google.com/run), [Cloud Scheduler](https://cloud.google.com/scheduler), and the [Cloud SQL Python Connector](https://github.com/googlecloudplatform/cloud-sql-python-connector) to consistently update and sync Cloud SQL instances based on IAM groups. It will create missing database IAM users, GRANT roles to database IAM users based on their IAM groups, and REVOKE roles from database IAM users no longer in IAM groups.
55

66
## Supported Databases
7-
Currently only **MySQL 8.0** databases are supported.
7+
Currently only the following databases are supported:
8+
- **MySQL 8.0**
9+
- **PostgreSQL 13**
10+
- **PostgreSQL 12**
11+
- **PostgreSQL 11**
12+
- **PostgreSQL 10**
13+
- **PostgreSQL 9.6**
814

915
## Overview
1016
The Cloud SQL IAM Database Authentication for Groups service at an overview is made of Cloud Scheduler Job(s) and Cloud Run instance(s).
@@ -141,7 +147,7 @@ To properly manage the database users on each Cloud SQL instance that is configu
141147
Add the service account as an IAM authenticated database user on each Cloud SQL instance that needs managing through IAM groups. Can be done both manually through the Google Cloud Console or through the following `gcloud` command.
142148

143149
Replace the following values:
144-
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account.
150+
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account. (**NOTE**: For Postgres instances, remove the `.gserviceaccount.com` suffix from service account email.)
145151
- `INSTANCE_NAME`: The name of a Cloud SQL instance.
146152
```
147153
gcloud sql users create <SERVICE_ACCOUNT_EMAIL> \
@@ -156,7 +162,8 @@ Connect to all Cloud SQL instances in question with an admin user or another dat
156162

157163
Once connected, grant the service account IAM database user the following permissions:
158164

159-
Replace the following values in the above commands:
165+
#### MySQL Instance
166+
Replace the following values in the below commands:
160167
- `SERVICE_ACCOUNT_ID`: The ID (name) for the service account (everything before the **@** portion of email)
161168
Allow the service account to read database users and their roles.
162169
```
@@ -173,6 +180,15 @@ Allow the service account to **GRANT/REVOKE** roles to users through being a **R
173180
GRANT ROLE_ADMIN ON *.* TO '<SERVICE_ACCOUNT_ID>';
174181
```
175182

183+
#### PostgreSQL Instance
184+
Postgres allows a role or user to easily be granted the appropriate permissions for **CREATE**, and **GRANT/REVOKE** that are needed for creating and managing the group roles for IAM groups with one single command.
185+
186+
Replace the following values:
187+
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account with the `.gserviceaccount.com` suffix removed.
188+
```
189+
ALTER ROLE "<SERVICE_ACCOUNT_EMAIL>" WITH CREATEROLE;
190+
```
191+
176192
## Deploying to Cloud Run
177193
To build and deploy the service to Cloud Run, run the following commands:
178194

@@ -189,7 +205,7 @@ gcloud builds submit \
189205
Deploy Cloud Run Service from container image:
190206

191207
Replace the following values:
192-
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account.
208+
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account created above.
193209
- `PROJECT_ID`: The Google Cloud project ID.
194210
```
195211
gcloud run deploy iam-db-authn-groups \
@@ -234,7 +250,7 @@ An example command creating a Cloud Scheduler job to run the IAM database authen
234250
Replace the following values:
235251
- `JOB_NAME`: The name for the Cloud Scheduler job.
236252
- `SERVICE_URL`: The service URL of the Cloud Run service.
237-
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account.
253+
- `SERVICE_ACCOUNT_EMAIL`: The email address for the service account created above.
238254
- `PATH_TO_PAYLOAD`: Path to payload JSON file.
239255
```
240256
gcloud scheduler jobs create http \
@@ -264,7 +280,7 @@ The name of the mapped IAM group database role is the email of the IAM group wit
264280

265281
The service verifies that a group role exists or creates one on the database if it does not exist. It is recommended to configure the Cloud Scheduler job(s) and after having it triggered **at least** once, have a Database Administrator or project admin verify the creation of the group roles and **GRANT** the group roles the appropriate privileges on each Cloud SQL instance that should be inherited by database users of those IAM groups on all consecutive Cloud Scheduler runs.
266282

267-
To verify the creation of group roles after Cloud Scheduler has triggered at least once, the following command can be run:
283+
To verify the creation of group roles after Cloud Scheduler has triggered at least once, the following command can be run for **MySQL** instances (**PostgreSQL** instances require connecting to the database to verify):
268284

269285
Replace the following values:
270286
- `INSTANCE_NAME`: The name of a Cloud SQL instance that was configured in the Cloud Scheduler JSON payload.

app.py

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,15 +31,22 @@
3131
InstanceConnectionName,
3232
)
3333
from iam_groups_authn.iam_admin import get_iam_users
34-
from iam_groups_authn.mysql import init_connection_engine, RoleService, mysql_username
34+
from iam_groups_authn.utils import DatabaseVersion
35+
from iam_groups_authn.mysql import (
36+
init_mysql_connection_engine,
37+
MysqlRoleService,
38+
mysql_username,
39+
)
40+
from iam_groups_authn.postgres import (
41+
init_postgres_connection_engine,
42+
PostgresRoleService,
43+
)
3544

3645
# define scopes
3746
SCOPES = [
3847
"https://www.googleapis.com/auth/admin.directory.group.member.readonly",
3948
"https://www.googleapis.com/auth/sqlservice.admin",
4049
]
41-
# supported database types
42-
SUPPORTED_DATABASES = ["MYSQL_8_0"]
4350

4451
app = Quart(__name__)
4552

@@ -125,10 +132,12 @@ async def run_groups_authn():
125132

126133
# get database version of instance and check if supported
127134
database_version = await instance_tasks[instance][1]
128-
if database_version not in SUPPORTED_DATABASES:
135+
try:
136+
database_version = DatabaseVersion(database_version)
137+
except ValueError as e:
129138
raise ValueError(
130-
f"Unsupported database version for instance `{instance}`. Current supported versions are: {SUPPORTED_DATABASES}"
131-
)
139+
f"Unsupported database version for instance `{instance}`. Current supported versions are: {list(DatabaseVersion.__members__.keys())}"
140+
) from e
132141

133142
# add missing IAM group members to database
134143
add_users_task = asyncio.create_task(
@@ -137,12 +146,21 @@ async def run_groups_authn():
137146
group_tasks[group],
138147
instance_tasks[instance][0],
139148
instance,
149+
database_version,
140150
)
141151
)
142152

143-
# initialize database engine
144-
db = init_connection_engine(instance, updated_creds, ip_type)
145-
role_service = RoleService(db)
153+
# initialize database connection pool
154+
if database_version.is_mysql():
155+
db = init_mysql_connection_engine(instance, updated_creds, ip_type)
156+
role_service = MysqlRoleService(db)
157+
else:
158+
db = init_postgres_connection_engine(instance, updated_creds, ip_type)
159+
role_service = PostgresRoleService(db)
160+
logging.debug(
161+
"[%s][%s] Initialized a %s connection pool."
162+
% (instance, group, database_version.value)
163+
)
146164

147165
# verify role for IAM group exists on database, create if does not exist
148166
role = mysql_username(group)
@@ -176,6 +194,7 @@ async def run_groups_authn():
176194
role,
177195
users_with_roles_task,
178196
group_tasks[group],
197+
database_version,
179198
)
180199
)
181200

@@ -186,6 +205,7 @@ async def run_groups_authn():
186205
role,
187206
users_with_roles_task,
188207
group_tasks[group],
208+
database_version,
189209
)
190210
)
191211
results = await asyncio.gather(

iam_groups_authn/mysql.py

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
import sqlalchemy
1919
from google.cloud.sql.connector import connector
2020
from google.cloud.sql.connector.instance_connection_manager import IPTypes
21-
from iam_groups_authn.utils import async_wrap
21+
from iam_groups_authn.utils import RoleService, async_wrap
2222
from google.auth.transport.requests import Request
2323

2424

@@ -38,11 +38,11 @@ def mysql_username(iam_email):
3838
return username
3939

4040

41-
class RoleService:
42-
"""Class for managing a DB user's role grants."""
41+
class MysqlRoleService(RoleService):
42+
"""Class for managing a MySQL DB user's role grants."""
4343

4444
def __init__(self, db):
45-
"""Initialize a RoleService object.
45+
"""Initialize a MysqlRoleService object.
4646
4747
Args:
4848
db: Database connection object.
@@ -59,12 +59,13 @@ def fetch_role_grants(self, group_name):
5959
Returns:
6060
results: List of results for given query.
6161
"""
62+
# mysql query to get users with group role
63+
stmt = sqlalchemy.text(
64+
"SELECT FROM_USER, TO_USER FROM mysql.role_edges WHERE FROM_USER= :group_name"
65+
)
6266
# create connection to db instance
6367
with self.db.connect() as db_connection:
64-
# query role_edges table
65-
stmt = sqlalchemy.text(
66-
"SELECT FROM_USER, TO_USER FROM mysql.role_edges WHERE FROM_USER= :group_name"
67-
)
68+
# query users with roles
6869
results = db_connection.execute(stmt, {"group_name": group_name}).fetchall()
6970
return results
7071

@@ -78,9 +79,8 @@ def create_group_role(self, role):
7879
Args:
7980
role: Name of group role to be verified or created as new role.
8081
"""
81-
# create connection to db instance
82+
stmt = sqlalchemy.text("CREATE ROLE IF NOT EXISTS :role")
8283
with self.db.connect() as db_connection:
83-
stmt = sqlalchemy.text("CREATE ROLE IF NOT EXISTS :role")
8484
db_connection.execute(stmt, {"role": role})
8585

8686
@async_wrap
@@ -116,8 +116,10 @@ def revoke_group_role(self, role, users):
116116
db_connection.execute(stmt, {"role": role, "user": user})
117117

118118

119-
def init_connection_engine(instance_connection_name, creds, ip_type=IPTypes.PUBLIC):
120-
"""Configure and initialize database connection pool.
119+
def init_mysql_connection_engine(
120+
instance_connection_name, creds, ip_type=IPTypes.PUBLIC
121+
):
122+
"""Configure and initialize MySQL database connection pool.
121123
122124
Configures the parameters for the database connection pool. Initiliazes the
123125
database connection pool using the Cloud SQL Python Connector.
@@ -145,7 +147,6 @@ def init_connection_engine(instance_connection_name, creds, ip_type=IPTypes.PUBL
145147

146148
# service account email to access DB, mysql truncates usernames to before '@' sign
147149
service_account_email = mysql_username(creds.service_account_email)
148-
149150
# build connection for db using Python Connector
150151
connection = lambda: connector.connect(
151152
instance_connection_name,

iam_groups_authn/postgres.py

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# postgres.py contains all database specific functions for connecting
16+
# and querying a postgreSQL database
17+
18+
import sqlalchemy
19+
from google.cloud.sql.connector import connector
20+
from google.cloud.sql.connector.instance_connection_manager import IPTypes
21+
from iam_groups_authn.utils import RoleService, async_wrap
22+
from google.auth.transport.requests import Request
23+
24+
25+
class PostgresRoleService(RoleService):
26+
"""Class for managing a Postgres DB user's role grants."""
27+
28+
def __init__(self, db):
29+
"""Initialize a PostgresRoleService object.
30+
31+
Args:
32+
db: Database connection object.
33+
"""
34+
self.db = db
35+
36+
@async_wrap
37+
def fetch_role_grants(self, group_name):
38+
"""Fetch mappings of group roles granted to DB users.
39+
40+
Args:
41+
group_name: IAM group name prefix of email that is used as group role.
42+
43+
Returns:
44+
results: List of results for given query.
45+
"""
46+
# postgres query to get users with group role
47+
stmt = sqlalchemy.text(
48+
"SELECT pg_roles.rolname, (SELECT pg_roles.rolname FROM pg_roles WHERE oid = pg_auth_members.member) FROM pg_roles, pg_auth_members WHERE pg_auth_members.roleid = (SELECT oid FROM pg_roles WHERE rolname= :group_name) and pg_roles.rolname= :group_name"
49+
)
50+
# create connection to db instance
51+
with self.db.connect() as db_connection:
52+
# query users with roles
53+
results = db_connection.execute(stmt, {"group_name": group_name}).fetchall()
54+
return results
55+
56+
@async_wrap
57+
def create_group_role(self, role):
58+
"""Verify or create DB role.
59+
60+
Given a group role, verify existance of role on DB or create new role
61+
to manage DB users.
62+
63+
Args:
64+
role: Name of group role to be verified or created as new role.
65+
"""
66+
# check if group role exists, otherwise create it
67+
check_stmt = sqlalchemy.text("SELECT 1 FROM pg_roles WHERE rolname= :role")
68+
stmt = sqlalchemy.text(f'CREATE ROLE "{role}"')
69+
# create connection to db instance
70+
with self.db.connect() as db_connection:
71+
# check if role already exists
72+
role_check = db_connection.execute(check_stmt, {"role": role}).fetchone()
73+
# if role does not exist, create it
74+
if not role_check:
75+
db_connection.execute(stmt)
76+
77+
@async_wrap
78+
def grant_group_role(self, role, users):
79+
"""Grant DB group role to DB users.
80+
81+
Given a DB group role and a list of DB users, grant the DB role to each user.
82+
83+
Args:
84+
role: Name of DB role to grant to users.
85+
users: List of DB users' usernames.
86+
"""
87+
with self.db.connect() as db_connection:
88+
# if there are users to grant group role to, grant role to users
89+
if users:
90+
users = '"' + '", "'.join(users) + '"'
91+
stmt = sqlalchemy.text(f'GRANT "{role}" TO {users}')
92+
db_connection.execute(stmt)
93+
94+
@async_wrap
95+
def revoke_group_role(self, role, users):
96+
"""Revoke DB group role to DB users.
97+
98+
Given a DB group role and a list of DB users, revoke the DB role from each user.
99+
100+
Args:
101+
role: Name of DB role to revoke from users.
102+
users: List of DB users' usernames.
103+
"""
104+
# create connection to db instance
105+
with self.db.connect() as db_connection:
106+
# if there are users to revoke group role from, revoke role from users
107+
if users:
108+
users = '"' + '", "'.join(users) + '"'
109+
stmt = sqlalchemy.text(f'REVOKE "{role}" FROM {users}')
110+
db_connection.execute(stmt)
111+
112+
113+
def init_postgres_connection_engine(
114+
instance_connection_name, creds, ip_type=IPTypes.PUBLIC
115+
):
116+
"""Configure and initialize Postgres database connection pool.
117+
118+
Configures the parameters for the database connection pool. Initiliazes the
119+
database connection pool using the Cloud SQL Python Connector.
120+
121+
Args:
122+
instance_connection_name: Instance connection name of Cloud SQL instance.
123+
(e.g. "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>")
124+
creds: Credentials to get OAuth2 access token from, needed for IAM service
125+
account authentication to DB.
126+
ip_type: IP address type for instance connection.
127+
(IPTypes.PUBLIC or IPTypes.PRIVATE)
128+
Returns:
129+
A database connection pool instance.
130+
"""
131+
db_config = {
132+
"pool_size": 2,
133+
"max_overflow": 2,
134+
"pool_timeout": 30, # 30 seconds
135+
"pool_recycle": 1800, # 30 minutes
136+
}
137+
# refresh credentials if not valid
138+
if not creds.valid:
139+
request = Request()
140+
creds.refresh(request)
141+
142+
# service account to access DB, postgres removes suffix
143+
service_account_email = (creds.service_account_email).removesuffix(
144+
".gserviceaccount.com"
145+
)
146+
# build connection for db using Python Connector
147+
connection = lambda: connector.connect(
148+
instance_connection_name,
149+
"pg8000",
150+
ip_types=ip_type,
151+
user=service_account_email,
152+
db="postgres",
153+
enable_iam_auth=True,
154+
)
155+
156+
# create connection pool
157+
pool = sqlalchemy.create_engine(
158+
"postgresql+pg8000://", creator=connection, **db_config
159+
)
160+
return pool

0 commit comments

Comments
 (0)