feat: add connection string secret annotations (#119)

# Summary

Fixes
mongodb/mongodb-kubernetes-operator#1522.

This is a port of
mongodb/mongodb-kubernetes-operator#1582.

In this PR, I've added the ability to add custom annotations to the
generated connection string secrets in MongoDB Community Operator.

This is useful to handle more deployment scenarios, in particular,
scenarios where the operator is not deployed cluster-wide, but to a
specific namespace. In these scenarios, the
`connectionStringSecretNamespace` property becomes useless because, as
stated in the [Kubernetes
docs](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#owners-dependents),
cross-namespace owner references are disallowed, thus allowing for the
secrets to be immediately garbage-collected, as stated in
mongodb/mongodb-kubernetes-operator#1578. For
the owner references to be valid, the secrets need to be generated in
the namespace of the MDBC resource. However, if the user needs the
secrets to be present in other namespaces, they can use
[reflector](https://github.com/emberstack/kubernetes-reflector), for
instance, which allows for the secrets to be copied to other namespaces.
The problem is that reflector and other similar controllers require the
source secrets to be annotated with specific properties.

As such, I've implemented a `connectionStringSecretAnnotations` property
that allows MongoDB Community Operator users to specify per-user
connection string secret annotations.

## Proof of Work

I've added a unit test and an e2e test. The unit test is passing.
Regarding the e2e test, it was passing in the
mongodb-kubernetes-operator repository, but I couldn't figure out how to
run the e2e tests in this repository.

## Checklist
- [ ] Have you linked a jira ticket and/or is the ticket in the title?
  - No, should I create an issue in this repository for that?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
  - What does this mean?
- [ ] Have you checked for release_note changes?

## Reminder (Please remove this when merging)
- Please try to Approve or Reject Changes the PR, keep PRs in review as
short as possible
- Our Short Guide for PRs:
[Link](https://docs.google.com/document/d/1T93KUtdvONq43vfTfUt8l92uo4e4SEEvFbIEKOxGr44/edit?tab=t.0)
  - I can't access this link
- Remember the following Communication Standards - use comment prefixes
for clarity:
  * **blocking**: Must be addressed before approval.
  * **follow-up**: Can be addressed in a later PR or ticket.
  * **q**: Clarifying question.
  * **nit**: Non-blocking suggestions.
  * **note**: Side-note, non-actionable. Example: Praise 
  * --> no prefix is considered a question

Author: limwa

config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

@@ -577,6 +577,13 @@ spec:
                       nullable: true
                       type: object
                       x-kubernetes-preserve-unknown-fields: true
+                    connectionStringSecretAnnotations:
+                      additionalProperties:
+                        type: string
+                      description: ConnectionStringSecretAnnotations is the annotations
+                        of the secret object created by the operator which exposes
+                        the connection strings for the user.
+                      type: object
                     connectionStringSecretName:
                       description: |-
                         ConnectionStringSecretName is the name of the secret object created by the operator which exposes the connection strings for the user.

docs/mongodbcommunity/users.md

@@ -42,6 +42,8 @@ You cannot disable SCRAM authentication.
    | `spec.users.roles` | array of objects | Configures roles assigned to the user. | Yes |
    | `spec.users.roles.role.name` | string | Name of the role. Valid values are [built-in roles](https://www.mongodb.com/docs/manual/reference/built-in-roles/#built-in-roles) and [custom roles](deploy-configure.md#define-a-custom-database-role) that you have defined. | Yes |
    | `spec.users.roles.role.db` | string | Database that the role applies to. | Yes |
+   | `spec.users.connectionStringSecretAnnotations` | object | Annotations of the secret object created by the operator which exposes the connection strings for the user. | No |
+
 
    ```yaml
    ---

helm_chart/crds/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

@@ -577,6 +577,13 @@ spec:
                       nullable: true
                       type: object
                       x-kubernetes-preserve-unknown-fields: true
+                    connectionStringSecretAnnotations:
+                      additionalProperties:
+                        type: string
+                      description: ConnectionStringSecretAnnotations is the annotations
+                        of the secret object created by the operator which exposes
+                        the connection strings for the user.
+                      type: object
                     connectionStringSecretName:
                       description: |-
                         ConnectionStringSecretName is the name of the secret object created by the operator which exposes the connection strings for the user.

mongodb-community-operator/api/v1/mongodbcommunity_types.go

@@ -449,6 +449,10 @@ type MongoDBUser struct {
 	// +optional
 	ConnectionStringSecretNamespace string `json:"connectionStringSecretNamespace,omitempty"`
 
+	// ConnectionStringSecretAnnotations is the annotations of the secret object created by the operator which exposes the connection strings for the user.
+	// +optional
+	ConnectionStringSecretAnnotations map[string]string `json:"connectionStringSecretAnnotations,omitempty"`
+
 	// Additional options to be appended to the connection string.
 	// These options apply only to this user and will override any existing options in the resource.
 	// +kubebuilder:validation:Type=object
@@ -748,12 +752,13 @@ func (m *MongoDBCommunity) GetAuthUsers() []authtypes.User {
 		}
 
 		users[i] = authtypes.User{
-			Username:                        u.Name,
-			Database:                        u.DB,
-			Roles:                           roles,
-			ConnectionStringSecretName:      u.GetConnectionStringSecretName(m.Name),
-			ConnectionStringSecretNamespace: u.GetConnectionStringSecretNamespace(m.Namespace),
-			ConnectionStringOptions:         u.AdditionalConnectionStringConfig.Object,
+			Username:                          u.Name,
+			Database:                          u.DB,
+			Roles:                             roles,
+			ConnectionStringSecretName:        u.GetConnectionStringSecretName(m.Name),
+			ConnectionStringSecretNamespace:   u.GetConnectionStringSecretNamespace(m.Namespace),
+			ConnectionStringSecretAnnotations: u.ConnectionStringSecretAnnotations,
+			ConnectionStringOptions:           u.AdditionalConnectionStringConfig.Object,
 		}
 
 		if u.DB != constants.ExternalDB {

mongodb-community-operator/api/v1/zz_generated.deepcopy.go

@@ -74,6 +74,7 @@ func (r ReplicaSetReconciler) updateConnectionStringSecrets(ctx context.Context,
 		connectionStringSecret := secret.Builder().
 			SetName(secretName).
 			SetNamespace(secretNamespace).
+			SetAnnotations(user.ConnectionStringSecretAnnotations).
 			SetField("connectionString.standard", mdb.MongoAuthUserURI(user, pwd, clusterDomain)).
 			SetField("connectionString.standardSrv", mdb.MongoAuthUserSRVURI(user, pwd, clusterDomain)).
 			SetField("username", user.Username).

mongodb-community-operator/controllers/mongodb_users.go

@@ -687,6 +687,43 @@ func assertStatefulsetReady(ctx context.Context, t *testing.T, mgr manager.Manag
 	assert.True(t, statefulset.IsReady(sts, expectedReplicas))
 }
 
+func TestService_connectionStringSecretAnnotationsAreApplied(t *testing.T) {
+	ctx := context.Background()
+	secretAnnotations := map[string]string{
+		"tests.first-annotation":  "some-value",
+		"tests.second-annotation": "other-value",
+	}
+
+	mdb := newScramReplicaSet(mdbv1.MongoDBUser{
+		Name: "testuser",
+		PasswordSecretRef: mdbv1.SecretKeyReference{
+			Name: "password-secret-name",
+		},
+		ScramCredentialsSecretName:        "scram-credentials",
+		ConnectionStringSecretAnnotations: secretAnnotations,
+	})
+
+	mgr := client.NewManager(ctx, &mdb)
+
+	err := createUserPasswordSecret(ctx, mgr.Client, mdb, "password-secret-name", "pass")
+	assert.NoError(t, err)
+
+	r := NewReconciler(mgr, "fake-mongodbRepoUrl", "fake-mongodbImage", "ubi8", AgentImage, "fake-versionUpgradeHookImage", "fake-readinessProbeImage")
+	res, err := r.Reconcile(ctx, reconcile.Request{NamespacedName: types.NamespacedName{Namespace: mdb.Namespace, Name: mdb.Name}})
+	assertReconciliationSuccessful(t, res, err)
+	assertConnectionStringSecretAnnotations(ctx, t, mgr.Client, mdb, secretAnnotations)
+}
+
+func assertConnectionStringSecretAnnotations(ctx context.Context, t *testing.T, c k8sClient.Client, mdb mdbv1.MongoDBCommunity, expectedAnnotations map[string]string) {
+	connectionStringSecret := corev1.Secret{}
+	scramUsers := mdb.GetAuthUsers()
+	require.Len(t, scramUsers, 1)
+	secretNamespacedName := types.NamespacedName{Name: scramUsers[0].ConnectionStringSecretName, Namespace: scramUsers[0].ConnectionStringSecretNamespace}
+	err := c.Get(ctx, secretNamespacedName, &connectionStringSecret)
+	require.NoError(t, err)
+	assert.Subset(t, connectionStringSecret.Annotations, expectedAnnotations)
+}
+
 func TestService_configuresPrometheusCustomPorts(t *testing.T) {
 	ctx := context.Background()
 	mdb := newTestReplicaSet()

mongodb-community-operator/controllers/replicaset_controller_test.go

@@ -74,6 +74,9 @@ type User struct {
 	// ConnectionStringSecretNamespace is the namespace of the secret object created by the operator which exposes the connection strings for the user.
 	ConnectionStringSecretNamespace string `json:"connectionStringSecretNamespace,omitempty"`
 
+	// ConnectionStringSecretAnnotations is the annotations of the secret object created by the operator which exposes the connection strings for the user.
+	ConnectionStringSecretAnnotations map[string]string
+
 	// ConnectionStringOptions contains connection string options for this user
 	// These options will be appended at the end of the connection string and
 	// will override any existing options from the resources.

mongodb-community-operator/pkg/authentication/authtypes/authtypes.go

@@ -11,6 +11,7 @@ type builder struct {
 	labels          map[string]string
 	name            string
 	namespace       string
+	annotations     map[string]string
 	ownerReferences []metav1.OwnerReference
 }
 
@@ -24,6 +25,11 @@ func (b *builder) SetNamespace(namespace string) *builder {
 	return b
 }
 
+func (b *builder) SetAnnotations(annotations map[string]string) *builder {
+	b.annotations = annotations
+	return b
+}
+
 func (b *builder) SetField(key, value string) *builder {
 	b.data[key] = []byte(value)
 	return b
@@ -73,6 +79,7 @@ func (b builder) Build() corev1.Secret {
 			Namespace:       b.namespace,
 			OwnerReferences: b.ownerReferences,
 			Labels:          b.labels,
+			Annotations:     b.annotations,
 		},
 		Data: b.data,
 		Type: b.dataType,

mongodb-community-operator/pkg/kube/secret/secret_builder.go

@@ -203,6 +203,7 @@ func ConnectionStringSecretsAreConfigured(ctx context.Context, mdb *mdbv1.MongoD
 
 			assert.NoError(t, err)
 			assertEqualOwnerReference(t, "Secret", secretNamespacedName, secret.GetOwnerReferences(), expectedOwnerReference)
+			containsMetadata(t, secret.ObjectMeta, map[string]string{}, user.ConnectionStringSecretAnnotations, "secret "+secretNamespacedName.Name)
 		}
 	}
 }
@@ -683,6 +684,18 @@ func AddConnectionStringOptionToUser(ctx context.Context, mdb *mdbv1.MongoDBComm
 	}
 }
 
+func AddConnectionStringAnnotationsToUser(ctx context.Context, mdb *mdbv1.MongoDBCommunity, annotations map[string]string) func(t *testing.T) {
+	return func(t *testing.T) {
+		t.Logf("Adding %v to connection string annotations", annotations)
+		err := e2eutil.UpdateMongoDBResource(ctx, mdb, func(db *mdbv1.MongoDBCommunity) {
+			db.Spec.Users[0].ConnectionStringSecretAnnotations = annotations
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
 func StatefulSetContainerConditionIsTrue(ctx context.Context, mdb *mdbv1.MongoDBCommunity, containerName string, condition func(c corev1.Container) bool) func(*testing.T) {
 	return func(t *testing.T) {
 		sts := appsv1.StatefulSet{}

mongodb-community-operator/test/e2e/mongodbtests/mongodbtests.go

@@ -106,4 +106,24 @@ func TestReplicaSetWithConnectionString(t *testing.T) {
 		t.Run("Test SRV Connectivity with generated connection string secret",
 			tester.ConnectivityRejected(ctx, WithURI(mongodbtests.GetSrvConnectionStringForUser(ctx, mdb, scramUser))))
 	})
+
+	/**
+	Connection String Annotations options.
+	*/
+	t.Run("Connection String With Annotations", func(t *testing.T) {
+		t.Run("Resetting Connection String Options", mongodbtests.ResetConnectionStringOptions(ctx, &mdb))
+		t.Run("Test Add New Connection String Annotations to Resource", mongodbtests.AddConnectionStringAnnotationsToUser(ctx, &mdb, map[string]string{"mongodbcommunity.mongodb.com/test-annotation": "test-value"}))
+		t.Run("Test Secrets Are Updated", mongodbtests.MongoDBReachesRunningPhase(ctx, &mdb))
+
+		scramUser = mdb.GetAuthUsers()[0]
+		t.Run("Test Basic Connectivity", tester.ConnectivitySucceeds())
+		t.Run("Test SRV Connectivity", tester.ConnectivitySucceeds(WithURI(mdb.MongoSRVURI("")), WithoutTls(), WithReplicaSet(mdb.Name)))
+		t.Run("Test Basic Connectivity with generated connection string secret",
+			tester.ConnectivitySucceeds(WithURI(mongodbtests.GetConnectionStringForUser(ctx, mdb, scramUser))))
+		t.Run("Test SRV Connectivity with generated connection string secret",
+			tester.ConnectivitySucceeds(WithURI(mongodbtests.GetSrvConnectionStringForUser(ctx, mdb, scramUser))))
+
+		ownerRef := mdb.GetOwnerReferences()[0]
+		t.Run("Test Connection String Annotations are as expected", mongodbtests.ConnectionStringSecretsAreConfigured(ctx, &mdb, ownerRef))
+	})
 }

mongodb-community-operator/test/e2e/replica_set_connection_string_options/replica_set_connection_string_options_test.go

@@ -7569,6 +7569,13 @@ spec:
                       nullable: true
                       type: object
                       x-kubernetes-preserve-unknown-fields: true
+                    connectionStringSecretAnnotations:
+                      additionalProperties:
+                        type: string
+                      description: ConnectionStringSecretAnnotations is the annotations
+                        of the secret object created by the operator which exposes
+                        the connection strings for the user.
+                      type: object
                     connectionStringSecretName:
                       description: |-
                         ConnectionStringSecretName is the name of the secret object created by the operator which exposes the connection strings for the user.