DWO sometimes creates a deployment before serviceaccount gets pull secrets

[amisevsk]

Description

In some cases, the DevWorkspace Operator creates a Deployment for the DevWorkspace before the associated serviceaccount is assigned an ImagePullSecret by the cluster. When this occurs, the Deployment requires a rollout, which in some cases can take significant time.

How To Reproduce

1. Create a DevWorkspace
2. Watch pods while DevWorkspace is starting

If the bug occurs, shortly after DevWorkspace creation, a pod will be created and then immediately enter the Terminating state. A new deployment is rolled out and the pod restarts.

If the enableExperimentalFeatures config option is enabled in DWO, the diff for the Deployment is logged:

&v1.Deployment{
  	... // 2 ignored fields
  	Spec: v1.DeploymentSpec{
  		Replicas: &1,
  		Selector: &{MatchLabels: {\"controller.devfile.io/devworkspace_id\": \"test\"}},
  		Template: v1.PodTemplateSpec{
  			ObjectMeta: {Name: \"test\", Namespace: \"dw\", Labels: {\"controller.devfile.io/creator\": \"\", \"controller.devfile.io/devworkspace_id\": \"test\", \"controller.devfile.io/devworkspace_name\": \"test-contrib-code\"}},
  			Spec: v1.PodSpec{
  				... // 2 ignored and 14 identical fields
  				ShareProcessNamespace: nil,
  				SecurityContext:       &{},
- 				ImagePullSecrets:      []v1.LocalObjectReference{{Name: \"test-sa-dockercfg-htj6x\"}},
+ 				ImagePullSecrets:      nil,
  				Hostname:              \"\",
  				Subdomain:             \"\",
  				... // 1 ignored and 13 identical fields
  			},
  		},
  		Strategy:        {Type: \"Recreate\"},
  		MinReadySeconds: 0,
  		... // 2 ignored and 1 identical fields
  	},
  	... // 1 ignored field
  }

(This is logged as a json string -- to view it formatted correctly it's necessary to e.g. echo -e it)

Expected behavior

The DevWorkspace Operator should ideally wait a short while if necessary to give a chance for the ImagePullSecrets to be populated (in my experience, it's within a second or two, maximum).

Additional context