[5/n] [reconfigurator-planning] gate adds/updates on zone image sources being known (#8921)

Expand the set of gates for adds/updates to include the fact that zone
image sources should be known. Add tests for this:

* `cmds-mupdate-update-flow` contains the bulk of testing for this
scenario.
* I had to make tweaks to some tests, particularly to
`cmds-target-release.txt`, in order to start running the test in earnest
from the Artifact state rather than the InstallDataset state.

Author: sunshowers

dev-tools/reconfigurator-cli/tests/input/cmds-add-sled-no-disks.txt

@@ -12,6 +12,10 @@ sled-add --ndisks 0
 # Generate a new inventory collection that includes that sled.
 inventory-generate
 
+# Set the add_zones_with_mupdate_override planner config to ensure that zone
+# adds happen despite zone image sources not being Artifact.
+set planner-config --add-zones-with-mupdate-override true
+
 # Try to plan a new blueprint; this should be okay even though the sled
 # we added has no disks.
 blueprint-plan dbcbd3d6-41ff-48ae-ac0b-1becc9b2fd21 eb0796d5-ab8a-4f7b-a884-b4aeacb8ab51

dev-tools/reconfigurator-cli/tests/input/cmds-expunge-newly-added-external-dns.txt

@@ -9,6 +9,11 @@ blueprint-edit 3f00b694-1b16-4aaa-8f78-e6b3a527b434 expunge-zones 8429c772-07e8-
 blueprint-diff 3f00b694-1b16-4aaa-8f78-e6b3a527b434 366b0b68-d80e-4bc1-abd3-dc69837847e0
 
 blueprint-show 366b0b68-d80e-4bc1-abd3-dc69837847e0
+
+# Set the add_zones_with_mupdate_override planner config to ensure that zone
+# adds happen despite zone image sources not being Artifact.
+set planner-config --add-zones-with-mupdate-override true
+
 # blueprint-plan will place a new external DNS zone, diff DNS to see the new zone has `ns<N>` and NS records.
 blueprint-plan 366b0b68-d80e-4bc1-abd3-dc69837847e0
 blueprint-diff 366b0b68-d80e-4bc1-abd3-dc69837847e0 9c998c1d-1a7b-440a-ae0c-40f781dea6e2

dev-tools/reconfigurator-cli/tests/input/cmds-expunge-newly-added-internal-dns.txt

@@ -12,6 +12,10 @@ blueprint-diff dbcbd3d6-41ff-48ae-ac0b-1becc9b2fd21 8da82a8e-bf97-4fbd-8ddd-9f64
 blueprint-edit 8da82a8e-bf97-4fbd-8ddd-9f6462732cf1 mark-for-cleanup 99e2f30b-3174-40bf-a78a-90da8abba8ca
 blueprint-diff 8da82a8e-bf97-4fbd-8ddd-9f6462732cf1 58d5e830-0884-47d8-a7cd-b2b3751adeb4
 
+# Set the add_zones_with_mupdate_override planner config to ensure that zone
+# adds happen despite zone image sources not being Artifact.
+set planner-config --add-zones-with-mupdate-override true
+
 # Planning a new blueprint will now replace the expunged zone, with new records for its replacement.
 blueprint-plan 58d5e830-0884-47d8-a7cd-b2b3751adeb4
 blueprint-diff 58d5e830-0884-47d8-a7cd-b2b3751adeb4 af934083-59b5-4bf6-8966-6fb5292c29e1

dev-tools/reconfigurator-cli/tests/input/cmds-mupdate-update-flow.txt

@@ -104,9 +104,6 @@ sled-set serial2 mupdate-override unset
 # Run the planner again. This will cause sled 2's blueprint
 # remove_mupdate_override to be unset. But no further planning steps will
 # happen because the target release generation is not new enough.
-#
-# TODO: we want to block further planning steps until the target release is
-# uploaded and all install-dataset zones have been converted to artifact ones.
 inventory-generate
 blueprint-plan latest latest
 blueprint-show latest
@@ -119,10 +116,19 @@ blueprint-plan latest latest
 blueprint-show latest
 blueprint-diff latest
 
-# Now clear the mupdate override error. At this point, the rest of the
-# planner starts working.
+# Now clear the mupdate override error. At this point, we're *still* blocked
+# on serial1's install dataset not being known (so cannot be noop converted).
 sled-set serial1 mupdate-override unset
 inventory-generate
+blueprint-plan latest latest
+blueprint-diff latest
+
+# Update serial1's install dataset. Finally the planner is unblocked. (Note that
+# we do noop conversions of serial1 to Artifact and the unblocking of further
+# steps within the same blueprint. In other words, the code that considers
+# whether add/update are blocked takes into account noop conversions.)
+sled-update-install-dataset serial1 --to-target-release
+inventory-generate
 
 # This will attempt to update the RoT bootloader on the first sled.
 blueprint-plan latest latest

dev-tools/reconfigurator-cli/tests/input/cmds-target-release.txt

@@ -7,6 +7,42 @@ show
 # Create a TUF repository from a fake manifest. (The output TUF repo is
 # written to a temporary directory that this invocation of `reconfigurator-cli`
 # is running out of as its working directory.)
+#
+# This is used to simulate the initial version of the system.
+tuf-assemble ../../update-common/manifests/fake-0.0.1.toml
+
+# Load the target release from the assembled TUF repository.
+set target-release repo-0.0.1.zip
+
+# Print the default target release.
+show
+
+# Update the install dataset on all sleds to the target release.
+# This will cause zones to be noop converted over to Artifact,
+# unblocking upgrades.
+sled-update-install-dataset serial0 --to-target-release
+sled-update-install-dataset serial1 --to-target-release
+sled-update-install-dataset serial2 --to-target-release
+
+# Generate inventory, then do a planning run to ensure that all zones
+# are set to Artifact.
+inventory-generate
+blueprint-plan latest latest
+blueprint-diff latest
+# The above blueprint includes a pending MGS update, which we should delete
+# (we want to start from a fresh state).
+blueprint-edit latest delete-sp-update serial0
+# Also set the Omicron config for all sleds to reflect the
+# corresponding image sources.
+sled-set serial0 omicron-config latest
+sled-set serial1 omicron-config latest
+sled-set serial2 omicron-config latest
+# Generate inventory once more to reflect the omicron config changes.
+inventory-generate
+inventory-show latest
+
+# Setup is now done -- create another TUF repository which will act as the
+# target release being updated to.
 tuf-assemble ../../update-common/manifests/fake.toml
 
 # Load the target release from the assembled TUF repository.
@@ -17,14 +53,13 @@ show
 
 # Test that this state survives a save/load operation.
 save saved.out
-wipe all
+wipe system
 # This should NOT show the target release.
 show
-load saved.out
+load saved.out 61f451b3-2121-4ed6-91c7-a550054f6c21
 # This should show the target release.
 show
 
-
 # Great.  Now, let's run through an upgrade!
 # First, print out what we've got.
 sled-list

dev-tools/reconfigurator-cli/tests/output/cmds-add-sled-no-disks-stdout

@@ -33,16 +33,34 @@ added sled 00320471-945d-413c-85e7-03e091a70b3c (serial: serial3)
 generated inventory collection eb0796d5-ab8a-4f7b-a884-b4aeacb8ab51 from configured sleds
 
 
+> # Set the add_zones_with_mupdate_override planner config to ensure that zone
+> # adds happen despite zone image sources not being Artifact.
+> set planner-config --add-zones-with-mupdate-override true
+planner config updated:
+*   add zones with mupdate override:   false -> true
+
+
+
 > # Try to plan a new blueprint; this should be okay even though the sled
 > # we added has no disks.
 > blueprint-plan dbcbd3d6-41ff-48ae-ac0b-1becc9b2fd21 eb0796d5-ab8a-4f7b-a884-b4aeacb8ab51
 INFO skipping noop image source check for all sleds, reason: no target release is currently set
-WARN cannot issue more MGS-driven updates (no current artifacts)
 generated blueprint 8da82a8e-bf97-4fbd-8ddd-9f6462732cf1 based on parent blueprint dbcbd3d6-41ff-48ae-ac0b-1becc9b2fd21
 blueprint source: planner with report:
 planning report for blueprint 8da82a8e-bf97-4fbd-8ddd-9f6462732cf1:
+planner config:
+    add zones with mupdate override:   true
+
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c: 9 zones
+    - sled 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6: 8 zones
+    - sled d81c6a84-79b8-4958-ae41-ea46c9b19763: 8 zones
+
+* adding zones despite being blocked, as specified by the `add_zones_with_mupdate_override` planner config option
 * no zpools in service for NTP zones on sleds: 00320471-945d-413c-85e7-03e091a70b3c
 * discretionary zone placement waiting for NTP zones on sleds: 00320471-945d-413c-85e7-03e091a70b3c
+* zone updates waiting on zone add blockers
 
 
 
@@ -279,8 +297,19 @@ parent:    dbcbd3d6-41ff-48ae-ac0b-1becc9b2fd21
 
 blueprint source: planner with report:
 planning report for blueprint 8da82a8e-bf97-4fbd-8ddd-9f6462732cf1:
+planner config:
+    add zones with mupdate override:   true
+
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c: 9 zones
+    - sled 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6: 8 zones
+    - sled d81c6a84-79b8-4958-ae41-ea46c9b19763: 8 zones
+
+* adding zones despite being blocked, as specified by the `add_zones_with_mupdate_override` planner config option
 * no zpools in service for NTP zones on sleds: 00320471-945d-413c-85e7-03e091a70b3c
 * discretionary zone placement waiting for NTP zones on sleds: 00320471-945d-413c-85e7-03e091a70b3c
+* zone updates waiting on zone add blockers
 
 
 

dev-tools/reconfigurator-cli/tests/output/cmds-example-stdout

@@ -599,10 +599,15 @@ T ENA ID                                   PARENT
 
 > blueprint-plan ade5749d-bdf3-4fab-a8ae-00bea01b3a5a
 INFO skipping noop image source check for all sleds, reason: no target release is currently set
-WARN cannot issue more MGS-driven updates (no current artifacts)
 generated blueprint 86db3308-f817-4626-8838-4085949a6a41 based on parent blueprint ade5749d-bdf3-4fab-a8ae-00bea01b3a5a
 blueprint source: planner with report:
-empty planning report for blueprint 86db3308-f817-4626-8838-4085949a6a41.
+planning report for blueprint 86db3308-f817-4626-8838-4085949a6a41:
+* zone adds waiting on blockers
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 89d02b1b-478c-401a-8e28-7a26f74fa41b: 18 zones
+
+* zone updates waiting on zone add blockers
 
 
 
@@ -1839,10 +1844,16 @@ INTERNAL DNS STATUS
 > # sled to be expunged.
 > blueprint-plan latest
 INFO skipping noop image source check for all sleds, reason: no target release is currently set
-WARN cannot issue more MGS-driven updates (no current artifacts)
 generated blueprint 86db3308-f817-4626-8838-4085949a6a41 based on parent blueprint ade5749d-bdf3-4fab-a8ae-00bea01b3a5a
 blueprint source: planner with report:
-empty planning report for blueprint 86db3308-f817-4626-8838-4085949a6a41.
+planning report for blueprint 86db3308-f817-4626-8838-4085949a6a41:
+* zone adds waiting on blockers
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 2eb69596-f081-4e2d-9425-9994926e0832: 4 zones
+    - sled 89d02b1b-478c-401a-8e28-7a26f74fa41b: 17 zones
+
+* zone updates waiting on zone add blockers
 
 
 

dev-tools/reconfigurator-cli/tests/output/cmds-expunge-newly-added-external-dns-stdout

@@ -837,13 +837,31 @@ blueprint source: edited directly with reconfigurator-cli
 
 
 
+
+> # Set the add_zones_with_mupdate_override planner config to ensure that zone
+> # adds happen despite zone image sources not being Artifact.
+> set planner-config --add-zones-with-mupdate-override true
+planner config updated:
+*   add zones with mupdate override:   false -> true
+
+
+
 > # blueprint-plan will place a new external DNS zone, diff DNS to see the new zone has `ns<N>` and NS records.
 > blueprint-plan 366b0b68-d80e-4bc1-abd3-dc69837847e0
 INFO skipping noop image source check for all sleds, reason: no target release is currently set
-WARN cannot issue more MGS-driven updates (no current artifacts)
 generated blueprint 9c998c1d-1a7b-440a-ae0c-40f781dea6e2 based on parent blueprint 366b0b68-d80e-4bc1-abd3-dc69837847e0
 blueprint source: planner with report:
 planning report for blueprint 9c998c1d-1a7b-440a-ae0c-40f781dea6e2:
+planner config:
+    add zones with mupdate override:   true
+
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 711ac7f8-d19e-4572-bdb9-e9b50f6e362a: 15 zones
+    - sled 9dc50690-f9bf-4520-bf80-051d0f465c2c: 15 zones
+    - sled a88790de-5962-4871-8686-61c1fd5b7094: 15 zones
+
+* adding zones despite being blocked, as specified by the `add_zones_with_mupdate_override` planner config option
 * discretionary zones placed:
   * external_dns zone on sled 711ac7f8-d19e-4572-bdb9-e9b50f6e362a from source install dataset
 * zone updates waiting on discretionary zones
@@ -1343,6 +1361,16 @@ parent:    366b0b68-d80e-4bc1-abd3-dc69837847e0
 
 blueprint source: planner with report:
 planning report for blueprint 9c998c1d-1a7b-440a-ae0c-40f781dea6e2:
+planner config:
+    add zones with mupdate override:   true
+
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 711ac7f8-d19e-4572-bdb9-e9b50f6e362a: 15 zones
+    - sled 9dc50690-f9bf-4520-bf80-051d0f465c2c: 15 zones
+    - sled a88790de-5962-4871-8686-61c1fd5b7094: 15 zones
+
+* adding zones despite being blocked, as specified by the `add_zones_with_mupdate_override` planner config option
 * discretionary zones placed:
   * external_dns zone on sled 711ac7f8-d19e-4572-bdb9-e9b50f6e362a from source install dataset
 * zone updates waiting on discretionary zones

dev-tools/reconfigurator-cli/tests/output/cmds-expunge-newly-added-internal-dns-stdout

@@ -645,13 +645,30 @@ external DNS:
 
 
 
+> # Set the add_zones_with_mupdate_override planner config to ensure that zone
+> # adds happen despite zone image sources not being Artifact.
+> set planner-config --add-zones-with-mupdate-override true
+planner config updated:
+*   add zones with mupdate override:   false -> true
+
+
+
 > # Planning a new blueprint will now replace the expunged zone, with new records for its replacement.
 > blueprint-plan 58d5e830-0884-47d8-a7cd-b2b3751adeb4
 INFO skipping noop image source check for all sleds, reason: no target release is currently set
-WARN cannot issue more MGS-driven updates (no current artifacts)
 generated blueprint af934083-59b5-4bf6-8966-6fb5292c29e1 based on parent blueprint 58d5e830-0884-47d8-a7cd-b2b3751adeb4
 blueprint source: planner with report:
 planning report for blueprint af934083-59b5-4bf6-8966-6fb5292c29e1:
+planner config:
+    add zones with mupdate override:   true
+
+* zone adds and updates are blocked:
+  - sleds have deployment units with image sources not set to Artifact:
+    - sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c: 15 zones
+    - sled 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6: 15 zones
+    - sled d81c6a84-79b8-4958-ae41-ea46c9b19763: 15 zones
+
+* adding zones despite being blocked, as specified by the `add_zones_with_mupdate_override` planner config option
 * discretionary zones placed:
   * internal_dns zone on sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c from source install dataset
 * zone updates waiting on discretionary zones