How to deal with different Nexus versions having different types?

[jmpesp]

During an upgrade, any request to Nexus will hit a load balancer and could be sent to two different Nexus versions. There are many ways that this could fail but one that was discussed in relation to PR #1092 was the following:

1. Nexus version X has a database enum that has variants A, B, C.

2. Nexus version Y introduces a new enum variant D through a schema upgrade.

3. A user request creates a row with that new enum variant.

4. Another user request does a GET to list all objects that include that enum

This can be reproduced (using PR #1092) by POSTing to the /images endpoint:

{
  "name": "crucible-tester-sparse",
  "description": "boot from a url!",
  "distribution": "debian",
  "version": "9",
  "block_size": 512,
  "source": {
    "type": "url",
    "src": "http://127.0.0.1:8080/crucible-tester-sparse.img"
  }
}

then, using the CockroachDB cli to perform a schema upgrade that adds a variant to the enum:

ALTER TYPE distribution ADD VALUE 'freebsd';
update global_image set distribution = 'freebsd' where name = 'crucible-tester-sparse';

then, performing a GET /images. Nexus currently returns a 500 error:

May 18 20:37:36.123 INFO request completed

error_message_external: Internal Server Error
error_message_internal: unexpected database error: Unrecognized enum variant for Distribution
response_code: 500
uri: /images
method: GET
req_id: a53847a4-1456-4f08-8d8f-b41378bda09a
remote_addr: 127.0.0.1:36764
local_addr: 127.0.0.1:12220
component: dropshot_external
name: e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c

PR #1092 introduces InterregnumError - specifically, the database enum contains the "freebsd" variant but the external enum does not. A TryFrom is required to convert from the database enum to the external enum, with an Interregnum error being thrown if freebsd is found in the DB.

Note that's not to say that this is the way we should do it, just that it was an attempt at addressing this issue's question. A safe upgrade scheme would be to add a variant to the database enum first, let that upgrade propagate out to all Nexus instances, and then proceed with a second upgrade to add the variant to the external enum. This implicitly relies on every Nexus being upgraded from version X -> Y before being upgraded from Y -> Z.

I'm opening this issue to track work related to the issue's title question: how are we going to deal with different Nexus versions having different types? There's a versioning RFD in the works that should also address this question (@smklein @ahl)

[ahl]

cc @iliana who is taking the first swing at the versioning RFD

[davepacheco]

Thanks for raising this!

This may have already been done, but I think it'd really help the discussion to write down the operational and developer constraints/goals. Things like: we can limit the range of versions that are compatible, but we must support rollback of software versions (which can also be constrained, and doesn't necessarily apply to schema changes), and we must not have to update the entire fleet of Nexuses atomically, etc.

It might be worth codifying what to do in the meantime. I think we've generally tried to make sure the path from the database to a Rust type that we're actually going to work with accounts for the possibility that we don't understand the database type. I've usually used InternalError for this, since I think it's basically a bug (and I don't know how Nexus could know the problem was an intentional upgrade and not, say, corruption). I expect that in the normal upgrade path we do not expect a Nexus instance to receive something it doesn't understand. (This doesn't just apply to the database. I think the same applies to calls to other APIs, too.)

[davepacheco]

Another piece of "what we're doing in the meantime": I don't think we've been taking any steps to try to accommodate version mismatches today, either between database schema <-> Nexus or between services. It feels to me too early to start doing that.

[davepacheco]

I believe this question is now answered via RFD 527 Online database schema updates and RFD 531 Managing external API versions for online upgrade. The short answer is that the online upgrade system will only ever have all the old Nexus instances or all the new Nexus instances in service at a time. The schema update will be applied during the handoff from the old to the new.

Closing this because this is now tracked elsewhere but I'm happy to keep the discussion open if that's useful!