[omicron-dev] Verify the version of the databases on startup

[bnaecker]

A few people have run into failures that boiled down to an outdated version of the CockroachDB binary. I hit the most recent, which was a failure like this:

bnaecker@shale : ~/omicron $ PATH="$OMICRON_PATH:$PATH" cargo test
    Finished test [unoptimized + debuginfo] target(s) in 0.43s
     Running unittests (target/debug/deps/db_macros-76f9e17f75c2b8bf)

running 5 tests
test tests::test_derive_metadata_identity_fails_for_enums ... ok
test tests::test_derive_metadata_identity_fails_without_embedded_identity ... ok
test tests::test_derive_metadata_identity_fails_without_table_name ... ok
test tests::test_derive_metadata_identity_fails_with_wrong_table_name_type ... ok
test tests::test_derive_metadata_identity_minimal_example_compiles ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/omicron_common-e28515a47bb80ccb)

running 17 tests
test api::external::error::test::test_bail_unless ... ok
test api::external::test::test_bytecount ... ok
test api::external::test::test_ip_port_range_from_str ... ok
test api::external::http_pagination::test::test_scan_by_nameid_generic ... ok
test api::external::test::test_ip_port_range_into_str ... ok
test api::external::test::test_name_parse_from_param ... ok
test api::external::test::test_name_parse ... ok
test api::external::http_pagination::test::test_pagination_examples ... ok
test api::external::test::test_firewall_deserialization ... ok
test api::external::http_pagination::test::test_scan_by_nameid_id ... ok
test api::external::http_pagination::test::test_scan_by_id ... ok
test api::external::http_pagination::test::test_scan_by_nameid_name ... ok
test api::external::http_pagination::test::test_scan_by_name ... ok
test config::test::test_bad_url ... ok
test config::test::test_example_url ... ok
test api::external::http_pagination::test::test_pagination_schemas ... ok
test api::external::test::test_networktarget_parsing ... ok

test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running unittests (target/debug/deps/omicron_package-18aabcaeda753e2d)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests (target/debug/deps/omicron_nexus-90da724fb34d3605)

running 37 tests
test authn::external::cookies::test::test_parse_cookies_empty_headers ... ok
test authn::external::cookies::test::test_parse_cookies_one_cookie ... ok
test authn::external::cookies::test::test_parse_cookies_two_cookie_headers_same_name ... ok
test authn::external::cookies::test::test_parse_cookies_two_cookie_headers ... ok
test authn::external::cookies::test::test_parse_cookies_two_cookies ... ok
test authn::external::cookies::test::test_parse_cookies_ignore_other_headers ... ok
test authn::external::session_cookie::test::test_get_token_no_header ... ok
test authn::external::session_cookie::test::test_get_token_other_cookie_present ... ok
test authn::external::spoof::test::test_spoof_header_missing ... ok
test authn::external::session_cookie::test::test_get_token ... ok
test authn::external::spoof::test::test_spoof_header_valid ... ok
test authn::test::test_internal_users ... ok
test authn::external::spoof::test::test_spoof_reserved_values ... ok
test authn::external::spoof::test::test_spoof_header_bad_uuids ... ok
test config::test::test_config_nonexistent ... ok
test config::test::test_config_empty ... ok
test config::test::test_config_bad_toml ... ok
test authn::external::session_cookie::test::test_expired_cookie_idle ... ok
test authn::external::session_cookie::test::test_other_cookie ... ok
test authn::external::session_cookie::test::test_garbage_cookie ... ok
test authn::external::test::test_authn_sequence ... ok
test authn::external::session_cookie::test::test_missing_cookie ... ok
test authn::external::session_cookie::test::test_expired_cookie_absolute ... ok
test authn::external::session_cookie::test::test_valid_cookie ... ok
test config::test::test_bad_authn_schemes ... ok
test db::collection_insert::test::test_verify_query ... ok
test config::test::test_valid ... ok
Nov 23 23:13:30.441 DEBG authorize result, result: Err(Unauthenticated { internal_message: "authorization failed for unauthenticated request" }), resource: Database, action: Query, actor: None
test context::test::test_background_context ... ok
Nov 23 23:13:30.443 DEBG authorize result, result: Ok(()), resource: Database, action: Query, actor: Some(Actor(001de000-05e4-0000-0000-000000004007))
test context::test::test_test_context ... ok
test authz::test::test_database ... ok
test authz::test::test_organization ... ok
test db::datastore::test::test_project_creation ... ok
test db::datastore::test::test_session_methods ... ok
test db::saga_recovery::test::test_failure_during_saga_can_be_recovered ... ok
test db::collection_insert::test::test_collection_present ... ok
test db::saga_recovery::test::test_successful_saga_does_not_replay_during_recovery ... ok
test db::collection_insert::test::test_collection_not_present ... ok

test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.33s

     Running unittests (target/debug/deps/nexus-88d67932f333622a)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_authn_http.rs (target/debug/deps/test_authn_http-ab021e6cc7680911)

running 3 tests
test test_authn_spoof_unconfigured ... ok
test test_authn_session_cookie ... ok
test test_authn_spoof_allowed ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running tests/test_authz.rs (target/debug/deps/test_authz-c5e39fb540b6b3a2)

running 1 test
test test_authz_basic ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.92s

     Running tests/test_basic.rs (target/debug/deps/test_basic-053624b266e278f4)

running 4 tests
test test_sleds_list ... ok
test test_basic_failures ... ok
test test_projects ... FAILED
test test_projects_list ... FAILED

failures:

---- test_projects stdout ----
log file: "/tmp/test_basic-053624b266e278f4-test_projects.20002.3.log"
note: configured to log to "/tmp/test_basic-053624b266e278f4-test_projects.20002.3.log"
thread 'test_projects' panicked at 'assertion failed: `(left == right)`
  left: `201`,
 right: `500`', /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/a1f980d/dropshot/src/test_util.rs:193:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dropped ClickHouseInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)

---- test_projects_list stdout ----
log file: "/tmp/test_basic-053624b266e278f4-test_projects_list.20002.1.log"
note: configured to log to "/tmp/test_basic-053624b266e278f4-test_projects_list.20002.1.log"
thread 'test_projects_list' panicked at 'assertion failed: `(left == right)`
  left: `201`,
 right: `500`', /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/a1f980d/dropshot/src/test_util.rs:193:9
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dropped ClickHouseInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)


failures:
    test_projects
    test_projects_list

test result: FAILED. 2 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.34s

The log file shows this as the received error from CRDB:

...
{"msg":"client request","v":0,"name":"test_projects","level":30,"time":"2021-11-23T23:13:35.549509529+00:00","hostname":"shale","pid":20002,"component":"external client test context","body":"Body(Full(b\"{\\\"name\\\":\\\"simproject1\\\",\\\"description\\\":\\\"<auto-generated by test suite>\\\"}\"))","uri":"http://127.0.0.1:60793/organizations/test-org/projects","method":"POST"}
{"msg":"incoming request","v":0,"name":"test_projects","level":10,"time":"2021-11-23T23:13:35.549713881+00:00","hostname":"shale","pid":20002,"uri":"/organizations/test-org/projects","method":"POST","req_id":"a0c8b460-e8ca-4cf4-bb2c-c068d949c679","remote_addr":"127.0.0.1:50534","local_addr":"127.0.0.1:60793","component":"dropshot_external","name":"0f2aa402-fa00-4f32-9d53-a5569d243756"}
{"msg":"request completed","v":0,"name":"test_projects","level":30,"time":"2021-11-23T23:13:35.897322908+00:00","hostname":"shale","pid":20002,"uri":"/organizations/test-org/projects","method":"POST","req_id":"a0c8b460-e8ca-4cf4-bb2c-c068d949c679","remote_addr":"127.0.0.1:50534","local_addr":"127.0.0.1:60793","component":"dropshot_external","name":"0f2aa402-fa00-4f32-9d53-a5569d243756","error_message_external":"Internal Server Error","error_message_internal":"database error (kind = Unknown): internal error: error in argument for $11: unsupported user defined type family ArrayFamily\nDETAILS: stack trace:\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:804: DecodeOidDatum()\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:360: execBind()\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1558: execCmd()\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1391: run()\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:508: ServeConn()\n/ws/gc/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:626: func1()\n/ws/gc/cockroach/cache/go1.16.2/src/runtime/asm_amd64.s:1371: goexit()\n\nHINT: You have encountered an unexpected error.\n\nPlease check the public issue tracker to check whether this problem is\nalready tracked. If you cannot find it there, please report the error\nwith details by creating a new issue.\n\nIf you would rather not post publicly, please contact us directly\nusing the support form.\n\nWe appreciate your feedback.\n\n","response_code":"500"}
{"msg":"client received response","v":0,"name":"test_projects","level":30,"time":"2021-11-23T23:13:35.897792021+00:00","hostname":"shale","pid":20002,"component":"external client test context","status":"500"}

It's not totally obvious what's happening, but the relevant portion of the error is unsupported user defined type family ArrayFamily. This is ultimately fixed by downloading CockroachDB again with ./tools/ci_download_cockroach, which did indeed pull a new version.

We should check the version number is what we expect when running cargo run --bin omicron-dev, to fail early in case there's a version mismatch.

[davepacheco]

Thanks for filing this!

I haven't looked too closely, but it looks superficially similar to cockroach#61046. v21.1 fixed a different issue that was blocking our use of enums. We recently added an enum, so that may be why we're hitting it.

[teisenbe]

This will be resolved by #427