`sic_allocate_instance_snat_ip` always uses the default pool

[jmpesp]

I created an IP pool with:

oxide api /system/ip-pools --method POST --input - <<EOF
{
  "name": "mypool",
  "description": "an IP pool"
}
EOF

oxide api /system/ip-pools/mypool/ranges/add --method POST --input - <<EOF
{
  "first": "10.1.0.100",
  "last": "10.1.0.200"
}
EOF

and submitted an instance create request with:

  "external_ips": [
    {
      "type": "ephemeral",
      "pool_name": "mypool"
    }
  ],

This doesn't work - I'm seeing No external IP addresses available and a "400 Bad Request", yet no external IPs are being used yet:

root@[fd00:1122:3344:101::5]:32221/omicron> select * from external_ip;
  id | name | description | time_created | time_modified | time_deleted | ip_pool_id | ip_pool_range_id | instance_id | kind | ip | first_port | last_port
-----+------+-------------+--------------+---------------+--------------+------------+------------------+-------------+------+----+------------+------------
(0 rows)

root@[fd00:1122:3344:101::5]:32221/omicron> select * from ip_pool_range;
                   id                  |         time_created          |         time_modified         | time_deleted | first_address | last_address |              ip_pool_id              | rcgen
---------------------------------------+-------------------------------+-------------------------------+--------------+---------------+--------------+--------------------------------------+--------
  e6124f3b-99cf-4e46-a7dc-6c4d0484f0c9 | 2023-01-24 20:56:33.056055+00 | 2023-01-24 20:56:33.056055+00 | NULL         | 10.1.0.100    | 10.1.0.200   | c3470a78-603e-4b9e-95fa-8a2c5f5bb52b |     0
(1 row)

The Nexus log shows:

{
  "msg": "saga finished",
  "v": 0,
  "name": "879812ec-d2c2-45e2-a70e-b54a22190178",
  "level": 30,
  "time": "2023-01-24T21:22:27.167849886Z",
  "hostname": "oxz_nexus",
  "pid": 6165,
  "saga_name": "instance-create",
  "saga_id": "e3da8980-319d-4c5a-b5f2-ec68c71ebed2",
  "sec_id": "879812ec-d2c2-45e2-a70e-b54a22190178",
  "component": "ServerContext"
}
{
  "msg": "request completed",
  "v": 0,
  "name": "879812ec-d2c2-45e2-a70e-b54a22190178",
  "level": 30,
  "time": "2023-01-24T21:22:27.167927904Z",
  "hostname": "oxz_nexus",
  "pid": 6165,
  "uri": "//organizations/myorg/projects/myproj/instances",
  "method": "POST",
  "req_id": "cd859cd3-d016-4384-a657-fd191ec578fb",
  "remote_addr": "[fd01:1122:3344:101::15]:53702",
  "local_addr": "[fd02:1122:3344:101::3]:80",
  "component": "dropshot_external",
  "error_message_external": "No external IP addresses available",
  "error_message_internal": "No external IP addresses available",
  "response_code": "400"
}

So what failed?

root@[fd00:1122:3344:101::5]:32221/omicron> select event_type,node_id from saga_node_event where saga_id = 'e3da8980-319d-4c5a-b5f2-ec68c71ebed2' and event_type = 'failed';
  event_type | node_id
-------------+----------
  failed     |      38
(1 row)

root@[fd00:1122:3344:101::5]:32221/omicron> select saga_dag->'graph'->'nodes'->38 from saga where id = 'e3da8980-319d-4c5a-b5f2-ec68c71ebed2';
                                                  ?column?
-------------------------------------------------------------------------------------------------------------
  {"Action": {"action_name": "instance_create.create_snat_ip", "label": "CreateSnatIp", "name": "snat_ip"}}
(1 row)

Looking at sic_allocate_instance_snat_ip, it's using the "default" pool instead of "mypool":

    let (.., pool) = datastore                                                                             
        .ip_pools_fetch_default_for(&opctx, authz::Action::CreateChild)                                    
        .await                                                                                             
        .map_err(ActionError::action_failed)?;                                                             
    let pool_id = pool.identity.id;

There doesn't seem to be any way to instruct Nexus to use "mypool" here? If I instead add the ranges to "default", everything works correctly.

#2056 mentions that it adds "a default IP pool named default, which is used for address allocation unless a more specific IP pool is provided", but I don't see where that could occur.

[jmpesp]

@smklein tagging you as you worked on #2056

[bnaecker]

Interesting. There's a test that seems to pass checking this exact behavior, integration_tests::instances::test_instance_ephemeral_ip_from_correct_pool.

[smklein]

I believe the ephemeral IP you are allocating is being correctly allocated from the requested pool, and only uses the default pool if one is not supplied.

I do not think we currently expose an API to allow users to choose where their SNAT IP comes from. I think that actually pre-dated #2056 - let's take a look at the allocate_instance_snat_ip from before my change:

omicron/nexus/src/db/datastore/external_ip.rs

Lines 33 to 47 in cc1ef4f

	pub async fn allocate_instance_snat_ip(
	&self,
	opctx: &OpContext,
	ip_id: Uuid,
	project_id: Uuid,
	instance_id: Uuid,
	) -> CreateResult<ExternalIp> {
	let data = IncompleteExternalIp::for_instance_source_nat(
	ip_id,
	project_id,
	instance_id,
	/* pool_id = */ None,
	);
	self.allocate_external_ip(opctx, data).await
	}

(See that pool_id is always set to None!)

This used to mean, "Just use whatever IP pool is the default for the project". However, we no longer have project-scoped IP pools, so this was removed. In lieu of that, we're using the default pool.

All this is to say:

• I don't really know if we want to add something explicitly through the API to allow instance creation to specify the pool for SNAT IPs. This was not exposed, and continues to be not exposed.
• I think having project-wide settings to select a "preferred" pool, if not default, would be a fine addition?
• However, without that feature, I think the behavior that "SNAT IP allocation always comes from the default pool" is currently expected.

[jmpesp]

There's a test that seems to pass checking this exact behavior, integration_tests::instances::test_instance_ephemeral_ip_from_correct_pool.

The key difference with what I was doing and with this passing is that IP ranges are added to both the default and other pool, where I was only populating the other pool.

I don't really know if we want to add something explicitly through the API to allow instance creation to specify the pool for SNAT IPs. This was not exposed, and continues to be not exposed.

Yeah, reading a little more about SNAT IPs it seems like they're meant to be anonymous and non-discoverable:

omicron/nexus/db-model/src/external_ip.rs

Lines 41 to 47 in b277d5d

	/// This encompasses the three flavors of external IPs: automatic source NAT
	/// IPs, Ephemeral IPs, and Floating IPs. The first two are similar in that they
	/// are anonymous, lacking a name or description, which a Floating IP is a named
	/// API resource. The second two are similar in that they are externally-visible
	/// addresses and port ranges, while source NAT IPs are not discoverable in the
	/// API at all, and only provide outbound connectivity to instances, not
	/// inbound.

It seems that what I was doing before #2056 worked by accident :) I'll update my scripts and close this issue. Thanks for the quick responses!