better flesh out operator privileges in other Silos

[davepacheco]

#1340 proposed that users with fleet-level privileges would have no privileges to access siloed resources in Silos other than their own. (This phrasing wasn't really fleshed out until RFD 297, but I believe that's essentially what #1340 meant.) The principle was essentially: we shouldn't be deciding who's allowed to cross Silo lines -- if an operator wants to access another Silo, they can do that, but they do it by having an account in that Silo's IdP, which makes it noisy and auditable.

RFD 309 raises a number of user stories that cast some doubt on this approach. To be clear, I'm not sure yet what the right answer is, but there are enough questions that I think we want to revisit this before committing too far one way or the other.

CC @plotnick @rmustacc @kc8apf

[zephraph]

I don't think it's quite clear to me how 309 casts doubt on the separation. We've always acknowledged that operators are going to need some lens into silo resources to be able to debug escalated issues, but my understand is that we still want to preserve the separation so that operators don't have unnecessary access to what might be confidential details.

#3092 is an example of where we're bending the boundaries a bit by making project/instance names visible to the operator. While this gives operators more insight into silo specific resources it does so to aid in establishing shared communication between the operator and the developer. I chose very specifically to only display the minimal context that the operator needs in this case.

[davepacheco]

I understood #1340 (and my summary in this comment) to be proposing that Fleet Administrators would have no privileges to see just about anything inside other Silos. Not the list of projects, instances, or anything like that. I agree that's too rigid and RFD 309 explains why. This ticket is about figuring out what these boundaries really are and ideally what principles guide them: e.g., provide access when there's particular value provided by a holistic, cross-Silo view; or where it's not even possible to correlate things without it: e.g., if a Fleet Admin can see sleds but not instances, and the Silo Admin can see instances but not sleds, then literally nobody can tell you what instances are on what sleds.

[davepacheco]

When we do this, we should re-evaluate:

• whether it makes sense to allow Silo and Project policies to refer to identities outside the Silo. See Fleet privileges should not cascade into Silos #1580.
• disallow fleet administrators from reading or modifying the policy on the Silo. See Fleet privileges should not cascade into Silos #1580 for details. Since fleet admins need to be able to modify which group is the "admin" group for a Silo, they may wind up changing the policy, but would not have carte blanche to change it however they like.

[askfongjojo]

A couple of us met today to try putting this long-standing open item to rest.
@davepacheco @smklein @inickles. Please feel free to make edits or additional comments as you see fit.

Here are the key determinations we've made (and I'll follow up with a short RFD that essentially collates these decisions with meeting notes, background info, related tickets to make them more visible/formal):

1. Stick to the current model of strict tenancy separation as described in omicron#1340 for all silo resource API. User with fleet-level roles (admin or viewer - whom we'll reference as "operators" below) will need to be explicitly granted the appropriate silo role if they want to view or act on resources in certain silos.
2. For other use cases that allow a user to consume fleet-level information for audit or support purposes (e.g., someone in SecOps), we'll come up with new IAM roles to make the access highly constrained and explicit. This overrides the design stated in RFD 523 (audit logs) and RFD 496 (support bundle) that makes these fleet-wide artifacts accessible to operators by default. In other words, operators will get audit log and support bundle access only if they explicitly grant themselves the corresponding new roles; users who have these new roles only will not have access to other operator/system API or UI. We'll not attempt to redact resource names or other potentially sensitive information (e.g., IP addresses) until this is mandated by customers.
3. For the special case of enumerating resources residing on hardware components to determine maintenance or fault impact (today, the only such example is sled_instance_list but we may have similar APIs for disks and IP addresses in the future), we should use IDs everywhere as resource identifiers to avoid leaking any sensitive information conveyed in resource names.
4. For any future use cases that present a strong need to allow operator access to silo resources, we'll likely enhance the IAM model to provide more fine-grained roles that govern what specific actions/attributes users can take or see for each resource.

[david-crespo]

This will be interesting for the web console. We only have system and silo sides. We’ll have to either add a third section that only shows up for users with this role, or we reuse the system section for audit role-only users but the only thing in the sidebar is the thing they have access to.