Address quota issues for IAM create/delete custom role issue

[leahecole]

We are seeing test failures we have seen before in IAM in #9525 and #9526 as a result of #9491. We asked the IAM team about this years ago on YAQs and never heard back. We ended up choosing to use multiple test projects to solve the issue then and likely will need to do that again now, unless circumstances have changed and someone knows something I don't! (I suspect they haven't though - someone actually pinged me about that YAQs a month or so ago asking what we ended up doing)

[m-strzelczyk]

I think that the documentation is quite clear about the conditions of deleted roles.

When a role is deleted, any role bindings that refer to the role remain in your allow policies, but they have no effect. You can undelete a role within 7 days. During this 7-day period, the Google Cloud console shows that the role was deleted. You can also list deleted roles programmatically, but they are omitted by default.

After 7 to 14 days, the role is scheduled for permanent deletion. At this point, the role no longer counts towards the limit of 300 custom roles per organization or 300 custom roles per project.

The permanent deletion process takes 30 days. During the 30-day window, the role and all associated bindings are permanently removed, and you cannot create a new role with the same role ID.

After the role has been permanently deleted, up to 44 days after the initial deletion request, you can create a new role using the same role ID.

I guess we need to revert back to using separate projects, to avoid this quota.

[leahecole]

This whole 30 day window really is a pain