Allow customization of the DataProtection PersistKeysToDbContext key ring store

[JoeMarkov]

As a good developer I should configure the Data Protection API so that I store the key ring outside my service. So obviously you try to

            services.AddDataProtection()
                .PersistKeysToDbContext<DataProtectionContext>();

Great! it works! Simple and awesome! You end up with a table and an entry like this:

But, if I now want to do a micro-service style architecture with a shared database for infrastructure things like this, I get into trouble because the Data Protection API just takes the first available key ring in this table. So clearly we have a race condition if I want to let multiple applications share this table.

I am not using my database for much else and creating a separate database just to keep the key rings apart feels a bit stupid.

So my proposal is to allow the user to define the name of the key ring to look for. So that I can keep multiple independent key rings in the same shared table.

            services.AddDataProtection()
                .PersistKeysToDbContext<DataProtectionContext>(applicationName: "PaymentAPI" );

and another app like

            services.AddDataProtection()
                .PersistKeysToDbContext<DataProtectionContext>(applicationName: "InvoiceAPI" );

This would then store two rows in the [DataProtectionKeys] table.

Is your feature request related to a problem? Please describe.

Yes, I try to keep all my Data Protection key rings in one table. But that does not work under current implementation. I either have to create a separate context per service or hack the implementation of PersistKeysToDbContext.

Describe the solution you'd like

Passing a key ring name to the PersistKeysToDbContext would help me a lot.

[blowdart]

Keyrings are meant to be shared between multiple apps, why do you think there's a race condition in just taking the first key from the table? App isolation is enabled seperate to the key in use.

[JoeMarkov]

I agree that the key-ring should be sharable between services and I am looking for guidance on how to properly share the keys across different services and across similar services.

My point is that is that, for example:

• At first deployment, the database table is empty and now three services starts up at the same time and find the table empty, and wants to insert their own key ring to that table. All overwriting each other with different KeyId's.

• At key rotation after 90 days, multiple services might want to do the key rotation at the same time and adding their new key to the key ring.

Yes, the risk that they happen exactly at the same is small, but I think it can be a potential issue creating random weird bugs in production .

Or I should not worry about this issue (just trying to be a good developer here :) )?

So, what I wanted in my scenario is that if I have three independent services in production, that does not need to share the key, then I would like to be able to isolate them in the same "dataProtection" table. So that I can have only one table and database, holding multiple keys. Perhaps be able to "name" the key to look for? like InvoiceKey, PaymentKey, WebFrontendKey?

What I did in my application/setup was to create three IDataProtectionKeyContext that overrides the table name like

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<DataProtectionKey>().ToTable("DataProtectionKeys_BillingAPI");
        }

Or perhaps worth mention how to reason about this on the Key storage providers in ASP.NET Core page.

[JoeMarkov]

Also, don't know if the Write to the table should have some optimistic concurrency check using a timestamp to detect if the key-ring was charged in-between the read and write to the table. As currently the provider assumes its the sole owner of the data.

[blowdart]

Ah got you, yea, the rush to create keys in an empty keyring is something that really needs addressing. Right now you can create a master node which is the one responsible for creating keys and have the rest read only,

However renewal isn't so much a problem, renewal starts a few days before keys expire, staggers with a bit of randomness, and the keyring is going to be reloaded a couple of times before keys expire, so if multiple instances create keys they'll become consistent before the need to rotate actually happens.

[JoeMarkov]

Thanks! I am glad to hear that you agree with my concerns 👍

Well, I think we can both agree that the issue here can cause various random intermittent problems in production and for the average developer who don't understand the inner working of the Data Protection API storage model will have a hard time troubleshooting these errors.

So, I was also considering the master node idea, but that still needs that I hack my own storage model I guess, because there's not feature today to control "read/write" against the key-ring. then there is always the issue what a read-only node should do if there is no key ring, should it wait and retry (polly?) or just crash at startup?

I just want that we have clear guidance in the documentation on how do I configure data protection in a proper way in a typical (for today) micro-service or load balanced container environment.

I also think the feature I asked for in the beginning, to be able to store multiple key-rings in the same table is a good feature too. Perhaps be able to name the keys?

While talking about the data protection API, I really wish there was an official package from Microsoft to store the key ring in Azure Key Vault. I found a sample implementation here https://github.com/edumentab/AzureKeyVaultKeyRingRepository
that I have been looking at to use. Just to have the data protection key and the key-ring in the same "place".

[blowdart]

We can't store the key ring in keyvault, the API from keyvault doesn't lend itself to reading multiple keys very well, plus the permissions aren't granular enough for apps for us to feel comfortable with it.

Naming keys etc gets rather away from the purpose of data projection which is mean to be an opaque box where you don't configure anything beyond storage.

But yes, the container story is rather lacking right now. I'll put it into next sprint planning.

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[ZOXEXIVO]

@JoeMarkov How do you solve problem, when some service renew key, but other services use old versions and it need load new key without restart?

[JoeMarkov]

What I did was that I hacked the data context class and created one context for each service, so that each service has its own database table in a shared database. As each service has its own table, then they will automatically get their own keys. However this does not solve the issue if you need to load-balance and use the same key in multiple services.

Like
[
public class PaymentApiDataProtectionContext : DbContext, IDataProtectionKeyContext
{
public PaymentApiDataProtectionContext(DbContextOptions options)
: base(options) { }

    // This maps to the table that stores keys.
    public DbSet<DataProtectionKey>? DataProtectionKeys { get; set; }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        //Rename the table name 
        modelBuilder.Entity<DataProtectionKey>().ToTable("DataProtectionKeys_PaymentAPI");
    }
}](url)

Alternatively, you hack the data protection "storage" model and then do a "reload of the keys before you add new keys to it" as a way to minimize the risk that you overwrite data. Like how they do it here:
https://github.com/edumentab/AzureKeyVaultKeyRingRepository/blob/master/AzureKeyVaultKeyRingRepository%20demo/AzureKeyVaultKeyRingRepository.cs#L158

Get a fresh copy of the keyring before you add more to the keyring, asa way to reduce issues of overwrites of keys.

[ZOXEXIVO]

@JoeMarkov thanks. We have problem with same service under load-balancer.
I think DataProtection layer looks like old legacy system from early 2000 and need to be fully rewrited.
In new projects we skip this library and make own, robust and simple implementation