DataProtection - Wrong activation time

[georgelivingston]

Describe the bug

Activation time is ahead of the creation time and causing (Cookies was not authenticated. Failure message: Unprotect ticket failed)

To Reproduce

Multiple service is deployed with the below code to protect the cookies. Everything works as expected. From yesterday, suddenly facing this error (Cookies was not authenticated. Failure message: Unprotect ticket failed) in one service, were the other services works as expected. On further analysis, found activate date is ahead of the creation date in the blob stored XML.

var dataProtectionBuilder = services.AddDataProtection()
                .PersistKeysToAzureBlobStorage(environmentVariable.AzureBlobStorage, "secret", "secretkey.xml")
                .SetApplicationName("cookiename");
dataProtectionBuilder?.ProtectKeysWithCertificate(x509ClientCertificate);

Below is the active key in the blob stored xml file.

<creationDate>2021-05-26T09:35:20.9324376Z</creationDate>
<activationDate>2021-05-26T09:35:20.078957Z</activationDate
<expirationDate>2021-08-24T09:35:20.078957Z</expirationDate>

Exceptions (if any)

Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[7]
Cookies was not authenticated. Failure message: Unprotect ticket failed

Further technical details

• ASP.NET Core version - 3.1
• Microsoft.AspNetCore.DataProtection, Version=3.1.0.0

[blowdart]

That's very odd. To clarify the other services sharing the keyring work just fine, with the same cookies?

[georgelivingston]

That's very odd. To clarify the other services sharing the keyring work just fine, with the same cookies?

Yes, the 3 services are sharing the same keyring. It works for 2 services, and not for only one service, with the same cookies.

[blowdart]

If it's working for two apps, then the date doesn't have any effect. Do you have any way to repo this for us?

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[georgelivingston]

If it's working for two apps, then the date doesn't have any effect. Do you have any way to repo this for us?

Sorry, I cannot be able to share the repo. But I have prepared the ConfigureServices details from startup, and data protection blob stored key.xml for your analysis. On further analysis, we found that, in one of our service has also faced this error for a second and it got recovered automatically. The other services didn't recover. After we manual refreshed the pod, it works as expected, and it occurred in our production, and we are down for more than an hour. We scared about this bug, that it may occur anytime again. We need your valuable suggestion or workaround to resolve this.

Additional note: Also, I have noted one more thing, the default rotation time is 90 days, but it is rotated irrespective of that, please refer the blob key.xml for more details.

public void ConfigureServices(IServiceCollection services)
 {

     services.AddAzureClients(builder =>
     {
         builder.AddBlobServiceClient(environmentVariable.AzureBlobStorage);
     });

     var keyVaultCache = new KeyVaultProxy();
     var clientSecretCredential = new ClientSecretCredential(environmentVariable.AzureADVaultTenantId, environmentVariable.AzureADVaultClientId, environmentVariable.AzureADVaultClientSecret);
     var cyptoOption = new CryptographyClientOptions();
     cyptoOption.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     IKeyEncryptionKeyResolver keyEncryptionKeyResolver = new KeyResolver(clientSecretCredential, cyptoOption);
     services.AddSingleton(keyEncryptionKeyResolver);

     KeyClientOptions keyClientOptions = new KeyClientOptions();
     keyClientOptions.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     SecretClientOptions secretClientOptions = new SecretClientOptions();
     secretClientOptions.AddPolicy(keyVaultCache, HttpPipelinePosition.PerCall);

     var secretClient = new SecretClient(new Uri(environmentVariable.VaultUri), clientSecretCredential, secretClientOptions);
     var keyClient = new KeyClient(new Uri(environmentVariable.VaultUri), clientSecretCredential, keyClientOptions);
     services.AddSingleton(secretClient);
     services.AddSingleton(keyClient);
     var dataProtectionBuilder = services.AddDataProtection()
         .PersistKeysToAzureBlobStorage(environmentVariable.AzureBlobStorage, "secret", "secretkey.xml")
         .SetApplicationName("cookiename");

     if (!webHostEnvironment.IsDevelopment())
     {
         KeyVaultSecret certificate = secretClient.GetSecret(environmentVariable.DataProtectionKeyVaultSecret);
         var x509ClientCertificate = new X509Certificate2(Convert.FromBase64String(certificate.Value));
         dataProtectionBuilder?.ProtectKeysWithCertificate(x509ClientCertificate);
         dataProtectionBuilder.DisableAutomaticKeyGeneration();
     }


     services.Configure<CookiePolicyOptions>(options =>
     {
         options.CheckConsentNeeded = context => false;
         options.MinimumSameSitePolicy = SameSiteMode.None;
     });

     services.AddAuthentication(options =>
     {
         options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
         options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
         options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
     })
     .AddCookie()
     .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options =>
     {
         var handler = new JwtSecurityTokenHandler();
         handler.InboundClaimTypeMap.Clear();
         options.SecurityTokenValidator = handler;
         options.RequireHttpsMetadata = true;
         options.Authority = environmentVariable.LoginAuthority;
         options.SignInScheme = "Cookies";
         options.GetClaimsFromUserInfoEndpoint = true;

         // Configure the Auth0 Client ID and Client Secret.
         options.ClientId = environmentVariable.LoginClientId;
         options.ClientSecret = environmentVariable.LoginClientSecret;
         options.CallbackPath = new PathString("/signin-oidc");

         // options.Scope.Clear();
         options.Scope.Add("openid");
         options.Scope.Add("profile");
         options.ClaimActions.MapUniqueJsonKey("id", "id");
         options.ClaimActions.MapUniqueJsonKey("EmailId", "EmailId");
         options.ResponseType = OpenIdConnectResponseType.Code;
         options.UsePkce = true;
         options.ResponseMode = "query";

         options.Events.OnRemoteFailure = context =>
         {
             if (context.Failure.Message.Contains("Correlation failed", StringComparison.InvariantCultureIgnoreCase))
             {
                 context.Response.Redirect("/");
             }
             else
             {
                 context.Response.Redirect("/Account/Error");
             }

             context.HandleResponse();
             return Task.CompletedTask;
         };

         options.Events.OnRedirectToIdentityProvider = context =>
         {
             if (context.Request.Headers.ContainsKey("X-Requested-With"))
             {
                 context.Response.StatusCode = StatusCodes.Status401Unauthorized;
                 context.Response.WriteAsync(JsonConvert.SerializeObject(new { loginLink = environmentVariable.RedirectHomePageUrl, isPageRedirection = true }));
                 context.HandleResponse();
             }

             return Task.CompletedTask;
         };
     });
	
     services.AddJwtAuthentication(AppAuthenticationSchemes.IdJwtScheme);

     services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
         .AddCertificate(options =>
         {
             options.AllowedCertificateTypes = CertificateTypes.All;
         });

}

Key.XML

<?xml version="1.0" encoding="utf-8"?><repository><key id="36c1e414-5b3c-438a-8b07-73ec5a528486" version="1"><creationDate>2020-04-23T12:31:14.818651Z</creationDate><activationDate>2020-04-23T12:31:13.543231Z</activationDate><expirationDate>2020-07-22T12:31:13.543231Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="099f105a-e590-4e66-90d7-0883e8e2db72" version="1"><creationDate>2020-07-21T08:05:10.9149068Z</creationDate><activationDate>2020-07-22T12:31:13.543231Z</activationDate><expirationDate>2020-10-19T08:05:10.704182Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</X509Certificate></X509Data></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedKey></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="78f05c65-38d9-4b9e-bf26-7a4577a0f5f1" version="1"><creationDate>2020-09-03T09:05:48.0849065Z</creationDate><activationDate>2020-09-03T09:05:45.1695927Z</activationDate><expirationDate>2020-12-02T09:05:45.1695927Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="2a89281f-9fd0-453c-ac37-c9aced5a0759" version="1"><creationDate>2020-11-30T10:07:35.2706401Z</creationDate><activationDate>2020-12-02T09:05:45.1695927Z</activationDate><expirationDate>2021-02-28T10:07:34.8158656Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.9.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.9.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="e000fae6-6a3b-46ae-b476-a95f30a522be" version="1"><creationDate>2021-02-15T06:18:04.1338347Z</creationDate><activationDate>2021-02-15T06:18:02.2434726Z</activationDate><expirationDate>2021-05-16T06:18:02.2434726Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.12.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key><key id="cf7e1a32-3a8a-4f34-8eca-367e1aa06f79" version="1"><creationDate>2021-05-14T14:07:38.0258627Z</creationDate><activationDate>2021-05-16T06:18:02.2434726Z</activationDate><expirationDate>2021-08-12T14:07:37.7895481Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.14.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=3.1.14.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection"><EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#"><EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" /><KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#"><X509Data><X509Certificate>***Redacted***</X509Certificate></X509Data></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedKey></KeyInfo><CipherData><CipherValue>***Redacted***</CipherValue></CipherData></EncryptedData></encryptedSecret></descriptor></descriptor></key><key id="8b561d98-e699-4a50-8ab9-c4a70d13061a" version="1"><creationDate>2021-05-26T09:35:20.9324376Z</creationDate><activationDate>2021-05-26T09:35:20.078957Z</activationDate><expirationDate>2021-08-24T09:35:20.078957Z</expirationDate><descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=3.1.15.0, Culture=neutral, PublicKeyToken=adb9793829ddae60"><descriptor><encryption algorithm="AES_256_CBC" /><validation algorithm="HMACSHA256" /><masterKey p5:requiresEncryption="true" xmlns:p5="http://schemas.asp.net/2015/03/dataProtection"><!-- Warning: the key below is in an unencrypted form. --><value>***Redacted***</value></masterKey></descriptor></descriptor></key></repository>

[georgelivingston]

Due to no propagation time between the created and activation time, this issue occurs. I will explain the same with the below illustration.

The new key is generated from POD 1. The User A logs in and visits the first pod, and he claims the new protected cookie. Again, the same user visits using POD 2 (Old key), now the request failed with this “Cookies was not authenticated. Failure message: Unprotect ticket failed” and redirects to login.

User A again logs in and redirected to POD 1, it got successful, now again the consecutive request sends to POD 2. Again, it redirects to login. It became deadlock, POD 2 become useless, and the user facing repeated login.

As explained in the documentation it should have a provided an enough propagation time to get sync'd with the new key for all POD’s.

https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-management?view=aspnetcore-5.0

[ajbeaven]

Thanks for reporting this one @georgelivingston.

I believe I am experiencing the same problem on my .NET5 applications. I have two apps storing the Data Protection keys in Azure Blob storage where the activation date of the key is >1 second after the creation date. One app is responsible for creating a cookie, which is then sent to the second app to authenticate the user there. This issue presents as the second app failing to authenticate.

Similarly the keys seem to be rotated extremely frequently (not every 90 days). This would happen at least once every 24-48 hours, after which time the second app would start to fail with an authentication error.

Both apps use the same code in Startup.cs:

var dataProtection = services.AddDataProtection()
	.SetApplicationName(siteSettings.SiteName)
	.PersistKeysToAzureBlobStorage(
		configuration.GetConnectionString("StorageConnectionString"),
		dataProtectionSettings.KeyStoreContainerName,
		dataProtectionSettings.KeyStoreBlobName)
	.ProtectKeysWithAzureKeyVault(
		new Uri(dataProtectionSettings.KeyVaultIdentifier),
		new DefaultAzureCredential());

Unlike @georgelivingston, I'm protecting the keys via Azure Key Vault. MS support seemed fairly confident it wasn't related to the Azure Key Vault side of things. That would seem to be supported by the fact that we both have the same problem despite using different protection mechanisms.

MS support is reportedly looking in to this for me now. Fingers crossed they identify the issue and fix shortly as this is causing real problems on my production application.

In the meantime I have set up a auto-heal rule to restart the second app when it sees a few 401 errors in quick succession. Restarting the app appears to resolve the error, presumably because it retrieves the new key at startup.

[ajbeaven]

The support team was able to identify the issue which was that multiple apps were using the same storage location to persist the Data Protection keys. Both of my apps were storing these keys in the same Azure Storage blob.

I figured this to be OK due to my reading of the docs relating to the SetApplicationName() method. This states that using different application names across apps will isolate those apps from one another. Unfortunately this isolation does not extend to the actual physical data protection keys rather, just the payload stored in them. Pointing to the same location with multiple apps using different application names will causes these issues with the activation date and subsequent problems with cross-app data protection.

@georgelivingston I'm unsure if this is the same setup you've got, but I'd recommend checking for this same thing. Using a different location to store the data protection keys immediately resolved all my issues.

[adityamandaleeka]

This is hard to investigate without a repro. If someone can provide one, we'd be happy to investigate.