V13 AppDomain (terminating) due to Unhandled exception

[rammi987]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

13.5.2

Bug summary

We have a live production site witch suddenly just went down without notis. This is not an exception i have seen before, but look to relate to the EF core implenetion in Umbraco member part.

Specifics

No response

Steps to reproduce

{"@t":"2025-03-05T08:07:33.3778760Z","@mt":"Unhandled exception in AppDomain (terminating).","@l":"Error","@x":"OpenIddict.Abstractions.OpenIddictExceptions+ConcurrencyException: The application was concurrently updated and cannot be persisted in its current state.\r\nReload the application from the database and retry the operation.\r\n ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.\r\n at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithRowsAffectedOnlyAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)\r\n at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)\r\n at OpenIddict.EntityFrameworkCore.OpenIddictEntityFrameworkCoreApplicationStore5.UpdateAsync(TApplication application, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at OpenIddict.EntityFrameworkCore.OpenIddictEntityFrameworkCoreApplicationStore5.UpdateAsync(TApplication application, CancellationToken cancellationToken)\r\n at OpenIddict.Core.OpenIddictApplicationManager1.UpdateAsync(TApplication application, CancellationToken cancellationToken)\r\n at OpenIddict.Core.OpenIddictApplicationManager1.UpdateAsync(TApplication application, OpenIddictApplicationDescriptor descriptor, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Infrastructure.Security.OpenIdDictApplicationManagerBase.CreateOrUpdate(OpenIddictApplicationDescriptor clientDescriptor, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Api.Delivery.Security.MemberApplicationManager.EnsureMemberApplicationAsync(IEnumerable1 loginRedirectUrls, IEnumerable1 logoutRedirectUrls, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Api.Delivery.Handlers.InitializeMemberApplicationNotificationHandler.HandleAsync(UmbracoApplicationStartingNotification notification, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Core.Events.INotificationAsyncHandler1.HandleAsync(IEnumerable1 notifications, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Core.Events.EventAggregator.PublishCoreAsync[TNotification](IEnumerable1 allHandlers, IEnumerable1 notifications, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Core.Events.NotificationAsyncHandlerWrapperImpl1.HandleAsync[TNotification,TNotificationHandler](IEnumerable1 notifications, CancellationToken cancellationToken, ServiceFactory serviceFactory, Func4 publish)\r\n at Umbraco.Cms.Core.Events.EventAggregator.PublishNotificationsAsync[TNotification,TNotificationHandler](IEnumerable1 notifications, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Core.Events.EventAggregator.PublishAsync[TNotification,TNotificationHandler](IEnumerable1 notifications, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Core.Events.EventAggregator.PublishAsync[TNotification](TNotification notification, CancellationToken cancellationToken)\r\n at Umbraco.Cms.Infrastructure.Runtime.CoreRuntime.StartAsync(CancellationToken cancellationToken, Boolean isRestarting)\r\n at Umbraco.Cms.Infrastructure.Runtime.CoreRuntime.StartAsync(CancellationToken cancellationToken)\r\n at Umbraco.Extensions.WebApplicationExtensions.BootUmbracoAsync(WebApplication app)\r\n at Program.

$(String[] args) in D:\a\1\s\backend\umbraco\Program.cs:line 32\r\n at Program.(String[] args)","SourceContext":"Umbraco.Cms.Infrastructure.Runtime.CoreRuntime","ProcessId":2440,"ProcessName":"w3wp","ThreadId":1,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn0sdwk0002WO","Log4NetLevel":"ERROR"}`

Expected result / actual result

The site should not go down.

---

This item has been added to our backlog AB#54804

[github-actions]

Hi there @rammi987!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[kjac]

Hi @rammi987

Thanks for reporting this 💪 is this something you encounter regularly?

[kjac]

...and a follow-up question: Did this happen on a load balanced site?

[rammi987]

Hi @kjac We have expirence it twice, where we just get a notification from the customer says, there site went down.
So not regularly, but it is a mysterious.

The setup is as followed.
We have Azure Container app to run a NextJs application in the frontend, witch get its data from the delivery api.
The umbraco intances is running on a Azure web app with linux kernel, and a mssql, with a blob storage.
The Umbraco intances is a single running server, and don´t have any scaling rules, the server is 1 P1V3.

[kjac]

Thanks @rammi987

Any chance multiple servers could be connecting to the production DB? I'm thinking different environments, or even devs hooking up to the production DB for debugging purposes?

[rammi987]

@kjac No. The dev and test server is locatede in another resccouse group where prod i located (i was prod we see this error on (never seen it on other env.)) The only connection to the production DB is the app server for prod itself.

[kjac]

This "should not happen" 🙈

The InitializeMemberApplicationNotificationHandler is responsible for updating the member application for OpenID auth at boot time. The notification handler is only ever invoked once by the core.

Something really strange is afoot here.

I have created #19760 which adds defensive coding around the notification handler to guard against concurrent DB writes. Hopefully that solves the issue.