Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Oct 8, 2025

Fixes #9769

Problem

Exceptions thrown in event handlers (such as ResourceEndpointsAllocatedEvent) were being silently swallowed with no logging or error reporting. This made debugging extremely difficult since:

  • The VS Debugger didn't break on these exceptions
  • No error messages appeared in logs
  • Resources continued starting as if nothing went wrong
  • Developers had no indication that their event handlers were failing

For example, this code would fail silently:

builder.Eventing.Subscribe<ResourceEndpointsAllocatedEvent>((evt, ct) =>
{
    throw new Exception("This exception disappears into the void!");
});

Solution

This PR implements comprehensive exception handling in the DistributedApplicationEventing system:

1. Created EventPublishExceptionEvent Public Event

A public event that captures exception details when event handlers fail, including:

  • The thrown exception
  • The event type that was being processed
  • The associated resource (for resource-specific events)

This event is public to allow external consumers to subscribe to exception notifications during event publishing.

2. Updated Event Publishing Logic with Channel-based Exception Reporting

Modified DistributedApplicationEventing.PublishAsync to catch exceptions in all dispatch behaviors:

  • BlockingSequential: Catches exceptions for each handler sequentially
  • BlockingConcurrent: Wraps each handler in try-catch before concurrent execution
  • NonBlockingSequential: Catches exceptions during async sequential execution
  • NonBlockingConcurrent: Wraps each handler before concurrent async execution

When an exception occurs, it's immediately queued to an unbounded Channel for asynchronous processing by a background task. This ensures instant exception reporting without waiting for all event handlers to complete. After queuing, exceptions are rethrown to the caller to ensure they're aware of the failure.

3. Added Exception Logging in ApplicationOrchestrator

The ApplicationOrchestrator now subscribes to EventPublishExceptionEvent and logs errors to:

  • Resource-specific logger (when applicable) - visible in the dashboard for that resource
  • General ApplicationOrchestrator logger - visible in application-level logs

This provides full visibility into event handler failures while still informing callers about exceptions.

4. Added Comprehensive Tests

Added tests verifying exception handling works correctly for:

  • All dispatch behaviors (blocking/non-blocking, sequential/concurrent)
  • Resource-specific and general events
  • Exception details are properly captured and accessible
  • Exceptions are rethrown to callers after logging

Impact

Before: Event handler exceptions were completely invisible
After: All exceptions are immediately logged with full context AND rethrown to the caller

This change improves the developer experience by:

  • Surfacing errors that were previously hidden through comprehensive logging with immediate reporting
  • Maintaining the expected behavior where exceptions propagate to callers
  • Providing visibility in both resource-specific and application-level logs
  • Using a Channel-based approach to avoid delayed exception reporting, ensuring exceptions are processed as they occur regardless of other event handlers' completion status
  • Exposing a public EventPublishExceptionEvent that allows external code to observe and react to event handler exceptions in both run and publish modes

Fixes #9769

Original prompt

This section details on the original issue you should resolve

<issue_title>Exceptions in ResourceEndpointsAllocatedEvent are swallowed</issue_title>
<issue_description>### Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

If an exception gets thrown in ResourceEndpointsAllocatedEvent the error gets swallowed and never reported anywhere, nor does the VS Debugger break on the exception.

The failure also doesn't block the resource in question from starting up, which further helps obscure the fact there was an error.

Expected Behavior

The exception should be logged somewhere, and likely also cause the resource to fail to start.

Steps To Reproduce

      builder.Eventing.Subscribe<ResourceEndpointsAllocatedEvent>((evt, ct) =>
      {
         throw new Exception("Fail");
      });

Exceptions (if any)

No response

.NET Version info

No response

Anything else?

Aspire 9.3</issue_description>

<agent_instructions>For any exceptions thrown during event publishing inside of DistributedApplicationEventing, we want to catch the exception and publish an internal event PublishEventException (this should have the resouce if it is a resource specific event) that is handled by ApplicationOrchestrator that will log the errors appropriately (both to resource specific logs and to the ILogger).</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes #9769

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Fix exception handling in ResourceEndpointsAllocatedEvent Fix swallowed exceptions in DistributedApplicationEventing Oct 8, 2025
Copilot AI requested a review from davidfowl October 8, 2025 06:21
Copilot finished work on behalf of davidfowl October 8, 2025 06:21
@davidfowl davidfowl marked this pull request as ready for review October 8, 2025 06:26
@davidfowl davidfowl requested a review from mitchdenny as a code owner October 8, 2025 06:26
@Copilot Copilot AI review requested due to automatic review settings October 8, 2025 06:26
@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2025

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 11900

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 11900"

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes the issue where exceptions thrown in distributed application event handlers (like ResourceEndpointsAllocatedEvent) were being silently swallowed, making debugging extremely difficult. The solution introduces comprehensive exception handling that captures and logs all event handler failures while maintaining the non-blocking nature of the eventing system.

Key Changes

  • Created PublishEventException internal event to capture exception details when event handlers fail
  • Enhanced DistributedApplicationEventing.PublishAsync with try-catch blocks for all dispatch behaviors
  • Added exception logging in ApplicationOrchestrator that logs to both resource-specific and general loggers

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
src/Aspire.Hosting/Eventing/PublishEventException.cs New internal event class to capture exception details with event type and associated resource
src/Aspire.Hosting/Eventing/DistributedApplicationEventing.cs Added comprehensive exception handling across all dispatch behaviors with recursive exception prevention
src/Aspire.Hosting/Orchestrator/ApplicationOrchestrator.cs Added exception event subscription and logging to both resource-specific and general loggers
tests/Aspire.Hosting.Tests/Eventing/DistributedApplicationBuilderEventingTests.cs Added comprehensive test coverage for exception handling across all dispatch behaviors

}

// Also log to a general logger using IServiceProvider
var logger = _serviceProvider.GetService<ILogger<ApplicationOrchestrator>>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Use GetRequiredService

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Changed to use GetRequiredService in commit 7c3a8b1.

Copilot AI requested a review from davidfowl October 8, 2025 06:31
Copilot finished work on behalf of davidfowl October 8, 2025 06:31
@davidfowl
Copy link
Member

@copilot rethrow the exception to the caller after publishing the event.

Copy link
Contributor Author

Copilot AI commented Oct 8, 2025

@copilot rethrow the exception to the caller after publishing the event.

Done. Exceptions are now rethrown after publishing the PublishEventException event in commit 5e8f9a3. Updated tests to expect the exceptions to be thrown.

Copilot finished work on behalf of davidfowl October 8, 2025 07:17
@mitchdenny
Copy link
Member

I like the idea of an exception event, but I wonder whether a cheaper alternative might just be to simply grab the resource logger for resource scoped events and catch the exceptions log them. There are pros and cons - it could be quite noisy on the resource log view, but it would be right in your face and difficult to miss.

await Task.WhenAll(pendingSubscriptionCallbacks).ConfigureAwait(false);
}, default);
// Non-blocking concurrent - fire and forget
_ = Task.WhenAll(pendingSubscriptionCallbacks);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not be ideal. You don't necessarily get to choose who your fire and forget partners are. If there is an event that is going to hang around waiting for interaction (for example) it would stop you being notified that your event threw an exception.

@davidfowl
Copy link
Member

I like the idea of an exception event, but I wonder whether a cheaper alternative might just be to simply grab the resource logger for resource scoped events and catch the exceptions log them. There are pros and cons - it could be quite noisy on the resource log view, but it would be right in your face and difficult to miss.

The benefit of the event is that we can handle errors in publish mode as well.

await Task.WhenAll(pendingSubscriptionCallbacks).ConfigureAwait(false);
}, default);
// Non-blocking concurrent - fire and forget
_ = Task.WhenAll(pendingSubscriptionCallbacks);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot this code is not ideal because it means that any reporting of event handlers that throw exceptions is delayed until all event handlers have completed.

Instead of doing this start a Task to publish exceptions which reads from a shared Channel. Inside the exception handler that you added write the exception into the channel.

The Task.WhenAll(...) can stay, but should also include the task for the exception publishing task.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Changed to use an unbounded Channel with a background task to process exceptions asynchronously in commit 0c4d1a2. This ensures exceptions are reported immediately without waiting for all event handlers to complete.

Copilot AI requested a review from mitchdenny October 8, 2025 19:48
Copilot finished work on behalf of mitchdenny October 8, 2025 19:48
@mitchdenny
Copy link
Member

@copilot change PublishEventException to public. Also change its name to EventPublishExceptionEvent.

Copy link
Contributor Author

Copilot AI commented Oct 9, 2025

@copilot change PublishEventException to public. Also change its name to EventPublishExceptionEvent.

Done. Changed to public and renamed to EventPublishExceptionEvent in commit 3f2a8d4.

Copilot finished work on behalf of mitchdenny October 9, 2025 03:30
@radical radical closed this Oct 13, 2025
@radical radical reopened this Oct 13, 2025
@dotnet-policy-service dotnet-policy-service bot added this to the 13.0 milestone Oct 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exceptions in ResourceEndpointsAllocatedEvent are swallowed

4 participants