Load Balancing: Implement distributed background jobs

[Zeegaan]

Notes

Introduces distributed background job support for load-balanced Umbraco environments by implementing database-backed job coordination to ensure certain jobs run only once across all servers.
Problem
Umbraco's RecurringBackgroundJobs currently lack the ability to distinguish between:

• Per-server jobs - Must run on every server instance
• Distributed jobs - Should run only once across the entire cluster

In load-balanced scenarios, this causes distributed jobs to execute redundantly on all servers, leading to unnecessary processing and potential conflicts.

Solution

This PR introduces a new IDistributedBackgroundJob abstraction with database-backed coordination:

IDistributedBackgroundJob Interface

• New abstraction for jobs that should run once across all servers
• Jobs are persisted and coordinated via database

DistributedBackgroundJobsHostedService

• Periodically checks for pending distributed jobs (configurable interval)
• Uses database locking to prevent concurrent execution:
• Acquires lock on distributed jobs table
• Fetches available job and marks IsRunning = true
• Releases lock
• Executes the job
• Re-acquires lock and updates LastRun timestamp and IsRunning = false

DistributedJobSettings

• CheckPeriod - How frequently to poll for jobs
• StartupDelay - Staggered startup delay per server to reduce initial load

IDistributedJobService

• Manages distributed job lifecycle
• UpdateAllChangedAsync() - Reconciles registered jobs with database on startup:
• Compares ServiceCollection registrations against database records
• Updates job periods and configuration when changed
• Ensures database stays in sync with code

Testing

Single server testing

• Run the site and observe the distributed jobs database table
• Verify jobs execute according to their configured periods
• Confirm jobs with longer periods remain pending until due
• You can also configure some of these periods (Like webhook period in webhook settings), and try to restart the site, assert the period gets updated.

Load-Balanced testing:

• Run two sites locally against the same database
• Verify each job executes on only one server (not both)

[Copilot]

Pull Request Overview

Implements distributed (cluster-wide, single-run) background jobs with database-backed coordination so that specific recurring jobs execute only once across all servers in a load-balanced Umbraco environment.

• Introduces IDistributedBackgroundJob and a coordinating DistributedBackgroundJobHostedService with locking and persistence.
• Adds repository/service layer, schema (table + lock), migration, install seed data, and configuration settings (DistributedJobSettings).
• Converts selected former per-server IRecurringBackgroundJob implementations to distributed jobs and registers them accordingly.

Reviewed Changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tools/Umbraco.JsonSchema/UmbracoCmsSchema.cs	Adds DistributedJobSettings to schema definition.
Umbraco.Web.Common/DependencyInjection/UmbracoBuilderExtensions.cs	Replaces previous recurring job registrations with distributed job registrations and hosted service.
DistributedJobService.cs	Service implementing coordination logic for claiming and finishing jobs.
IDistributedJobService.cs	Interface for distributed job coordination.
DistributedJobRepository.cs	Data access for distributed job records (selection, update, finish).
IDistributedJobRepository.cs	Repository contract for distributed jobs.
DistributedJobDto.cs	NPoco DTO mapping for distributed job table.
DistributedBackgroundJobModel.cs	Domain model for distributed jobs.
AddDistributedJobs.cs	Migration creating table, lock entry, and seeding jobs on upgrade.
UmbracoPlan.cs	Adds migration step for distributed jobs.
DatabaseSchemaCreator.cs	Includes DistributedJobDto in install schema creation.
DatabaseDataCreator.cs	Seeds initial distributed job data and lock on fresh install.
UmbracoBuilder.Services.cs	Registers DistributedJobService.
UmbracoBuilder.Repositories.cs	Registers DistributedJobRepository.
WebhookLoggingCleanup.cs	Converted to distributed job pattern.
WebhookFiring.cs	Converted to distributed job pattern.
TemporaryFileCleanupJob.cs	Converted to distributed job pattern.
ScheduledPublishingJob.cs	Converted to distributed job pattern.
OpenIddictCleanupJob.cs	Converted to distributed job pattern.
LongRunningOperationsCleanupJob.cs	Converted to distributed job pattern.
LogScrubberJob.cs	Converted to distributed job pattern.
HealthCheckNotifierJob.cs	Converted to distributed job pattern, removes dynamic delay logic.
ContentVersionCleanupJob.cs	Converted to distributed job pattern.
CacheInstructionsPruningJob.cs	Converted to distributed job pattern.
IDistributedBackgroundJob.cs	Defines distributed job contract.
DistributedBackgroundJobHostedService.cs	Timer-driven executor for distributed jobs.
Constants-Locks.cs	Adds DistributedJobs lock constant.
Constants-DatabaseSchema.cs	Adds DistributedJob table constant.
UmbracoBuilder.Configuration.cs	Registers DistributedJobSettings options.
Constants-Configuration.cs	Adds configuration key for distributed jobs.
DistributedJobSettings.cs	Adds configurable polling period and startup delay.
UmbracoBuilderAuthExtensions.cs	Registers OpenIddictCleanupJob as distributed job.

[nikolajlauridsen]

Overall looks really cool. I have two concerns

One about responsibilities, I've written more about that in the Repository file :D

The other thing that comes to mind is that I see you add the jobs in the database in the data creator, which begs the question, if I, as an implementor, want to add my own job, do I also manually need to create a migration which adds this row the the table? I would ideally like to avoid people having to do this, so can we instead do this automatically on boot? I mention this in the UpdateAllChangedAsync as well

[nikolajlauridsen]

Looks good now, only have one minor naming nitpick