Ctrlk
GitHub
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Runtime Administration and Graceful Drain

Release-backed operator guide for pausing, resuming, draining, and checking an Elsa 3.8.1 workflow runtime.

Use runtime administration when you need to stop new workflow work on a host without immediately terminating the process. Elsa exposes status, pause, resume, and force-drain endpoints through the workflow runtime API. These operations control runtime admission; they do not change the state of individual workflow definitions or instances unless a force-drain has to interrupt active execution cycles. The admission boundary is implemented by each ingress adapter, so a paused runtime flag is not a promise that every queued or scheduled dispatch has stopped.

Choose the operation

Situation
Operation
Result

Inspect whether the host accepts work

GET /admin/workflow-runtime/status

Returns runtime state, ingress-source state, and active execution-cycle count

Temporarily stop new work

POST /admin/workflow-runtime/pause

Sets an administrative pause; existing work can finish

Return a paused host to normal operation

POST /admin/workflow-runtime/resume

Clears the administrative pause, unless the runtime is draining

Escalate an unsafe or stuck shutdown

POST /admin/workflow-runtime/force-drain

Immediately force-cancels active execution cycles and leaves the runtime draining

The route prefix is relative to the Elsa API base configured by your host. The release endpoints use the workflows/runtime:control permission for pause, resume, and force-drain. Status requires workflows/runtime:view. See Elsa API Permissions for role design.

Configure graceful shutdown

The workflow runtime feature registers graceful-shutdown behavior. Configure it through ConfigureGracefulShutdown when the default timings or pause persistence policy do not fit your deployment:

using Elsa.Extensions;
using Elsa.Workflows.Runtime.Options;

builder.Services.AddElsa(elsa =>
{
    elsa.UseWorkflowRuntime(runtime =>
    {
        runtime.ConfigureGracefulShutdown(options =>
        {
            options.DrainDeadline = TimeSpan.FromSeconds(30);
            options.IngressPauseTimeout = TimeSpan.FromSeconds(5);
            options.PausePersistence = PausePersistencePolicy.SessionScoped;
        });
    });
});

The release defaults are a 30-second overall drain deadline, a 5-second per-ingress-source pause timeout, a 10,000-item paused stimulus-queue depth, Buffer overflow behavior, session-scoped pause state, and a cap of 100 force-cancelled instance IDs in the force-drain response. Elsa clamps the effective drain deadline to the host shutdown timeout minus a small safety margin.

DrainDeadline and IngressPauseTimeout must be greater than zero. A source may provide its own positive pause timeout; the orchestrator still caps it at the remaining overall drain budget.

Inspect runtime status

The response contains the composite quiescence state, each registered ingress source, and the number of active execution cycles:

reason is a flags value. None means the runtime is accepting new work; AdministrativePause means an operator pause is active; Drain means the runtime is draining. The flags can coexist. A drain is forward-only for the current runtime generation and is cleared only when a new generation starts.

Pause and resume safely

Pause is idempotent. A repeated pause returns the current post-request state without publishing another effective-transition audit event. The optional reason is human-readable and is included in the status response.

While paused, HTTP-triggered workflow requests are rejected with 503 Service Unavailable and a reason-aware Retry-After header (60 seconds for an administrative pause, 5 seconds while draining). Other ingress adapters have their own admission and buffering behavior. Do not infer from the runtime status alone that scheduled, broker, or already queued internal work has stopped.

The paused stimulus-queue policy is Buffer by default. If you configure a maximum paused depth and choose Reject, new stimuli beyond the threshold are rejected instead. In the 3.8.1 Core source, the internal bookmark-queue processor does not itself consult IQuiescenceSignal, even though its diagnostic ingress-source entry reports Paused; treat queued bookmark processing as a separate operational concern and verify it before using pause as a hard freeze.

The readiness probe becomes degraded while the runtime is paused or draining, so a load balancer or orchestrator should stop routing new traffic to that host.

Resume clears only the administrative-pause flag:

If a drain is active, resume returns 409 Conflict with the code runtime-draining. Restart or reactivate the runtime to create a new generation; do not treat resume as a way to reverse a drain.

Persist a pause across reactivation

PausePersistencePolicy.SessionScoped is the default: a new runtime generation starts unpaused. Select AcrossReactivations when an operator pause must survive a host restart or shell reactivation:

This policy uses Elsa's IKeyValueStore. Configure a durable, shared provider for a multi-node deployment; the in-memory store cannot preserve the pause through a process exit or coordinate it across nodes. If no key-value store is registered, the release implementation tolerates the missing store and the configured policy does not persist anything, so verify the provider during deployment checks. The persisted pause is re-applied on each node during activation, and an explicit resume clears it. See Runtime coordination storage for provider selection.

Graceful drain and force-drain

When the host stops, Elsa's drain service runs a graceful drain. It marks the runtime as draining, pauses ingress sources in parallel, waits for active execution cycles until the effective deadline, and force-cancels remaining cycles if the deadline is exceeded. A source that times out or throws is reported as PauseFailed; if it supports force-stop, Elsa attempts that escalation within the remaining budget.

Use force-drain only as an operator escalation:

Force-drain uses a zero deadline. Elsa force-cancels active execution cycles, persists affected instances with SubStatus = Interrupted, and writes a WorkflowInterrupted execution-log entry for each affected instance. The response reports the overall result, phase durations, per-source outcomes, the true number of force-cancelled cycles, and up to the configured number of instance IDs.

Force-drain does not terminate the host process. The runtime remains in the Drain state until the next runtime generation. If a competing drain is already in progress and no prior outcome is available, the endpoint returns 409 Conflict with the code drain-in-progress; a repeated operator-force call can instead receive the cached outcome. Force-drain does not delete queued bookmark records.

Recover interrupted execution

Core 3.8.1 separates interruption recording from recovery. When a force-drain cancels an active execution cycle, Elsa waits briefly for the runner to settle and then conditionally marks the persisted instance as:

It also writes a WorkflowInterrupted execution-log record. The reason is OperatorForce for an operator force-drain, DeadlineBreach when a graceful drain exceeds its deadline, or PersistenceFailure when the interruption could not be recorded normally. If the instance row does not exist yet, Elsa writes a forensic log entry with the persistence-failure reason instead.

The conditional write protects real terminal outcomes. A naturally finished, faulted, or already user-cancelled instance is not turned back into an interrupted instance. A Finished + Cancelled row is promoted only when a short pre-cancel snapshot showed that the cancellation was caused by this drain. This prevents a late drain write from overwriting a completed workflow.

On the next shell activation, the startup recovery task scans only Running + Interrupted instances and requeues them immediately. It restores the tenant context before restarting a tenant-scoped instance. The scan is bounded by RuntimeOptions.RestartInterruptedWorkflowsBatchSize, which is 100 by default. A separate recurring recovery task looks for stale Running + executing instances using RuntimeOptions.InactivityThreshold, which defaults to five minutes; this liveness-based path is distinct from the startup scan.

You can tune these recovery settings with the runtime options:

For recovery after a process restart, configure durable workflow-instance, bookmark, and execution-log storage. In-memory stores cannot provide the records that the recovery scan needs after the process exits. The 3.8.1 release implements the conditional interruption write for EF Core, Dapper, MongoDB, and Elasticsearch; a custom workflow-instance store must implement IWorkflowInstanceStore.TryMarkInterruptedAsync with equivalent terminal-state protection. See Persistence: interrupted-workflow recovery for the provider matrix and verification checklist.

Treat WorkflowInterrupted records as an operational trail: they identify the interruption and its reason, but recovery still depends on the persisted workflow instance and its runtime state being available to restart it.

Verify readiness after an operation

The runtime readiness check creates a workflow client and reads the quiescence state. It is healthy only when the runtime is accepting new work; paused or draining is reported as degraded, and a client-creation failure is unhealthy. With the standard endpoint mapping, degraded and unhealthy readiness produce HTTP 503 while liveness remains a separate process check.

Use readiness to remove a host from traffic and liveness to decide whether an orchestrator should restart the process. For endpoint registration and the persistence and distributed-lock probes, see Readiness and Health Checks.

What Studio shows

The Elsa 3.8.1 Studio dashboard renders the backend dashboard's runtime status as a compact, read-only chip:

API status
Studio label
Visual meaning

AcceptingWork

Accepting work

Success

Paused

Paused

Warning

Draining

Draining

Informational

missing, unauthorized, unavailable, or unknown

Unavailable

Error

The dashboard status chip helps a process designer or operator understand why new work is not arriving when the backend dashboard data is available. An Unavailable chip can mean a missing dashboard feature, denied access, a backend failure, or an unknown status; it is not proof that the runtime itself is unhealthy. The chip does not replace the API permission check or provide the pause/resume/force-drain controls. Use the runtime-admin API from an authorized operational tool for state changes.

Operator checklist

  1. Call /admin/workflow-runtime/status and record reason, source states, and activeExecutionCycleCount.

  2. For planned maintenance, pause the runtime and, if your host maps Elsa's readiness check, confirm /health/ready returns 503 before changing infrastructure.

  3. Wait for active cycles to settle, or use force-drain only when interruption is acceptable and the interrupted-workflow recovery path is understood.

  4. After maintenance, resume a paused runtime and verify status plus readiness. A drained runtime needs a new generation.

  5. In a multi-node deployment, perform the check on every node and use shared durable coordination storage. Pause state is per runtime or shell; even with AcrossReactivations, persistence reapplies state on activation but does not broadcast a live pause to other nodes. A single node's status does not describe the cluster.

Release source

This page is grounded in Elsa Core release/3.8.1 and Elsa Studio release/3.8.1:

PreviousReadiness and Health Checks
NextDistributed Tracing

Last updated