> For the complete documentation index, see [llms.txt](https://docs.elsaworkflows.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.elsaworkflows.io/guides/troubleshooting.md).

# Troubleshooting

Symptom-first entry point for diagnosing Elsa workflow, host, and Studio issues.

Use this guide to choose a first diagnostic signal and find its focused runbook. An individual workflow's state, host readiness, and process-level diagnostics answer different questions.

For short answers and links to the right task-specific guide, see the [Frequently Asked Questions](/guides/faq.md).

## Choose the first diagnostic

Use workflow-instance records for one workflow, readiness checks for the Elsa probes configured on the host, and logs or traces for process-level behavior. A healthy readiness result does not prove that every external dependency is reachable or that a particular business workflow will complete.

| Symptom                                                                     | First check and next guide                                                                                                                                                                                                                                                                                                                                 |
| --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Instance appears idle or stopped                                            | Read status, sub-status, and the latest journal entry in [Investigate a Workflow Instance](/operate/workflow-state-and-journal.md). A suspended instance may be waiting; if an expected event or timer is missing, check [triggers](/guides/running-workflows/using-a-trigger.md) or [timers](/guides/running-workflows/timer-and-scheduled-workflows.md). |
| Expected event did not resume an instance or scheduled trigger did not fire | Check the matching bookmark or trigger and the scheduler path: [Using a Trigger](/guides/running-workflows/using-a-trigger.md), [Timer and Scheduled Workflows](/guides/running-workflows/timer-and-scheduled-workflows.md), [Quartz Scheduling](/guides/running-workflows/quartz-scheduling.md).                                                          |
| Instance faulted or journal shows an activity failure                       | Start with any incident and the journal; inspect the activity execution record if the host captured it. See [Investigate a Workflow Instance](/operate/workflow-state-and-journal.md), then [Structured Logs](/operate/structured-logs.md) or [Distributed Tracing](/operate/distributed-tracing.md).                                                      |
| New work is not accepted across workflows                                   | Check the Elsa readiness probes registered by the host in [Readiness and Health Checks](/operate/readiness-and-health-checks.md). For a paused or draining runtime, see [Runtime Administration and Graceful Drain](/operate/runtime-administration.md).                                                                                                   |
| You need backend evidence                                                   | Use [Structured Logs](/operate/structured-logs.md) for `ILogger` events, [Console Logs](/operate/console-logs.md) for managed `Console.Out` / `Console.Error`, or configured [Distributed Tracing](/operate/distributed-tracing.md) for workflow/activity spans. These complement, but do not replace, the journal.                                        |
| A diagnostics page is unavailable in Studio                                 | Confirm the backend feature and matching Studio module are enabled, then check the API URL and required access. Start with [Structured Logs](/operate/structured-logs.md), [Console Logs](/operate/console-logs.md), or [Distributed Tracing](/operate/distributed-tracing.md), as applicable.                                                             |

Use the detailed symptom playbooks below for checks specific to triggers, bookmarks, scheduling, and runtime configuration. Avoid assuming an endpoint path or persistence schema: both depend on the host and provider configuration.

***

## Symptom Playbooks

### Workflows Don't Start

**Symptoms:**

* Creating a workflow instance via API returns success, but no execution occurs
* Workflow appears in "Pending" or never changes to "Running"
* No logs indicating workflow execution

**Step-by-Step Diagnosis:**

1. **Check workflow definition is published:**

   ```bash
   curl http://localhost:5000/elsa/api/workflow-definitions/<definition-id>
   ```

   Verify `isPublished: true` in the response.
2. **Check for trigger conditions:**
   * If the workflow uses a trigger (HTTP, Timer, etc.), ensure the trigger is configured correctly
   * For HTTP triggers, verify the endpoint is listening: check logs for `Mapped endpoint` messages
3. **Verify workflow runtime is started:** Look for startup logs:

   ```
   [INF] Elsa workflow runtime started
   ```
4. **Check for exceptions in logs:** Increase log level to Debug for Elsa namespaces (see [Logging Configuration](https://github.com/elsa-workflows/elsa-gitbook/tree/main/guides/troubleshooting/examples/logging-config.md))
5. **Verify database connectivity:**

   ```sql
   -- Check if workflow instance was persisted
   SELECT * FROM elsa.workflow_instances ORDER BY created_at DESC LIMIT 5;
   ```

**Common Fixes:**

* **Unpublished definition:** Publish the workflow via Elsa Studio or API
* **Missing trigger registration:** Ensure `UseHttp()`, `UseScheduling()`, or other trigger modules are configured in Program.cs
* **Startup order issues:** Verify Elsa services are registered before `app.Run()`

**Code Reference:**

* `src/modules/Elsa.Workflows.Core/Middleware/Activities/DefaultActivityInvokerMiddleware.cs` - Entry point for activity execution. This middleware orchestrates the execution of activities and is where the workflow execution pipeline begins.

***

### Workflows Don't Resume (Bookmark Issues)

**Symptoms:**

* Workflow is suspended at a blocking activity but never resumes
* External events (HTTP callbacks, signals) don't wake the workflow
* Logs show "Bookmark not found" errors

**Step-by-Step Diagnosis:**

1. **Verify the bookmark exists:**

   ```sql
   SELECT * FROM elsa.bookmarks 
   WHERE workflow_instance_id = '<instance-id>';
   ```
2. **Check bookmark hash matches stimulus:** Bookmarks are matched using a deterministic hash based on activity type and stimulus data. Mismatched stimulus (e.g., different payload, correlation ID) won't find the bookmark.
3. **Verify distributed lock provider is configured:** The `WorkflowResumer` acquires a distributed lock before resuming. Without a proper lock provider in multi-node setups, resume operations may fail silently.
4. **Check for lock acquisition failures:** Enable Debug logging for `Elsa.Workflows.Runtime`:

   ```
   [DBG] Attempting to acquire lock for workflow instance <id>
   [INF] Lock acquisition failed; resume already in progress
   ```
5. **Verify stimulus hashing:** The bookmark filter uses a hash of (activity type + stimulus payload). If your resume payload doesn't exactly match what was stored when the bookmark was created, the bookmark won't be found.

**Common Fixes:**

* **Configure distributed lock provider:**

  ```csharp
  elsa.UseWorkflowRuntime(runtime =>
  {
      runtime.UseDistributedRuntime();
      runtime.DistributedLockProvider = sp => 
          new RedisDistributedSynchronizationProvider(/*...*/);
  });
  ```
* **Match stimulus exactly:** Ensure the resume payload matches the bookmark's expected stimulus data
* **Check lock timeouts:** Increase lock acquisition timeout for high-latency environments

**Code References:**

* `src/modules/Elsa.Workflows.Runtime/Services/WorkflowResumer.cs` - Uses `IDistributedLockProvider` to serialize resume operations. The `ResumeWorkflowAsync` method acquires a lock on the workflow instance before loading and resuming, preventing concurrent modifications.
* `src/modules/Elsa.Workflows.Core/Contexts/ActivityExecutionContext.cs` - Contains `CreateBookmark()` which generates bookmarks with deterministic hashes. Bookmarks auto-suspend the workflow when created.

***

### Duplicate Resumes (Concurrency Issues)

**Symptoms:**

* Same workflow step executes multiple times
* Duplicate side effects (emails sent twice, records created twice)
* Logs show multiple nodes processing the same resume request

**Step-by-Step Diagnosis:**

1. **Verify distributed locking is enabled:** Check for `UseDistributedRuntime()` in configuration:

   ```csharp
   elsa.UseWorkflowRuntime(runtime => runtime.UseDistributedRuntime());
   ```
2. **Check lock provider connectivity:**

   ```bash
   # For Redis
   redis-cli -h redis-host PING

   # For PostgreSQL
   psql -h postgres-host -U user -c "SELECT 1"
   ```
3. **Review logs for concurrent lock acquisitions:** In a correctly configured cluster, only one node should successfully acquire the lock:

   ```
   [INF] Acquired distributed lock for workflow <id>
   [INF] Lock acquisition failed for workflow <id> - already in progress
   ```
4. **Verify Quartz clustering (for scheduled tasks):** Check that all nodes share the same Quartz database and clustering is enabled:

   ```
   quartz.jobStore.clustered = true
   ```

**Common Fixes:**

* **Single scheduler pattern:** In multi-node deployments, either:
  * Use Quartz clustering with a shared database
  * Designate a single scheduler node
* **Configure distributed locking:**

  ```csharp
  runtime.DistributedLockProvider = sp =>
      new PostgresDistributedSynchronizationProvider(connectionString);
  ```
* **Ensure idempotent activities:** Design activities to be safe for replay (check before insert, use idempotency keys)

**Code References:**

* `src/modules/Elsa.Workflows.Runtime/Services/WorkflowResumer.cs` - Implements distributed locking pattern using `IDistributedLockProvider` from Medallion.Threading library.
* `src/modules/Elsa.Workflows.Core/Middleware/Activities/DefaultActivityInvokerMiddleware.cs` - Handles "bookmark burning" via the `AutoBurn` option. When a bookmark is consumed successfully, it's deleted to prevent re-use.

***

### Timers Fire Multiple Times or Not at All

**Symptoms:**

* Scheduled workflows run multiple times per trigger
* Timers never fire despite being scheduled
* Cron schedules execute at wrong times or skip executions

**Step-by-Step Diagnosis:**

1. **Check Quartz clustering configuration:**

   ```sql
   -- Verify Quartz sees all nodes
   SELECT * FROM qrtz_scheduler_state;
   ```

   All nodes should appear with recent `last_checkin_time`.
2. **Verify scheduled bookmarks exist:**

   ```sql
   -- Check for scheduled triggers
   SELECT * FROM qrtz_triggers 
   WHERE trigger_group = 'Elsa.Scheduling'
   ORDER BY next_fire_time;
   ```
3. **Check clock synchronization:** All nodes must use synchronized time (NTP). Clock drift causes misfires or duplicates.

   ```bash
   timedatectl status  # Check if NTP is active
   ```
4. **Review DefaultBookmarkScheduler behavior:** This service categorizes bookmarks by type (Timer, Delay, Cron) and schedules them via Quartz:
   * **Timer/Delay:** One-time execution at specific time
   * **Cron:** Recurring based on cron expression
5. **Check for misfires:**

   ```sql
   SELECT * FROM qrtz_triggers WHERE misfire_instr != 0;
   ```

**Common Fixes:**

* **Enable Quartz clustering:**

  ```csharp
  elsa.UseQuartz(quartz =>
  {
      quartz.UsePostgreSql(connectionString);
      // Clustering is enabled automatically
  });
  ```
* **Adjust misfire thresholds:**

  ```properties
  quartz.jobStore.misfireThreshold = 60000
  ```
* **Use consistent time zones:**

  ```yaml
  env:
    - name: TZ
      value: "UTC"
  ```

**Code References:**

* `src/modules/Elsa.Scheduling/Services/DefaultBookmarkScheduler.cs` - Categorizes and schedules Timer/Delay/Cron bookmarks. Uses Quartz to schedule `ResumeWorkflowTask` jobs for future execution.
* `src/modules/Elsa.Workflows.Runtime/Middleware/Activities/BackgroundActivityInvokerMiddleware.cs` - Handles background/job execution for long-running activities. Activities marked for background execution are dispatched to a job queue.

***

### Stuck/Running Workflows and Incident Handling

**Symptoms:**

* Workflows stay in "Running" state indefinitely
* Workflow appears stuck at a specific activity
* Incidents are created but not handled

**Step-by-Step Diagnosis:**

1. **Identify stuck workflows:**

   ```sql
   SELECT id, definition_id, status, sub_status, created_at, updated_at
   FROM elsa.workflow_instances
   WHERE status = 'Running'
   AND updated_at < NOW() - INTERVAL '1 hour';
   ```
2. **Inspect the workflow instance's incidents:** Open the instance in Elsa Studio or retrieve `GET /workflow-instances/<instance-id>` and inspect `workflowState.incidents`.
3. **Review the workflow journal and activity executions:** Check `GET /workflow-instances/<instance-id>/journal` or the filtered `POST` variant, then inspect the failing activity execution in Studio if you need per-activity state or retry metadata.
4. **Check for orphaned locks:** If a node crashes while holding a lock, the workflow may appear stuck:

   ```bash
   # Redis: Check for workflow locks
   redis-cli --scan --pattern "workflow:*"

   # These should auto-expire based on TTL
   ```
5. **Verify background jobs are processing:** For activities executed in background, check the job queue:

   ```sql
   SELECT * FROM elsa.workflow_execution_log
   WHERE workflow_instance_id = '<instance-id>'
   ORDER BY timestamp DESC;
   ```

**Common Fixes:**

* **Cancel stuck workflows:**

  ```bash
  curl -X POST http://localhost:5000/elsa/api/workflow-instances/<id>/cancel
  ```
* **Retry faulted activities:** Use `POST /alterations/workflows/retry` with one or more `workflowInstanceIds`, and optionally `activityIds`, after fixing the root cause.
* **Configure incident strategies:**

  ```csharp
  builder.Services.Configure<IncidentOptions>(options =>
  {
      options.DefaultIncidentStrategy = typeof(ContinueWithIncidentsStrategy);
  });
  ```
* **Implement graceful shutdown:** Ensure pods wait for in-progress workflows before terminating

**Code Reference:**

* `src/modules/Elsa.Workflows.Core/Middleware/Activities/DefaultActivityInvokerMiddleware.cs` - Manages activity lifecycle including fault handling. Exceptions during activity execution are caught and can trigger incident creation.

***

### High Database Load / Slow Persistence

**Symptoms:**

* Slow workflow execution times
* Database CPU/IO consistently high
* Timeout errors during workflow operations

**Step-by-Step Diagnosis:**

1. **Identify slow queries:**

   ```sql
   -- PostgreSQL: Enable query logging temporarily
   SET log_min_duration_statement = 100;  -- Log queries > 100ms
   ```
2. **Check table sizes:**

   ```sql
   SELECT relname, n_live_tup, n_dead_tup
   FROM pg_stat_user_tables
   WHERE schemaname = 'elsa'
   ORDER BY n_live_tup DESC;
   ```
3. **Check for missing indexes:** Common queries that need indexes:
   * `workflow_instances` by `status`, `definition_id`, `correlation_id`
   * `bookmarks` by `workflow_instance_id`, `hash`
4. **Review connection pool utilization:**

   ```sql
   SELECT count(*) FROM pg_stat_activity 
   WHERE datname = 'elsa';
   ```
5. **Check for long-running transactions:**

   ```sql
   SELECT pid, now() - pg_stat_activity.query_start AS duration, query
   FROM pg_stat_activity
   WHERE (now() - pg_stat_activity.query_start) > INTERVAL '5 minutes';
   ```

**Common Fixes:**

* **Increase connection pool size:**

  ```
  Server=host;Database=elsa;MaxPoolSize=200
  ```
* **Add indexes for common queries:**

  ```sql
  CREATE INDEX idx_bookmarks_hash ON elsa.bookmarks(hash);
  CREATE INDEX idx_instances_status ON elsa.workflow_instances(status);
  ```
* **Implement data retention (see below)**
* **Use read replicas for queries:** Configure Elsa to use read replicas for workflow history queries

***

## Logging & Diagnostics

### Increasing Log Level

For troubleshooting, increase Elsa log verbosity. See [Logging Configuration Examples](https://github.com/elsa-workflows/elsa-gitbook/tree/main/guides/troubleshooting/examples/logging-config.md) for complete snippets.

**Quick Configuration (appsettings.json):**

```json
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Elsa": "Debug",
      "Elsa.Workflows.Runtime": "Debug",
      "Elsa.Scheduling": "Debug"
    }
  }
}
```

### Key Log Messages to Look For

| Scenario         | Log Pattern                                           | Level       |
| ---------------- | ----------------------------------------------------- | ----------- |
| Workflow started | `Starting workflow instance {Id}`                     | Information |
| Bookmark created | `Created bookmark {Hash} for activity {ActivityId}`   | Debug       |
| Lock acquired    | `Acquired distributed lock for workflow {Id}`         | Debug       |
| Lock failed      | `Lock acquisition failed; resume already in progress` | Information |
| Timer scheduled  | `Scheduled timer for bookmark {Id} at {Time}`         | Debug       |
| Activity faulted | `Activity {ActivityType} faulted: {Exception}`        | Error       |
| Bookmark burned  | `Burned bookmark {Hash}`                              | Debug       |

### Including Correlation/Workflow IDs in Logs

Enable structured logging with workflow context:

```csharp
Log.Logger = new LoggerConfiguration()
    .Enrich.FromLogContext()
    .Enrich.WithProperty("Application", "ElsaServer")
    .WriteTo.Console(new CompactJsonFormatter())
    .CreateLogger();
```

In workflows, the workflow instance ID and correlation ID are automatically added to the log scope when using Elsa's logging infrastructure.

***

## OpenTelemetry tracing and diagnostics

In `release/3.8.2`, core instrumentation emits baseline workflow and activity spans through the `Elsa.Workflows` activity source. The optional `Elsa.OpenTelemetry` extension adds a second span layer and error handlers. Separate Studio diagnostics modules provide structured logs, console logs, and OTLP collection.

**Quick Setup:**

```csharp
using Elsa.Extensions;
using Elsa.OpenTelemetry.Middleware;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddAspNetCoreInstrumentation();
        tracing.AddHttpClientInstrumentation();
        tracing.AddSource("Elsa.Workflows");
        tracing.AddOtlpExporter();
    });

builder.Services.AddElsa(elsa => elsa
    .UseWorkflows(workflows =>
    {
        workflows.WithDefaultWorkflowExecutionPipeline(pipeline =>
            pipeline.UseWorkflowExecutionTracing());
        workflows.WithDefaultActivityExecutionPipeline(pipeline =>
            pipeline.UseActivityExecutionTracing());
    })
    .UseOpenTelemetry());
```

**What's Traced:**

* Workflow execution spans
* Activity execution spans
* Workflow and activity status events, incidents, and identity tags

For the complete release-backed setup and error-handler behavior, see [OpenTelemetry workflow and activity tracing](/extensibility/opentelemetry-tracing.md).

**Code Reference:**

* `src/modules/Elsa.Workflows.Core/Telemetry/WorkflowInstrumentation.cs` - Defines core workflow/activity spans and metrics.
* `elsa-extensions/src/modules/diagnostics/Elsa.OpenTelemetry/*` - Defines the optional additional workflow/activity tracing middleware and error-span handlers.
* `src/modules/Elsa.Diagnostics.OpenTelemetry/*` - Provides OTLP ingestion, query APIs, and the Studio diagnostics collector surface.

***

## Data Retention & Cleanup

### Workflow Instance Retention

Old workflow instances consume database space and slow queries. Configure the separate retention extension for automatic cleanup:

```csharp
elsa.UseRetention(retention =>
{
    retention.SweepInterval = TimeSpan.FromHours(1);
    retention.AddDeletePolicy("Finished workflows after 30 days", sp =>
    {
        var clock = sp.GetRequiredService<ISystemClock>();
        return new RetentionWorkflowInstanceFilter
        {
            WorkflowStatus = WorkflowStatus.Finished,
            TimestampFilters = new[]
            {
                new TimestampFilter
                {
                    Column = nameof(WorkflowInstance.FinishedAt),
                    Operator = TimestampFilterOperator.LessThanOrEqual,
                    Timestamp = clock.UtcNow.AddDays(-30)
                }
            }
        };
    });
});
```

See [Retention](/optimize/retention.md) before enabling deletion in a production environment.

### Manual cleanup

Avoid deleting workflow rows directly with SQL. Elsa's retention job also cleans related bookmarks and runtime records, while table names, status storage, and transaction behavior vary by persistence provider. Use a narrow retention policy or the provider's documented maintenance procedure instead. See [Retention](/optimize/retention.md) for the supported filter and cleanup behavior.

### Quartz Cleanup

Quartz manages its own cleanup, but old triggers can accumulate:

```sql
-- Remove old fired triggers
DELETE FROM qrtz_fired_triggers
WHERE fired_time < EXTRACT(EPOCH FROM NOW() - INTERVAL '7 days') * 1000;
```

***

## Production Checklist

Before going to production, verify:

### Infrastructure

* [ ] Database connection pooling configured (50-200 connections)
* [ ] Database backups scheduled and tested
* [ ] Redis/lock provider accessible from all nodes
* [ ] Message broker (RabbitMQ/Azure Service Bus) configured for cache invalidation

### Elsa Configuration

* [ ] `UseDistributedRuntime()` enabled
* [ ] Distributed lock provider configured
* [ ] Quartz clustering enabled (if multi-node)
* [ ] Workflow instance retention configured
* [ ] Health checks exposed (`/health/ready`, `/health/live`)

### Observability

* [ ] Structured logging enabled (JSON format recommended)
* [ ] Log aggregation configured (ELK, Loki, CloudWatch)
* [ ] OpenTelemetry tracing enabled (optional but recommended)
* [ ] Alerts configured for:
  * High error rates
  * Long-running workflows
  * Lock acquisition failures
  * Database connection issues

### Security

* [ ] API authentication configured
* [ ] HTTPS enforced for resume URLs
* [ ] Database credentials in secrets (not config files)
* [ ] Network policies restricting access to Elsa services

***

## Escalation Tips

When troubleshooting doesn't resolve the issue:

1. **Gather Information:**
   * Elsa version (`dotnet list package | grep Elsa`)
   * .NET version (`dotnet --version`)
   * Database type and version
   * Logs from the time of the issue (with Debug level enabled)
   * Workflow definition JSON (if applicable)
2. **Reproduce in Isolation:**
   * Create a minimal workflow that demonstrates the issue
   * Test on a single node to rule out clustering issues
3. **Search Existing Issues:**
   * [Elsa GitHub Issues](https://github.com/elsa-workflows/elsa-core/issues)
   * [GitHub Discussions](https://github.com/elsa-workflows/elsa-core/discussions)
4. **Create a Bug Report:** Include:
   * Steps to reproduce
   * Expected vs actual behavior
   * Relevant logs
   * Minimal reproduction project (if possible)
5. **Community Resources:**
   * Discord community
   * Stack Overflow (tag: `elsa-workflows`)

***

## Related Documentation

* [Clustering Guide](/guides/clustering.md) - Multi-node deployment configuration
* [Distributed Hosting](/hosting/distributed-hosting.md) - Core distributed concepts
* [Incidents Configuration](/operate/incidents/configuration.md) - Incident handling setup
* [Retention](/optimize/retention.md) - Data retention policies
* [Logging Framework](/features/logging-framework.md) - Activity logging

## Supplementary Files

* [Logging Configuration Examples](https://github.com/elsa-workflows/elsa-gitbook/tree/main/guides/troubleshooting/examples/logging-config.md)
* [Troubleshooting Checklist](https://github.com/elsa-workflows/elsa-gitbook/tree/main/guides/troubleshooting/examples/checklist.md)
* [Source File References](https://github.com/elsa-workflows/elsa-gitbook/tree/main/guides/troubleshooting/README-REFERENCES.md)

***

**Last Updated:** 2025-11-25