# INC-1423: Checkout latency and webhook delays

Fictional public demo record

| Field | Value |
| --- | --- |
| Severity | SEV-1 |
| Status | Resolved |
| Service | Checkout API |
| Regions | EU West, US East |
| Impact window | 2026-06-18 09:00-10:24 UTC |
| Incident commander | Ivo Nuuskmõmmik |
| Responders | Andres Kask, Edward Green |
| Review owner | Timo Maasik |
| Submitted | 2026-06-19 15:00 UTC |

## Executive summary

At 09:00 UTC on 2026-06-18, checkout latency increased in EU West and US East while outbound order webhooks accumulated behind the checkout worker fleet. A deployment completed eight minutes earlier changed the webhook batch size from 50 to 500. The larger batches held worker leases longer than the configured headroom allowed. Timed-out deliveries retried as new large batches, which amplified queue depth and left fewer workers available for synchronous checkout work.

Responders declared a SEV-1 at 09:05 UTC, correlated the queue growth with the deployment at 09:18 UTC, and approved a rollback at 09:31 UTC. Worker saturation began falling at 09:42 UTC. Checkout latency returned below the recovery threshold at 09:58 UTC, webhook backlog cleared at 10:12 UTC, and the three affected fictional customers confirmed recovery by 10:24 UTC. The incident was resolved at 10:31 UTC.

The accepted root cause is the unbounded webhook batch-size increase combined with retry behavior that requeued whole timed-out batches. The release passed existing functional checks because staging traffic did not exercise production-scale queue depth and no guardrail compared batch size with worker lease headroom.

## Impact

From 09:00 UTC until customer recovery was confirmed at 10:24 UTC:

- Checkout API p95 latency peaked at 4.8 seconds in EU West and 4.2 seconds in US East.
- Some checkout requests timed out, while successful orders could receive confirmations several minutes late.
- Outbound order webhooks accumulated to a peak queue depth of 184,200.
- Meridian Outfitters, Lantern Home, and Acorn Supply are fictional customer records used in this public demo. Their support reports were linked to the incident and recovery was confirmed.
- No evidence showed payment capture loss, duplicate charges, or data corruption.

Customer impact was reviewed and marked confirmed. The impact narrative is limited to the stored Grafana, Kubernetes, GitHub, timeline, and support evidence described below. Intercom was unavailable during the investigation, so responders recorded that source gap instead of assuming completeness.

## Chronology

All times are fixed UTC on 2026-06-18.

| Time | Elapsed | Event |
| --- | ---: | --- |
| 09:00 | +00m | Grafana alert fired after Checkout API p95 exceeded 2 seconds and webhook queue depth crossed 40,000. |
| 09:05 | +05m | Ivo Nuuskmõmmik declared INC-1423 as SEV-1 and took incident command. |
| 09:11 | +11m | Scope was confirmed across EU West and US East. Customer impact moved from unknown to confirmed. |
| 09:18 | +18m | GitHub deployment `deploy-2026-06-18.1` was correlated with the first sustained queue rise. |
| 09:24 | +24m | Fictional customer reports from Meridian Outfitters and Lantern Home were linked to delayed order confirmations. |
| 09:31 | +31m | Responders approved rollback of the webhook batch-size change. The decision favored the reversible mitigation over adding workers while retries were still amplifying load. |
| 09:42 | +42m | Kubernetes worker saturation fell below 80 percent and queue growth stopped. |
| 09:58 | +58m | Checkout p95 stayed below 900 milliseconds for ten minutes. |
| 10:12 | +72m | Webhook queue depth returned below 2,000 and retry volume normalized. |
| 10:24 | +84m | All three fictional affected customers confirmed order confirmations had recovered. |
| 10:31 | +91m | Incident resolved. Monitoring continued through 11:00 UTC. |
| 11:08 | +128m | Incident record moved to review with evidence and actions preserved. |

## Detection and diagnosis

The investigation asked: "Did the 08:52 UTC checkout deployment increase webhook batch size enough to saturate workers, amplify retries, and cause checkout latency?"

The local investigation completed checks against:

- Grafana: checkout p95, worker saturation, retry rate, and webhook queue depth for 08:40-10:20 UTC.
- Kubernetes: worker readiness, CPU saturation, restarts, lease duration, and horizontal scaling events for the Checkout API namespace.
- GitHub: the deployment commit and configuration diff for `deploy-2026-06-18.1`.
- Intercom: unavailable because the fictional connector was not configured. Customer reports were assessed from the stored incident records instead.

Stored artifact lineage:

- `artifact:demo:v1:INC-1423:grafana-checkout` with a content-addressed SHA-256 digest.
- `artifact:demo:v1:INC-1423:kubernetes-workers` with a content-addressed SHA-256 digest.
- `artifact:demo:v1:INC-1423:github-deployment` with a content-addressed SHA-256 digest.
- `artifact:demo:v1:INC-1423:intercom-readiness` records the unavailable-source result and its reason.

The Grafana and Kubernetes checks support the deployment correlation. The GitHub diff confirms the batch-size change from 50 to 500. A negative finding records that pod restarts did not precede the latency increase. A gap finding records the unavailable Intercom source. These findings were reviewed before the root cause was accepted.

## Root cause

The 08:52 UTC deployment increased outbound webhook batch size from 50 to 500 without adjusting worker lease duration or concurrency limits. Large batches held worker leases long enough to exceed request and delivery timeouts. The retry path requeued the entire timed-out batch, so each failure created more queued work. Shared worker capacity shifted away from synchronous checkout requests, increasing checkout latency while the webhook backlog grew.

The root cause was accepted with high confidence because three independent checks align:

1. GitHub confirms the only relevant deployment change was the tenfold batch-size increase.
2. Grafana shows queue depth and retry rate rising immediately after the deployment and falling after rollback.
3. Kubernetes shows worker lease saturation during the same window without a preceding restart or node-capacity event.

Rollback reversed the configuration change and recovery followed without another infrastructure change. This temporal sequence supports causality while the negative restart finding reduces a competing explanation.

## Contributing factors

- Staging load tests used less than 10 percent of production peak webhook volume.
- The release check validated delivery correctness but did not measure worker lease headroom.
- Retry behavior requeued whole batches instead of only failed deliveries.
- Checkout and webhook processing shared a worker capacity pool.
- The queue-depth alert fired after synchronous checkout latency had already started rising.
- The customer-support connector was unavailable to the investigation, which made impact confirmation more manual.

## What went well

- The incident was declared five minutes after the first alert.
- Ownership remained explicit: Ivo Nuuskmõmmik led command, Andres Kask owned rollback, and Edward Green owned customer recovery checks.
- Responders preserved the deployment, metrics, workload, decision, and customer evidence in the incident record as work happened.
- The rollback was reversible, approved with a written decision, and executed before speculative scaling changes.
- Recovery required no data repair and no evidence indicated duplicate charges or lost payment captures.
- The postmortem carries the accepted findings and evidence references from the investigation instead of reconstructing them later.

## What made response harder

- The first alert grouped checkout latency and webhook queue symptoms without naming their shared worker pool.
- There was no dashboard view comparing batch size, lease duration, and available concurrency.
- Whole-batch retries made queue depth grow nonlinearly, which complicated early recovery estimates.
- The unavailable support connector prevented a complete automated customer-report search.
- The deployment checklist did not identify webhook batch size as a capacity-sensitive setting.

## Detection gap

Tellagen had alerts for checkout latency and absolute webhook queue depth, but no alert measured queue growth rate against available worker lease headroom. The existing queue alert crossed its threshold after customer-facing latency had already degraded. There was also no release guard that rejected a batch-size increase when the load-test profile was below a defined fraction of production peak.

The prevention plan adds both controls: a pre-deployment batch-size headroom test and an alert on queue growth rate plus worker lease saturation.

## Actions

| Owner | Due | Status | Action |
| --- | --- | --- | --- |
| Andres Kask | 2026-06-18 | Completed | Roll back `deploy-2026-06-18.1` and restore webhook batch size to 50. |
| Edward Green | 2026-06-26 | Open | Change retries to requeue only failed webhook deliveries, with an idempotency regression test. |
| Ivo Nuuskmõmmik | 2026-06-27 | In progress | Add queue-growth and lease-headroom alerts for the shared checkout worker pool. |
| Timo Maasik | 2026-07-02 | Open | Raise staging webhook load to at least 35 percent of production peak and block batch-size changes without the capacity check. |
| Andres Kask | 2026-07-05 | Open | Separate synchronous checkout and outbound webhook concurrency budgets. |

## Evidence and review record

The submitted version includes the incident record, key timeline events, all four investigation artifacts, the accepted root-cause finding, supporting and negative findings, the source-gap finding, the prevention recommendation, and the owned actions. The reviewer accepted the root cause after checking the deployment diff, metric window, workload state, and rollback recovery sequence.

This document contains only fictional names, systems, customers, identifiers, and fixed timestamps for Tellagen's public demo. It contains no reusable credential, customer data, internal host, or live integration link.