# The pipeline that reported success while doing nothing

**An identification outage that lasted thirteen days, three failed fixes, and
what we changed about how we watch things.**

---

## The symptom

A customer's dashboard said identification had stopped *yesterday*. The truth
was **thirteen days**, and every system involved was reporting healthy.

```
tracking_events         8,191 in 24h, most recent seconds ago   ← pixel fine
liveintent_resolutions  ~1,200/day, most recent minutes ago     ← vendor fine
signal_account_visitors last created 2026-08-29 13:43           ← nobody new
unified_profiles        last created 2026-08-29 13:43           ← nobody new
```

Traffic arriving, resolutions arriving, and not one new identified person in
thirteen days.

## Three things that were all green

**The schedule.** The hourly job was not paused, was firing on time, had
nothing stuck, and showed ten recent actions. Looking at the scheduler told you
everything was fine.

**The `updated_at` column.** Rows were being written every hour, so any monitor
watching row freshness saw a live system. Those writes were a *different* job —
an hourly intent recompute touching existing rows. `updated_at` moved;
`created_at` had not moved in thirteen days.

**The workflow status.** In an earlier round of the same outage, the workflow
caught its own exception and returned a result object with
`status='failed'`. Temporal saw a **COMPLETED** workflow. A caught failure
returned as a value is a completion.

## The actual cause

Every hourly run failed in two seconds. The error surfaced as a circuit-breaker
trip, which was itself a mask — the first attempt raised the real error, the
second returned `CircuitOpenError`, and only the second was reported:

```
CheckViolationError: new row for relation "liveintent_ingestion_runs"
violates check constraint "liveintent_ingestion_runs_status_check"
DETAIL: Failing row contains (…, no_data, 0, 0,
        discovery found no dated partitions under the configured prefix).
```

The job had a branch for "the vendor delivered nothing", which recorded
`status='no_data'`. The CHECK constraint permitted `running | success | partial
| failed`. Every one of those writes was rejected, the retries exhausted, the
breaker opened, **and the workflow died on its bookkeeping call — before the
step that turns resolutions into people.**

`SELECT DISTINCT status` returned `{success, failed}`. The value `no_data` had
never once landed.

## Each fix planted the next one

| | What broke | Duration |
|---|---|---|
| 1 | Discovery hit an S3 `AccessDenied`; the workflow caught it and returned a value, so the scheduler saw COMPLETED | 3d 7h |
| 2 | The fix for #1 recorded the failure with an empty date; the recorder died on `date.fromisoformat('')` and tripped the breaker | 4 days |
| 3 | The fix for #2 added the `no_data` branch, and nobody widened the constraint | **13 days** |

All three were in the same recorder. The common factor was not carelessness:
**the status vocabulary had two homes** — string literals in the job, a CHECK
constraint in SQL — and no test read both.

## The design error underneath

> **Recording what happened must not be able to prevent what happens.**

The write that logged the run was awaited on the critical path, ahead of the
step that produced the product's entire value. A logging concern was a hard
dependency of the business. It took identification down twice before anyone
named it.

## What we changed

- **One vocabulary, one home.** The statuses are declared once in code, and a
  test asserts that set equals the CHECK constraint as written in the
  migration. Adding a status without a migration now fails a test instead of
  production.
- **The recorder cannot fail the run.** It logs loudly and returns. Safety
  comes from the *absence* of the row: a monitor watches run freshness with a
  three-hour budget and goes stale precisely when this breaks.
- **The banner measures the output, not the job.** It had read "when did our
  ingest last run", which is a different fact from "when did we last identify
  someone" — and a crashed run refreshed it, which is how thirteen days
  displayed as one. It now reads the last actual identification, on a threshold
  derived from the measured gap distribution (p50 1.0h, p90 2.0h) rather than
  chosen.

## What to take from it

**Ask what would make the number move.** A monitor that cannot go red is
decoration. Before trusting one, work out which specific failure it would catch
— and check that a *failure* cannot refresh the thing it measures. Our banner,
our `updated_at`, and our workflow status all had that defect in different
forms.

**Distinguish "the job ran" from "the job did something."** They are different
questions with different right answers, and fusing them lets a healthy job
speak for a dead pipeline.

**A test that reads one side of a contract is not a test of the contract.** The
constraint and the code each said something true about themselves. Nothing read
both.