Fail-Closed State: One Verb, One Writer, One Delivery

Introduction

A smart-lock platform is, underneath the UI, a pile of state machines: a stay is reserved or admitted, a key is issued or revoked, a membership is applying, active, past_due, scheduled-to-end, or ended. Every one of those states is also mirrored somewhere outside our database — in a payment provider, in a lock controller, in a messaging channel.

The failure mode that actually hurts is not a crash. It is divergence: our database says the membership ends at the end of the month, and the billing provider has never heard of it. Or the guest's browser says "payment complete", and the ledger disagrees. Or the same approval email goes out three times because a retry re-entered a handler that was not idempotent.

This post walks through the patterns we converged on while consolidating a membership lifecycle, a payment lane, and a notification pipeline into something we are willing to put in front of physical doors.

1. One verb per transition

The first problem we found was not a bug — it was arity. A membership could be ended from a self-cancel screen, from an admin force-cancel, from a rejected application, and from a retired plan. Four screens, four code paths, four slightly different definitions of "ended".

Four writers means four places to forget to stop billing, four places to forget to cancel downstream reservations, four places where the state table drifts.

The fix is boring and effective: collapse every path into one verb, and make the reason a parameter.

export type MembershipEndReason =
  | { kind: 'self_cancel'; effectiveAt: 'period_end' | 'month_boundary' }
  | { kind: 'force_cancel'; actorId: string; note: string }
  | { kind: 'application_rejected'; actorId: string }
  | { kind: 'plan_retired'; planId: string };
export async function endMembership(
  id: string,
  reason: MembershipEndReason,
): Promise<TransitionOutcome> {
  const current = await loadMembership(id);
  const next = resolveEndState(current.status, reason);
  if (!next) return { status: 'noop', code: 'MEMBERSHIP_ALREADY_ENDED' };
  await stopExternalBilling(current, reason);
  const applied = await casMembershipState(id, current.status, next);
  if (!applied) return { status: 'conflict', code: 'MEMBERSHIP_STATE_CHANGED', current: current.status };
  await cancelFutureMemberReservations(id, next.endsAt);
  return { status: 'applied', to: next.status };
}

Two details matter more than the shape of the function:

• stopExternalBilling runs before the local write. If the external world refuses, we have not yet lied in our own database.
• The write is a compare-and-set, not a blind update:

update memberships
   set status = $3,
       ends_at = $4,
       updated_at = now()
 where id = $1
   and status = $2
returning id;

If zero rows come back, someone else moved the row between our read and our write. That is a conflict, not a success.

Write the state table down, then guard it

We wrote the allowed transitions into an ADR as a literal table (from-state × event → to-state), then added a guard on who is allowed to write membership state at all. Application code calls verbs; nothing else touches the status column directly. Database-level rules make that enforceable rather than aspirational:

create or replace function guard_membership_status()
returns trigger language plpgsql as $$
begin
  if new.status is distinct from old.status
     and current_setting('app.membership_verb', true) is null then
    raise exception 'membership status must change through a membership verb';
  end if;
  return new;
end $$;

The point is not the specific mechanism. The point is that "only this module writes this column" should be checkable at runtime, not a convention in a code review.

2. Fail closed when the source of truth is unreachable

The sharpest lesson of the whole batch: a member asked to cancel while the payment provider was unavailable, and the old code happily scheduled an end date locally. The subscription kept billing. The database said "ending"; the provider said "active". Nobody noticed until the next invoice.

When a transition has an external leg, an unreachable dependency must produce a refusal, not a local-only success.

export async function requestSelfCancel(membershipId: string): Promise<TransitionOutcome> {
  const membership = await loadMembership(membershipId);
  if (membership.billing === 'stripe') {
    const schedule = await billing.scheduleCancelAtPeriodEnd(membership.subscriptionId)
      .catch(() => null);
    if (!schedule) {
      return { status: 'refused', code: 'BILLING_UNAVAILABLE' };
    }
    return endMembership(membershipId, { kind: 'self_cancel', effectiveAt: 'period_end' });
  }
  return endMembership(membershipId, { kind: 'self_cancel', effectiveAt: 'month_boundary' });
}

The same principle showed up in the admin list: an application still under review, or with a checkout in flight, must not be "force-cancelled" into a terminal state — because the money may land a second later. Refusing with a code (APPLICATION_UNDER_REVIEW, CHECKOUT_IN_PROGRESS) is a better outcome than a cleanup job nobody wrote.

Rule of thumb: if a transition cannot be completed end-to-end, do not complete half of it. A refusal is recoverable; a split brain is not.

3. Outcomes, not English error strings

Stale clicks are normal. An admin opens the member list, goes to lunch, comes back, and approves an application that someone else already rejected. The old behaviour was throw new Error('membership already processed') — unlocalizable, untestable, and indistinguishable from a real fault.

Model the result as data:

export type OutcomeCode =
  | 'MEMBERSHIP_ALREADY_ENDED'
  | 'MEMBERSHIP_STATE_CHANGED'
  | 'APPLICATION_UNDER_REVIEW'
  | 'CHECKOUT_IN_PROGRESS'
  | 'BILLING_UNAVAILABLE';
export type TransitionOutcome =
  | { status: 'applied'; to: MembershipState }
  | { status: 'noop'; code: OutcomeCode }
  | { status: 'conflict'; code: OutcomeCode; current: MembershipState }
  | { status: 'refused'; code: OutcomeCode };

Now the UI can decide: noop means refresh quietly, conflict means reload and show what actually happened, refused means show a localized reason and keep the button enabled. Tests assert on codes instead of substrings. And when the SDK surface is typed this way, a caller that forgets to handle refused fails type-checking rather than production.

4. Idempotency at every external boundary

Anything that touches money or a lock gets the same treatment: a stable key, a recorded purpose, and a dedupe table.

Creating charges through one contract. Scattered paymentIntents.create calls produced charges nobody could attribute afterwards. One factory, one shape:

export type PaymentPurpose =
  | 'walk_in'
  | 'reservation_balance'
  | 'membership_initial'
  | 'membership_renewal';
export function buildIdempotencyKey(input: {
  purpose: PaymentPurpose;
  subjectId: string;
  attempt: number;
}): string {
  return `${input.purpose}:${input.subjectId}:${input.attempt}`;
}
export async function createIntent(input: CreateIntentInput) {
  return stripe.paymentIntents.create(
    { amount: input.amount, currency: input.currency, metadata: { purpose: input.purpose, subject_id: input.subjectId } },
    { idempotencyKey: buildIdempotencyKey(input) },
  );
}

The metadata.purpose stamp is what later lets a refund lane, a reconciliation report, or an audit answer "what was this charge for?" without guessing from the amount.

Consuming webhooks exactly once. Providers retry; that is the contract. So dedupe on the provider's event id, and separate "seen" from "processed":

create table payment_webhook_events (
  event_id    text primary key,
  event_type  text not null,
  received_at timestamptz not null default now(),
  processed_at timestamptz,
  attempts    int not null default 0,
  last_error  text
);

const claimed = await db.query(
  `insert into payment_webhook_events (event_id, event_type)
   values ($1, $2) on conflict (event_id) do nothing returning event_id`,
  [event.id, event.type],
);
if (claimed.rowCount === 0) return ok('duplicate');
try {
  await handle(event);
  await markProcessed(event.id);
} catch (err) {
  await recordFailure(event.id, err);
  if (isTransient(err)) throw err; // let the provider retry
  return ok('permanent-failure-recorded');
}

Note the distinction between transient and permanent failures. Re-throwing a transient error asks for a retry; swallowing a permanent one stops an infinite redelivery loop while leaving a row an operator can inspect.

5. The transactional outbox: notify once, in the right language

Sending email or chat messages inline with a state change is a trap. If the send succeeds and the transaction rolls back, you have notified a guest about something that did not happen. If the transaction commits and the send throws, the guest never hears about a key they now hold.

So every guest-facing message became one verb writing to an outbox, with direct sends fenced off:

create table notification_outbox (
  id          uuid primary key default gen_random_uuid(),
  dedupe_key  text not null unique,
  event_type  text not null,
  recipient   jsonb not null,
  locale      text not null,
  payload     jsonb not null,
  claimed_at  timestamptz,
  sent_at     timestamptz,
  attempts    int not null default 0,
  last_error  text
);

The dedupe_key (for example entry_key_issued:{stayId}) is what guarantees a key card is dispatched once even if the issuing path runs twice. The dispatcher claims rows so concurrent workers never double-send:

update notification_outbox
   set claimed_at = now(), attempts = attempts + 1
 where id in (
   select id from notification_outbox
    where sent_at is null
      and (claimed_at is null or claimed_at < now() - interval '5 minutes')
    order by id
    for update skip locked
    limit 50
 )
returning *;

And the part that is easy to get wrong: release the claim when the send did not happen. A flag set optimistically before an awaited send will silently suppress the retry forever.

for (const row of claimed) {
  try {
    await channel.send(row);
    await db.query('update notification_outbox set sent_at = now() where id = $1', [row.id]);
  } catch (err) {
    await db.query(
      'update notification_outbox set claimed_at = null, last_error = $2 where id = $1',
      [row.id, String(err)],
    );
  }
}

Locale is resolved at enqueue time, not at send time, with an explicit fallback chain so the stored row is self-describing:

const locale = guest.preferredLocale ?? facility.defaultLocale ?? 'ja';

6. The server advances state; the client only renders it

A redirect-based payment return is an untrusted input. The old flow had the return page confirm the payment — effectively letting a URL advance a money state. We wrote this down as an ADR: payment state is advanced by the server ledger only.

// Anti-pattern: the client asserts the outcome.
// await api.confirmPayment({ stayId, status: 'succeeded' });
// Pattern: the return page re-reads the server's view and renders it.
const stay = await server.getStay(stayId);
if (stay.settlement === 'pending') return renderHolding(stay); // poll or wait for webhook
if (stay.balanceDue > 0) return renderBalanceDue(stay);
return renderCompleted(stay);

The same reasoning applies to door access: entry keys are persisted server-side and the privileged issuance path is gated, so a client cannot mint admission by calling the right endpoint with the right shape. Admission is a server decision; the client receives a rendering of it.

7. Authorization is a server decision, too

Several fixes in this batch share one shape: the UI already hid something, and the server did not refuse it.

• A plan marked members-only was greyed out in the picker — but a crafted walk-in request still priced and sold it.
• A reservation-only entrance showed no walk-in button — but the endpoint accepted walk-ins.
• A status-update RPC trusted the row id without checking the caller's facility.

Hiding is presentation. Refusing is access control. Every one of these became a server-side gate returning a code:

export function evaluateCheckinRequest(input: CheckinRequest): CheckinDecision {
  if (input.config.mode === 'reservation_only' && !input.reservationId) {
    return { allowed: false, code: 'WALK_IN_NOT_ALLOWED' };
  }
  if (input.plan.membersOnly && !input.membership?.isActive) {
    return { allowed: false, code: 'MEMBERS_ONLY_PLAN' };
  }
  if (!input.config.planIds.includes(input.plan.id)) {
    return { allowed: false, code: 'PLAN_NOT_IN_CONFIG' };
  }
  if (input.blacklist.matches(input.guest)) {
    return { allowed: false, code: 'GUEST_BLOCKED' };
  }
  return { allowed: true };
}

Routing bans through one shared gate also means the refusal always carries its code, so the audit trail records why entry was denied rather than a generic failure.

At the data layer, scoping belongs in a policy, not in a handler someone might forget:

create policy reservations_update_own_facility on reservations
  for update
  using (facility_id = any (current_user_facility_ids()))
  with check (facility_id = any (current_user_facility_ids()));

8. Accounting invariants deserve the same rigor

Quota and money are state machines wearing a different hat. Two invariants we had to make explicit:

• Symmetry: a monthly quota slot is counted against the month the slot is used, and returned to that same month on cancellation — never to "now". Otherwise a late cancellation silently mints credit in the current month.
• Policy-bound refunds: quota is returned only when the cancellation policy says so, exactly as a ticket book behaves, and a refused payment is refunded in full exactly once.

export function releaseQuota(usage: QuotaUsage, policy: CancellationPolicy, now: Date): QuotaDelta | null {
  if (!policy.refundsQuota(usage.startsAt, now)) return null;
  return { month: usage.usageMonth, delta: +1 };
}

And for deposits: charge only what the deposit does not cover. Double-charging is a state bug, not a pricing bug.

9. Guardrails in CI

Finally, a small but unglamorous piece: the pipeline itself got always-on guards — failing the build on committed merge-conflict markers (we had shipped a set into a membership approval path), and flagging a renumbered migration whose old version still lingers in a local database. Both are cheap checks for failure modes that are expensive and confusing in production.

Summary

If you are building a system where a bug opens or refuses a physical door, these are the invariants worth enforcing structurally rather than by discipline:

1. One verb per transition. Many callers, one writer, one compare-and-set.
2. Write the state table down and guard the column so nothing can bypass the verbs.
3. Fail closed. If the external leg cannot complete, refuse; never record a local-only truth.
4. Return outcomes, not strings. applied | noop | conflict | refused with stable codes.
5. Idempotency keys everywhere money or access is involved, plus webhook dedupe on provider event ids.
6. Transactional outbox for every notification, with a dedupe key, claim/release semantics, and locale resolved at enqueue.
7. The server advances state; the client renders it. Redirect returns are untrusted input.
8. Hiding is not refusing. Every UI restriction needs a server-side gate with a code and a policy behind it.

None of these are clever. All of them are the difference between a system you can reason about at 3 a.m. and one you can only apologize for.