investor-flow/docs/adr/0009-rate-limit-first-data-plane.md

# ADR-0009: Rate-limit-first data plane

Date: 2026-07-19  
Status: Accepted

## Context

Every external vendor we use has a short rate limit:

| Source | Typical limit / failure mode |
|--------|------------------------------|
| Yahoo Finance (`yahoo-finance2`) | Edge 429 / "Too Many Requests" under concurrent chart+quote+summary |
| X (bird CLI / cookie session) | HTTP 429 on search / timeline |
| FRED | API key quota; burst-sensitive |
| SEC EDGAR | Fair-access pacing (~10 req/s official guidance) |
| Reddit | OAuth / public endpoint throttles |

The product already had **shared cache + queue dedupe** (ADR-0004) and **stale-while-revalidate**, but request handlers still opened **live vendor calls** (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.

## Decision

Design the data plane around rate limits as a first-class constraint:

### 1. Request path never stampede

tRPC handlers **prefer local state**:

1. SQLite / `kv_cache` / typed tables (quotes, candles, …)
2. Stale-while-revalidate via `CacheRepository.get` (schedule background refresh)
3. **Static / curated fallbacks** for slow-changing composition (e.g. ETF top holdings)
4. Live vendor only when nothing local exists, with a **short timeout** and graceful empty/stale result

UI clicks must not fan out N charts or N quoteSummary calls.

### 2. One shared queue owns outbound pacing

All background refreshes go through `AdapterQueue`:

- **Per-source min-interval** between fetches (steady-state throttle)
- **Source-wide cool-down** on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
- While cool-down is active: **skip all jobs for that source**; do not enqueue schedule floods
- Job exponential backoff remains for *ordinary* failures only
- Rate-limit hits **do not burn** `MAX_ATTEMPTS` into permanent `failed` without a long cool-down first

### 3. Demand set bounds work

Only symbols in `symbol_demand` (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).

### 4. Stale is better than empty

Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.

### 5. Observability

Queue health exposes active **source cool-downs** so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.

## Implementation map

| Piece | Location |
|-------|----------|
| Policy helpers (detect 429, ladders) | `app/server/src/queue/sourceRatePolicy.ts` |
| Cool-down + drain skip | `app/server/src/queue/AdapterQueue.ts` |
| ETF holdings cache + static fallback | `market.sectorHoldings` + `etfHoldingsFallback.ts` |
| FRED series write-through | `kv_cache` in condition path |
| Shared market cache | ADR-0004, `CacheRepository` |

## Consequences

- **Positive:** Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
- **Follow-ups:** Surface per-family cool-downs in admin queue UI (process-wide `vendorGate` + queue_state).

## Implementation (2026-08 hardening)

All vendors share `app/server/src/services/vendorGate.ts` with **runtime registration** so future integrations do not require editing a closed union:

1. `registerVendorIntegration({ family, sourceKinds, policy })` before traffic  
2. `VendorSourceAdapter` / `defineVendorAdapter` auto-wrap `fetchOne`  
3. `AdapterQueue` constructor **throws** if any adapter `source_kind` lacks a family  
4. CI: `vendorHttpGuard.test.ts` fails on bare `fetch(` outside the gate modules  
5. Handbook: `docs/VENDOR_INTEGRATIONS.md`

| Family | Call style | Drain budget |
|--------|------------|--------------|
| yfinance | `withVendorGate` (yahoo-finance2 + options) | 3 |
| sec | `secHttp` → vendorGate | 1 |
| fred / finra / nasdaq / reddit | `vendorFetch` | 1 |
| x | `withVendorGate` around bird CLI | 1 |
| *(future)* | `registerVendorIntegration` + `VendorSourceAdapter` | policy.drainJobBudget |

A 429 cools the **family** (every `source_kind` in that family), not just the one job. Per-request min gaps are process-wide; job min-intervals in AdapterQueue are coarser backup.

## Rejected alternatives

- **Retry harder on 429:** Makes thrash worse.
- **Per-user fetch with no shared cool-down:** Multiplies load; violates ADR-0004.
- **Block UI until live succeeds:** Timeouts and empty states; bad UX for a terminal-style education app.