80 lines
3.9 KiB
Markdown
80 lines
3.9 KiB
Markdown
|
# ADR-0009: Rate-limit-first data plane
|
|||
|
|
|
|||
|
|
Date: 2026-07-19
|
|||
|
|
Status: Accepted
|
|||
|
|
|
|||
|
|
## Context
|
|||
|
|
|
|||
|
|
Every external vendor we use has a short rate limit:
|
|||
|
|
|
|||
|
|
| Source | Typical limit / failure mode |
|
|||
|
|
|--------|------------------------------|
|
|||
|
|
| Yahoo Finance (`yahoo-finance2`) | Edge 429 / "Too Many Requests" under concurrent chart+quote+summary |
|
|||
|
|
| X (bird CLI / cookie session) | HTTP 429 on search / timeline |
|
|||
|
|
| FRED | API key quota; burst-sensitive |
|
|||
|
|
| SEC EDGAR | Fair-access pacing (~10 req/s official guidance) |
|
|||
|
|
| Reddit | OAuth / public endpoint throttles |
|
|||
|
|
|
|||
|
|
The product already had **shared cache + queue dedupe** (ADR-0004) and **stale-while-revalidate**, but request handlers still opened **live vendor calls** (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.
|
|||
|
|
|
|||
|
|
## Decision
|
|||
|
|
|
|||
|
|
Design the data plane around rate limits as a first-class constraint:
|
|||
|
|
|
|||
|
|
### 1. Request path never stampede
|
|||
|
|
|
|||
|
|
tRPC handlers **prefer local state**:
|
|||
|
|
|
|||
|
|
1. SQLite / `kv_cache` / typed tables (quotes, candles, …)
|
|||
|
|
2. Stale-while-revalidate via `CacheRepository.get` (schedule background refresh)
|
|||
|
|
3. **Static / curated fallbacks** for slow-changing composition (e.g. ETF top holdings)
|
|||
|
|
4. Live vendor only when nothing local exists, with a **short timeout** and graceful empty/stale result
|
|||
|
|
|
|||
|
|
UI clicks must not fan out N charts or N quoteSummary calls.
|
|||
|
|
|
|||
|
|
### 2. One shared queue owns outbound pacing
|
|||
|
|
|
|||
|
|
All background refreshes go through `AdapterQueue`:
|
|||
|
|
|
|||
|
|
- **Per-source min-interval** between fetches (steady-state throttle)
|
|||
|
|
- **Source-wide cool-down** on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
|
|||
|
|
- While cool-down is active: **skip all jobs for that source**; do not enqueue schedule floods
|
|||
|
|
- Job exponential backoff remains for *ordinary* failures only
|
|||
|
|
- Rate-limit hits **do not burn** `MAX_ATTEMPTS` into permanent `failed` without a long cool-down first
|
|||
|
|
|
|||
|
|
### 3. Demand set bounds work
|
|||
|
|
|
|||
|
|
Only symbols in `symbol_demand` (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).
|
|||
|
|
|
|||
|
|
### 4. Stale is better than empty
|
|||
|
|
|
|||
|
|
Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.
|
|||
|
|
|
|||
|
|
### 5. Observability
|
|||
|
|
|
|||
|
|
Queue health exposes active **source cool-downs** so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.
|
|||
|
|
|
|||
|
|
## Implementation map
|
|||
|
|
|
|||
|
|
| Piece | Location |
|
|||
|
|
|-------|----------|
|
|||
|
|
| Policy helpers (detect 429, ladders) | `app/server/src/queue/sourceRatePolicy.ts` |
|
|||
|
|
| Cool-down + drain skip | `app/server/src/queue/AdapterQueue.ts` |
|
|||
|
|
| ETF holdings cache + static fallback | `market.sectorHoldings` + `etfHoldingsFallback.ts` |
|
|||
|
|
| FRED series write-through | `kv_cache` in condition path |
|
|||
|
|
| Shared market cache | ADR-0004, `CacheRepository` |
|
|||
|
|
|
|||
|
|
## Consequences
|
|||
|
|
|
|||
|
|
- **Positive:** Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
|
|||
|
|
- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
|
|||
|
|
- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
|
|||
|
|
- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
|
|||
|
|
- **Follow-ups:** Route remaining live Yahoo calls in `market.condition` / peer resolution through the same cache-or-queue pattern; surface cool-downs in admin queue UI.
|
|||
|
|
|
|||
|
|
## Rejected alternatives
|
|||
|
|
|
|||
|
|
- **Retry harder on 429:** Makes thrash worse.
|
|||
|
|
- **Per-user fetch with no shared cool-down:** Multiplies load; violates ADR-0004.
|
|||
|
|
- **Block UI until live succeeds:** Timeouts and empty states; bad UX for a terminal-style education app.
|