Investor Flow Build
e262187c3c
- Add await ctx.cache.subscribe() to addSymbol mutation so symbols added via the sidebar get registered in symbol_demand and yfinance jobs are queued immediately - Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue - Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false to quoteSummary() to handle Yahoo schema drift - Add error detail logging for analyst ratings schema failures - Update .gitignore with common ignores
80 lines
3.9 KiB
Markdown
80 lines
3.9 KiB
Markdown
# ADR-0009: Rate-limit-first data plane
|
||
|
||
Date: 2026-07-19
|
||
Status: Accepted
|
||
|
||
## Context
|
||
|
||
Every external vendor we use has a short rate limit:
|
||
|
||
| Source | Typical limit / failure mode |
|
||
|--------|------------------------------|
|
||
| Yahoo Finance (`yahoo-finance2`) | Edge 429 / "Too Many Requests" under concurrent chart+quote+summary |
|
||
| X (bird CLI / cookie session) | HTTP 429 on search / timeline |
|
||
| FRED | API key quota; burst-sensitive |
|
||
| SEC EDGAR | Fair-access pacing (~10 req/s official guidance) |
|
||
| Reddit | OAuth / public endpoint throttles |
|
||
|
||
The product already had **shared cache + queue dedupe** (ADR-0004) and **stale-while-revalidate**, but request handlers still opened **live vendor calls** (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.
|
||
|
||
## Decision
|
||
|
||
Design the data plane around rate limits as a first-class constraint:
|
||
|
||
### 1. Request path never stampede
|
||
|
||
tRPC handlers **prefer local state**:
|
||
|
||
1. SQLite / `kv_cache` / typed tables (quotes, candles, …)
|
||
2. Stale-while-revalidate via `CacheRepository.get` (schedule background refresh)
|
||
3. **Static / curated fallbacks** for slow-changing composition (e.g. ETF top holdings)
|
||
4. Live vendor only when nothing local exists, with a **short timeout** and graceful empty/stale result
|
||
|
||
UI clicks must not fan out N charts or N quoteSummary calls.
|
||
|
||
### 2. One shared queue owns outbound pacing
|
||
|
||
All background refreshes go through `AdapterQueue`:
|
||
|
||
- **Per-source min-interval** between fetches (steady-state throttle)
|
||
- **Source-wide cool-down** on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
|
||
- While cool-down is active: **skip all jobs for that source**; do not enqueue schedule floods
|
||
- Job exponential backoff remains for *ordinary* failures only
|
||
- Rate-limit hits **do not burn** `MAX_ATTEMPTS` into permanent `failed` without a long cool-down first
|
||
|
||
### 3. Demand set bounds work
|
||
|
||
Only symbols in `symbol_demand` (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).
|
||
|
||
### 4. Stale is better than empty
|
||
|
||
Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.
|
||
|
||
### 5. Observability
|
||
|
||
Queue health exposes active **source cool-downs** so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.
|
||
|
||
## Implementation map
|
||
|
||
| Piece | Location |
|
||
|-------|----------|
|
||
| Policy helpers (detect 429, ladders) | `app/server/src/queue/sourceRatePolicy.ts` |
|
||
| Cool-down + drain skip | `app/server/src/queue/AdapterQueue.ts` |
|
||
| ETF holdings cache + static fallback | `market.sectorHoldings` + `etfHoldingsFallback.ts` |
|
||
| FRED series write-through | `kv_cache` in condition path |
|
||
| Shared market cache | ADR-0004, `CacheRepository` |
|
||
|
||
## Consequences
|
||
|
||
- **Positive:** Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
|
||
- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
|
||
- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
|
||
- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
|
||
- **Follow-ups:** Route remaining live Yahoo calls in `market.condition` / peer resolution through the same cache-or-queue pattern; surface cool-downs in admin queue UI.
|
||
|
||
## Rejected alternatives
|
||
|
||
- **Retry harder on 429:** Makes thrash worse.
|
||
- **Per-user fetch with no shared cool-down:** Multiplies load; violates ADR-0004.
|
||
- **Block UI until live succeeds:** Timeouts and empty states; bad UX for a terminal-style education app.
|