Investor Flow Build
ac94acf9e3
CI / Test & Type-Check (push) Canceled after 0s
Snapshot of in-progress module work across multiple slices: - Dealer Flow: dealerExposureEngine, dealerMapService, dealerMapExplain, dealerMapIntegrity, dealerMapReplay, dealerStudyEngine, hanStyleLevels - Mirror Portfolio (M21): fundRepository, captureIngest, mirrorAlertProducers, fund holdings strip, live book, position capture ingest - Options: BSM, NormalizedOptionSurface types, OptionsChainRouter, ConvexityGate, option legs panel - Alert producers: vixLevel, rotation, thesis, unlock, portfolioRisk, mirror (fund_capture, fund_13f, mirror_diff) - FINRA short interest adapter + queue integration - SEC company tickers adapter + ingest (symbol search index seed) - Vendor gate (rate-limit-first data plane, ADR-0009) - CUSIP registry, reverse 13F refresh, stock float service - LRU cache, portfolio backtest engine - Frontend: dealer-flow, funds, journal, lab, monitor, plan, portfolio, reports, screener, strategies, theses, guided-start, exits, more pages - Volume profile, workspace profile, visibility-aware poll - ADRs 0010 (mirror math not advice), 0011 (symbol search index) - VENDOR_INTEGRATIONS.md, END_USER_TEST.md - .gitignore: exclude DBs, .DS_Store, local config, agent scratch
100 lines
4.9 KiB
Markdown
100 lines
4.9 KiB
Markdown
# ADR-0009: Rate-limit-first data plane
|
||
|
||
Date: 2026-07-19
|
||
Status: Accepted
|
||
|
||
## Context
|
||
|
||
Every external vendor we use has a short rate limit:
|
||
|
||
| Source | Typical limit / failure mode |
|
||
|--------|------------------------------|
|
||
| Yahoo Finance (`yahoo-finance2`) | Edge 429 / "Too Many Requests" under concurrent chart+quote+summary |
|
||
| X (bird CLI / cookie session) | HTTP 429 on search / timeline |
|
||
| FRED | API key quota; burst-sensitive |
|
||
| SEC EDGAR | Fair-access pacing (~10 req/s official guidance) |
|
||
| Reddit | OAuth / public endpoint throttles |
|
||
|
||
The product already had **shared cache + queue dedupe** (ADR-0004) and **stale-while-revalidate**, but request handlers still opened **live vendor calls** (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.
|
||
|
||
## Decision
|
||
|
||
Design the data plane around rate limits as a first-class constraint:
|
||
|
||
### 1. Request path never stampede
|
||
|
||
tRPC handlers **prefer local state**:
|
||
|
||
1. SQLite / `kv_cache` / typed tables (quotes, candles, …)
|
||
2. Stale-while-revalidate via `CacheRepository.get` (schedule background refresh)
|
||
3. **Static / curated fallbacks** for slow-changing composition (e.g. ETF top holdings)
|
||
4. Live vendor only when nothing local exists, with a **short timeout** and graceful empty/stale result
|
||
|
||
UI clicks must not fan out N charts or N quoteSummary calls.
|
||
|
||
### 2. One shared queue owns outbound pacing
|
||
|
||
All background refreshes go through `AdapterQueue`:
|
||
|
||
- **Per-source min-interval** between fetches (steady-state throttle)
|
||
- **Source-wide cool-down** on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
|
||
- While cool-down is active: **skip all jobs for that source**; do not enqueue schedule floods
|
||
- Job exponential backoff remains for *ordinary* failures only
|
||
- Rate-limit hits **do not burn** `MAX_ATTEMPTS` into permanent `failed` without a long cool-down first
|
||
|
||
### 3. Demand set bounds work
|
||
|
||
Only symbols in `symbol_demand` (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).
|
||
|
||
### 4. Stale is better than empty
|
||
|
||
Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.
|
||
|
||
### 5. Observability
|
||
|
||
Queue health exposes active **source cool-downs** so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.
|
||
|
||
## Implementation map
|
||
|
||
| Piece | Location |
|
||
|-------|----------|
|
||
| Policy helpers (detect 429, ladders) | `app/server/src/queue/sourceRatePolicy.ts` |
|
||
| Cool-down + drain skip | `app/server/src/queue/AdapterQueue.ts` |
|
||
| ETF holdings cache + static fallback | `market.sectorHoldings` + `etfHoldingsFallback.ts` |
|
||
| FRED series write-through | `kv_cache` in condition path |
|
||
| Shared market cache | ADR-0004, `CacheRepository` |
|
||
|
||
## Consequences
|
||
|
||
- **Positive:** Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
|
||
- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
|
||
- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
|
||
- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
|
||
- **Follow-ups:** Surface per-family cool-downs in admin queue UI (process-wide `vendorGate` + queue_state).
|
||
|
||
## Implementation (2026-08 hardening)
|
||
|
||
All vendors share `app/server/src/services/vendorGate.ts` with **runtime registration** so future integrations do not require editing a closed union:
|
||
|
||
1. `registerVendorIntegration({ family, sourceKinds, policy })` before traffic
|
||
2. `VendorSourceAdapter` / `defineVendorAdapter` auto-wrap `fetchOne`
|
||
3. `AdapterQueue` constructor **throws** if any adapter `source_kind` lacks a family
|
||
4. CI: `vendorHttpGuard.test.ts` fails on bare `fetch(` outside the gate modules
|
||
5. Handbook: `docs/VENDOR_INTEGRATIONS.md`
|
||
|
||
| Family | Call style | Drain budget |
|
||
|--------|------------|--------------|
|
||
| yfinance | `withVendorGate` (yahoo-finance2 + options) | 3 |
|
||
| sec | `secHttp` → vendorGate | 1 |
|
||
| fred / finra / nasdaq / reddit | `vendorFetch` | 1 |
|
||
| x | `withVendorGate` around bird CLI | 1 |
|
||
| *(future)* | `registerVendorIntegration` + `VendorSourceAdapter` | policy.drainJobBudget |
|
||
|
||
A 429 cools the **family** (every `source_kind` in that family), not just the one job. Per-request min gaps are process-wide; job min-intervals in AdapterQueue are coarser backup.
|
||
|
||
## Rejected alternatives
|
||
|
||
- **Retry harder on 429:** Makes thrash worse.
|
||
- **Per-user fetch with no shared cool-down:** Multiplies load; violates ADR-0004.
|
||
- **Block UI until live succeeds:** Timeouts and empty states; bad UX for a terminal-style education app.
|