Files

T

Investor Flow Build e262187c3c fix: backfill symbol_demand for sidebar-added symbols + analyst ratings schema fix

- Add await ctx.cache.subscribe() to addSymbol mutation so symbols
  added via the sidebar get registered in symbol_demand and yfinance
  jobs are queued immediately
- Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue
- Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false
  to quoteSummary() to handle Yahoo schema drift
- Add error detail logging for analyst ratings schema failures
- Update .gitignore with common ignores

2026-07-23 18:02:24 -04:00

3.9 KiB

Raw Blame History

ADR-0009: Rate-limit-first data plane

Date: 2026-07-19
Status: Accepted

Context

Every external vendor we use has a short rate limit:

Source	Typical limit / failure mode
Yahoo Finance (`yahoo-finance2`)	Edge 429 / "Too Many Requests" under concurrent chart+quote+summary
X (bird CLI / cookie session)	HTTP 429 on search / timeline
FRED	API key quota; burst-sensitive
SEC EDGAR	Fair-access pacing (~10 req/s official guidance)
Reddit	OAuth / public endpoint throttles

The product already had shared cache + queue dedupe (ADR-0004) and stale-while-revalidate, but request handlers still opened live vendor calls (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.

Decision

Design the data plane around rate limits as a first-class constraint:

1. Request path never stampede

tRPC handlers prefer local state:

SQLite / kv_cache / typed tables (quotes, candles, …)
Stale-while-revalidate via CacheRepository.get (schedule background refresh)
Static / curated fallbacks for slow-changing composition (e.g. ETF top holdings)
Live vendor only when nothing local exists, with a short timeout and graceful empty/stale result

UI clicks must not fan out N charts or N quoteSummary calls.

2. One shared queue owns outbound pacing

All background refreshes go through AdapterQueue:

Per-source min-interval between fetches (steady-state throttle)
Source-wide cool-down on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
While cool-down is active: skip all jobs for that source; do not enqueue schedule floods
Job exponential backoff remains for ordinary failures only
Rate-limit hits do not burn MAX_ATTEMPTS into permanent failed without a long cool-down first

3. Demand set bounds work

Only symbols in symbol_demand (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).

4. Stale is better than empty

Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.

5. Observability

Queue health exposes active source cool-downs so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.

Implementation map

Piece	Location
Policy helpers (detect 429, ladders)	`app/server/src/queue/sourceRatePolicy.ts`
Cool-down + drain skip	`app/server/src/queue/AdapterQueue.ts`
ETF holdings cache + static fallback	`market.sectorHoldings` + `etfHoldingsFallback.ts`
FRED series write-through	`kv_cache` in condition path
Shared market cache	ADR-0004, `CacheRepository`

Consequences

Positive: Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
Positive: New features have a clear rule: “cache first, queue refresh, static if needed.”
Trade-off: After a cool-down, data may be minutes-to-hours stale until the next successful drain.
Trade-off: Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
Follow-ups: Route remaining live Yahoo calls in market.condition / peer resolution through the same cache-or-queue pattern; surface cool-downs in admin queue UI.

Rejected alternatives

Retry harder on 429: Makes thrash worse.
Per-user fetch with no shared cool-down: Multiplies load; violates ADR-0004.
Block UI until live succeeds: Timeouts and empty states; bad UX for a terminal-style education app.

3.9 KiB Raw Blame History Unescape Escape