Files

T

CI / Test & Type-Check (push) Canceled after 0s

Details

feat: dealer flow, mirror portfolio (M21), options convexity, FINRA short interest, alert producers, vendor gate

Snapshot of in-progress module work across multiple slices:

- Dealer Flow: dealerExposureEngine, dealerMapService, dealerMapExplain,
  dealerMapIntegrity, dealerMapReplay, dealerStudyEngine, hanStyleLevels
- Mirror Portfolio (M21): fundRepository, captureIngest, mirrorAlertProducers,
  fund holdings strip, live book, position capture ingest
- Options: BSM, NormalizedOptionSurface types, OptionsChainRouter,
  ConvexityGate, option legs panel
- Alert producers: vixLevel, rotation, thesis, unlock, portfolioRisk,
  mirror (fund_capture, fund_13f, mirror_diff)
- FINRA short interest adapter + queue integration
- SEC company tickers adapter + ingest (symbol search index seed)
- Vendor gate (rate-limit-first data plane, ADR-0009)
- CUSIP registry, reverse 13F refresh, stock float service
- LRU cache, portfolio backtest engine
- Frontend: dealer-flow, funds, journal, lab, monitor, plan, portfolio,
  reports, screener, strategies, theses, guided-start, exits, more pages
- Volume profile, workspace profile, visibility-aware poll
- ADRs 0010 (mirror math not advice), 0011 (symbol search index)
- VENDOR_INTEGRATIONS.md, END_USER_TEST.md
- .gitignore: exclude DBs, .DS_Store, local config, agent scratch

2026-08-10 13:36:26 -04:00

4.9 KiB

Raw Permalink Blame History

ADR-0009: Rate-limit-first data plane

Date: 2026-07-19
Status: Accepted

Context

Every external vendor we use has a short rate limit:

Source	Typical limit / failure mode
Yahoo Finance (`yahoo-finance2`)	Edge 429 / "Too Many Requests" under concurrent chart+quote+summary
X (bird CLI / cookie session)	HTTP 429 on search / timeline
FRED	API key quota; burst-sensitive
SEC EDGAR	Fair-access pacing (~10 req/s official guidance)
Reddit	OAuth / public endpoint throttles

The product already had shared cache + queue dedupe (ADR-0004) and stale-while-revalidate, but request handlers still opened live vendor calls (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.

Decision

Design the data plane around rate limits as a first-class constraint:

1. Request path never stampede

tRPC handlers prefer local state:

SQLite / kv_cache / typed tables (quotes, candles, …)
Stale-while-revalidate via CacheRepository.get (schedule background refresh)
Static / curated fallbacks for slow-changing composition (e.g. ETF top holdings)
Live vendor only when nothing local exists, with a short timeout and graceful empty/stale result

UI clicks must not fan out N charts or N quoteSummary calls.

2. One shared queue owns outbound pacing

All background refreshes go through AdapterQueue:

Per-source min-interval between fetches (steady-state throttle)
Source-wide cool-down on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
While cool-down is active: skip all jobs for that source; do not enqueue schedule floods
Job exponential backoff remains for ordinary failures only
Rate-limit hits do not burn MAX_ATTEMPTS into permanent failed without a long cool-down first

3. Demand set bounds work

Only symbols in symbol_demand (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).

4. Stale is better than empty

Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.

5. Observability

Queue health exposes active source cool-downs so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.

Implementation map

Piece	Location
Policy helpers (detect 429, ladders)	`app/server/src/queue/sourceRatePolicy.ts`
Cool-down + drain skip	`app/server/src/queue/AdapterQueue.ts`
ETF holdings cache + static fallback	`market.sectorHoldings` + `etfHoldingsFallback.ts`
FRED series write-through	`kv_cache` in condition path
Shared market cache	ADR-0004, `CacheRepository`

Consequences

Positive: Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
Positive: New features have a clear rule: “cache first, queue refresh, static if needed.”
Trade-off: After a cool-down, data may be minutes-to-hours stale until the next successful drain.
Trade-off: Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
Follow-ups: Surface per-family cool-downs in admin queue UI (process-wide vendorGate + queue_state).

Implementation (2026-08 hardening)

All vendors share app/server/src/services/vendorGate.ts with runtime registration so future integrations do not require editing a closed union:

registerVendorIntegration({ family, sourceKinds, policy }) before traffic
VendorSourceAdapter / defineVendorAdapter auto-wrap fetchOne
AdapterQueue constructor throws if any adapter source_kind lacks a family
CI: vendorHttpGuard.test.ts fails on bare fetch( outside the gate modules
Handbook: docs/VENDOR_INTEGRATIONS.md

Family	Call style	Drain budget
yfinance	`withVendorGate` (yahoo-finance2 + options)	3
sec	`secHttp` → vendorGate	1
fred / finra / nasdaq / reddit	`vendorFetch`	1
x	`withVendorGate` around bird CLI	1
(future)	`registerVendorIntegration` + `VendorSourceAdapter`	policy.drainJobBudget

A 429 cools the family (every source_kind in that family), not just the one job. Per-request min gaps are process-wide; job min-intervals in AdapterQueue are coarser backup.

Rejected alternatives

Retry harder on 429: Makes thrash worse.
Per-user fetch with no shared cool-down: Multiplies load; violates ADR-0004.
Block UI until live succeeds: Timeouts and empty states; bad UX for a terminal-style education app.

4.9 KiB Raw Permalink Blame History Unescape Escape