fix: backfill symbol_demand for sidebar-added symbols + analyst ratings schema fix

- Add await ctx.cache.subscribe() to addSymbol mutation so symbols added via the sidebar get registered in symbol_demand and yfinance jobs are queued immediately - Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue - Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false to quoteSummary() to handle Yahoo schema drift - Add error detail logging for analyst ratings schema failures - Update .gitignore with common ignores
2026-07-23 18:02:24 -04:00
parent 5ef2b2f060
commit e262187c3c
204 changed files with 25014 additions and 2934 deletions
@@ -0,0 +1,194 @@
+# Investor Flow — Functional Design (as built)
+
+**Audit date:** 2026-07-18  
+**Source of truth for status:** this file + code under `app/`. Older slice SPECs, HANDOFF.md, and Automaton WORKFLOW.md are historical.  
+**Product intent:** Beginner-first multi-tenant educational research terminal for conviction investing (not day trading). Legal posture: educational publisher (ADR-0007).
+
+---
+
+## Status legend
+
+| Status | Meaning |
+|--------|---------|
+| **Live** | Backend + UI end-to-end usable for the intended job |
+| **Backend-ready** | Engine and/or tRPC exist with tests; little or no product UI |
+| **UI-local** | Screen exists but state is browser-only (localStorage / Zustand), not multi-device / multi-tenant |
+| **Stub** | Surface or procedure exists but incomplete (empty timeline, thin universe, etc.) |
+| **Missing** | Designed in domain model; not implemented as a product path |
+
+---
+
+## 1. User jobs and surfaces
+
+### 1.1 Identity, access, onboarding
+
+| Capability | Status | Notes |
+|------------|--------|-------|
+| Email/password signup | **Live** | New users get `status=pending_approval`; no session until admin approves |
+| Login + session cookie | **Live** | Non-active users rejected at login and `protectedProcedure` |
+| TOTP 2FA + backup codes | **Live** | Settings page |
+| GitHub / Google OAuth | **Backend-ready** | Router complete; env-dependent; OAuth users default `active` |
+| Onboarding wizard | **Live** | Complexity, risk, drawdown, starter watchlist, optional portfolio; queues SEC fetch |
+| Admin approve/reject | **Live** | `/admin/users` |
+| Settings / profile | **Live** | `/settings` |
+
+### 1.2 Research workbench (desktop)
+
+| Surface | Route | Status | Data path |
+|---------|-------|--------|-----------|
+| Symbol overview | `/` | **Live** | `market.snapshot`, company meta, X feed, collapsible filings + options |
+| Charts | `/chart-lab` | **Live** | candles, indicators, institutional buy markers |
+| Institutional dashboard | `/institutional` | **Live** | dashboard rollup, flow, insider stream, filings; Q-o-Q / M-o-M |
+| Filings | `/filings` | **Live** | EDGAR index + 13F + Form 4 detail |
+| Options DD | `/options-dd` | **Live** | chain + greeks (read-only teaching surface) |
+| Market outlook | `/market-outlook` | **Live (Phase 1–3 core)** | Beginner language. Condition strip; stronger/weaker-vs-market map; seasonality + calendar; rank snapshots; auto leadership check; truck/factory; macro notes. True ETF flow/COT still later. |
+| Social / X feed | on overview | **Live** | DB-backed 30d history + admin X credentials/accounts |
+| Command palette + nav | shell | **Live** | ⌘K, g-key shortcuts, theme presets |
+
+### 1.3 Process / journal workflow
+
+| Surface | Route | Status | Gap |
+|---------|-------|--------|-----|
+| Trade plan | `/trade-plan` | **UI-local** | Zustand only; not `theses` / `trades` / sizing APIs |
+| Execution | `/execution` | **UI-local** | Playbook + checklist local; not `trade_executions` |
+| Trade closure | `/trade-closure` | **UI-local** | localStorage |
+| Daily focus / goals | `/daily-focus` | **UI-local** | localStorage goals |
+| Emotion logger | on execution | **Partial** | UI + `emotionLogger` API exist; UI still primarily local path |
+| Sizing math | `/risk` (calculator) | **Live** | `sizing.compute` + UI cascade (math only, ADR-0007) |
+| Risk posture (M20) | `/risk` | **Live** | `risk.posture` / `risk.haltStatus` + RiskPosturePanel; halt persisted on drawdown breach |
+| Derisking suggestions | (no screen) | **Backend-ready** | `derisking.suggest` |
+| Thesis monitor | (no screen) | **Stub** | `thesisMonitor.assess` / `timeline` (empty events / empty timeline) |
+
+### 1.4 Discovery & strategy lab
+
+| Capability | Status | Notes |
+|------------|--------|-------|
+| Filter screener (M15a) | **Backend-ready** | `screener.filter` over watchlist quotes only; **no UI**; not in frontend `api` client |
+| Strategy screener (M15b) | **Backend-ready** | `screener.strategy`; thin universe; no UI |
+| Strategy authoring | **Backend-ready** | `strategies.list/create`; no Strategy Lab UI |
+| Backtest | **Backend-ready** | `backtest.run` / `evaluateLatest`; no equity-curve UI |
+| Sector confirmation cross-link | **Stub** | `sectorCrosslink.confirm` with empty universe in practice |
+| Screener saved filters | **Schema only** | tables exist; incomplete product loop |
+
+### 1.5 Options convexity sleeve (M17)
+
+| Capability | Status | Notes |
+|------------|--------|-------|
+| Options DD (read-only) | **Live** | M3 teaching panel |
+| Unlock ladder + payoff API | **Backend-ready** | `optionsConvexity.unlock` / `getPayoff` |
+| Option legs book + risk contribution | **Live (MVP)** | `portfolio.optionLegs` / add / remove; Risk posture premium-at-risk + CSP cash reserve |
+| Portfolio sleeve workflow (M17 full) | **Missing** | Unlock UX + sleeve risk budget not enforced in product path |
+
+### 1.6 Macro (M18)
+
+| Capability | Status | Notes |
+|------------|--------|-------|
+| FRED series / key admin | **Live** | Admin X Accounts page also manages FRED key |
+| Truck sales + manufacturing PMI charts | **Live** | Market routes |
+| Sector rotation heatmap-style panel | **Live** | Relative-strength style rotation, not full designed phase library UI |
+| Regime classify / history | **Backend-ready** | `macro.regimeClassify`, `regimeHistory` |
+| Portfolio-impact commentary | **Live** (partial) | `macro.commentary` via Market Outlook; **2 unit tests failing** on wording assertions |
+| Full economic calendar UI | **Stub** | `macro.calendar` |
+
+### 1.7 Alerts & mobile
+
+| Capability | Status | Notes |
+|------------|--------|-------|
+| Alert list / ack / unacked count | **Live** | Used on mobile + rotation alert chip |
+| Rotation signal → alert create | **Live** | `market.rotationCheckForAlert` |
+| Full hybrid AlertEngine (Form4, thesis, cluster, halt…) | **Partial** | Engine types/tests exist; not all event producers wired continuously |
+| SSE push | **Missing** | Design called for SSE; poll/list only |
+| Mobile companion `/mobile` | **Live (thin)** | Auth, holdings glance, unacked alerts — not full M19 job A vision |
+
+### 1.8 Admin / operator
+
+| Capability | Status |
+|------------|--------|
+| Users, sessions, reset password | **Live** |
+| Pending approval queue | **Live** |
+| Adapter queue health, pause/resume, retry, schedules, error stacks | **Live** |
+| SEC queue fetch + lint holders/insiders + data quality | **Live** |
+| X credentials, accounts, prune | **Live** |
+| FRED key | **Live** |
+| Audit log | **Live** |
+| Server restart control | **Live** |
+| GDPR export | **Backend-ready** (admin API) |
+
+### 1.9 Reports
+
+| Capability | Status |
+|------------|--------|
+| HTML research note generator | **Backend-ready** | `reports.generate` scopes include symbol/watchlist/portfolio/rotation/sizing/risk — **no UI**, not in frontend client |
+
+---
+
+## 2. Functional architecture (intended jobs)
+
+```
+Job A — Monitor (mobile + alerts)
+  alerts.list/ack  ·  portfolio.holdings  ·  rotation chip
+
+Job B — Research (desktop workbench)
+  watchlist → active symbol → market + SEC + institutional + options + social
+  market outlook (macro + rotation)
+
+Job C — Process (planned investor loop)
+  thesis + confluence + sizing → plan → execute → emotion → close → unlock
+  [mostly UI-local today; engines exist but not productized]
+
+Job D — Discover (planned)
+  filter/strategy screener → open in workbench
+  [API only]
+```
+
+---
+
+## 3. Design principles still in force
+
+Unchanged from domain model / ADRs:
+
+1. **Primary Rule (ADR-0007)** — education, not advice; Primary-Rule lint test on curated strings.
+2. **Analyst Voice (ADR-0005)** — Alfred/Druckenmiller blend where LLM copy is used.
+3. **P6 plain English / P7 teach mechanics** — UI redesign token system supports this.
+4. **Local-first multi-tenant (ADR-0001)** + shared cache dedupe (ADR-0004).
+5. **LLM data provenance (ADR-0006)** — sensitive data stays on local-class providers; full gateway module is thin/incomplete vs original design.
+6. **Anti-gamification** — no confetti / outcome celebration.
+
+---
+
+## 4. Explicit non-goals (still)
+
+- Brokerage execution or custody
+- Investment adviser recommendations
+- Mobile-first full research workbench
+- Undefined-risk options strategies for beginners
+- Training on user portfolio/thesis content via external LLMs
+
+---
+
+## 5. Gap priority (product value)
+
+| P | Gap | Why it matters |
+|---|-----|----------------|
+| P0 | Fix failing MacroRegime commentary tests | **Done 2026-07-18** — suite green (504 pass / 1 skip) |
+| P0 | Expose SizingEngine + RiskEngine on tRPC + M20 UI | **Done 2026-07-18** — `/risk` + `sizing`/`risk` routers |
+| P1 | Persist trade plan / thesis / execution / closure server-side | Today process screens are single-browser toys |
+| P1 | Frontend client coverage for backend-only routers | APIs unreachable from SPA client |
+| P1 | Wire emotion logger UI to `emotionLogger` API | Schema + API exist; dual paths confuse |
+| P2 | Strategy Lab + backtest UI | Engines exist; no learning surface |
+| P2 | Screener UI (filter + strategy) | Discovery job unfinished |
+| P2 | Derisking + thesis monitor product surfaces | Process depth |
+| P2 | Options convexity sleeve UI | M17 incomplete |
+| P3 | Reports UI | Export pedagogy |
+| P3 | Schema cleanup (duplicate `strategies` DDL) | Maintainability |
+| P3 | SSE alerts / continuous event producers | Ops polish |
+| P3 | Full LLM Gateway as designed | Provenance enforcement completeness |
+
+---
+
+## 6. Related docs
+
+- Tech design (modules, stack, API map): [`TECH_DESIGN.md`](./TECH_DESIGN.md)
+- Domain glossary: [`../CONTEXT.md`](../CONTEXT.md)
+- ADRs: [`adr/`](./adr/)
+- Living wiki page: Obsidian `investor-flow.md` (synced with this audit)
@@ -0,0 +1,268 @@
+# Investor Flow — Technical Design (as built)
+
+**Audit date:** 2026-07-18  
+**Companion:** [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md)  
+**Canonical code roots:** `app/src` (Next frontend), `app/server/src` (Node backend)
+
+---
+
+## 1. Stack
+
+| Layer | Actual technology | Notes vs older docs |
+|-------|-------------------|---------------------|
+| Frontend | Next.js **16.2**, React **19**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA |
+| Backend runtime | **Node ≥22** (dev on Node 26), native TS via `--experimental-strip-types` | DESIGN said Bun; runtime glue is Node + `node:sqlite` |
+| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`) |
+| DB | SQLite file (`app/server/data/investor-flow.db`) | Single file multi-tenant |
+| Market data | `yahoo-finance2` **v3** class API | |
+| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + `secDataFetcher` | Filer CIK from accession prefix (critical fix) |
+| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | |
+| Macro | FredAdapter | Key stored admin-side |
+| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers |
+| Tests | `node --test --experimental-strip-types` | Backend ~503 tests; **500 pass, 2 fail, 1 skip** as of audit |
+| Deploy | `docker-compose.yml` (frontend + backend + volume) | Present; operator-run |
+
+---
+
+## 2. Process topology
+
+```
+Browser (Next :3000)
+  │  fetch /api/trpc/<proc>  (credentials include)
+  ▼
+Next rewrites / proxy  →  Backend (:3001)
+  │
+  ├─ appRouter (tRPC)
+  ├─ CacheRepository + AdapterQueue
+  ├─ Source adapters (yfinance, sec-*, x, reddit, fred)
+  └─ SQLite
+```
+
+- Dev helpers: `restart-servers.sh`, admin `serverRestart`.
+- Queue drain loop + 30s schedule loop inside backend process.
+
+---
+
+## 3. Multi-tenant data tiers (still valid)
+
+| Tier | Content | Isolation |
+|------|---------|-----------|
+| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro | Shared, no owner |
+| B | threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_* | Shared infrastructure |
+| C | watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters | `owner_id` / `user_id` |
+| D | users, sessions, admin_audit, x_credentials (singleton), llm_* | System |
+
+**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes.
+
+**Stale-while-revalidate:** CacheRepository returns cached rows and schedules refresh when past TTL.
+
+### 3.1 Rate-limit-first data plane (ADR-0009)
+
+All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:
+
+| Rule | Mechanism |
+|------|-----------|
+| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; short timeout if live is unavoidable |
+| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
+| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
+| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
+| Demand-bounded work | Only `symbol_demand` symbols get scheduled refresh |
+| Observability | `queue.health().sourceCooldowns` |
+
+**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge.
+
+Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts`. Live upgrade is best-effort when the source is not cooling down.
+
+---
+
+## 4. Backend module map
+
+```
+app/server/src/
+  index.ts                 HTTP server entry
+  trpc/router.ts           ~2k LOC — all routers
+  trpc/context.ts          session, cookies, db, cache, queue injection
+  db/schema.sql            DDL (note: duplicate strategies blocks — debt)
+  db/*Repository.ts        watchlist, portfolio, emotion logs
+  cache/CacheRepository.ts content-addressed cache API
+  queue/AdapterQueue.ts    schedule, pause, retry, source cool-downs (ADR-0009)
+  queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
+  adapters/                YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
+  analysis/                indicators, rotation, seasonality, etfHoldingsFallback, tickerContext
+  admin/                   operator functions + CLI
+  auth/                    totp, oauth, backup codes
+  alerts/AlertEngine.ts
+  risk/RiskEngine.ts + haltCircuitBreaker.ts   ← pure; NOT on tRPC
+  sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix  ← pure; NOT on tRPC
+  strategy/BacktestEngine.ts
+  screener/UniverseEvaluator.ts + SectorCrosslink.ts
+  options/ConvexityGate.ts
+  derisking/DeriskingEngine.ts
+  thesis/ThesisMonitor.ts
+  macro/FredAdapter.ts + MacroRegime.ts
+  reports/ReportRunner.ts
+  onboarding/starter.ts
+  services/secDataFetcher.ts
+  x/backfill.ts
+```
+
+### 4.1 tRPC surface (mounted)
+
+| Router | Procedures (summary) |
+|--------|----------------------|
+| `auth` | signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback |
+| `onboarding` | starter, complete |
+| `market` | snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings |
+| `dashboard` | rollup |
+| `admin` | users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport |
+| `alerts` | list, acknowledge, acknowledgeAll, unackedCount |
+| `institutional` | flow, insiderStream, ownershipHistory, buyEvents |
+| `edgar` | filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx |
+| `watchlists` | list, addSymbol, removeSymbol |
+| `portfolio` | holdings, addHolding, removeHolding |
+| `options` | chain, greeks |
+| `reports` | generate |
+| `screener` | filter, strategy |
+| `strategies` | list, create |
+| `backtest` | run, evaluateLatest |
+| `sectorCrosslink` | confirm |
+| `optionsConvexity` | unlock, getPayoff |
+| `derisking` | suggest |
+| `macro` | series, calendar, regimeClassify, commentary, regimeHistory |
+| `thesisMonitor` | assess, timeline |
+| `x` | feed, cashtag_search, timeline, accountsForSymbol |
+| `reddit` | subreddit, search |
+| `emotionLogger` | add, getByTrade, delete |
+
+### 4.2 Sizing + risk (mounted 2026-07-18)
+
+| Router | Procedures |
+|--------|------------|
+| `sizing` | `compute` — pure `sizePosition` + portfolio/unlock/halt context |
+| `risk` | `posture` — pure `assessRisk` + portfolio load + halt persist; `haltStatus` |
+
+Still deferred: journal create path that throws `HaltedError` when plans are server-persisted (`persist-trade-plan-execution-loop`).
+
+---
+
+## 5. Frontend architecture
+
+```
+app/src/
+  app/                     Next App Router pages
+  components/              Panels + ui kit + layout
+  stores/                  Zustand (several persist to localStorage)
+  lib/trpc.ts              Partial API client (subset of routers)
+  lib/chart-theme.ts       CSS-variable themed Recharts
+  lib/strings.ts           Primary-rule sensitive copy
+```
+
+### 5.1 Routes
+
+| Path | Role |
+|------|------|
+| `/` | Workbench overview |
+| `/chart-lab` | Charts |
+| `/institutional` | Institutional + filings |
+| `/filings` | Filings only |
+| `/options-dd` | Options DD |
+| `/market-outlook` | Macro / rotation / notes |
+| `/trade-plan`, `/execution`, `/trade-closure`, `/daily-focus` | Process (local) |
+| `/settings` | Auth + onboarding |
+| `/mobile` | Thin companion |
+| `/admin`, `/users`, `/queue`, `/audit-logs`, `/x-accounts` | Admin |
+
+### 5.2 Client coverage gap
+
+`lib/trpc.ts` exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.
+
+**Not exposed in client (but on server):** screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under `market.commentary`), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).
+
+---
+
+## 6. Caching & adapters
+
+| Source kind | Role | Typical freshness |
+|-------------|------|-------------------|
+| yfinance | quote, candles, sector, options | minutes / EOD |
+| sec / sec-fetch | filings, 13F, Form 4 bulk | daily schedule default |
+| sec-lint-* | gap detection / backfill | on demand admin |
+| x | timelines / cashtags | demand + prune |
+| reddit | subreddit/search | on demand |
+| fred / macro | series + commentary inputs | slower |
+| llm | summaries (tables present) | sparse product use |
+
+**AdapterQueue features (built):** pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.
+
+---
+
+## 7. AuthZ model
+
+| Gate | Behavior |
+|------|----------|
+| `publicProcedure` | No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens) |
+| `protectedProcedure` | Session + `users.status === 'active'` |
+| `adminProcedure` | `is_admin` flag |
+
+Watchlist list currently falls back to `userId ?? 'anonymous'` — convenient for local dev, weak isolation if exposed beyond localhost.
+
+---
+
+## 8. Testing
+
+| Suite | Command | Audit result |
+|-------|---------|--------------|
+| Backend | `cd app/server && npm test` | **504 pass / 0 fail / 1 skip** (after P0 2026-07-18) |
+| Frontend lite | Primary-Rule lint + options pure tests | Present |
+
+Known red: `MacroRegime` commentary tests (`generateMacroCommentary` short/long-term + no macro-trade recommendation).
+
+---
+
+## 9. Schema debt
+
+1. **`strategies` defined twice** in `schema.sql` (legacy `regime_gate/setup/risk_policy` vs `components/unlocked`). SQLite keeps first-created shape depending on migration history — dangerous drift.
+2. Parallel filter tables: `screener_filters` vs `saved_filters`.
+3. Process tables (`trades`, `trade_executions`, `theses`, `emotion_logs`) exist ahead of UI persistence.
+
+---
+
+## 10. LLM / provenance (as built vs design)
+
+| Designed | As built |
+|----------|----------|
+| Full LLMGateway with classifyPayload gate | Tables `llm_providers`, `llm_dispatch_audit`, `llm_summaries`; no complete gateway module under `src/` |
+| Ornith default `is_local=true` | Env-driven URL in docker-compose (`LLM_PROVIDER_URL`) |
+| Sensitive thesis data never leaves host | Thesis content mostly not yet flowing through LLM product paths |
+
+ADR-0006/0008 still govern intent; implementation is incomplete.
+
+---
+
+## 11. Deployment
+
+```bash
+# Local dev (typical)
+cd app/server && npm run dev   # :3001
+cd app && npm run dev          # :3000
+
+# Compose
+docker compose up --build
+```
+
+Secrets: `.env.example` documents session secret, SEC operator email, X cookie fields, Reddit, FRED, LLM URL.
+
+---
+
+## 12. Documentation map (after this audit)
+
+| Doc | Role |
+|-----|------|
+| `docs/FUNCTIONAL_DESIGN.md` | What users can do / pending |
+| `docs/TECH_DESIGN.md` | This file — how it is built |
+| `CONTEXT.md` | Ubiquitous language (glossary only) |
+| `docs/adr/*` | Durable decisions |
+| Obsidian `investor-flow.md` | Session orientation wiki |
+| `.automaton/tasks/*` | Work items; many `complete/` slices are historical |
+
+Deprecated as status sources: root `HANDOFF.md` (2026-06-30 orchestrator snapshot), Automaton WORKFLOW “21 tasks” pool list, slice DECOMPOSITION checkbox state.
@@ -0,0 +1,79 @@
+# ADR-0009: Rate-limit-first data plane
+
+Date: 2026-07-19  
+Status: Accepted
+
+## Context
+
+Every external vendor we use has a short rate limit:
+
+| Source | Typical limit / failure mode |
+|--------|------------------------------|
+| Yahoo Finance (`yahoo-finance2`) | Edge 429 / "Too Many Requests" under concurrent chart+quote+summary |
+| X (bird CLI / cookie session) | HTTP 429 on search / timeline |
+| FRED | API key quota; burst-sensitive |
+| SEC EDGAR | Fair-access pacing (~10 req/s official guidance) |
+| Reddit | OAuth / public endpoint throttles |
+
+The product already had **shared cache + queue dedupe** (ADR-0004) and **stale-while-revalidate**, but request handlers still opened **live vendor calls** (ETF top holdings charts, condition VIX, peers) and the drain loop treated 429 like a normal error with multi-second job backoff. Result: thrash → empty UI panels → worse rate limits.
+
+## Decision
+
+Design the data plane around rate limits as a first-class constraint:
+
+### 1. Request path never stampede
+
+tRPC handlers **prefer local state**:
+
+1. SQLite / `kv_cache` / typed tables (quotes, candles, …)
+2. Stale-while-revalidate via `CacheRepository.get` (schedule background refresh)
+3. **Static / curated fallbacks** for slow-changing composition (e.g. ETF top holdings)
+4. Live vendor only when nothing local exists, with a **short timeout** and graceful empty/stale result
+
+UI clicks must not fan out N charts or N quoteSummary calls.
+
+### 2. One shared queue owns outbound pacing
+
+All background refreshes go through `AdapterQueue`:
+
+- **Per-source min-interval** between fetches (steady-state throttle)
+- **Source-wide cool-down** on rate-limit signals (2 → 5 → 15 → 30 → 60 minutes escalating)
+- While cool-down is active: **skip all jobs for that source**; do not enqueue schedule floods
+- Job exponential backoff remains for *ordinary* failures only
+- Rate-limit hits **do not burn** `MAX_ATTEMPTS` into permanent `failed` without a long cool-down first
+
+### 3. Demand set bounds work
+
+Only symbols in `symbol_demand` (watchlist ∪ holdings) get scheduled yfinance/sec/x refresh. Breadth of interest, not user count, drives cost (ADR-0001 / CONTEXT demand set).
+
+### 4. Stale is better than empty
+
+Showing yesterday’s holdings weights or a 10-minute-old quote with a “cached” affordance beats a blank panel that hammers the vendor. Education product (ADR-0007) does not require millisecond freshness for composition / macro context.
+
+### 5. Observability
+
+Queue health exposes active **source cool-downs** so operators can see “Yahoo paused 4m” instead of a pile of failed jobs.
+
+## Implementation map
+
+| Piece | Location |
+|-------|----------|
+| Policy helpers (detect 429, ladders) | `app/server/src/queue/sourceRatePolicy.ts` |
+| Cool-down + drain skip | `app/server/src/queue/AdapterQueue.ts` |
+| ETF holdings cache + static fallback | `market.sectorHoldings` + `etfHoldingsFallback.ts` |
+| FRED series write-through | `kv_cache` in condition path |
+| Shared market cache | ADR-0004, `CacheRepository` |
+
+## Consequences
+
+- **Positive:** Under vendor stress the UI keeps serving cache/static; queue self-throttles instead of amplifying 429s; operators see cool-downs.
+- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
+- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
+- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
+- **Follow-ups:** Route remaining live Yahoo calls in `market.condition` / peer resolution through the same cache-or-queue pattern; surface cool-downs in admin queue UI.
+
+## Rejected alternatives
+
+- **Retry harder on 429:** Makes thrash worse.
+- **Per-user fetch with no shared cool-down:** Multiplies load; violates ADR-0004.
+- **Block UI until live succeeds:** Timeouts and empty states; bad UX for a terminal-style education app.