feat: dealer flow, mirror portfolio (M21), options convexity, FINRA short interest, alert producers, vendor gate
CI / Test & Type-Check (push) Canceled after 0s
CI / Test & Type-Check (push) Canceled after 0s
Snapshot of in-progress module work across multiple slices: - Dealer Flow: dealerExposureEngine, dealerMapService, dealerMapExplain, dealerMapIntegrity, dealerMapReplay, dealerStudyEngine, hanStyleLevels - Mirror Portfolio (M21): fundRepository, captureIngest, mirrorAlertProducers, fund holdings strip, live book, position capture ingest - Options: BSM, NormalizedOptionSurface types, OptionsChainRouter, ConvexityGate, option legs panel - Alert producers: vixLevel, rotation, thesis, unlock, portfolioRisk, mirror (fund_capture, fund_13f, mirror_diff) - FINRA short interest adapter + queue integration - SEC company tickers adapter + ingest (symbol search index seed) - Vendor gate (rate-limit-first data plane, ADR-0009) - CUSIP registry, reverse 13F refresh, stock float service - LRU cache, portfolio backtest engine - Frontend: dealer-flow, funds, journal, lab, monitor, plan, portfolio, reports, screener, strategies, theses, guided-start, exits, more pages - Volume profile, workspace profile, visibility-aware poll - ADRs 0010 (mirror math not advice), 0011 (symbol search index) - VENDOR_INTEGRATIONS.md, END_USER_TEST.md - .gitignore: exclude DBs, .DS_Store, local config, agent scratch
This commit is contained in:
Investor Flow Build
Binary file not shown.
|
After Width: | Height: | Size: 252 KiB |
@@ -0,0 +1,78 @@
|
||||
# End-user test guide — Investor Flow
|
||||
|
||||
**Status:** Ready for manual end-user testing (local)
|
||||
|
||||
**Product framing:** Investment management workbench (portfolio, risk, monitor, research, decision process). Adaptive workspace density. Educational publisher legal posture (no buy/sell advice).
|
||||
|
||||
---
|
||||
|
||||
## Start the stack
|
||||
|
||||
From repo root (or your usual scripts):
|
||||
|
||||
```bash
|
||||
# Backend (example)
|
||||
cd app/server && npm run dev # or your start command on :3001
|
||||
|
||||
# Frontend
|
||||
cd app && npm run dev # typically :3000
|
||||
```
|
||||
|
||||
Or use `./restart-servers.sh` if that is your operator path.
|
||||
|
||||
Ensure admin can approve new signups if accounts start as `pending_approval`.
|
||||
|
||||
---
|
||||
|
||||
## Suggested test path (new user)
|
||||
|
||||
1. **Sign up / sign in** → Settings
|
||||
2. **Workspace interview** (if not onboarded): experience → goal → horizon → terms
|
||||
- Expect: liquid starter symbols, redirect to **Portfolio**
|
||||
3. **Settings → Workspace** — change density focused ↔ standard ↔ full
|
||||
- Expect: nav items appear/disappear; Overview panels change
|
||||
4. **Portfolio** — add a holding; confirm marks, cost, P&L on the same page (no separate Monitor)
|
||||
5. **Risk** — set equity / peak, refresh posture, run sizing math
|
||||
7. **Get Started** — pick a strategy template (options templates only at full density)
|
||||
8. **Decision plan** (`/plan`) — save a planned/active decision + optional thesis
|
||||
9. **Theses** — create/update status (intact / weakening / broken)
|
||||
10. **Journal** — close a decision with P&L + reflection
|
||||
11. **Research** (`/`) — management strip + symbol overview
|
||||
12. **Market Outlook** — collapsible sections + tooltips
|
||||
13. **Screener** (standard+) — filter watchlist `price > 50`
|
||||
14. **Strategy lab** — backtest if strategy forked and candles cached
|
||||
15. **Reports** — generate HTML note
|
||||
15. **Mobile width** (<768px) — bottom tabs: Research · Portfolio · Risk · Alerts · More
|
||||
16. **Nav** — Research section first in desktop sidebar; no Monitor entry
|
||||
|
||||
---
|
||||
|
||||
## Density checklist
|
||||
|
||||
| Density | Should see |
|
||||
|---------|------------|
|
||||
| focused | Portfolio (with marks), Risk, Symbol, Market, Charts, Plan, Theses, Journal, Alerts; limited Overview panels |
|
||||
| standard | + Institutional, Strategies, Screener, Lab, Reports, Exits |
|
||||
| full | + Filings, Options DD, Goals, advanced strategy templates |
|
||||
|
||||
---
|
||||
|
||||
## Known limits (acceptable for this test pass)
|
||||
|
||||
- Emotion logger UI not fully productized (reflection on journal close covers close notes)
|
||||
- SSE live alert push not required (poll/list works)
|
||||
- Backtest/screener quality depends on cache/watchlist data
|
||||
- Some admin TypeScript noise pre-existed; does not block user flows
|
||||
- Brokerage connect is out of scope (manual holdings only)
|
||||
|
||||
---
|
||||
|
||||
## Pass criteria for this release
|
||||
|
||||
- [ ] New user can finish interview and land on Portfolio
|
||||
- [ ] Holdings appear in Monitor and Risk
|
||||
- [ ] Density change updates nav without re-login
|
||||
- [ ] Decision plan saves and shows in Journal
|
||||
- [ ] Thesis CRUD works
|
||||
- [ ] Mobile 5-tab shell usable for core management
|
||||
- [ ] No “Recommended buy” style language on Guided Start / starters
|
||||
+29
-15
@@ -2,7 +2,7 @@
|
||||
|
||||
**Audit date:** 2026-07-23
|
||||
**Source of truth for status:** this file + code under `app/`. Older slice SPECs and HANDOFF.md are historical.
|
||||
**Product intent:** Beginner-first multi-tenant educational research terminal for conviction investing (not day trading). Legal posture: educational publisher (ADR-0007).
|
||||
**Product intent:** Multi-tenant **investment management** workbench (portfolio, risk, monitor, research context) that helps users become better at market awareness and risk management *while managing*. Not a course app. Legal posture for copy/outputs: educational publisher (ADR-0007) - process and math, never buy/sell advice.
|
||||
|
||||
---
|
||||
|
||||
@@ -28,7 +28,8 @@
|
||||
| Login + session cookie | **Live** | Non-active users rejected at login and `protectedProcedure` |
|
||||
| TOTP 2FA + backup codes | **Live** | Settings page |
|
||||
| GitHub / Google OAuth | **Backend-ready** | Router complete; env-dependent; OAuth users default `active` |
|
||||
| Onboarding wizard | **Live** | Complexity, risk, drawdown, starter watchlist, optional portfolio; queues SEC fetch |
|
||||
| Onboarding wizard | **Live** | Workspace interview (experience, goal, horizon, jargon) → density + risk defaults + starter pack; optional portfolio; queues SEC fetch |
|
||||
| Workspace density | **Live** | `focused` \| `standard` \| `full` filters nav, symbol Overview panels, strategy templates; editable in Settings |
|
||||
| Admin approve/reject | **Live** | `/admin/users` |
|
||||
| Settings / profile | **Live** | `/settings` |
|
||||
|
||||
@@ -38,10 +39,13 @@
|
||||
|---------|-------|--------|-----------|
|
||||
| Symbol overview | `/` | **Live** | `market.snapshot`, company meta, X feed, collapsible filings + options |
|
||||
| Multiple watchlists | sidebar | **Live** | Create/delete/rename/reorder, dropdown selector in sidebar, per-watchlist symbols |
|
||||
| Charts | `/chart-lab` | **Live** | candles, indicators, institutional buy markers |
|
||||
| Charts | `/chart-lab` | **Live** | candles, indicators, institutional buy markers; equity volume-by-price profile (visible range, from OHLC bars) |
|
||||
| Institutional dashboard | `/institutional` | **Live** | dashboard rollup, flow, insider stream, filings; Q-o-Q / M-o-M |
|
||||
| Filings | `/filings` | **Live** | EDGAR index + 13F + Form 4 detail |
|
||||
| Options DD | `/options-dd` | **Live** | chain + greeks (read-only teaching surface) |
|
||||
| Dealer Flow | `/dealer-flow` | **Live (MVP)** | GEX/VEX/OI metric modes + strike profile; delayed Yahoo OK; **integrity gates** (incomplete vs degraded vs complete); IV hygiene; keep last good map; as-of ET; `dealer-map-replay` CLI + Admin → Queue **Dealer map integrity** buttons; Layer-0 + optional L1 |
|
||||
| Study Desk | (on `/dealer-flow`) | **Live** | Educational setups from map; hist rank; log; auto-grade; scorecard; promote to journal draft; CSV export |
|
||||
| Mentor ledger | (Study Desk tab) | **Live (MVP)** | Local import from harvest → confirm drafts → path-match grade + mentor scorecard (not buy signals) |
|
||||
| Market outlook | `/market-outlook` | **Live (Phase 1–3 core)** | Beginner language. Condition strip; stronger/weaker-vs-market map; seasonality + calendar; rank snapshots; auto leadership check; truck/factory; macro notes. True ETF flow/COT still later. |
|
||||
| Social / X feed | on overview | **Live** | DB-backed 30d history + admin X credentials/accounts |
|
||||
| Command palette + nav | shell | **Live** | ⌘K, g-key shortcuts, theme presets |
|
||||
@@ -50,11 +54,14 @@
|
||||
|
||||
| Surface | Route | Status | Gap |
|
||||
|---------|-------|--------|-----|
|
||||
| Trade plan | `/trade-plan` | **UI-local** | Zustand only; not `theses` / `trades` / sizing APIs |
|
||||
| Execution | `/execution` | **UI-local** | Playbook + checklist local; not `trade_executions` |
|
||||
| Trade closure | `/trade-closure` | **UI-local** | localStorage |
|
||||
| Decision plan | `/plan` | **Live** | Server `trades.save`; optional thesis create |
|
||||
| Theses | `/theses` | **Live** | CRUD on `theses` table |
|
||||
| Journal / close | `/journal` | **Live** | `trades.list` / `close` + reflection |
|
||||
| Screener | `/screener` | **Live** | Filter + strategy screen over watchlist |
|
||||
| Strategy lab | `/lab` | **Live** | Backtest equity curve + evaluate latest |
|
||||
| Reports | `/reports` | **Live** | HTML research notes |
|
||||
| Daily focus / goals | `/daily-focus` | **UI-local** | localStorage goals |
|
||||
| Emotion logger | on execution | **Partial** | UI + `emotionLogger` API exist; UI still primarily local path |
|
||||
| Emotion logger | API | **Partial** | API live; journal reflection covers close notes |
|
||||
| Sizing math | `/risk` (calculator) | **Live** | `sizing.compute` + UI cascade (math only, ADR-0007) |
|
||||
| Risk posture (M20) | `/risk` | **Live** | `risk.posture` / `risk.haltStatus` + RiskPosturePanel; halt persisted on drawdown breach |
|
||||
| Derisking suggestions | (no screen) | **Backend-ready** | `derisking.suggest` |
|
||||
@@ -64,10 +71,10 @@
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Filter screener (M15a) | **Backend-ready** | `screener.filter` over watchlist quotes only; **no UI**; not in frontend `api` client |
|
||||
| Strategy screener (M15b) | **Backend-ready** | `screener.strategy`; thin universe; no UI |
|
||||
| Strategy authoring | **Backend-ready** | `strategies.list/create`; no Strategy Lab UI |
|
||||
| Backtest | **Backend-ready** | `backtest.run` / `evaluateLatest`; no equity-curve UI |
|
||||
| Filter screener (M15a) | **Live** | `/screener` + `screener.filter` over watchlist quotes |
|
||||
| Strategy screener (M15b) | **Live** | `/screener` strategy mode when density ≥ standard |
|
||||
| Strategy authoring | **Partial** | presets + fork; custom authoring still light |
|
||||
| Backtest | **Live** | `/lab` equity curve + evaluate latest |
|
||||
| Sector confirmation cross-link | **Stub** | `sectorCrosslink.confirm` with empty universe in practice |
|
||||
| Screener saved filters | **Schema only** | tables exist; incomplete product loop |
|
||||
|
||||
@@ -77,7 +84,7 @@
|
||||
|------------|--------|-------|
|
||||
| Options DD (read-only) | **Live** | M3 teaching panel |
|
||||
| Unlock ladder + payoff API | **Backend-ready** | `optionsConvexity.unlock` / `getPayoff` |
|
||||
| Option legs book + risk contribution | **Live (MVP)** | `portfolio.optionLegs` / add / remove; Risk posture premium-at-risk + CSP cash reserve |
|
||||
| Option legs book + risk contribution | **Live (MVP)** | `OptionLegsPanel` on `/portfolio` + `/risk`; `portfolio.optionLegs` / add / remove; posture folds premium-at-risk + CSP cash reserve into exposure/flags |
|
||||
| Portfolio sleeve workflow (M17 full) | **Missing** | Unlock UX + sleeve risk budget not enforced in product path |
|
||||
|
||||
### 1.6 Macro (M18)
|
||||
@@ -102,7 +109,13 @@
|
||||
| Alert producers: 7 more types (rotation, regime, conviction, thesis, cluster, drawdown, asymmetry) | **Stub** | Engine types exist; producers not wired |
|
||||
| Email delivery (SMTP) | **Live** | Nodemailer, admin SMTP config page, rate limit 1/hr per type/symbol/user |
|
||||
| SSE push | **Missing** | Design called for SSE; poll/list only |
|
||||
| Mobile companion `/mobile` | **Live (thin)** | Auth, holdings glance, unacked alerts — not full M19 job A vision |
|
||||
| Mobile companion `/mobile` | **Redirect** | Redirects to Portfolio; main shell is phone-optimized companion (Job A) |
|
||||
| Mobile shell (iPhone Air) | **Live** | Fixed `dvh` chrome, safe-area insets, bottom tabs (Portfolio · Risk · Research · Alerts · More), slim header + search overlay; no desktop layout flash |
|
||||
| Mobile lists sheet | **Live** | Header list icon opens portfolio + watchlist bottom sheet for symbol switch |
|
||||
| Portfolio on phone | **Live** | Card list + 2×2 snapshot under `md`; wide table from `md` up |
|
||||
| Rotation / institutional on phone | **Live** | Card rows with key horizons; full tables from `md` up with sticky first column |
|
||||
| Mobile polling hygiene | **Live** | Snapshot / alerts / market condition pause while document is hidden |
|
||||
| Portfolio book | **Live** | Unified holdings + live marks + add/remove (`HoldingsBookView`). `/monitor` redirects here. |
|
||||
|
||||
### 1.8 Admin / operator
|
||||
|
||||
@@ -123,7 +136,7 @@
|
||||
|
||||
| Capability | Status |
|
||||
|------------|--------|
|
||||
| HTML research note generator | **Backend-ready** | `reports.generate` scopes include symbol/watchlist/portfolio/rotation/sizing/risk — **no UI**, not in frontend client |
|
||||
| HTML research note generator | **Live** | `/reports` + `reports.generate` |
|
||||
|
||||
---
|
||||
|
||||
@@ -154,7 +167,8 @@ Unchanged from domain model / ADRs:
|
||||
|
||||
1. **Primary Rule (ADR-0007)** — education, not advice; Primary-Rule lint test on curated strings.
|
||||
2. **Analyst Voice (ADR-0005)** — Alfred/Druckenmiller blend where LLM copy is used.
|
||||
3. **P6 plain English / P7 teach mechanics** — UI redesign token system supports this.
|
||||
3. **P6 plain English / P7 teach mechanics** — in-context cockpit labels (not a curriculum product).
|
||||
3b. **Adaptive workspace density** — experience interview sets focused/standard/full; user can change anytime.
|
||||
4. **Local-first multi-tenant (ADR-0001)** + shared cache dedupe (ADR-0004).
|
||||
5. **LLM data provenance (ADR-0006)** — sensitive data stays on local-class providers; full gateway module is thin/incomplete vs original design.
|
||||
6. **Anti-gamification** — no confetti / outcome celebration.
|
||||
|
||||
+30
-6
@@ -62,16 +62,34 @@ All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is
|
||||
|
||||
| Rule | Mechanism |
|
||||
|------|-----------|
|
||||
| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; short timeout if live is unavoidable |
|
||||
| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; **no live Yahoo on tRPC** (queue only) |
|
||||
| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
|
||||
| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
|
||||
| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
|
||||
| Demand-bounded work | Only `symbol_demand` symbols get scheduled refresh |
|
||||
| Observability | `queue.health().sourceCooldowns` |
|
||||
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` (rotation universe) |
|
||||
| TTL-aware tiers | `yfinance-quote` (5m), `yfinance-eod` (6h candles), `yfinance-meta` (daily), `yfinance-holdings` (weekly) |
|
||||
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
|
||||
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
|
||||
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
|
||||
| Observability | `queue.health()`: cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||||
|
||||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge.
|
||||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow.
|
||||
|
||||
Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts`. Live upgrade is best-effort when the source is not cooling down.
|
||||
### Dealer Flow data plane (2026-08)
|
||||
|
||||
- Engine: pure `dealerExposureEngine` on `NormalizedOptionSurface` only (no vendor imports).
|
||||
- Default provider: Yahoo via `composeYFinanceWithOptions` + `OPTIONS_CHAIN_PROVIDER` (default `yfinance`).
|
||||
- Paid switch later: implement SourceFetch for `tradier`/`polygon`, register in `sourceRatePolicy`, set env - engine unchanged.
|
||||
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
|
||||
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
|
||||
- SEC institutional (alert-critical): SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`); `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done); `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. UI toggle: Classic | Dealer (HS).
|
||||
- **Vendor rate-limit enforcement (hard requirement, all sources + future):** process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||||
- LLM: per-user OpenAI-compatible `base_url` + encrypted key + model (`userLlm.*`); not OpenAI-only.
|
||||
- Dealer Flow plain-English notes: in-app `dealerFlowExplainNotes.ts` only (L0/L1). Optional offline scripts harvest X handles and distill into that file **and** write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
|
||||
- Study Desk: pure `dealerStudyEngine` propose/grade; table `dealer_study_setups`; tRPC `dealerStudy.*` (propose, log, list, grade, gradeDue, scorecard). Grades use `price_candles` 1d barrier logic (target before invalidation). Complementary to strategy `backtest.*` - not the same surface.
|
||||
|
||||
Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts` + queued `yfinance:topHoldings:*` refresh.
|
||||
|
||||
---
|
||||
|
||||
@@ -88,7 +106,9 @@ app/server/src/
|
||||
queue/AdapterQueue.ts schedule, pause, retry, source cool-downs (ADR-0009)
|
||||
queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
|
||||
adapters/ YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
|
||||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback, tickerContext
|
||||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback, tickerContext, dealerExposureEngine, dealerMapService, dealerMapExplain, dealerFlowExplainNotes, dealerStudyEngine
|
||||
options/ bsm, types (NormalizedOptionSurface), OptionsChainRouter (paid-ready provider seam)
|
||||
llm/ openaiCompatible client, userLlmEndpoint
|
||||
admin/ operator functions + CLI
|
||||
auth/ totp, oauth, backup codes
|
||||
alerts/AlertEngine.ts
|
||||
@@ -123,6 +143,10 @@ app/server/src/
|
||||
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol |
|
||||
| `portfolio` | holdings, addHolding, removeHolding |
|
||||
| `options` | chain, greeks |
|
||||
| `dealerMap` | get, levels, scenario, velocity, explain — cache-only reads; schedule options chains via queue (ADR-0009) |
|
||||
| `dealerStudy` | propose (hist + optional mentor rank), log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
|
||||
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard — local mentor path-match; claim types map to study hypotheses for Phase-3 blend |
|
||||
| `userLlm` | status, upsertEndpoint, clear, test — per-user OpenAI-compatible base_url + encrypted key + model |
|
||||
| `reports` | generate |
|
||||
| `screener` | filter, strategy |
|
||||
| `strategies` | list, create |
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Adding a vendor integration
|
||||
|
||||
Rate limits are a **first-class product constraint** (ADR-0009). Every external
|
||||
vendor - existing and future - must share the same process-wide gate.
|
||||
|
||||
If a new integration can call the network without `vendorGate`, that is a bug.
|
||||
|
||||
## Required steps
|
||||
|
||||
### 1. Register the family (before any traffic)
|
||||
|
||||
```ts
|
||||
import { registerVendorIntegration } from '../services/vendorGate.ts';
|
||||
|
||||
registerVendorIntegration({
|
||||
family: 'polygon', // unique budget name
|
||||
sourceKinds: ['polygon'], // AdapterQueue source_kind(s)
|
||||
policy: {
|
||||
minIntervalMs: 200, // gap between completed calls
|
||||
maxInflight: 1, // 1 = single-flight
|
||||
drainJobBudget: 2, // max jobs per drain cycle
|
||||
hostPattern: 'polygon\\.io', // docs / optional default allowlist
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Call this at process startup (e.g. next to adapter registration in `index.ts`)
|
||||
or inside the adapter module top-level so importing the adapter registers it.
|
||||
|
||||
### 2. Extend SourceKind if needed
|
||||
|
||||
Add the string to `SourceKind` in `app/server/src/cache/CacheRepository.ts`.
|
||||
|
||||
### 3. Implement the adapter (prefer base class)
|
||||
|
||||
```ts
|
||||
import { VendorSourceAdapter, type FetchResult } from './SourceAdapter.ts';
|
||||
import { vendorFetch } from '../services/vendorGate.ts';
|
||||
|
||||
export class PolygonAdapter extends VendorSourceAdapter {
|
||||
readonly sourceKind = 'polygon' as const;
|
||||
|
||||
protected async fetchOneUngated(key: CacheKey): Promise<FetchResult> {
|
||||
// All HTTP:
|
||||
const resp = await vendorFetch('polygon', url, {
|
||||
hostAllowlist: /polygon\.io/i,
|
||||
});
|
||||
// Library SDKs:
|
||||
// return withVendorGate('polygon', () => client.get(...));
|
||||
// but VendorSourceAdapter already gates the whole job — do not double-gate
|
||||
// unless you make additional calls from a service outside fetchOne.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Helpers:
|
||||
|
||||
| API | Use when |
|
||||
|-----|----------|
|
||||
| `VendorSourceAdapter` | New queue adapter (auto-gates `fetchOne`) |
|
||||
| `defineVendorAdapter({...})` | Tiny one-off adapter without a class |
|
||||
| `withVendorGate(family, fn)` | SDK / subprocess calls |
|
||||
| `vendorFetch(family, url, opts)` | Raw HTTP |
|
||||
| `secHttp` / `secFetch` | SEC only (UA + host rules) |
|
||||
|
||||
### 4. Register with AdapterQueue
|
||||
|
||||
```ts
|
||||
adapters.set('polygon', new PolygonAdapter());
|
||||
// Constructor throws if 'polygon' was not bindSourceKind'd.
|
||||
```
|
||||
|
||||
### 5. Schedule (optional)
|
||||
|
||||
Add default interval in `SCHEDULE_INTERVALS` / `sourceRatePolicy` if the source
|
||||
should refresh on a timer.
|
||||
|
||||
## What AdapterQueue enforces
|
||||
|
||||
- **Construction:** every adapter `source_kind` must have a family binding.
|
||||
- **Drain:** at most `drainJobBudget` jobs per family per cycle.
|
||||
- **429/403:** cools **all** source_kinds in that family + process-wide gate.
|
||||
- **Heal / schedules:** skip family while cooling.
|
||||
|
||||
## What CI enforces
|
||||
|
||||
`src/services/__tests__/vendorHttpGuard.test.ts`:
|
||||
|
||||
- Fails if bare `fetch(` appears under `adapters/`, `services/`, `macro/`, `mirror/`
|
||||
outside allowlisted gate modules.
|
||||
- Covers runtime registration of a fictional future vendor.
|
||||
|
||||
## Checklist for PR review
|
||||
|
||||
- [ ] `registerVendorIntegration` (or family + bind) present
|
||||
- [ ] `SourceKind` updated
|
||||
- [ ] No bare `fetch` / ungated SDK in the adapter
|
||||
- [ ] `hostAllowlist` set for HTTP
|
||||
- [ ] Rate-limit errors surface as thrown messages matching `isRateLimitError`
|
||||
- [ ] Stale/cached path preferred on UI (never stampede from clicks)
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
| Don't | Do |
|
||||
|-------|-----|
|
||||
| `await fetch(vendorUrl)` in an adapter | `vendorFetch(family, url, { hostAllowlist })` |
|
||||
| New `TokenBucket` per adapter | Shared `vendorGate` policy |
|
||||
| Cool only one `source_kind` on 429 | Family cool-down (automatic if gated) |
|
||||
| Call vendor from tRPC handler live | Cache / queue / static fallback |
|
||||
| Skip registration "just for a prototype" | Register with strict policy even for prototypes |
|
||||
@@ -70,7 +70,27 @@ Queue health exposes active **source cool-downs** so operators can see “Yahoo
|
||||
- **Positive:** New features have a clear rule: “cache first, queue refresh, static if needed.”
|
||||
- **Trade-off:** After a cool-down, data may be minutes-to-hours stale until the next successful drain.
|
||||
- **Trade-off:** Static ETF weights drift until the next successful Yahoo refresh (acceptable for education peek panels).
|
||||
- **Follow-ups:** Route remaining live Yahoo calls in `market.condition` / peer resolution through the same cache-or-queue pattern; surface cool-downs in admin queue UI.
|
||||
- **Follow-ups:** Surface per-family cool-downs in admin queue UI (process-wide `vendorGate` + queue_state).
|
||||
|
||||
## Implementation (2026-08 hardening)
|
||||
|
||||
All vendors share `app/server/src/services/vendorGate.ts` with **runtime registration** so future integrations do not require editing a closed union:
|
||||
|
||||
1. `registerVendorIntegration({ family, sourceKinds, policy })` before traffic
|
||||
2. `VendorSourceAdapter` / `defineVendorAdapter` auto-wrap `fetchOne`
|
||||
3. `AdapterQueue` constructor **throws** if any adapter `source_kind` lacks a family
|
||||
4. CI: `vendorHttpGuard.test.ts` fails on bare `fetch(` outside the gate modules
|
||||
5. Handbook: `docs/VENDOR_INTEGRATIONS.md`
|
||||
|
||||
| Family | Call style | Drain budget |
|
||||
|--------|------------|--------------|
|
||||
| yfinance | `withVendorGate` (yahoo-finance2 + options) | 3 |
|
||||
| sec | `secHttp` → vendorGate | 1 |
|
||||
| fred / finra / nasdaq / reddit | `vendorFetch` | 1 |
|
||||
| x | `withVendorGate` around bird CLI | 1 |
|
||||
| *(future)* | `registerVendorIntegration` + `VendorSourceAdapter` | policy.drainJobBudget |
|
||||
|
||||
A 429 cools the **family** (every `source_kind` in that family), not just the one job. Per-request min gaps are process-wide; job min-intervals in AdapterQueue are coarser backup.
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# ADR-0010: Mirror module outputs replication math, not advice (M21)
|
||||
|
||||
The Primary Rule (ADR-0007) forbids buy/sell/hold advice, yet the Mirror Portfolio module (M21) computes exact buy/sell quantities and pushes them as alerts. This is a deliberate, scoped carve-out: the module is a calculator for a **user-stated replication goal**, not a recommendation engine. The user declares the intent ("mirror Alpine Fox"); the app computes the math to satisfy it — the same "math, not instructions" posture as the SizingEngine. Every M21 string that would read "you should buy X because the fund did" is lint-blocked; the actionable alert uses mechanical framing ("to match your mirror target, the delta is N shares").
|
||||
|
||||
**Status:** accepted (2026-07-28, grilling session).
|
||||
|
||||
**Considered Options**
|
||||
- Tracker-only module (facts without mirror math) — rejected: loses the module's core job (what/when/how-much replication).
|
||||
- Mirror without alerts (surface-only) — rejected: the diff stays hidden from the "close tabs" mobile loop the user asked for.
|
||||
- Mirror output as advice-with-disclaimer — rejected: the Primary Rule is a posture, not a disclaimer regime.
|
||||
|
||||
**Consequences**
|
||||
- The actionable mirror alert is the closest Investor Flow comes to instructions; the mechanical-string lint is mandatory on every M21 string.
|
||||
- The module never renders a judgment on the fund's moves (option-2 reconciliation: computed, displayed, no verdicts).
|
||||
- Future features must not let the mirror drift into "you should mirror this fund" framing; the intent must always originate with the user.
|
||||
@@ -0,0 +1,27 @@
|
||||
# ADR-0011: Symbol Search Index — issuer CIK via SEC seed, hybrid add-resolve
|
||||
|
||||
Symbols lacked issuer identity and the app's only autocomplete corpus was the demand set
|
||||
(already-tracked symbols), making new-symbol discovery impossible. We decided: the `symbols`
|
||||
table gains an issuer CIK column, seeded from the SEC's `company_tickers.json` bulk file
|
||||
(~12k US tickers, overwritten in place daily) via a scheduled queue job; autocomplete resolves
|
||||
local-first against this table; adding an unmatched string is a soft-block ("not in the SEC
|
||||
registry — add anyway?") followed by background hydration through the existing adapter queue;
|
||||
and the symbol page surfaces a two-tier "who holds this" strip (tracked funds with weight, then
|
||||
institution count). A symbol carries TWO CIK roles in this app — the issuer CIK (the company the
|
||||
ticker represents, on `symbols`) and the filer CIK (the fund that files 13Fs, on `tracked_funds`)
|
||||
— same identifier space, different edges.
|
||||
|
||||
**Status:** accepted (2026-07-28, grilling session).
|
||||
|
||||
**Considered Options**
|
||||
- Live yfinance search suggestions on keystroke — rejected: violates ADR-0009 (no live vendor call on the request path); rate limits make autocomplete unreliable.
|
||||
- Curated universe seed (S&P 500 / NASDAQ 100) — rejected: narrow, and carries no issuer CIKs.
|
||||
- Strict resolve (block unknown symbols) — rejected: users need fresh/post-file-update tickers; strictness is a question, not a wall.
|
||||
- Loose add (no confirmation) — rejected: silently accepting typos/delisted names pollutes the "known database" the autocomplete relies on.
|
||||
- Full-row upsert on daily seed — rejected: would clobber yfinance-hydrated `ticker_kind` (gates module applicability), `sector`, `industry`, `peers`; merge policy is column-scoped instead.
|
||||
|
||||
**Consequences**
|
||||
- Two ownership domains on `symbols`: SEC fills issuer identity (cik, name, exchange); yfinance hydration owns classification (ticker_kind, sector, industry, peers). No overwrite war.
|
||||
- Non-SEC rows (crypto, some ETFs, indexes) are never purged — the SEC universe is a subset, not the whole.
|
||||
- The daily seed makes the local index essentially complete for US equities, so live-search fallback is rarely needed.
|
||||
- Autocomplete correctness now depends on the scheduled materializer running; a missed day only delays new tickers by one day (stale-while-revalidate).
|
||||
Reference in New Issue
Block a user