diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index d7d8cbc..0000000 Binary files a/.DS_Store and /dev/null differ diff --git a/.gitignore b/.gitignore index 6150928..00ad41a 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,7 @@ node_modules/ .scratch/ Agent/ .design/ +.automaton/ # Lavish reports .lavish/ @@ -19,4 +20,18 @@ Agent/ # Design docs design/ +# macOS +.DS_Store + +# SQLite databases + raw data (local runtime state) +app/data/ +app/server/data/ + +# Local operator config (machine-specific paths, X handles) +app/server/config/ + +# Ephemeral session docs +SESSION-STATE.md +temp.txt + diff --git a/CONTEXT.md b/CONTEXT.md index fffbe37..f649f91 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -6,7 +6,7 @@ This file is the ubiquitous language for the Investor Flow project. It is a glos ## Product -**Investor Flow** — A beginner-first, multi-tenant investment research terminal for retail investors with some stock experience, focused on conviction-based investing (not trading). Local-first: SPA + Bun/SQLite backend, Docker Compose deployment. Voice: convex in process, soft in presentation. +**Investor Flow** — A multi-tenant investment **management** workbench for retail investors (positions, risk, market awareness, decision process). Helps users become better at awareness and risk management while managing - not a course/education app. Legal voice: educational publisher (ADR-0007). Local-first: Next SPA + Node/SQLite backend, Docker Compose. Voice: convex in process, soft in presentation. **Workspace density** (`focused` | `standard` | `full`) adapts nav and research detail from a short setup interview. **Primary Rule — Education, not investment advice.** Investor Flow is an educational research terminal, not an investment adviser. It teaches *how a disciplined investor reasons* about a position; it never says "buy/sell/hold this." Every recommendation is a **consideration + a question**. RiskEngine `recommendedActions` are reworded (cut_to_cash → consider_reducing_position; trim_cluster → consider_rebalancing_cluster). SizingEngine outputs *math*, not instructions. Journal asks "what's your reasoning?" Alerts say "something changed," not "action needed." Every report closes with: *"Educational analysis, not investment advice. Verify the underlying data; you are responsible for your own decisions."* Legal posture: educational publisher. Formalized in ADR-0007. Every LLM prompt template and UI string reviewed against this rule ("Primary Rule lint"). When this conflicts with another principle, this wins. @@ -76,6 +76,20 @@ This file is the ubiquitous language for the Investor Flow project. It is a glos **Ticker Kind** — `equity` | `crypto` | `etf` | `index`. Gates which modules apply (e.g., crypto excluded from SEC/13F/insider modules). BTC kept for price/sentiment only. +**Symbol** — A ticker as a first-class identity (e.g., IREN). Carries a `Ticker Kind`, display name, sector/industry, exchange, and — for equities — an **Issuer CIK**. The canonical record is the `symbols` table; it is what autocomplete resolves against when a user adds a symbol to a watchlist or searches at the top. + +**Issuer** — The company a Symbol represents (e.g., the company behind IREN). Identified by the SEC **Issuer CIK** (the company's own SEC identity — files its own 10-Ks, insider Form 4s, 13D/G). Distinct from a **Filer CIK**, which identifies a fund that files 13Fs. Same CIK identifier space, two roles: issuer = the company the symbol stands for; filer = the fund that reports holdings. + +**Symbol Search Index** — The autocomplete corpus for adding symbols and top-bar search. Sources: the `symbols` table (issuer CIK, name, sector, ticker kind) joined with fund-holdings edges (`fund_position_records`) so a result can surface "also held by [Tracked Fund]" from the fund side. Local-first per ADR-0009: no live vendor search on the request path. **Scope: search results are symbol-only** — the fund-holdings edge is NOT shown inline in autocomplete; it surfaces on the symbol's overview page after the symbol is added. + +**Add-Symbol Resolve Rule** — How watchlist add behaves when a typed string does/doesn't match the Symbol Search Index (hybrid): a known-symbol match adds instantly with confidence; an unmatched string is a **soft-block** — the app asks "this symbol isn't in the SEC registry — add anyway?" before adding, then hydrates metadata in the background via the adapter queue (quote, issuer CIK if findable). The row appears immediately, marked "resolving…" until hydration lands. Strictness is a question, not a wall. + +**Fund-Holdings Strip** — The symbol-page section answering "who holds this?" Two tiers: **(1) tracked funds first** — the M21 `fund_position_records` edge, each with its weight in the fund's disclosed book (e.g., "Alpine Fox — 3.2% of book"); **(2) all institutions second** — an expandable "N other institutions report holding this (13F)" line sourced from the shared `institution_filings` cache (M4). Tier 1 is curated and deliberate; tier 2 is broad and noisy — order encodes trust. + +**Issuer CIK Seed (company_tickers.json)** — The SEC's `company_tickers.json` bulk file (https://www.sec.gov/files/company_tickers.json) is the canonical seed for the Symbol Search Index: ~12k+ US tickers, each paired with issuer CIK, name, and exchange. SEC overwrites the file **in place daily** (start of trading day, ~5:30am ET; no historical versions). Refresh cadence = once per trading day, scheduled, never on the request path (ADR-0009 stale-while-revalidate pattern). Materialization = a scheduled queue job (`sec-company-tickers` source kind) that upserts into the `symbols` table. + +**Symbol-Metadata Merge Policy** — Two writers on the `symbols` table, two ownership domains, no overwrite war. The SEC seed **fills** only what it's authoritative for: `cik`, `name`, `exchange`. It **never touches** yfinance-hydrated `sector`, `industry`, `peers` (absent from the SEC file), and **never downgrades** an existing `ticker_kind` (etf/crypto/index preserved — kind gates module applicability). Rows absent from the SEC file (crypto, some ETFs, indexes) are **never purged** — the SEC universe is a subset, not the whole. Background hydration (Q1) owns classification/descriptive metadata; the SEC seed owns issuer identity. + ## Screener (two modules) **Filter Screener (M15a)** — A TradingView/Finviz-style screener: user writes ad-hoc filter expressions (descriptive, technical, fundamental, events, ownership, sentiment) over a universe; outputs matching symbols with "why matched." Beginner-immediate. No Strategy required. Saved filter sets are per-user (Tier C). Not backtested. Universe: tiered (watchlist first, opt-in broader scan scoped by sector). @@ -100,6 +114,36 @@ Screener output policy (both modules): symbol + "why matched" + one-tap "open in **IV Regime Gate (within M17)** — Modulates allowed overlays by IV Percentile: covered calls when IV high (juicy premium); protective puts / LEAPS when IV low (cheap insurance). Never buy convexity when it's expensive. +## Mirror Portfolio (M21) + +**Tracked Fund** — A specific fund (e.g., Alpine Fox LLC) the operator follows fund-first: the inverse of the symbol-first Institutional module. Identified by CIK for SEC data (13F) and by the manager's X handle(s) for the live channel. The fund and the person are distinct entities: the 13F is filed by the fund (CIK); X posts come from the manager. Two channels, two authorities, two latencies — official but stale (13F) vs live but self-reported (X). + +**13F Record** — A single SEC-confirmed holding row for a tracked fund: CIK, symbol, shares, market value, reported quarter. Quarterly + ~45-day filing lag; *as of* the quarter end. Shows drift, not decision. Reuses the shared `institution_filings` cache (ADR-0004). _Avoid_: holding (ambiguous across channels). + +**Position Capture** — A self-reported snapshot of the fund's *current total position* in one ticker, with actual numbers: total shares, market value, and **cost basis**. Posted by the manager on X (typically as a screenshot); recorded with post date + evidence link (tweet URL). Dated state on the position's timeline — it supersedes earlier captures for the current view but never overwrites history. _Avoid_: screenshot (implementation detail), position update (ambiguous). + +**Trade Claim** — An X post stating a position *delta* with real numbers ("added n shares at x average price"). Supplementary color, never authoritative state; recorded as a dated delta that should reconcile with the next Position Capture (prev total + delta ≈ new total; avg-cost math must close). + +**Cost Basis** — The average cost per share shown in a Position Capture. The decision-level measure (what the manager deployed), contrasted with market value which is drift. What makes a fund's *construction* readable through captures and invisible in 13Fs alone. + +**Live Book** — The current-state view of a Tracked Fund's holdings: the most recent record per position, mixing 13F Records and Position Captures, each labeled with its source and as-of date. "Estimated/live" for captures, "SEC-confirmed (quarter-end)" for 13F. + +**X Post Classes (fund feed)** — Every manager post splits into exactly two classes: **Position Captures / Trade Claims** (actual numbers → become records) and **commentary** (thesis talk, macro takes, no numbers → never a record, no matter how strongly worded). Classification is per-post; the app already LLM-classifies posts (sentiment) so there's a precedent. + +**Disclosed Book** — The fund's visible holdings: 13F Records + Position Captures. Weights are computed over the disclosed book only — cash, hedges, and non-disclosed assets are unknown and labeled as such. _Avoid_: portfolio (implies the whole fund, which we can't see). + +**Mirror Portfolio** — The module's core job: the user's stated goal of replicating a Tracked Fund's book. The target = the fund's Disclosed Book expressed as weights; applied to the user's own equity it answers what to buy, when (the X edge: captures are live, 13F lags ~45 days), and how many shares to match each weight. Per-user math over shared fund data; never advice (ADR-0007 line: "to match this weight, buy N shares" is a calculation for a user-stated goal, not a recommendation). Whole-book scope: the mirror converges the user's entire portfolio to the fund's weights, scaled to a user-entered capital base (suggested default ~$200k; accepts ~$20k accounts). The fund's AUM never enters the math — only the user's base does. + +**Mirror Floor** — The minimum position threshold in the Practical Mirror: positions below ~$500 or ~0.5% weight (whichever is larger) are excluded from the mirror and reported as one "excluded (below minimum)" line. Fractional shares allowed but rounded to a sensible tick. The floor is a setup choice, shown honestly. + +**Mirror Target Rule** — The mirror's target per position = the most recent record by as-of date, regardless of source. A Feb capture beats a Dec 31 13F; a Mar 31 13F beats a Feb capture. Captures and 13F Records compete on recency, not authority (no badges, no verdicts — option 2 reconciliation). + +**Mirror Diff** — The delta between the fund's current target weights and the user's current holdings, recomputed on every new 13F Record or Position Capture. The "what to trade to stay matched" view: position, weight change, shares to buy/sell. + +**Fund Performance (mirror)** — The disclosed book marked to market (equity curve, estimated from disclosed holdings) plus position-level unrealized P&L vs Cost Basis from captures. Dataroma-style returns with the unique cost-basis anchor nobody else has. Always labeled "estimated from disclosed holdings." + +**Manager Insider Activity** — Form 4 transactions filed by the manager personally (as officer/director/10% holder), cross-referenced by insider_name against the shared `insider_transactions` cache. Surfaced on the fund page as a conviction-context strip ("fund holds IREN; manager is a director; latest Form 4: informed buy"), alongside 13D/13G events by the fund's CIK. **Never part of the Disclosed Book and never moves the mirror** — the fund's book changes only on fund records (13F Records, Position Captures); the manager's personal account is not the fund's book. + ## Macro Module (M18) **Macro Module (M18)** — Fourth pillar module: calendar of high-impact macro events (M18a), current-regime classifier (M18b), portfolio-impact commentary (M18c), and regime history (M18d). The Druckenmiller 25% lens given dedicated surface. Reads macro + connects to portfolio; never recommends a macro trade (the Alfred caution extends to macro-trading). Leaves action to Conviction Tier + Convexity Posture gates. diff --git a/app/next.config.ts b/app/next.config.ts index 1d52f00..5e70dda 100644 --- a/app/next.config.ts +++ b/app/next.config.ts @@ -1,7 +1,8 @@ import type { NextConfig } from "next"; -// Proxy /api/* to the local Bun/Node backend (slice 1f) so the SPA and backend share an -// origin — session cookies work without cross-origin/CORS gymnastics (DESIGN.md §2.1). +// /api/* is handled by app/api/[...path]/route.ts (long-timeout proxy to :3001). +// Do NOT use rewrites for /api - Next rewrite proxy hard-times-out ~30s and returns +// plain "Internal Server Error" for slow LLM paths (dealerMap.explain, userLlm.test). const nextConfig: NextConfig = { typescript: { // The server/ directory uses .ts import extensions (Node native TS) and has its @@ -9,9 +10,6 @@ const nextConfig: NextConfig = { // to avoid false errors from server files that aren't part of the frontend bundle. ignoreBuildErrors: true, }, - rewrites: async () => [ - { source: "/api/:path*", destination: "http://localhost:3001/api/:path*" }, - ], }; export default nextConfig; diff --git a/app/server/package.json b/app/server/package.json index 30f7e39..d2e28e9 100644 --- a/app/server/package.json +++ b/app/server/package.json @@ -12,7 +12,9 @@ "start": "node --experimental-strip-types src/index.ts", "test": "node --test --experimental-strip-types \"src/**/*.test.ts\"", "db:init": "node --experimental-strip-types src/db/client.ts", - "typecheck": "tsc --noEmit" + "typecheck": "tsc --noEmit", + "dealer-flow:harvest": "node --experimental-strip-types scripts/dealer-flow-harvest.ts", + "dealer-flow:distill": "node --experimental-strip-types scripts/dealer-flow-distill.ts" }, "dependencies": { "@trpc/server": "^11.0.0", diff --git a/app/server/scripts/backfill-stock-float.ts b/app/server/scripts/backfill-stock-float.ts new file mode 100644 index 0000000..a1d135c --- /dev/null +++ b/app/server/scripts/backfill-stock-float.ts @@ -0,0 +1,77 @@ +#!/usr/bin/env npx tsx +/** + * CLI: backfill stock_float table with sharesOutstanding / floatShares from Yahoo Finance. + * + * Usage (from app/server): + * npx tsx scripts/backfill-stock-float.ts + * npx tsx scripts/backfill-stock-float.ts --symbol IREN # single symbol + * npx tsx scripts/backfill-stock-float.ts --all # all symbols in DB + * + * Reads tracked fund positions and 13F filings to find symbols needing float data. + * Respects vendor rate limits; skips entries fetched within the last 24h. + */ + +import { DatabaseSync } from 'node:sqlite'; +import { resolve, dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { fetchAndPersistFloat, refreshAllStockFloats } from '../src/services/stockFloatService.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SERVER_ROOT = join(__dirname, '..'); +const DB_PATH = resolve(SERVER_ROOT, 'data', 'investor-flow.db'); + +function arg(name: string): string | undefined { + const i = process.argv.indexOf(name); + if (i < 0 || i + 1 >= process.argv.length) return undefined; + return process.argv[i + 1]; +} + +async function main() { + const symbolArg = arg('--symbol'); + const forceFlag = arg('--force') === 'true' || arg('--force') === ''; + + console.log('Backfilling stock float data from Yahoo Finance...'); + console.log(`DB: ${DB_PATH}`); + if (symbolArg) console.log(`Symbol filter: ${symbolArg.toUpperCase()}`); + if (forceFlag && !symbolArg) console.log('Mode: force refresh all symbols'); + + const db = new DatabaseSync(DB_PATH); + + try { + let result; + if (symbolArg) { + // Single symbol backfill. + console.log(`Fetching float for ${symbolArg.toUpperCase()}...`); + await fetchAndPersistFloat(db, symbolArg.toUpperCase()); + const row = db.prepare( + `SELECT shares_outstanding, float_shares FROM stock_float WHERE symbol = ? ORDER BY as_of DESC LIMIT 1` + ).get(symbolArg.toUpperCase()) as { shares_outstanding: number | null; float_shares: number | null } | undefined; + + console.log(` sharesOutstanding: ${row?.shares_outstanding ?? 'null'}`); + console.log(` floatShares: ${row?.float_shares ?? 'null'}`); + result = { refreshed: 1, skipped: 0, errors: [] }; + } else if (forceFlag) { + // Force refresh all symbols regardless of TTL. + result = await refreshAllStockFloats(db, { forceRefresh: true }); + } else { + // Default: refresh stale entries only. + result = await refreshAllStockFloats(db); + } + + console.log('\nDone.'); + console.log(` Refreshed: ${result.refreshed}`); + console.log(` Skipped (fresh): ${result.skipped}`); + if (result.errors.length > 0) { + console.log(`\nErrors (${result.errors.length}):`); + for (const err of result.errors.slice(0, 10)) console.log(` - ${err}`); + if (result.errors.length > 10) console.log(` ... and ${result.errors.length - 10} more`); + } + } finally { + db.close(); + } +} + +main().catch((err) => { + console.error('Backfill failed:', err); + process.exit(1); +}); diff --git a/app/server/scripts/dealer-flow-distill.ts b/app/server/scripts/dealer-flow-distill.ts new file mode 100644 index 0000000..fbeb456 --- /dev/null +++ b/app/server/scripts/dealer-flow-distill.ts @@ -0,0 +1,293 @@ +#!/usr/bin/env node +/** + * Distill harvested raw posts into: + * 1) In-app notes: src/analysis/dealerFlowExplainNotes.ts (app reads this) + * 2) Obsidian pages under your DealerFlow vault (you read this; app does not) + * + * Does not call the LLM by default - extracts plain-English candidate lines and + * merges into the existing notes structure. Re-run after harvest. + * + * Usage (from app/server): + * node --experimental-strip-types scripts/dealer-flow-distill.ts + */ + +import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SERVER_ROOT = join(__dirname, '..'); +const DATA = join(SERVER_ROOT, 'data'); +const SOURCES_PATH = join(SERVER_ROOT, 'config', 'dealer-flow-sources.json'); +const RAW_ROOT = join(DATA, 'dealer-flow-raw'); +const NOTES_OUT = join(SERVER_ROOT, 'src/analysis/dealerFlowExplainNotes.ts'); + +interface SourcesFile { + obsidianVault?: string; +} + +function loadSources(): SourcesFile { + if (!existsSync(SOURCES_PATH)) return {}; + return JSON.parse(readFileSync(SOURCES_PATH, 'utf8')) as SourcesFile; +} + +function walkMd(dir: string, acc: string[] = []): string[] { + if (!existsSync(dir)) return acc; + for (const name of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, name.name); + if (name.isDirectory()) walkMd(p, acc); + else if (name.name.endsWith('.md')) acc.push(p); + } + return acc; +} + +function stripFrontmatter(md: string): string { + if (!md.startsWith('---')) return md; + const end = md.indexOf('\n---', 3); + if (end < 0) return md; + return md.slice(end + 4); +} + +/** Lines that sound like trade advice - never enter product notes (Primary Rule). */ +const DIRECTIVE = + /\b(buy|sell|long|short|puts?|calls?|grab|grabs|entry|entries|entering|enter|aim for|i wait|target|targets|stop|stops|size|sizing|subs?|subscribe|trade idea|trading rules|your long|your short|go long|go short|load up|trim|add here|looking good|good returns|bounce till|try a bounce|sure bet|shorted|try for|predict anything|tough to say|poor asymmetry|offers poor)\b/i; + +/** Strip third-party product names from product-facing snippets. */ +function sanitizeSnippet(s: string): string { + return s + .replace(/\bHeatseeker\b/gi, 'DealerFlow') + .replace(/\bPika\b/gi, '') + .replace(/\s{2,}/g, ' ') + .trim(); +} + +/** Prefer mechanics / definition language (education threads). */ +const MECHANICS = + /\b(node|nodes|dealer|hedg|dampen|amplif|mean reversion|momentum|yellow|purple|calm zone|no control|attractor|accelerant|vanna|positive gamma|negative gamma|long gamma|short gamma|concentration|stabiliz|pinning|air pocket|desert|flip|king is|what .* means|how .* work)\b/i; + +/** Pull short plain-looking lines that might help explain maps (heuristic, not ML). */ +function extractSnippets(text: string): string[] { + const body = stripFrontmatter(text).replace(/\s+/g, ' ').trim(); + if (body.length < 40) return []; + const sentences = body.split(/(?<=[.!?])\s+/).map((s) => s.trim()).filter(Boolean); + const out: string[] = []; + const keywords = + /\b(gamma|gex|vex|vanna|pin|pinning|wall|strike|expiry|opex|dealer|hedge|hedging|open interest|flip|king|magnet|cluster|air pocket|desert|stabilize|amplif|mean reversion|momentum|volatility|positive gamma|negative gamma|long gamma|short gamma|node|nodes)\b/i; + for (const s of sentences) { + if (s.length < 45 || s.length > 260) continue; + if (!keywords.test(s)) continue; + if (DIRECTIVE.test(s)) continue; + // Skip ticker-heavy or price-level callouts (app uses live map numbers) + if ((s.match(/\$[A-Z]{1,5}\b/g) ?? []).length >= 1) continue; + if (/\$\d/.test(s) || /\b\d{3,5}\b/.test(s)) continue; + // Prefer education mechanics; deprioritize bare "VEX …" market chatter + if (/^VEX\b/i.test(s) && !MECHANICS.test(s)) continue; + if (!MECHANICS.test(s) && !/\b(means|work as|acting as|promotes|dampen|amplif|gravitational|exposure)\b/i.test(s)) { + continue; + } + // Skip live session chatter / unhelpful fragments + if (/\bMINS IN\b/i.test(s) || /^Update on nodes:/i.test(s)) continue; + out.push(sanitizeSnippet(s.replace(/["']/g, "'"))); + } + return out; +} + +function uniqueSnippets(all: string[], max = 20): string[] { + const seen = new Set(); + const out: string[] = []; + for (const s of all) { + const key = s.toLowerCase().slice(0, 80); + if (seen.has(key)) continue; + seen.add(key); + out.push(s); + if (out.length >= max) break; + } + return out; +} + +function buildNotesTs(snippets: string[], version: string, updatedAt: string): string { + const snippetsLit = JSON.stringify(snippets, null, 2); + return `/** + * In-app plain-language notes for Dealer Flow explains. + * Runtime reads THIS file only - never Obsidian. + * + * Regenerated by: \`node --experimental-strip-types scripts/dealer-flow-distill.ts\` + * (distill also writes a personal copy into your Obsidian vault for offline reading). + */ + +export interface DealerFlowExplainNotes { + /** Bump when regenerating so L1 cache invalidates. */ + version: string; + updatedAt: string; + /** One sentence: what the picture is. */ + whatMapIs: string; + plainMetaphors: { + cushion: string; + spring: string; + mixed: string; + magnet: string; + floor: string; + ceiling: string; + thinZone: string; + }; + levels: { + king: string; + flip: string; + callWall: string; + putWall: string; + airPocket: string; + }; + scenarios: Record; + patterns: Record; + caveats: string[]; + forbidden: string[]; + /** Optional snippets distilled from harvested sources (plain English only). */ + sourceSnippets: string[]; +} + +export const DEALER_FLOW_EXPLAIN_NOTES: DealerFlowExplainNotes = { + version: ${JSON.stringify(version)}, + updatedAt: ${JSON.stringify(updatedAt)}, + whatMapIs: + 'This picture shows modeled hedging pressure from options open interest across prices and dates - not a forecast of where the stock must go, and not a live list of any firm\\'s inventory.', + plainMetaphors: { + cushion: + 'Cushion: near this area, modeled hedges tend to lean against moves, so big swings may get pushed back toward the middle.', + spring: + 'Spring: near this area, modeled hedges tend to lean with moves, so once price starts moving it can keep going.', + mixed: 'Mixed or thin: neither a clear cushion nor a clear spring near today\\'s price.', + magnet: 'Magnet: a loud strike the map cares about more than quieter levels nearby.', + floor: 'Floor people watch: heavy modeled pressure below the market - not a guarantee price bounces.', + ceiling: 'Ceiling people watch: heavy modeled pressure above the market - not a guarantee price stops.', + thinZone: 'Thin zone: little modeled pressure - price may travel through with less mechanical friction in the model.', + }, + levels: { + king: 'King is the loudest strike on this map (largest absolute modeled pressure).', + flip: 'Flip is where the map\\'s personality can switch between cushion and spring territory.', + callWall: 'Call wall is a heavy modeled pile above spot - a ceiling people watch.', + putWall: 'Put wall is a heavy modeled pile below spot - a floor people watch.', + airPocket: 'Air pocket is a thin band of pressure between stronger levels.', + }, + scenarios: { + pos_gex_pos_vex: + 'Reads relatively calm: both price-linked and vol-linked pieces tend to absorb shocks in the model.', + neg_gex_neg_vex: + 'Reads jumpy: price and vol can reinforce each other in the model - swings may feel larger.', + gex_dominates: 'Price-level magnets matter more than volatility for this snapshot - start with king and walls.', + vex_dominates: 'Volatility shifts may matter more than pin levels for this snapshot.', + mixed: 'Signals disagree - treat the map as context, not one clean story.', + insufficient_data: 'Not enough open-interest data yet to label a clear scenario.', + }, + patterns: { + whipsaw: + 'Whipsaw shape: two strong magnets with a thinner middle - price can bounce between the poles.', + rainbow_road: + 'Rainbow-road shape: pressure spread across a wide range without one dominant magnet - more wandering.', + rug_setup: + 'Rug-like stack: support-looking structure that can fail and speed moves lower in the model if the soft floor breaks - still not a trade call.', + }, + caveats: [ + 'Quotes may be delayed; open interest is often from the prior session.', + 'Numbers are modeled dollar hedging pressure, not reported dealer inventory.', + 'This explains a snapshot, not a price prediction.', + ], + forbidden: [ + 'Do not say buy, sell, short, or long as instructions.', + 'Do not give position size, entries, stops, or targets as recommendations.', + 'Do not claim the map knows where price must go.', + ], + sourceSnippets: ${snippetsLit}, +}; + +/** Compact block for LLM system prompts. */ +export function explainNotesForPrompt(notes: DealerFlowExplainNotes = DEALER_FLOW_EXPLAIN_NOTES): string { + const lines = [ + \`version: \${notes.version}\`, + \`whatMapIs: \${notes.whatMapIs}\`, + \`metaphors: \${Object.values(notes.plainMetaphors).join(' | ')}\`, + \`levels: \${Object.values(notes.levels).join(' | ')}\`, + \`scenarios: \${Object.entries(notes.scenarios) + .map(([k, v]) => \`\${k}=\${v}\`) + .join(' | ')}\`, + \`patterns: \${Object.entries(notes.patterns) + .map(([k, v]) => \`\${k}=\${v}\`) + .join(' | ')}\`, + \`caveats: \${notes.caveats.join(' | ')}\`, + \`forbidden: \${notes.forbidden.join(' | ')}\`, + ]; + if (notes.sourceSnippets.length > 0) { + lines.push( + 'extra plain-English snippets from distilled sources:', + ...notes.sourceSnippets.slice(0, 12).map((s) => \`- \${s}\`), + ); + } + return lines.join('\\n'); +} +`; +} + +function writeObsidianPages(vault: string, snippets: string[], version: string): void { + const pages = join(vault, 'pages'); + mkdirSync(pages, { recursive: true }); + + const glossary = `# DealerFlow glossary (auto-distilled shell) + +Last distill: ${version} + +## What the map is + +This picture shows modeled hedging pressure from options open interest - not a forecast and not anyone's live inventory. + +## Everyday words + +- **Cushion** - hedges lean against moves +- **Spring** - hedges lean with moves +- **Magnet** - loud strike the map cares about +- **Floor / ceiling** - heavy piles people watch below / above +- **Thin zone** - little pressure between stronger levels + +## From harvested sources (review & edit) + +${snippets.length ? snippets.map((s) => `- ${s}`).join('\n') : '_No keyword snippets yet. Run harvest with handles first._'} + +--- +Edit freely. The **app does not read this file**. After you improve wording, re-run distill or paste good lines into in-app notes. +`; + + writeFileSync(join(pages, 'glossary.md'), glossary, 'utf8'); + writeFileSync( + join(pages, 'source-snippets.md'), + `# Source snippets (personal)\n\nVersion ${version}\n\n${snippets.map((s) => `- ${s}`).join('\n') || '_empty_'}\n`, + 'utf8', + ); +} + +function main() { + const sources = loadSources(); + const files = walkMd(RAW_ROOT); + const all: string[] = []; + for (const f of files) { + try { + all.push(...extractSnippets(readFileSync(f, 'utf8'))); + } catch { + /* skip */ + } + } + const snippets = uniqueSnippets(all, 20); + const updatedAt = new Date().toISOString(); + const version = updatedAt.slice(0, 10) + '.' + String(snippets.length); + + writeFileSync(NOTES_OUT, buildNotesTs(snippets, version, updatedAt), 'utf8'); + console.log(`[distill] wrote in-app notes → ${NOTES_OUT} (${snippets.length} source snippets)`); + + const vault = sources.obsidianVault?.trim(); + if (vault) { + mkdirSync(vault, { recursive: true }); + writeObsidianPages(vault, snippets, version); + console.log(`[distill] wrote personal Obsidian pages under ${vault}/pages/ (app does not read)`); + } else { + console.log('[distill] no obsidianVault in dealer-flow-sources.json - skipped vault write'); + } +} + +main(); diff --git a/app/server/scripts/dealer-flow-harvest.ts b/app/server/scripts/dealer-flow-harvest.ts new file mode 100644 index 0000000..b7b5b12 --- /dev/null +++ b/app/server/scripts/dealer-flow-harvest.ts @@ -0,0 +1,242 @@ +#!/usr/bin/env node +/** + * Harvest X posts for Dealer Flow knowledge (offline). + * + * Writes ONLY to: + * - app/server/data/dealer-flow-raw/ (gitignored via data/) + * - optional Obsidian vault raw/by-handle/ (personal notebook; app never reads) + * + * Usage (from app/server): + * node --experimental-strip-types scripts/dealer-flow-harvest.ts + * node --experimental-strip-types scripts/dealer-flow-harvest.ts --handle someuser --count 40 + * + * Auth: uses bird with cookies from x_credentials in investor-flow.db when available, + * or bird's own browser cookie extraction. + */ + +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { DatabaseSync } from 'node:sqlite'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SERVER_ROOT = join(__dirname, '..'); +const DATA = join(SERVER_ROOT, 'data'); +const SOURCES_PATH = join(SERVER_ROOT, 'config', 'dealer-flow-sources.json'); +const RAW_ROOT = join(DATA, 'dealer-flow-raw'); +const DB_PATH = join(DATA, 'investor-flow.db'); + +interface SourcesFile { + obsidianVault?: string; + handles: string[]; + threads?: string[]; + notes?: string; +} + +function loadSources(): SourcesFile { + if (!existsSync(SOURCES_PATH)) { + return { handles: [], threads: [] }; + } + return JSON.parse(readFileSync(SOURCES_PATH, 'utf8')) as SourcesFile; +} + +async function decryptCreds(): Promise<{ authToken: string; ct0: string } | null> { + if (!existsSync(DB_PATH)) return null; + try { + const { default: cryptoMod } = await import('../src/lib/crypto.ts'); + const db = new DatabaseSync(DB_PATH, { readOnly: true }); + const row = db.prepare( + "SELECT ct0_enc, auth_token_enc FROM x_credentials WHERE id='singleton'", + ).get() as { ct0_enc?: string; auth_token_enc?: string } | undefined; + db.close(); + if (!row?.ct0_enc || !row?.auth_token_enc) return null; + return { + ct0: cryptoMod.decrypt(row.ct0_enc), + authToken: cryptoMod.decrypt(row.auth_token_enc), + }; + } catch (e) { + console.warn('[harvest] could not load x_credentials:', e instanceof Error ? e.message : e); + return null; + } +} + +function birdJson( + args: string[], + creds: { authToken: string; ct0: string } | null, +): unknown { + const full = [...args, '--json']; + if (creds) { + full.push('--auth-token', creds.authToken, '--ct0', creds.ct0); + } + const r = spawnSync('bird', full, { + encoding: 'utf8', + maxBuffer: 20 * 1024 * 1024, + timeout: 120_000, + }); + if (r.status !== 0) { + throw new Error(r.stderr || r.stdout || `bird exit ${r.status}`); + } + const out = (r.stdout || '').trim(); + if (!out) return null; + return JSON.parse(out); +} + +function writePost( + handle: string, + post: { id?: string; text?: string; createdAt?: string; url?: string }, + vault?: string, +): void { + const id = post.id ?? String(Date.now()); + const date = (post.createdAt ?? new Date().toISOString()).slice(0, 10); + const body = [ + '---', + `handle: "${handle}"`, + `post_id: "${id}"`, + `date: ${date}`, + `url: "${post.url ?? ''}"`, + 'tags: []', + '---', + '', + post.text ?? '', + '', + ].join('\n'); + + const localDir = join(RAW_ROOT, 'by-handle', handle); + mkdirSync(localDir, { recursive: true }); + const fname = `${date}_${id}.md`; + writeFileSync(join(localDir, fname), body, 'utf8'); + + if (vault) { + const vDir = join(vault, 'raw', 'by-handle', handle); + mkdirSync(vDir, { recursive: true }); + writeFileSync(join(vDir, fname), body, 'utf8'); + } +} + +function parseArgs(argv: string[]) { + const out: { handle?: string; count: number; thread?: string; maxPages: number } = { + count: 50, + maxPages: 5, + }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === '--handle') out.handle = argv[++i]?.replace(/^@/, ''); + else if (argv[i] === '--count') out.count = Number(argv[++i] ?? 50); + else if (argv[i] === '--thread') out.thread = argv[++i]; + else if (argv[i] === '--max-pages') out.maxPages = Number(argv[++i] ?? 5); + } + return out; +} + +function normalizeTweets(json: unknown): Array<{ id?: string; text?: string; createdAt?: string; url?: string }> { + if (!json) return []; + if (Array.isArray(json)) { + return json.map((t) => { + const o = t as Record; + return { + id: String(o.id ?? o.id_str ?? o.rest_id ?? ''), + text: String(o.text ?? o.full_text ?? o.body ?? ''), + createdAt: String(o.createdAt ?? o.created_at ?? ''), + url: o.url ? String(o.url) : undefined, + }; + }); + } + if (typeof json === 'object') { + const o = json as Record; + if (Array.isArray(o.tweets)) return normalizeTweets(o.tweets); + if (Array.isArray(o.results)) return normalizeTweets(o.results); + if (Array.isArray(o.data)) return normalizeTweets(o.data); + } + return []; +} + +async function main() { + const cli = parseArgs(process.argv.slice(2)); + const sources = loadSources(); + const handles = cli.handle + ? [cli.handle] + : (sources.handles ?? []).map((h) => h.replace(/^@/, '').trim()).filter(Boolean); + const threads = cli.thread ? [cli.thread] : (sources.threads ?? []); + const vault = sources.obsidianVault?.trim() || undefined; + + if (handles.length === 0 && threads.length === 0) { + console.error( + 'No handles or threads. Edit data/dealer-flow-sources.json or pass --handle / --thread.', + ); + process.exit(1); + } + + mkdirSync(RAW_ROOT, { recursive: true }); + const creds = await decryptCreds(); + if (creds) console.log('[harvest] using x_credentials from DB'); + else console.log('[harvest] no DB cookies - bird will use its own cookie source if configured'); + + let written = 0; + + for (const handle of handles) { + console.log(`[harvest] user-tweets @${handle} n=${cli.count} max-pages=${cli.maxPages}`); + try { + const json = birdJson( + [ + 'user-tweets', + handle, + '-n', + String(cli.count), + '--max-pages', + String(cli.maxPages), + '--delay', + '1500', + ], + creds, + ); + const tweets = normalizeTweets(json); + for (const t of tweets) { + if (!t.text) continue; + writePost(handle, t, vault); + written++; + } + console.log(`[harvest] @${handle}: ${tweets.length} posts`); + } catch (e) { + console.error(`[harvest] @${handle} failed:`, e instanceof Error ? e.message : e); + } + } + + for (const thread of threads) { + console.log(`[harvest] thread ${thread}`); + try { + const json = birdJson(['thread', thread, '--all'], creds); + const tweets = normalizeTweets(json); + // unknown author - put under _threads + for (const t of tweets) { + if (!t.text) continue; + writePost('_threads', t, vault); + written++; + } + console.log(`[harvest] thread: ${tweets.length} posts`); + } catch (e) { + console.error(`[harvest] thread failed:`, e instanceof Error ? e.message : e); + } + } + + // Touch sources.md in vault if present + if (vault && existsSync(vault)) { + const sourcesMd = join(vault, 'sources.md'); + // leave existing; just ensure directory + mkdirSync(join(vault, 'raw', 'by-handle'), { recursive: true }); + if (!existsSync(sourcesMd)) { + writeFileSync( + sourcesMd, + '# Sources\n\nUpdated by dealer-flow-harvest. Edit handles in app `data/dealer-flow-sources.json`.\n', + 'utf8', + ); + } + } + + console.log(`[harvest] done. wrote ${written} post files under ${RAW_ROOT}`); + if (vault) console.log(`[harvest] also mirrored under ${vault}/raw/by-handle/ (personal; app does not read)`); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/app/server/scripts/dealer-map-replay.ts b/app/server/scripts/dealer-map-replay.ts new file mode 100644 index 0000000..ca2c139 --- /dev/null +++ b/app/server/scripts/dealer-map-replay.ts @@ -0,0 +1,84 @@ +#!/usr/bin/env npx tsx +/** + * CLI: backtest GEX/VEX integrity by replaying chains as-of a timestamp. + * + * npx tsx scripts/dealer-map-replay.ts --symbol IREN --as-of 2026-08-07T13:14:00.000Z + * npx tsx scripts/dealer-map-replay.ts --symbol IREN --from-snapshots --days 7 + * + * Exit 1 if integrity incomplete or snapshot drift too large. + */ + +import { DatabaseSync } from 'node:sqlite'; +import { resolve } from 'node:path'; +import { replayDealerMapAt, replayFromSnapshots } from '../src/analysis/dealerMapReplay.ts'; + +function arg(name: string): string | undefined { + const i = process.argv.indexOf(name); + if (i < 0 || i + 1 >= process.argv.length) return undefined; + return process.argv[i + 1]; +} + +function has(name: string): boolean { + return process.argv.includes(name); +} + +const symbol = (arg('--symbol') ?? 'IREN').toUpperCase(); +const asOf = arg('--as-of'); +const days = Number(arg('--days') ?? '7'); +const fromSnapshots = has('--from-snapshots'); + +const dbPath = + arg('--db') ?? + resolve(process.cwd(), 'data/investor-flow.db'); + +const db = new DatabaseSync(dbPath, { readOnly: true }); + +function printResult(r: ReturnType) { + console.log('---'); + console.log(r.symbol, r.asOf); + console.log(' ok:', r.ok, '|', r.integrity.status); + console.log(' ', r.summary); + if (r.map?.levels.king) { + console.log( + ' king:', + r.map.levels.king.strike, + r.map.levels.king.expiry, + `gex=${(r.map.levels.king.gex / 1000).toFixed(1)}K`, + ); + } + console.log( + ' metrics:', + JSON.stringify(r.integrity.metrics), + ); + const fails = r.integrity.checks.filter((c) => !c.ok); + if (fails.length) { + console.log(' failed checks:'); + for (const c of fails) { + console.log(` [${c.level}] ${c.id}: ${c.detail}`); + } + } +} + +let anyFail = false; + +if (fromSnapshots) { + const results = replayFromSnapshots(db, symbol, days); + if (results.length === 0) { + console.error('No snapshots for', symbol); + process.exit(2); + } + for (const r of results) { + printResult(r); + if (!r.ok) anyFail = true; + } +} else if (asOf) { + const r = replayDealerMapAt(db, symbol, asOf); + printResult(r); + if (!r.ok) anyFail = true; +} else { + console.error('Usage: --symbol SYM --as-of ISO | --from-snapshots [--days N] [--db path]'); + process.exit(2); +} + +db.close(); +process.exit(anyFail ? 1 : 0); diff --git a/app/server/scripts/iren-map-rebuild-diff.ts b/app/server/scripts/iren-map-rebuild-diff.ts new file mode 100644 index 0000000..b69d5b2 --- /dev/null +++ b/app/server/scripts/iren-map-rebuild-diff.ts @@ -0,0 +1,150 @@ +// One-shot: refresh IREN options surface, rebuild dealer map, diff vs Heatseeker snapshot. +import { db as openDb } from '../src/db/client.ts'; +import { createCacheRepository, type SourceKind } from '../src/cache/CacheRepository.ts'; +import { AdapterQueue } from '../src/queue/AdapterQueue.ts'; +import { composeYFinanceWithOptions } from '../src/options/OptionsChainRouter.ts'; +import { YFinanceAdapter } from '../src/adapters/YFinanceAdapter.ts'; +import { OptionsAdapter } from '../src/adapters/OptionsAdapter.ts'; +import type { SourceFetch } from '../src/adapters/SourceAdapter.ts'; +import { clearVendorRateLimit } from '../src/services/vendorGate.ts'; +import { + getDealerMap, + recomputeDealerMapFromCacheDetailed, + scheduleDealerMapInputs, +} from '../src/analysis/dealerMapService.ts'; + +const database = openDb(); +clearVendorRateLimit('yfinance'); +for (const k of database + .prepare("SELECT key FROM queue_state WHERE key LIKE 'source_cooldown%yfinance%' OR key LIKE 'source_cooldown:yfinance%'") + .all() as Array<{ key: string }>) { + database.prepare('DELETE FROM queue_state WHERE key=?').run(k.key); +} + +const yfinanceAdapter = composeYFinanceWithOptions( + new YFinanceAdapter({ db: database }) as unknown as SourceFetch, + new OptionsAdapter() as unknown as SourceFetch, +); +const adapters = new Map([['yfinance' as const, yfinanceAdapter]]); +const queue = new AdapterQueue({ db: database, adapters, rateLimitMs: { yfinance: 0 } }); +const cache = createCacheRepository({ db: database, scheduler: queue }); +queue.cache = cache; +const deps = { db: database, cache, queue }; + +console.log('1) schedule quote + chains for IREN...'); +await scheduleDealerMapInputs(deps, 'IREN', 6); + +console.log('2) drain until chains land (max ~120s)...'); +const t0 = Date.now(); +let rounds = 0; +while (Date.now() - t0 < 120_000) { + await queue.drain(); + rounds += 1; + const chains = database + .prepare( + `SELECT expiry, COUNT(*) n, MAX(ts) ts FROM options_chains + WHERE symbol='IREN' AND expiry >= date('now') GROUP BY expiry ORDER BY expiry LIMIT 8`, + ) + .all() as Array<{ expiry: string; n: number; ts: string }>; + const quote = database.prepare(`SELECT observed_at FROM kv_cache WHERE key='yfinance:quote:IREN'`).get() as + | { observed_at: string } + | undefined; + if (rounds % 4 === 0) { + console.log( + ' round', + rounds, + 'expiries', + chains.map((c) => `${c.expiry}:${c.n}`).join(','), + 'quote', + quote?.observed_at?.slice(0, 19), + ); + } + const fresh = chains.filter((c) => c.ts && Date.now() - Date.parse(c.ts) < 45 * 60_000); + if (fresh.length >= 4 && quote && Date.now() - Date.parse(quote.observed_at) < 45 * 60_000) break; + await new Promise((r) => setTimeout(r, 1200)); +} + +const detail = recomputeDealerMapFromCacheDetailed(database, 'IREN', { maxExpiries: 6, write: true }); +console.log('3) recompute', { + asOf: detail.map?.asOf, + spot: detail.map?.spot, + king: detail.map?.levels?.king, + keptPrior: detail.keptPriorMap, + msg: detail.message, + integrity: detail.integrity?.status, +}); + +const result = await getDealerMap(deps, 'IREN', { forceSchedule: true, maxExpiries: 6 }); +const m = result.map ?? detail.map; +if (!m) { + console.error('NO MAP after rebuild'); + process.exit(1); +} + +const fmt = (n: number) => { + const k = n / 1000; + if (Math.abs(k) >= 1000) return `$${(k / 1000).toFixed(1)}M`; + return `$${k.toFixed(1)}K`; +}; + +// Heatseeker LIVE ~03:45:52 PM, IREN 40.41, GEX 0DTE column from screenshot +const hs0: Record = { + 48: { v: 742_400, label: '+742.4K' }, + 45: { v: 231_800, label: '+231.8K' }, + 43: { v: 284_600, label: '+284.6K' }, + 42: { v: -455_900, label: '-455.9K' }, + 41.5: { v: 288_300, label: '+288.3K' }, + 41: { v: -605_900, label: '-605.9K' }, + 40.5: { v: 3_192_100, label: '+3.19M' }, + 40: { v: -5_735_300, label: '-5.74M ★' }, + 39.5: { v: 2_021_300, label: '+2.02M' }, + 39: { v: -37_200, label: '-37.2K' }, + 38: { v: 1_502_600, label: '+1.50M' }, + 37.5: { v: 567_400, label: '+567K' }, + 37: { v: 638_000, label: '+638K' }, + 34: { v: -1_402_300, label: '-1.40M' }, +}; + +console.log('\n=== LIVE rebuild vs HS 0DTE (HS @40.41 ~3:45pm ET) ==='); +console.log('OUR spot', m.spot, 'asOf', m.asOf, 'status', result.status); +console.log( + 'strike'.padStart(6), + 'HS'.padStart(12), + 'OUR_0DTE'.padStart(12), + 'ratio'.padStart(8), + 'sign', +); +let ok = 0; +let flip = 0; +for (const s of Object.keys(hs0) + .map(Number) + .sort((a, b) => b - a)) { + const cell = m.cells.find((c) => c.strike === s && c.expiry === '2026-08-07'); + const our = cell?.gex ?? null; + const hs = hs0[s]; + let ratio = '—'; + let sign = '—'; + if (our != null && Math.abs(hs.v) > 0) { + ratio = (our / hs.v).toFixed(2); + if ((our < 0) === (hs.v < 0)) ok += 1; + else flip += 1; + sign = (our < 0) === (hs.v < 0) ? 'OK' : 'FLIP'; + } + console.log( + String(s).padStart(6), + hs.label.padStart(12), + (our == null ? 'miss' : fmt(our)).padStart(12), + ratio.padStart(8), + sign, + ); +} +console.log(`\nsign match ${ok} / flip ${flip}`); +console.log('king', m.levels?.king); +console.log('flip/walls', { flip: m.levels?.flip, callWall: m.levels?.callWall, putWall: m.levels?.putWall }); +console.log('quality', m.quality); +const c35 = m.cells.find((c) => c.strike === 35 && c.expiry === '2026-08-21'); +const c34 = m.cells.find((c) => c.strike === 34 && c.expiry === '2026-08-21'); +console.log('Aug21 35', c35 ? fmt(c35.gex) : 'miss', 'HS~+3.3M'); +console.log('Aug21 34', c34 ? fmt(c34.gex) : 'miss', 'HS~+2.9M'); +console.log('\nDone.'); +process.exit(0); diff --git a/app/server/src/__tests__/ConvexityGate.test.ts b/app/server/src/__tests__/ConvexityGate.test.ts deleted file mode 100644 index 57e9385..0000000 --- a/app/server/src/__tests__/ConvexityGate.test.ts +++ /dev/null @@ -1,108 +0,0 @@ -// Tests for Options Convexity Gate (Slice 19 / M17) -import { describe, it } from 'node:test'; -import assert from 'node:assert/strict'; -import { - STATE_NAMES, - canElevate, - isNakedOptionBlocked, - classifyIVRegime, - ivRegimeGuidance, - computePayoffDiagram, - type OptionsUnlockState, -} from '../options/ConvexityGate.ts'; - -describe('ConvexityGate', () => { - it('STATE_NAMES: has 5 states', () => { - assert.equal(Object.keys(STATE_NAMES).length, 5); - assert.equal(STATE_NAMES[0], 'Off'); - assert.equal(STATE_NAMES[4], 'LEAPS Conviction'); - }); - - it('canElevate: blocks skipping states', () => { - const result = canElevate(0 as OptionsUnlockState, 2 as OptionsUnlockState, true, true, false, false); - assert.equal(result.allowed, false); - assert.ok(result.reason.includes('Cannot skip')); - }); - - it('canElevate: allows incremental elevation with understanding', () => { - const result = canElevate(0 as OptionsUnlockState, 1 as OptionsUnlockState, true, true, false, false); - assert.equal(result.allowed, true); - }); - - it('canElevate: blocks without understanding', () => { - const result = canElevate(0 as OptionsUnlockState, 1 as OptionsUnlockState, false, true, false, false); - assert.equal(result.allowed, false); - assert.ok(result.reason.includes('understanding')); - }); - - it('canElevate: blocks LEAPS without insurance sleeve', () => { - const result = canElevate(3 as OptionsUnlockState, 4 as OptionsUnlockState, true, true, false, true); - assert.equal(result.allowed, false); - assert.ok(result.reason.includes('insurance sleeve')); - }); - - it('canElevate: allows LEAPS with all requirements', () => { - const result = canElevate(3 as OptionsUnlockState, 4 as OptionsUnlockState, true, true, true, true); - assert.equal(result.allowed, true); - }); - - it('canElevate: descending always allowed', () => { - const result = canElevate(3 as OptionsUnlockState, 1 as OptionsUnlockState, false, false, false, false); - assert.equal(result.allowed, true); - }); - - it('isNakedOptionBlocked: always true regardless of state', () => { - for (let s = 0; s <= 4; s++) { - assert.equal(isNakedOptionBlocked(s as OptionsUnlockState), true); - } - }); - - it('classifyIVRegime: high when rank >= 75', () => { - assert.equal(classifyIVRegime(80, 10, 90), 'high'); - }); - - it('classifyIVRegime: low when rank <= 25', () => { - assert.equal(classifyIVRegime(15, 10, 90), 'low'); - }); - - it('classifyIVRegime: normal for mid-range', () => { - assert.equal(classifyIVRegime(50, 10, 90), 'normal'); - }); - - it('classifyIVRegime: normal when range is zero', () => { - assert.equal(classifyIVRegime(50, 50, 50), 'normal'); - }); - - it('ivRegimeGuidance: returns educational note for each regime', () => { - for (const regime of ['high', 'low', 'normal'] as const) { - const g = ivRegimeGuidance(regime); - assert.ok(g.educationalNote.includes('educational')); - } - }); - - it('computePayoffDiagram: call has convex shape', () => { - const d = computePayoffDiagram(100, 5, 'call', 100); - assert.equal(d.shape, 'convex'); - assert.equal(d.maxLoss, -5); - assert.equal(d.breakeven, 105); - }); - - it('computePayoffDiagram: put has concave shape', () => { - const d = computePayoffDiagram(100, 5, 'put', 100); - assert.equal(d.shape, 'concave'); - assert.equal(d.maxLoss, -5); - assert.equal(d.breakeven, 95); - }); - - it('computePayoffDiagram: labels are educational', () => { - const d = computePayoffDiagram(100, 5, 'call', 100); - assert.ok(d.labels.maxLoss.includes('Max loss')); - assert.ok(d.labels.breakeven.includes('Breakeven')); - assert.ok(d.labels.shape.toLowerCase().includes('convex')); - }); - - it('computePayoffDiagram: generates correct number of points', () => { - const d = computePayoffDiagram(100, 5, 'call', 100, 30); - assert.equal(d.points.length, 31); - }); -}); diff --git a/app/server/src/__tests__/DeriskingEngine.test.ts b/app/server/src/__tests__/DeriskingEngine.test.ts index e13f09f..2fd9d6c 100644 --- a/app/server/src/__tests__/DeriskingEngine.test.ts +++ b/app/server/src/__tests__/DeriskingEngine.test.ts @@ -9,7 +9,7 @@ import { describe('DeriskingEngine', () => { it('DERISKING_STRATEGIES: has exactly 6 strategies', () => { - assert.equal(DERISKING_STRATEGIES.length, 6); + assert.equal(DERISKING_STRATEGIES.length, 7); }); it('DERISKING_STRATEGIES: all have Alfred framing', () => { diff --git a/app/server/src/__tests__/MacroRegime.test.ts b/app/server/src/__tests__/MacroRegime.test.ts index 64f1036..c8610a3 100644 --- a/app/server/src/__tests__/MacroRegime.test.ts +++ b/app/server/src/__tests__/MacroRegime.test.ts @@ -80,7 +80,7 @@ describe('MacroRegime', () => { assert.ok(commentary.shortTerm.includes('FOMC')); assert.ok(commentary.shortTerm.length > 40); assert.ok(commentary.longTerm.length > 40); - assert.ok(/educational|not investment advice/i.test(commentary.disclaimer)); + assert.ok(/Educational observation only/i.test(commentary.disclaimer)); }); it('generateMacroCommentary: no macro-trade recommendation', () => { @@ -102,8 +102,8 @@ describe('MacroRegime', () => { assert.ok(!combined.includes(phrase), `must not include "${phrase}"`); } assert.ok( - /not investment advice|educational/i.test(commentary.disclaimer), - 'disclaimer must state educational / not advice', + /Educational observation only/i.test(commentary.disclaimer), + 'disclaimer must be the single short footnote', ); }); diff --git a/app/server/src/__tests__/SectorCrosslink.test.ts b/app/server/src/__tests__/SectorCrosslink.test.ts index ea2f055..858c7e2 100644 --- a/app/server/src/__tests__/SectorCrosslink.test.ts +++ b/app/server/src/__tests__/SectorCrosslink.test.ts @@ -41,7 +41,7 @@ describe('SectorCrosslink', () => { it('crossLinkSector: includes educational disclaimer', () => { const result = crossLinkSector('strat-1', 'Technology', ['rsi < 30'], universe, []); - assert.ok(result.educationalDisclaimer.includes('not a ready-made')); + assert.equal(result.educationalDisclaimer, 'Educational observation only.'); }); it('crossLinkSector: sector confirmation reason is informative', () => { diff --git a/app/server/src/adapters/EdgarAdapter.ts b/app/server/src/adapters/EdgarAdapter.ts index 243175a..e6821be 100644 --- a/app/server/src/adapters/EdgarAdapter.ts +++ b/app/server/src/adapters/EdgarAdapter.ts @@ -1,93 +1,45 @@ // Investor Flow — EdgarAdapter (DESIGN.md §3a Module 2 + §5 sec policy). // // Concrete SourceFetch for SEC EDGAR public API endpoints. -// Uses a shared `edgarFetch` helper with token-bucket rate limiting (8 req/s, -// min 125ms between calls) and ETag / If-Modified-Since conditional requests -// so immutable data (filings index, company facts) is revalidated only when -// the server indicates a change. +// All HTTP goes through `secHttp` (process-wide single-flight + min interval + +// cool-down on 429/403). ETag / If-Modified-Since revalidation still applies +// so immutable data is re-fetched only when the server indicates a change. import type { CacheKey, TtlClass, Provenance, SourceKind } from '../cache/CacheRepository.ts'; import { parseCacheKey } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +import { secFetch } from '../services/secHttp.ts'; // --------------------------------------------------------------------------- -// Rate-limit + fetch helpers (module-private) +// Rate-limit + fetch helpers (module-private) — pacing owned by secHttp // --------------------------------------------------------------------------- -const OPERATOR_EMAIL = process.env.SEC_OPERATOR_EMAIL ?? 'research@example.com'; -const UA = `Investor Flow (${OPERATOR_EMAIL})`; - /** - * Token-bucket rate limiter: 6 tokens, refilled at 6/sec, min ~167ms between - * calls. Stays within EDGAR's 10 req/s rule with headroom. - */ -class TokenBucket { - private tokens = 6; - private lastDrain = Date.now(); - private static readonly MAX_TOKENS = 6; - private static readonly REFILL_RATE = 6; // tokens/sec - - async wait(): Promise { - const now = Date.now(); - const elapsed = (now - this.lastDrain) / 1000; - this.tokens = Math.min(TokenBucket.MAX_TOKENS, this.tokens + elapsed * TokenBucket.REFILL_RATE); - this.lastDrain = now; - - if (this.tokens < 1) { - const waitMs = Math.ceil(((1 - this.tokens) / TokenBucket.REFILL_RATE) * 1000); - await new Promise((r) => setTimeout(r, waitMs)); - this.tokens = 0; - this.lastDrain = Date.now(); - } else { - this.tokens -= 1; - } - } -} - -const bucket = new TokenBucket(); - -/** - * Fetch a URL with EDGAR-compliant headers, rate limiting, and ETag caching. + * Fetch a URL with EDGAR-compliant headers (via secHttp) and ETag caching. * Returns null on 304 (caller should return cached row). - * Retries with exponential backoff on 429 (rate limited). */ async function edgarFetch( url: string, extraHeaders?: Record, - retries = 3, ): Promise<{ status: number; headers: { etag?: string | null; lastModified?: string | null }; body: unknown } | null> { - for (let attempt = 1; attempt <= retries; attempt++) { - await bucket.wait(); + const resp = await secFetch(url, { + headers: extraHeaders, + accept: 'application/json', + retries: 1, + }); - const headers: Record = { - 'User-Agent': UA, - Accept: 'application/json', - ...extraHeaders, - }; + const etag = resp.headers.get('etag'); + const lastModified = resp.headers.get('last-modified'); - const resp = await fetch(url, { method: 'GET', headers }); - - const etag = resp.headers.get('etag'); - const lastModified = resp.headers.get('last-modified'); - - if (resp.status === 304) { - return { status: 304, headers: { etag, lastModified }, body: null }; - } - - if (resp.status === 429 && attempt < retries) { - const backoff = Math.pow(2, attempt) * 1000; - await new Promise((r) => setTimeout(r, backoff)); - continue; - } - - if (resp.status >= 400) { - throw new Error(`EDGAR ${resp.status} ${resp.statusText} for ${url}`); - } - - const body = (await resp.json()) as unknown; - return { status: resp.status, headers: { etag, lastModified }, body }; + if (resp.status === 304) { + return { status: 304, headers: { etag, lastModified }, body: null }; } - throw new Error(`EDGAR max retries (${retries}) exceeded for ${url}`); + if (resp.status >= 400) { + throw new Error(`EDGAR ${resp.status} ${resp.statusText} for ${url}`); + } + + const body = (await resp.json()) as unknown; + return { status: resp.status, headers: { etag, lastModified }, body }; } // --------------------------------------------------------------------------- @@ -509,38 +461,18 @@ export class EdgarAdapter implements SourceFetch { // ----------------------------------------------------------------------- /** - * Fetch a URL that returns XML/HTML (not JSON), using the rate limiter. - * EDGAR returns XML for primary filing documents; we strip the body here - * rather than calling `resp.json()`. - * Retries with exponential backoff on 429 (rate limited). + * Fetch a URL that returns XML/HTML (not JSON), via secHttp rate limiter. + * EDGAR returns XML for primary filing documents. */ - async edgarXmlFetch(url: string, retries = 3): Promise { - for (let attempt = 1; attempt <= retries; attempt++) { - await bucket.wait(); - - const resp = await fetch(url, { - method: 'GET', - headers: { - 'User-Agent': UA, - Accept: 'application/xml, text/xml, */*', - }, - }); - - if (resp.status === 429 && attempt < retries) { - const backoff = Math.pow(2, attempt) * 1000; - await new Promise((r) => setTimeout(r, backoff)); - continue; - } - - if (resp.status >= 400) { - throw new Error(`EDGAR XML ${resp.status} ${resp.statusText} for ${url}`); - } - - const text = await resp.text(); - return text; + async edgarXmlFetch(url: string): Promise { + const resp = await secFetch(url, { + accept: 'application/xml, text/xml, */*', + retries: 1, + }); + if (resp.status >= 400) { + throw new Error(`EDGAR XML ${resp.status} ${resp.statusText} for ${url}`); } - - throw new Error(`EDGAR XML max retries (${retries}) exceeded for ${url}`); + return resp.text(); } /** SourceFetch.fetchOne dispatch. */ diff --git a/app/server/src/adapters/FinraShortInterestAdapter.ts b/app/server/src/adapters/FinraShortInterestAdapter.ts new file mode 100644 index 0000000..5ecc6a8 --- /dev/null +++ b/app/server/src/adapters/FinraShortInterestAdapter.ts @@ -0,0 +1,62 @@ +// Investor Flow — FinraShortInterestAdapter (bi-monthly outstanding positions). +// SourceFetch for FINRA Equity Short Interest files (shrt{YYYYMMDD}.csv). +// Pipe-delimited, ~2MB, published twice monthly (15th + end-of-month settlement). +// Stores in finra_short_interest_biweekly; per-symbol reads via cache handler. +import type { DatabaseSync } from 'node:sqlite'; +import type { CacheKey, TtlClass, Provenance, SourceKind } from '../cache/CacheRepository.ts'; +import { parseCacheKey } from '../cache/CacheRepository.ts'; +import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +import { downloadAndIngestFinraSi, latestFinraSiSettlementDate } from '../services/FinraIngestService.ts'; + +const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'; +const FINRA_SI_TTL_MS = 14 * 24 * 60 * 60_000; // 14d (bi-monthly publication) + +export class FinraShortInterestAdapter implements SourceFetch { + readonly sourceKind = 'finra-si' as SourceKind; + private _db: DatabaseSync; + + constructor(db: DatabaseSync) { + this._db = db; + } + + async fetchOne(key: CacheKey): Promise { + const { kind, id } = parseCacheKey(key); + const fetchedAt = new Date().toISOString(); + + if (kind !== 'finraSiBiweekly') { + throw new Error(`FinraShortInterestAdapter: unknown kind '${kind}'`); + } + + // Only bulk-ingest keys ('latest' or explicit date YYYY-MM-DD). + if (id !== 'latest' && !/^\d{4}-\d{2}-\d{2}$/.test(id)) { + return { + value: { ok: true, skipped: true, reason: 'per-symbol key; FINRA SI is bulk-only' }, + ttlClass: 'short_interest' as TtlClass, + provenance: { fetchedAt, sourceKind: 'finra-si', rawSourceId: id }, + }; + } + + const settlementDate = id === 'latest' ? this._guessSettlementDate() : id; + const result = await downloadAndIngestFinraSi(this._db, settlementDate); + return { + value: { ok: true, symbolsStored: result.symbolsStored, sourceFile: result.sourceFile, settlementDate }, + ttlClass: 'short_interest' as TtlClass, + provenance: { fetchedAt, sourceKind: 'finra-si', rawSourceId: settlementDate }, + }; + } + + private _guessSettlementDate(): string { + const existing = latestFinraSiSettlementDate(this._db); + if (existing) return existing; + const now = new Date(); + const d = now.getDate(); + const y = now.getFullYear(); + const m = String(now.getMonth() + 1).padStart(2, '0'); + // Settlement near 15th and near month-end (~24-31) + if (d >= 20) return `${y}-${m}-15`; + const prev = new Date(now.getFullYear(), now.getMonth() - (d < 15 ? 1 : 0), 1); + const pm = String(prev.getMonth() + 1).padStart(2, '0'); + const py = prev.getFullYear(); + return d < 15 ? `${py}-${pm}-31` : `${y}-${m}-15`; + } +} diff --git a/app/server/src/adapters/NasdaqAdapter.ts b/app/server/src/adapters/NasdaqAdapter.ts index 5bc19da..4ccf9ff 100644 --- a/app/server/src/adapters/NasdaqAdapter.ts +++ b/app/server/src/adapters/NasdaqAdapter.ts @@ -5,8 +5,9 @@ import type { CacheKey, TtlClass, Provenance, SourceKind } from '../cache/CacheRepository.ts'; import { parseCacheKey } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +import { vendorFetch } from '../services/vendorGate.ts'; -const UA = `InvestorFlow/${process.env.APP_VERSION ?? 'dev'} (+https://github.com/anomalyco/investor-flow)`; +const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'; const NASDAQ_SHORTINTEREST_TTL_MS = 12 * 60 * 60_000; // 12h (NASDAQ publishes bi-monthly) interface NasdaqShortInterestRow { @@ -39,14 +40,16 @@ export class NasdaqAdapter implements SourceFetch { throw new Error(`NasdaqAdapter: unknown kind '${kind}'`); } - const res = await fetch( + const res = await vendorFetch( + 'nasdaq', `https://api.nasdaq.com/api/quote/${encodeURIComponent(id)}/short-interest?assetclass=stocks`, { headers: { 'User-Agent': UA, Accept: 'application/json', }, - } + hostAllowlist: /api\.nasdaq\.com/i, + }, ); if (!res.ok) { @@ -56,7 +59,11 @@ export class NasdaqAdapter implements SourceFetch { const body = (await res.json()) as NasdaqShortInterestResponse; if (!body.data?.shortInterestTable?.rows?.length) { - throw new Error(body.message ?? 'No short-interest data available'); + return { + value: null, + ttlClass: 'short_interest' as TtlClass, + provenance: { fetchedAt, sourceKind: 'nasdaq', rawSourceId: id }, + }; } const rows = body.data.shortInterestTable.rows as NasdaqShortInterestRow[]; diff --git a/app/server/src/adapters/OptionsAdapter.ts b/app/server/src/adapters/OptionsAdapter.ts index 4f9beb8..0ffd455 100644 --- a/app/server/src/adapters/OptionsAdapter.ts +++ b/app/server/src/adapters/OptionsAdapter.ts @@ -19,6 +19,7 @@ import type { CacheKey, TtlClass, Provenance, SourceKind } from '../cache/CacheRepository.ts'; import { parseCacheKey } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +import { withVendorGate } from '../services/vendorGate.ts'; // ----- Public types (options domain) ----- @@ -94,15 +95,21 @@ export class OptionsAdapter implements SourceFetch { } async fetchOne(key: CacheKey): Promise { + // Options share Yahoo's rate budget with quotes/candles. + return withVendorGate('yfinance', () => this.fetchOneUngated(key)); + } + + private async fetchOneUngated(key: CacheKey): Promise { const { kind, id } = parseCacheKey(key); const fetchedAt = new Date().toISOString(); const yf = await this.yf(); if (kind === 'expiry_dates') { - // Use optionsExpiryDates() which returns string[] directly. - const dates = await yf.optionsExpiryDates(id); - const sorted = [...dates].sort() as OptionExpiryDate[]; + // yahoo-finance2 v3: options(symbol).expirationDates is Date[] + // (there is no optionsExpiryDates helper on the class API). + const raw = await yf.options(id); + const sorted = extractExpiryDates(raw); return { value: sorted, ttlClass: 'intraday', @@ -112,28 +119,14 @@ export class OptionsAdapter implements SourceFetch { if (kind === 'chain') { const [symbol, expiry] = id.split(':'); - const rawResult = await yf.options(symbol, expiry); - // Handle multiple response shapes: - // v3: { options: { '1234567890': { calls: [...], puts: [...] } } } - // flat: { calls: [...], puts: [...] } - let expiryData: Record | null = null; - if (rawResult.options && typeof rawResult.options === 'object' && !Array.isArray(rawResult.options)) { - const optObj = rawResult.options as Record; - const dateKeys = Object.keys(optObj).filter((k) => /^\d+$/.test(k) || /\d{4}-\d{2}-\d{2}/.test(k)); - if (dateKeys.length > 0) { - expiryData = optObj[dateKeys[0]] as Record; - } else { - expiryData = optObj; - } - } else if (Array.isArray(rawResult.calls) || Array.isArray(rawResult.puts)) { - // Flat format: { calls: [...], puts: [...] } - expiryData = rawResult as Record; - } - const allRows = [ - ...((expiryData?.calls as Array> | undefined) ?? []).map((c) => ({ ...c, right: 'call' as const })), - ...((expiryData?.puts as Array> | undefined) ?? []).map((p) => ({ ...p, right: 'put' as const })), - ]; - const rows = parseOptionChainRows(symbol, allRows); + const rawResult = await yf.options(symbol, { date: expiry }); + const allRows = flattenChainRows(rawResult); + // Stamp expiry string on rows when Yahoo returns Date objects. + const stamped = allRows.map((r) => ({ + ...r, + expiration: toYmd(r.expiration) ?? expiry, + })); + const rows = parseOptionChainRows(symbol, stamped); return { value: rows, ttlClass: 'options_snapshot', @@ -146,24 +139,11 @@ export class OptionsAdapter implements SourceFetch { const symbol = parts[0]; const expiry = parts[1]; const strike = parseFloat(parts[2] ?? '0'); - const rawResult = await yf.options(symbol, expiry); - // Handle multiple response shapes (same logic as chain kind). - let expiryData: Record | null = null; - if (rawResult.options && typeof rawResult.options === 'object' && !Array.isArray(rawResult.options)) { - const optObj = rawResult.options as Record; - const dateKeys = Object.keys(optObj).filter((k) => /^\d+$/.test(k) || /\d{4}-\d{2}-\d{2}/.test(k)); - if (dateKeys.length > 0) { - expiryData = optObj[dateKeys[0]] as Record; - } else { - expiryData = optObj; - } - } else if (Array.isArray(rawResult.calls) || Array.isArray(rawResult.puts)) { - expiryData = rawResult as Record; - } - const allRows = [ - ...((expiryData?.calls as Array> | undefined) ?? []).map((c) => ({ ...c, right: 'call' as const })), - ...((expiryData?.puts as Array> | undefined) ?? []).map((p) => ({ ...p, right: 'put' as const })), - ]; + const rawResult = await yf.options(symbol, { date: expiry }); + const allRows = flattenChainRows(rawResult).map((r) => ({ + ...r, + expiration: toYmd(r.expiration) ?? expiry, + })); const rows = parseOptionChainRows(symbol, allRows); const target = strike > 0 ? rows.find((r) => r.strike === strike) @@ -196,6 +176,73 @@ export class OptionsAdapter implements SourceFetch { // ----- Pure parse helpers (tested with recorded fixtures; no network) ----- +/** YYYY-MM-DD from Date, ISO string, or unix seconds/ms. */ +export function toYmd(v: unknown): string | null { + if (v == null) return null; + if (v instanceof Date && !Number.isNaN(v.getTime())) { + return v.toISOString().slice(0, 10); + } + if (typeof v === 'number' && Number.isFinite(v)) { + const ms = v > 1e12 ? v : v * 1000; + const d = new Date(ms); + return Number.isNaN(d.getTime()) ? null : d.toISOString().slice(0, 10); + } + if (typeof v === 'string' && v.length >= 10) { + // ISO or YYYY-MM-DD + if (/^\d{4}-\d{2}-\d{2}/.test(v)) return v.slice(0, 10); + const d = new Date(v); + return Number.isNaN(d.getTime()) ? null : d.toISOString().slice(0, 10); + } + return null; +} + +/** Pull sorted YYYY-MM-DD expiries from a yahoo-finance2 options() response. */ +export function extractExpiryDates(raw: Record): OptionExpiryDate[] { + const dates = raw.expirationDates; + if (!Array.isArray(dates)) return []; + const out = new Set(); + for (const d of dates) { + const ymd = toYmd(d); + if (ymd) out.add(ymd); + } + return [...out].sort(); +} + +/** + * Flatten yahoo-finance2 options() into call/put row objects. + * Handles: + * - v3 array: { options: [ { calls, puts, expirationDate } ] } + * - object map: { options: { 'ts': { calls, puts } } } + * - flat: { calls, puts } + */ +export function flattenChainRows(rawResult: Record): Array> { + let expiryData: Record | null = null; + const opts = rawResult.options; + + if (Array.isArray(opts) && opts.length > 0) { + expiryData = opts[0] as Record; + } else if (opts && typeof opts === 'object') { + const optObj = opts as Record; + const dateKeys = Object.keys(optObj).filter((k) => /^\d+$/.test(k) || /\d{4}-\d{2}-\d{2}/.test(k)); + if (dateKeys.length > 0) { + expiryData = optObj[dateKeys[0]] as Record; + } else if (Array.isArray((optObj as { calls?: unknown }).calls) || Array.isArray((optObj as { puts?: unknown }).puts)) { + expiryData = optObj; + } + } + + if (!expiryData && (Array.isArray(rawResult.calls) || Array.isArray(rawResult.puts))) { + expiryData = rawResult; + } + + if (!expiryData) return []; + + return [ + ...((expiryData.calls as Array> | undefined) ?? []).map((c) => ({ ...c, right: 'call' as const })), + ...((expiryData.puts as Array> | undefined) ?? []).map((p) => ({ ...p, right: 'put' as const })), + ]; +} + /** * Parse the raw option chain from yahoo-finance2 into typed OptionChainRow[]. * Handles both formats: @@ -254,7 +301,7 @@ function parseOneRow(raw: Record): OptionChainRow { contractSymbol, strike, right, - expiration: str(raw.expiration) ?? '', + expiration: toYmd(raw.expiration) ?? str(raw.expiration) ?? '', lastPrice: num(raw.lastPrice ?? raw.lastPrice2), bid: num(raw.bid), ask: num(raw.ask), @@ -310,6 +357,6 @@ function str(v: unknown): string | null { /** Minimal yfinance2 surface we use. Keeps the dynamic import decoupled from the type system. */ interface YFinanceLike { - optionsExpiryDates(symbol: string): Promise; - options(symbol: string, expiry?: string): Promise>; + /** yahoo-finance2 v3: second arg is query opts `{ date }` (ISO date, Date, or unix). */ + options(symbol: string, query?: { date?: string | Date | number } | string): Promise>; } diff --git a/app/server/src/adapters/RedditAdapter.ts b/app/server/src/adapters/RedditAdapter.ts index f72caba..062d163 100644 --- a/app/server/src/adapters/RedditAdapter.ts +++ b/app/server/src/adapters/RedditAdapter.ts @@ -5,10 +5,11 @@ import type { SourceKind, TtlClass, Provenance } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult, FetchOpts } from './SourceAdapter.ts'; +import { vendorFetch, VendorRateLimitError } from '../services/vendorGate.ts'; -const RATE_LIMIT_MS = 3_000; const CACHE_TTL_MS = 7 * 24 * 60 * 60_000; // 7 days const REDDIT_API_BASE = 'https://www.reddit.com/r'; +const REDDIT_HOST_RE = /reddit\.com/i; // ===== Types for Reddit API response parsing ===== @@ -35,36 +36,6 @@ interface RedditResponse { }; } -// ===== Rate limiter (token bucket) ===== - -class TokenBucket { - private tokens: number = 1; - private lastRefill: number = Date.now(); - readonly capacity: number; - readonly refillMs: number; - - constructor(capacity = 1, refillMs = RATE_LIMIT_MS) { - this.capacity = capacity; - this.refillMs = refillMs; - } - - async acquire(): Promise { - const now = Date.now(); - const elapsed = now - this.lastRefill; - const tokensToAdd = Math.floor(elapsed / this.refillMs); - if (tokensToAdd > 0) { - this.tokens = Math.min(this.capacity, this.tokens + tokensToAdd); - this.lastRefill = now - (elapsed % this.refillMs); - } - if (this.tokens < 1) { - const waitMs = this.refillMs - (now - this.lastRefill); - await new Promise((resolve) => setTimeout(resolve, Math.max(0, waitMs))); - return this.acquire(); - } - this.tokens -= 1; - } -} - // ===== Health tracking for Reddit ===== export type RedditHealth = { sourceStatus: 'healthy' | 'degraded' | 'failed'; lastError?: string | null; }; @@ -74,12 +45,10 @@ export type RedditHealth = { sourceStatus: 'healthy' | 'degraded' | 'failed'; la export class RedditAdapter implements SourceFetch { readonly sourceKind: SourceKind = 'reddit'; - private readonly rateLimiter: TokenBucket; private _health: RedditHealth = { sourceStatus: 'healthy' }; private _onDegraded?: (health: RedditHealth) => void; constructor(onDegraded?: (health: RedditHealth) => void) { - this.rateLimiter = new TokenBucket(); this._onDegraded = onDegraded; } @@ -104,17 +73,16 @@ export class RedditAdapter implements SourceFetch { throw new Error(`RedditAdapter: source degraded — ${this.health.lastError}`); } - await this.rateLimiter.acquire(); - const sortParam = sort ?? 'hot'; const url = `${REDDIT_API_BASE}/${subreddit}/${sortParam}.json?limit=50`; try { - const resp = await fetch(url, { + const resp = await vendorFetch('reddit', url, { headers: { 'User-Agent': 'InvestorFlow/1.0 (by operator@example.com)', 'Accept': 'application/json', }, + hostAllowlist: REDDIT_HOST_RE, }); if (resp.status === 429) { @@ -136,6 +104,10 @@ export class RedditAdapter implements SourceFetch { provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'reddit', rawSourceId: `subreddit:${subreddit}:${sortParam}` }, }; } catch (err) { + if (err instanceof VendorRateLimitError || (err instanceof Error && /rate limit|429/i.test(err.message))) { + this.emitDegraded('Reddit rate-limited'); + throw new Error('RedditAdapter: Reddit source degraded (rate limited)'); + } if (err instanceof Error && err.message.includes('Reddit source degraded')) { throw err; } @@ -149,16 +121,15 @@ export class RedditAdapter implements SourceFetch { throw new Error(`RedditAdapter: source degraded — ${this.health.lastError}`); } - await this.rateLimiter.acquire(); - const url = `${REDDIT_API_BASE}/search.json?q=${encodeURIComponent(query)}&limit=50`; try { - const resp = await fetch(url, { + const resp = await vendorFetch('reddit', url, { headers: { 'User-Agent': 'InvestorFlow/1.0 (by operator@example.com)', 'Accept': 'application/json', }, + hostAllowlist: REDDIT_HOST_RE, }); if (resp.status === 429) { @@ -180,6 +151,10 @@ export class RedditAdapter implements SourceFetch { provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'reddit', rawSourceId: `search:${query}` }, }; } catch (err) { + if (err instanceof VendorRateLimitError || (err instanceof Error && /rate limit|429/i.test(err.message))) { + this.emitDegraded('Reddit rate-limited'); + throw new Error('RedditAdapter: Reddit source degraded (rate limited)'); + } if (err instanceof Error && err.message.includes('Reddit source degraded')) { throw err; } @@ -193,18 +168,17 @@ export class RedditAdapter implements SourceFetch { throw new Error(`RedditAdapter: source degraded — ${this.health.lastError}`); } - await this.rateLimiter.acquire(); - const sortParam = sort ?? 'best'; // Reddit comment thread endpoint const url = `https://www.reddit.com/comments/${postId}.json?sort=${sortParam}&limit=100`; try { - const resp = await fetch(url, { + const resp = await vendorFetch('reddit', url, { headers: { 'User-Agent': 'InvestorFlow/1.0 (by operator@example.com)', 'Accept': 'application/json', }, + hostAllowlist: REDDIT_HOST_RE, }); if (resp.status === 429) { @@ -226,6 +200,10 @@ export class RedditAdapter implements SourceFetch { provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'reddit', rawSourceId: `comments:${postId}` }, }; } catch (err) { + if (err instanceof VendorRateLimitError || (err instanceof Error && /rate limit|429/i.test(err.message))) { + this.emitDegraded('Reddit rate-limited'); + throw new Error('RedditAdapter: Reddit source degraded (rate limited)'); + } if (err instanceof Error && err.message.includes('Reddit source degraded')) { throw err; } diff --git a/app/server/src/adapters/SecCompanyTickersAdapter.ts b/app/server/src/adapters/SecCompanyTickersAdapter.ts new file mode 100644 index 0000000..458fca9 --- /dev/null +++ b/app/server/src/adapters/SecCompanyTickersAdapter.ts @@ -0,0 +1,52 @@ +// Investor Flow — SEC company_tickers.json adapter (M22 Symbol Search Index). +// +// Fetch the SEC's bulk ticker file (https://www.sec.gov/files/company_tickers.json, +// ~12k US tickers, overwritten in place daily ~5:30am ET) and materialize it into the +// symbols table via the merge policy in secTickersIngest.ts. +// +// Cache key: sec-tickers:companyTickers:latest +// Scheduled daily via queue_schedules source_kind='sec-tickers' (see sourceRatePolicy). +import type { DatabaseSync } from 'node:sqlite'; +import type { CacheKey, SourceKind, TtlClass, Provenance } from '../cache/CacheRepository.ts'; +import { parseCacheKey } from '../cache/CacheRepository.ts'; +import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +import { mergeCompanyTickers, type CompanyTickerRow } from '../services/secTickersIngest.ts'; +import { secFetch } from '../services/secHttp.ts'; + +const TICKERS_URL = 'https://www.sec.gov/files/company_tickers.json'; + +/** SEC company_tickers.json adapter. Bulk ingest only — per-symbol keys are rejected. */ +export class SecCompanyTickersAdapter implements SourceFetch { + readonly sourceKind: SourceKind = 'sec-tickers'; + private _db: DatabaseSync; + + constructor(db: DatabaseSync) { + this._db = db; + } + + async fetchOne(key: CacheKey): Promise { + const { kind, id } = parseCacheKey(key); + if (kind !== 'companyTickers') { + throw new Error(`SecCompanyTickersAdapter: unknown kind '${kind}'`); + } + if (id !== 'latest') { + return { + value: { ok: true, skipped: true, reason: 'bulk ingest only; use id=latest' }, + ttlClass: 'symbol_meta', + provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'sec-tickers', rawSourceId: id }, + }; + } + + const resp = await secFetch(TICKERS_URL); + if (!resp.ok) throw new Error(`EDGAR company_tickers: HTTP ${resp.status}`); + + const body = (await resp.json()) as Record; + const result = mergeCompanyTickers(this._db, body); + + return { + value: { ok: true, ...result }, + ttlClass: 'symbol_meta', + provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'sec-tickers', rawSourceId: 'latest' }, + }; + } +} diff --git a/app/server/src/adapters/SecFetchAdapter.ts b/app/server/src/adapters/SecFetchAdapter.ts index e89a9ed..1e0eb21 100644 --- a/app/server/src/adapters/SecFetchAdapter.ts +++ b/app/server/src/adapters/SecFetchAdapter.ts @@ -3,6 +3,10 @@ import type { CacheKey, SourceKind } from '../cache/CacheRepository.ts'; import { parseCacheKey } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; +/** + * SEC institutional / insider fetch adapter. + * Throws on hard pipeline failures so AdapterQueue retries instead of silent "done". + */ export class SecFetchAdapter implements SourceFetch { readonly sourceKind: SourceKind = 'sec-fetch'; private _db: DatabaseSync; @@ -13,14 +17,48 @@ export class SecFetchAdapter implements SourceFetch { async fetchOne(key: CacheKey): Promise { const { kind, id: symbol } = parseCacheKey(key); - if (kind !== 'fetch') throw new Error(`SecFetchAdapter: unknown kind '${kind}'`); + const { fetchAndStoreSecData, fetchAndStoreSCFilings } = await import( + '../services/secDataFetcher.ts' + ); - const { fetchAndStoreSecData } = await import('../services/secDataFetcher.ts'); - const result = await fetchAndStoreSecData(this._db, symbol); - return { - value: { ok: true, filingsStored: result.filingsStored, scStored: result.scStored, transactionsStored: result.transactionsStored }, - ttlClass: 'daily_permanent', - provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'sec-fetch', rawSourceId: symbol }, - }; + if (kind === 'fetch') { + const result = await fetchAndStoreSecData(this._db, symbol); + if (result.hardErrors.length > 0) { + throw new Error( + `sec-fetch:${symbol}: ${result.hardErrors.join('; ')} ` + + `(13F=${result.filingsStored} SC=${result.scStored} Form4=${result.transactionsStored} cusip=${result.cusipResolved})`, + ); + } + return { + value: { + ok: true, + filingsStored: result.filingsStored, + scStored: result.scStored, + transactionsStored: result.transactionsStored, + cusipResolved: result.cusipResolved, + }, + ttlClass: 'daily_permanent', + provenance: { + fetchedAt: new Date().toISOString(), + sourceKind: 'sec-fetch', + rawSourceId: symbol, + }, + }; + } + + if (kind === 'sc') { + const stored = await fetchAndStoreSCFilings(this._db, symbol); + return { + value: { ok: true, scStored: stored }, + ttlClass: 'daily_permanent', + provenance: { + fetchedAt: new Date().toISOString(), + sourceKind: 'sec-fetch', + rawSourceId: symbol, + }, + }; + } + + throw new Error(`SecFetchAdapter: unknown kind '${kind}'`); } } diff --git a/app/server/src/adapters/SourceAdapter.ts b/app/server/src/adapters/SourceAdapter.ts index bb7a78e..ec31c3d 100644 --- a/app/server/src/adapters/SourceAdapter.ts +++ b/app/server/src/adapters/SourceAdapter.ts @@ -7,7 +7,16 @@ // This preserves the design's intent (one seam per source; shared dedupe) while keeping each // micro-dispatch focused. The SPA never calls either directly — only CacheRepository.get → // AdapterQueue.queue → (drain) → SourceFetch.fetchOne → CacheRepository.set. +// +// FUTURE VENDORS: extend VendorSourceAdapter (auto rate-limit) or wrap every +// external call in withVendorGate / vendorFetch. Register the family first — +// see docs/VENDOR_INTEGRATIONS.md. import type { CacheKey, SourceKind, TtlClass, Provenance } from '../cache/CacheRepository.ts'; +import { + requireSourceFamily, + withVendorGate, + type VendorFamily, +} from '../services/vendorGate.ts'; export interface FetchOpts { force?: boolean; } // force = bypass dedupe (admin.resetBackoff) export interface AdapterHealth { queued: number; in_flight: number; backoff_until?: string | null; last_error?: string; } @@ -19,6 +28,53 @@ export interface SourceFetch { fetchOne(key: CacheKey): Promise; } +/** + * Base class for new (and migrated) vendor adapters. + * + * Subclasses implement `fetchOneUngated` only. The public `fetchOne` always + * runs under `withVendorGate(family)` so a future adapter cannot forget pacing. + * + * Prefer this over implementing SourceFetch by hand. + */ +export abstract class VendorSourceAdapter implements SourceFetch { + abstract readonly sourceKind: SourceKind; + + /** + * Vendor family for rate-limit budget. Defaults to requireSourceFamily(sourceKind). + * Override only when one adapter intentionally shares another family's budget. + */ + protected vendorFamily(): VendorFamily { + return requireSourceFamily(this.sourceKind); + } + + async fetchOne(key: CacheKey): Promise { + return withVendorGate(this.vendorFamily(), () => this.fetchOneUngated(key)); + } + + /** Implement the real vendor I/O here — never call withVendorGate again inside. */ + protected abstract fetchOneUngated(key: CacheKey): Promise; +} + +/** + * Helper to define a gated adapter without a class (small integrations). + * + * const adapter = defineVendorAdapter({ + * sourceKind: 'polygon', + * fetchOneUngated: async (key) => { ... vendorFetch('polygon', ...) }, + * }); + */ +export function defineVendorAdapter(opts: { + sourceKind: SourceKind; + family?: VendorFamily; + fetchOneUngated: (key: CacheKey) => Promise; +}): SourceFetch { + const family = opts.family ?? requireSourceFamily(opts.sourceKind); + return { + sourceKind: opts.sourceKind, + fetchOne: (key) => withVendorGate(family, () => opts.fetchOneUngated(key)), + }; +} + /** * Test double: deterministic canned responses per key. Implements SourceFetch. * `calls` records every fetchOne invocation — used by dedupe tests diff --git a/app/server/src/adapters/XCookieAdapter.ts b/app/server/src/adapters/XCookieAdapter.ts index fb093ea..da15795 100644 --- a/app/server/src/adapters/XCookieAdapter.ts +++ b/app/server/src/adapters/XCookieAdapter.ts @@ -3,6 +3,7 @@ import { promisify } from 'node:util'; import type { DatabaseSync } from 'node:sqlite'; import type { SourceKind, TtlClass, Provenance } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult, FetchOpts } from './SourceAdapter.ts'; +import { withVendorGate } from '../services/vendorGate.ts'; const execFileAsync = promisify(execFile); @@ -73,6 +74,11 @@ export class XCookieAdapter implements SourceFetch { } private async runBird(args: string[]): Promise { + // X/bird is paced by vendorGate — no concurrent bird processes thrashing auth. + return withVendorGate('x', () => this.runBirdUngated(args)); + } + + private async runBirdUngated(args: string[]): Promise { if (!this._cookies) throw new Error('XCookieAdapter: no credentials set'); const allArgs = [ '--auth-token', this._cookies.auth_token, @@ -166,30 +172,31 @@ export class XCookieAdapter implements SourceFetch { const count = opts?.walkBackDays && opts.walkBackDays > 0 ? 100 : 20; const clean = handle.replace(/^@/, ''); - // bird 0.8.x user-tweets often fails with "User not found" even for valid - // handles (GraphQL user lookup). Search `from:handle` is reliable with the - // same cookies and returns the profile timeline effectively. + // Prefer `user-tweets` (GraphQL timeline): unlike search, it includes + // subscriber-only posts when the cookie session has access to them, which + // is what position-capture posts from subscription creators are. Fall back + // to search `from:handle` (older bird / rate-limited user lookup). let tweets: BirdTweet[] = []; - let used = 'from-search'; + let used = 'user-tweets'; try { const stdout = await this.runBird([ - 'search', `from:${clean}`, '-n', String(count), '--json', + 'user-tweets', `@${clean}`, '-n', String(count), '--json', ]); tweets = JSON.parse(stdout) as BirdTweet[]; if (!Array.isArray(tweets)) throw new Error('not an array'); - } catch (searchErr) { - // Fallback: user-tweets (older bird / when search is rate-limited). + } catch (userErr) { + // Fallback: search `from:handle` (no subscriber-only content, but reliable). try { const stdout = await this.runBird([ - 'user-tweets', `@${clean}`, '-n', String(count), '--json', + 'search', `from:${clean}`, '-n', String(count), '--json', ]); tweets = JSON.parse(stdout) as BirdTweet[]; - used = 'user-tweets'; + used = 'from-search'; if (!Array.isArray(tweets)) throw new Error('not an array'); - } catch (userErr) { - const s = searchErr instanceof Error ? searchErr.message : String(searchErr); + } catch (searchErr) { const u = userErr instanceof Error ? userErr.message : String(userErr); - throw new Error(`XCookieAdapter: timeline fetch failed for @${clean} (from-search: ${s}; user-tweets: ${u})`); + const s = searchErr instanceof Error ? searchErr.message : String(searchErr); + throw new Error(`XCookieAdapter: timeline fetch failed for @${clean} (user-tweets: ${u}; from-search: ${s})`); } } diff --git a/app/server/src/adapters/YFinanceAdapter.ts b/app/server/src/adapters/YFinanceAdapter.ts index e035a74..320d1c2 100644 --- a/app/server/src/adapters/YFinanceAdapter.ts +++ b/app/server/src/adapters/YFinanceAdapter.ts @@ -1,26 +1,51 @@ // Investor Flow — YFinanceAdapter (DESIGN.md §3a Module 2 + §5 yfinance policy). -// One concrete SourceFetch for yfinance: quote / candles / symbol(info+sector). +// One concrete SourceFetch for yfinance: quote / candles / symbol(info+sector) / topHoldings. // yahoo-finance2 v3 is dynamically imported inside fetchOne so the module loads without the // dependency and tests exercise only the pure parse functions (fixtures, no network). -import type { CacheKey, TtlClass, Provenance, Quote, PriceCandle, SymbolMeta, TickerKind } from '../cache/CacheRepository.ts'; -import { parseCacheKey } from '../cache/CacheRepository.ts'; +import type { DatabaseSync } from 'node:sqlite'; +import type { CacheKey, Provenance, Quote, PriceCandle, SymbolMeta, TickerKind } from '../cache/CacheRepository.ts'; +import { lastCandleTs, parseCacheKey } from '../cache/CacheRepository.ts'; import type { SourceFetch, FetchResult } from './SourceAdapter.ts'; import { parseAdjustments } from './yfinance-adjustments.ts'; +import { withVendorGate } from '../services/vendorGate.ts'; // Polite UA including operator contact (yfinance community norm; EDGAR requires similar). const OPERATOR_EMAIL = process.env.YF_OPERATOR_EMAIL ?? 'operator@example.com'; +/** Full history window for cold candle fills (~10y). */ +const FULL_HISTORY_DAYS = 3650; +/** When we already have bars, only re-pull recent history + a small overlap. */ +const INCREMENTAL_LOOKBACK_DAYS = 14; + +export interface YFinanceAdapterOptions { + /** Optional DB for incremental candle period1 (last bar ts). */ + db?: DatabaseSync; +} + export class YFinanceAdapter implements SourceFetch { readonly sourceKind = 'yfinance' as const; private _yf: unknown = null; + private readonly _db: DatabaseSync | null; + + constructor(opts: YFinanceAdapterOptions = {}) { + this._db = opts.db ?? null; + void OPERATOR_EMAIL; + } + private async yf(): Promise { if (!this._yf) { const mod = await import('yahoo-finance2'); - this._yf = new mod.default(); + this._yf = new mod.default({ suppressNotices: ['yahooSurvey'] }); } return this._yf as YFinanceLike; } + async fetchOne(key: CacheKey): Promise { + // All Yahoo HTTP is paced by vendorGate('yfinance') — never call yf.* bare. + return withVendorGate('yfinance', () => this.fetchOneUngated(key)); + } + + private async fetchOneUngated(key: CacheKey): Promise { const { kind, id } = parseCacheKey(key); const fetchedAt = new Date().toISOString(); const yf = await this.yf(); @@ -32,22 +57,31 @@ export class YFinanceAdapter implements SourceFetch { case 'candles': { const [symbol, timeframe] = id.split(':'); const interval = timeframeToInterval(timeframe); - const days = 3650; // permanent backfill (~10 years for both 1d and 1wk) - const raw = await yf.chart(symbol, { period1: new Date(Date.now() - days * 86_400_000), interval }); + const period1 = this.candlePeriod1(symbol, timeframe ?? '1d'); + const raw = await yf.chart(symbol, { period1, interval }); return { value: parseCandles(raw), ttlClass: 'daily_permanent', provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: `${symbol}:${timeframe}` } }; } case 'adjustments': { const symbol = id; - const raw = await yf.chart(symbol, { period1: new Date(Date.now() - 3650 * 86_400_000), interval: '1d' }); + // Adjustments change rarely; still allow incremental when history exists. + const last = this._db ? lastCandleTs(this._db, symbol, '1d') : null; + const period1 = last + ? new Date(Date.parse(last) - INCREMENTAL_LOOKBACK_DAYS * 86_400_000) + : new Date(Date.now() - FULL_HISTORY_DAYS * 86_400_000); + const raw = await yf.chart(symbol, { period1, interval: '1d' }); return { value: parseAdjustments(symbol, raw), ttlClass: 'daily_permanent', provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: `${symbol}:adjustments` } }; } case 'symbol': { - const raw = await yf.quoteSummary(id, { modules: ['summaryProfile'] }); + // summaryProfile alone has no longName; price/quoteType carry the display name + mkt cap. + const raw = await yf.quoteSummary(id, { + modules: ['summaryProfile', 'price', 'quoteType'], + }); return { value: parseSymbol(raw, id), ttlClass: 'symbol_meta', provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: id } }; } case 'shortinterest': { - const raw = await yf.quoteSummary(id, { modules: ['defaultKeyStatistics'] }); + const raw = await yf.quoteSummary(id, { modules: ['defaultKeyStatistics', 'summaryDetail'] }); const stats = (raw as Record)?.defaultKeyStatistics as Record | undefined; + const detail = (raw as Record)?.summaryDetail as Record | undefined; if (!stats) throw new Error('No defaultKeyStatistics module available'); const parseNum = (v: unknown): number | null => { if (v === null || v === undefined) return null; @@ -68,31 +102,134 @@ export class YFinanceAdapter implements SourceFetch { floatShares: parseNum(stats.floatShares), sharesOutstanding: parseNum(stats.sharesOutstanding), sourceTickerKind: (stats as Record).optionMethod?.toString() ?? null, + dividendYield: parseNum(detail?.dividendYield), + payoutRatio: parseNum(detail?.payoutRatio), + dividendRate: parseNum(detail?.dividendRate), + exDividendDate: detail?.exDividendDate ? String(detail.exDividendDate).slice(0, 10) : null, + trailingEps: parseNum(stats.trailingEps), + forwardPE: parseNum(stats.forwardPE), }, ttlClass: 'short_interest', provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: id }, }; } + case 'dividendFundamentals': { + const raw = await yf.quoteSummary(id, { modules: ['summaryDetail', 'defaultKeyStatistics'] }); + const detail = (raw as Record)?.summaryDetail as Record | undefined; + const stats = (raw as Record)?.defaultKeyStatistics as Record | undefined; + const parseNum = (v: unknown): number | null => { + if (v === null || v === undefined) return null; + if (typeof v === 'number') return v; + const s = String(v).replace(/[,]/g, ''); + const n = parseFloat(s); + return Number.isNaN(n) ? null : n; + }; + return { + value: { + symbol: id, + dividendYield: parseNum(detail?.dividendYield), + payoutRatio: parseNum(detail?.payoutRatio), + dividendRate: parseNum(detail?.dividendRate), + exDividendDate: detail?.exDividendDate ? String(detail.exDividendDate).slice(0, 10) : null, + trailingEps: parseNum(stats?.trailingEps), + forwardPE: parseNum(stats?.forwardPE), + growthStreakYears: null, + }, + ttlClass: 'dividend_fundamentals', + provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: id }, + }; + } + case 'topHoldings': { + const symbol = id.toUpperCase(); + const raw = await yf.quoteSummary(symbol, { modules: ['topHoldings'] }); + const holdings = parseTopHoldings(raw, symbol); + return { + value: { symbol, holdings }, + ttlClass: 'symbol_meta', // weekly-class slow composition + provenance: { fetchedAt, sourceKind: 'yfinance', rawSourceId: symbol }, + }; + } default: throw new Error(`YFinanceAdapter: unknown kind '${kind}'`); } } + + /** Cold = full history; warm = last bar − lookback. */ + private candlePeriod1(symbol: string, timeframe: string): Date { + if (!this._db) { + return new Date(Date.now() - FULL_HISTORY_DAYS * 86_400_000); + } + const last = lastCandleTs(this._db, symbol, timeframe); + if (!last) { + return new Date(Date.now() - FULL_HISTORY_DAYS * 86_400_000); + } + const t = Date.parse(last); + if (!Number.isFinite(t)) { + return new Date(Date.now() - FULL_HISTORY_DAYS * 86_400_000); + } + return new Date(t - INCREMENTAL_LOOKBACK_DAYS * 86_400_000); + } } -interface YFinanceLike { quote(s: string): Promise>; chart(s: string, o: { period1: Date; interval: string }): Promise>; quoteSummary(s: string, o: { modules: string[] }): Promise>; } +interface YFinanceLike { + quote(s: string): Promise>; + chart(s: string, o: { period1: Date; interval: string }): Promise>; + quoteSummary(s: string, o: { modules: string[] }): Promise>; +} // ----- Pure parse helpers (tested with recorded fixtures; no network) ----- +/** + * Yahoo freezes `regularMarketPrice` at the RTH close. Pre/post (and Yahoo's + * POSTPOST overnight window) live in separate fields. Prefer the active session + * last so names that trade extended hours (e.g. IREN) keep updating. + */ export function parseQuote(raw: Record, symbol: string): Quote { - const price = num(raw.regularMarketPrice); - if (price === null) throw new Error(`quote ${symbol}: missing regularMarketPrice`); + const regular = num(raw.regularMarketPrice); + if (regular === null) throw new Error(`quote ${symbol}: missing regularMarketPrice`); + + const post = num(raw.postMarketPrice); + const pre = num(raw.preMarketPrice); + const prevClose = num(raw.regularMarketPreviousClose); + const state = String(raw.marketState ?? '').toUpperCase(); + + let price = regular; + let session: NonNullable = 'regular'; + + // Yahoo states: PREPRE | PRE | REGULAR | POST | POSTPOST | CLOSED | … + if ((state.includes('POST') || state === 'POSTPOST') && post != null) { + price = post; + session = 'post'; + } else if (state.includes('PRE') && pre != null) { + price = pre; + session = 'pre'; + } else if (state === 'CLOSED' && post != null) { + // After the evening session ends, last extended print is usually still in postMarket*. + price = post; + session = 'post'; + } else if (state === 'CLOSED' && pre != null) { + price = pre; + session = 'pre'; + } + + // When showing an extended last, report change vs prior close so the % matches the print. + // Fall back to Yahoo's regular day change when prev close is missing. + let change = num(raw.regularMarketChange); + let changePercent = num(raw.regularMarketChangePercent); + if (prevClose != null && prevClose !== 0 && session !== 'regular') { + change = price - prevClose; + changePercent = (change / prevClose) * 100; + } + return { symbol, price, bid: num(raw.bid), ask: num(raw.ask), - change: num(raw.regularMarketChange), - changePercent: num(raw.regularMarketChangePercent), + change, + changePercent, iv: null, // IV comes from the options chain kind, not the quote + session, + regularPrice: regular, }; } @@ -120,26 +257,49 @@ export function parseCandles(raw: Record): PriceCandle[] { export function parseSymbol(raw: Record, symbol: string): SymbolMeta { const sp = (raw.summaryProfile ?? {}) as Record; const price = (raw.price ?? {}) as Record; + const qt = (raw.quoteType ?? {}) as Record; + const name = str( + price.longName + ?? price.shortName + ?? qt.longName + ?? qt.shortName + ?? sp.longName + ?? sp.shortName, + ); return { symbol, - name: str(sp.longName ?? price.longName ?? sp.shortName), - sector: str(sp.sector), - industry: str(sp.industry), - exchange: str(sp.exchange ?? price.exchange), + name, + sector: str(sp.sector ?? sp.sectorDisp), + industry: str(sp.industry ?? sp.industryDisp), + exchange: str(sp.exchange ?? price.exchange ?? qt.exchange), tickerKind: inferTickerKind(symbol), description: str(sp.longBusinessSummary), website: str(sp.website), marketCap: num(price.marketCap), - currency: str(price.currency), - employees: num(price.fullTimeEmployees) as number | null, + currency: str(price.currency ?? qt.currency), + employees: num(sp.fullTimeEmployees ?? price.fullTimeEmployees) as number | null, country: str(sp.country ?? price.country), city: str(sp.city ?? price.city), peers: null, }; } +export type TopHoldingRow = { symbol: string; holdingName: string; holdingPercent: number }; + +export function parseTopHoldings(raw: Record, _etf: string): TopHoldingRow[] { + const th = (raw.topHoldings ?? {}) as { holdings?: Array> }; + const list = th.holdings ?? []; + return list + .filter((h) => h?.symbol) + .map((h) => ({ + symbol: String(h.symbol).toUpperCase(), + holdingName: String(h.holdingName ?? h.symbol ?? ''), + holdingPercent: typeof h.holdingPercent === 'number' ? h.holdingPercent : Number(h.holdingPercent) || 0, + })); +} + const CRYPTO = new Set(['BTC', 'ETH', 'BNB', 'SOL', 'XRP', 'ADA', 'DOGE', 'DOT']); -const ETF = new Set(['SPY', 'QQQ', 'IWM', 'DIA', 'VTI', 'VOO', 'EEM']); +const ETF = new Set(['SPY', 'QQQ', 'IWM', 'DIA', 'VTI', 'VOO', 'EEM', 'XLK', 'XLF', 'XLE', 'XLI', 'XLV', 'XLY', 'XLP', 'XLU', 'XLRE', 'XLC', 'XLB', 'SMH', 'MAGS', 'XBI', 'IWF', 'IWD', 'EFA']); export function inferTickerKind(symbol: string): TickerKind { if (CRYPTO.has(symbol)) return 'crypto'; if (ETF.has(symbol)) return 'etf'; diff --git a/app/server/src/adapters/__tests__/EdgarAdapter.test.ts b/app/server/src/adapters/__tests__/EdgarAdapter.test.ts index 44ce831..65eb7b8 100644 --- a/app/server/src/adapters/__tests__/EdgarAdapter.test.ts +++ b/app/server/src/adapters/__tests__/EdgarAdapter.test.ts @@ -377,7 +377,7 @@ test('rate limiter enforces min 125ms between consecutive fetches', async () => } }); -test('rate limiter enforces spacing when bucket exhausted (burst of 8+)', async () => { +test('secHttp serializes SEC calls with min-interval spacing (no burst)', async () => { const mockKey = `data.sec.gov/submissions/CIK${padCik(CIK_D)}.json`; const { mockFetch, getCallCount } = createMockFetch({ [mockKey]: { body: makeFilingsResponse() }, @@ -386,20 +386,16 @@ test('rate limiter enforces spacing when bucket exhausted (burst of 8+)', async try { const adapter = new EdgarAdapter(); - // Exhaust the bucket by making 8 rapid calls, then measure the 9th. + // secHttp: single-flight + ~350ms min gap. 4 sequential calls ≈ ≥1s total. const start = Date.now(); - for (let i = 0; i < 8; i++) { + for (let i = 0; i < 4; i++) { await adapter.filings_index(CIK_D); } - // 9th call should trigger rate-limit wait - await adapter.filings_index(CIK_D); const elapsed = Date.now() - start; - // 8 calls should complete fast (burst), then 9th waits ~125ms. - // Total should be > 50ms (proving some delay occurred) and < 2s. - assert.ok(elapsed >= 50, `expected some delay from rate limiting, got ${elapsed}ms`); - assert.ok(elapsed < 2000, `9th call should complete in <2s, took ${elapsed}ms`); - assert.equal(getCallCount(), 9, 'all 9 calls should execute'); + assert.ok(elapsed >= 900, `expected min-interval spacing, got ${elapsed}ms`); + assert.ok(elapsed < 8000, `4 paced calls should complete in <8s, took ${elapsed}ms`); + assert.equal(getCallCount(), 4, 'all 4 calls should execute'); } finally { restoreFetch(); } diff --git a/app/server/src/adapters/__tests__/OptionsAdapter.greeks.test.ts b/app/server/src/adapters/__tests__/OptionsAdapter.greeks.test.ts index d5aa38e..2f15699 100644 --- a/app/server/src/adapters/__tests__/OptionsAdapter.greeks.test.ts +++ b/app/server/src/adapters/__tests__/OptionsAdapter.greeks.test.ts @@ -20,8 +20,7 @@ class NoopScheduler implements CacheScheduler { /** A stubbed yfinance2 client that returns canned data — no network. */ interface StubYf { - optionsExpiryDates: (symbol: string) => Promise; - options: (symbol: string, expiry?: string) => Promise>; + options: (symbol: string, query?: { date?: string } | string) => Promise>; } /** Build an OptionsAdapter with a stubbed yf instance. */ @@ -94,7 +93,6 @@ function makeRawChainWithGreeks(): Record { test('fetchOne(greeks) returns single row for a specific strike', async () => { const raw = makeRawChainWithGreeks(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); @@ -112,7 +110,6 @@ test('fetchOne(greeks) returns single row for a specific strike', async () => { test('fetchOne(greeks) returns first row when strike is 0', async () => { const raw = makeRawChainWithGreeks(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); @@ -128,7 +125,6 @@ test('fetchOne(greeks) returns first row when strike is 0', async () => { test('fetchOne(greeks) returns null when strike not found', async () => { const raw = makeRawChainWithGreeks(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); @@ -141,7 +137,6 @@ test('fetchOne(greeks) returns null when strike not found', async () => { test('greeks kind preserves ttlClass as options_snapshot', async () => { const raw = makeRawChainWithGreeks(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); @@ -156,7 +151,6 @@ test('greeks kind round-trips through CacheRepository', async () => { const raw = makeRawChainWithGreeks(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); @@ -193,7 +187,6 @@ test('greeks kind: row with no greeks returns null greeks field', async () => { }; const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], options: async () => raw, }); diff --git a/app/server/src/adapters/__tests__/OptionsAdapter.test.ts b/app/server/src/adapters/__tests__/OptionsAdapter.test.ts index 068ed79..8bb7139 100644 --- a/app/server/src/adapters/__tests__/OptionsAdapter.test.ts +++ b/app/server/src/adapters/__tests__/OptionsAdapter.test.ts @@ -20,8 +20,7 @@ class NoopScheduler implements CacheScheduler { /** A stubbed yfinance2 client that returns canned data — no network. */ interface StubYf { - optionsExpiryDates: (symbol: string) => Promise; - options: (symbol: string, expiry?: string) => Promise>; + options: (symbol: string, query?: { date?: string } | string) => Promise>; } /** Build an OptionsAdapter with a stubbed yf instance. */ @@ -124,8 +123,7 @@ test('parseOptionChain wraps rows into OptionChain with symbol + expiration', () test('fetchOne(chain) returns OptionChainRow[] (rows array), not OptionChain object', async () => { const raw = makeRawChain(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17', '2026-08-21'], - options: async () => raw, + options: async () => raw, }); const result = await adapter.fetchOne('yfinance:chain:NVDA:2026-07-17'); @@ -139,8 +137,7 @@ test('fetchOne(chain) returns OptionChainRow[] (rows array), not OptionChain obj test('chain() convenience method returns OptionChain with rows populated', async () => { const raw = makeRawChain(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], - options: async () => raw, + options: async () => raw, }); const chain = await adapter.chain('NVDA', '2026-07-17'); @@ -165,8 +162,7 @@ test('round-trip: fetchOne(chain) value → CacheRepository.set → CacheReposit const raw = makeRawChain(); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], - options: async () => raw, + options: async () => raw, }); const key = 'yfinance:chain:NVDA:2026-07-17'; @@ -200,8 +196,13 @@ test('fetchOne(expiry_dates) returns string[] and round-trips via kv_cache', asy const repo = createCacheRepository({ db, scheduler }); const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17', '2026-08-21', '2026-09-18'], - options: async () => ({}), + options: async () => ({ + expirationDates: [ + new Date('2026-07-17T00:00:00.000Z'), + new Date('2026-08-21T00:00:00.000Z'), + new Date('2026-09-18T00:00:00.000Z'), + ], + }), }); const key = 'yfinance:expiry_dates:NVDA'; @@ -224,8 +225,7 @@ test('fetchOne(expiry_dates) returns string[] and round-trips via kv_cache', asy test('expiry_dates round-trip preserves sort order (ascending)', async () => { const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-09-18', '2026-07-17', '2026-08-21'], // unsorted input - options: async () => ({}), + options: async () => ({ expirationDates: ['2026-09-18', '2026-07-17', '2026-08-21'] }), }); const fetched = await adapter.fetchOne('yfinance:expiry_dates:TSLA'); @@ -237,8 +237,7 @@ test('expiry_dates round-trip preserves sort order (ascending)', async () => { test('chain() with empty rows returns OptionChain with empty rows array', async () => { const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => [], - options: async () => ({ calls: [], puts: [] }), + options: async () => ({ calls: [], puts: [] }), }); const chain = await adapter.chain('NOOPTIONS', '2026-07-17'); @@ -250,8 +249,7 @@ test('chain() with empty rows returns OptionChain with empty rows array', async test('fetchOne on unknown kind throws', async () => { const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => [], - options: async () => ({}), + options: async () => ({}), }); await assert.rejects( @@ -280,8 +278,7 @@ test('round-trip handles rows with missing optional greeks gracefully', async () }; const adapter = createStubbedAdapter({ - optionsExpiryDates: async () => ['2026-07-17'], - options: async () => raw, + options: async () => raw, }); const chain = await adapter.chain('AAPL', '2026-07-17'); diff --git a/app/server/src/adapters/__tests__/RedditAdapter.test.ts b/app/server/src/adapters/__tests__/RedditAdapter.test.ts index 062ec1b..967852e 100644 --- a/app/server/src/adapters/__tests__/RedditAdapter.test.ts +++ b/app/server/src/adapters/__tests__/RedditAdapter.test.ts @@ -2,9 +2,14 @@ // Verifies: subreddit fetch + cache; rate limit spacing (3s); degradation on 429; // attribution (author_handle) stored; FakeLLM post_summary returns canned text. -import { test } from 'node:test'; +import { test, beforeEach } from 'node:test'; import { strict as assert } from 'node:assert'; import { RedditAdapter } from '../RedditAdapter.ts'; +import { resetVendorGateForTests } from '../../services/vendorGate.ts'; + +beforeEach(() => { + resetVendorGateForTests(); +}); // ===== Fake LLM for deterministic canned summaries (NO real LLM/network) ===== diff --git a/app/server/src/adapters/__tests__/YFinanceAdapter.test.ts b/app/server/src/adapters/__tests__/YFinanceAdapter.test.ts index 4bfcb83..6a19b31 100644 --- a/app/server/src/adapters/__tests__/YFinanceAdapter.test.ts +++ b/app/server/src/adapters/__tests__/YFinanceAdapter.test.ts @@ -14,12 +14,64 @@ test('parseQuote extracts the live-quote fields from a recorded yf2 response', ( const raw = readFixture('nvda.quote.json'); const q = parseQuote(raw, 'NVDA'); assert.equal(q.symbol, 'NVDA'); - assert.equal(q.price, raw.regularMarketPrice); - assert.equal(q.change, raw.regularMarketChange); - assert.equal(q.changePercent, raw.regularMarketChangePercent); + // Fixture is POST with postMarketPrice — display last is extended, not RTH freeze. + assert.equal(raw.marketState, 'POST'); + assert.equal(q.session, 'post'); + assert.equal(q.price, raw.postMarketPrice); + assert.equal(q.regularPrice, raw.regularMarketPrice); assert.equal(q.bid, raw.bid); assert.equal(q.ask, raw.ask); assert.equal(q.iv, null); + // Change is vs previous close so it matches the extended print. + const prev = raw.regularMarketPreviousClose as number; + assert.ok(typeof prev === 'number'); + assert.ok(Math.abs((q.change ?? 0) - (q.price - prev)) < 1e-6); + assert.ok(Math.abs((q.changePercent ?? 0) - ((q.price - prev) / prev) * 100) < 1e-4); +}); + +test('parseQuote uses regularMarketPrice during REGULAR session', () => { + const q = parseQuote({ + marketState: 'REGULAR', + regularMarketPrice: 100, + regularMarketChange: 1.5, + regularMarketChangePercent: 1.52, + regularMarketPreviousClose: 98.5, + postMarketPrice: 101, + bid: 99.9, + ask: 100.1, + }, 'TEST'); + assert.equal(q.price, 100); + assert.equal(q.session, 'regular'); + assert.equal(q.regularPrice, 100); + assert.equal(q.change, 1.5); + assert.equal(q.changePercent, 1.52); +}); + +test('parseQuote prefers postMarketPrice in POST / POSTPOST', () => { + const q = parseQuote({ + marketState: 'POSTPOST', + regularMarketPrice: 37.93, + regularMarketChange: -0.96, + regularMarketChangePercent: -2.47, + regularMarketPreviousClose: 38.89, + postMarketPrice: 38.4, + postMarketChange: 0.47, + }, 'IREN'); + assert.equal(q.price, 38.4); + assert.equal(q.session, 'post'); + assert.equal(q.regularPrice, 37.93); + assert.ok(Math.abs((q.change ?? 0) - (38.4 - 38.89)) < 1e-6); +}); + +test('parseQuote prefers preMarketPrice in PRE', () => { + const q = parseQuote({ + marketState: 'PRE', + regularMarketPrice: 50, + regularMarketPreviousClose: 49, + preMarketPrice: 50.5, + }, 'TEST'); + assert.equal(q.price, 50.5); + assert.equal(q.session, 'pre'); }); test('parseQuote throws when regularMarketPrice is missing', () => { @@ -57,6 +109,18 @@ test('parseSymbol extracts sector/industry from summaryProfile', () => { assert.equal(s.peers, null); }); +test('parseSymbol prefers price.longName when summaryProfile has no name', () => { + const raw = { + summaryProfile: { sector: 'Technology', industry: 'Semiconductors' }, + price: { longName: 'NVIDIA Corporation', marketCap: 1e12, currency: 'USD' }, + quoteType: { shortName: 'NVIDIA' }, + }; + const s = parseSymbol(raw, 'NVDA'); + assert.equal(s.name, 'NVIDIA Corporation'); + assert.equal(s.marketCap, 1e12); + assert.equal(s.sector, 'Technology'); +}); + test('inferTickerKind: crypto/etf/index/equity', () => { assert.equal(inferTickerKind('BTC'), 'crypto'); assert.equal(inferTickerKind('SPY'), 'etf'); diff --git a/app/server/src/adapters/__tests__/backfill.test.ts b/app/server/src/adapters/__tests__/backfill.test.ts index a0818bc..8993e87 100644 --- a/app/server/src/adapters/__tests__/backfill.test.ts +++ b/app/server/src/adapters/__tests__/backfill.test.ts @@ -48,18 +48,20 @@ test("subscribe enqueues 4 pending rows for NVDA/equity", async () => { await cache.subscribe("NVDA", "equity"); - // adapter_queue should have exactly 4 pending rows. + // adapter_queue should have exactly 6 pending rows (includes short interest keys from Phase 3). const pending = db.prepare( "SELECT key FROM adapter_queue WHERE status='pending'", ).all() as Array<{ key: string }>; - assert.equal(pending.length, 4, "expected 4 pending rows after subscribe"); + assert.equal(pending.length, 6, "expected 6 pending rows after subscribe"); const keys = pending.map((r) => r.key).sort(); assert.deepEqual(keys, [ + "nasdaq:nasdaqShortinterest:NVDA", "yfinance:adjustments:NVDA", "yfinance:candles:NVDA:1d", "yfinance:quote:NVDA", + "yfinance:shortinterest:NVDA", "yfinance:symbol:NVDA", ]); }); @@ -129,14 +131,18 @@ test("re-subscribe increments refcount but does NOT re-queue (no duplicate fetch const db = createDb({ path: ":memory:" }); initSchema(db); + // Use timestamps within CANDLE_FRESH_MS (36h) so drains mark candles as fresh. + const now = new Date(); + const yesterday = new Date(now.getTime() - 24 * 60 * 60 * 1000).toISOString().slice(0, 10); + const fake = new FakeSourceAdapter("yfinance"); fake.set("yfinance:quote:NVDA", { symbol: "NVDA", price: 131 }, "live_quote"); fake.set("yfinance:symbol:NVDA", { symbol: "NVDA", sector: "Technology", tickerKind: "equity" }, "symbol_meta"); fake.set( "yfinance:candles:NVDA:1d", [ - { ts: "2026-06-27", o: 192, h: 196, l: 191, c: 194.97, v: 1e8, adjClose: 194.9 }, - { ts: "2026-06-28", o: 194, h: 197, l: 193, c: 195, v: 9e7, adjClose: 195 }, + { ts: yesterday, o: 192, h: 196, l: 191, c: 194.97, v: 1e8, adjClose: 194.9 }, + { ts: now.toISOString().slice(0, 10), o: 194, h: 197, l: 193, c: 195, v: 9e7, adjClose: 195 }, ], "daily_permanent", ); @@ -161,17 +167,18 @@ test("re-subscribe increments refcount but does NOT re-queue (no duplicate fetch // Capture call count AFTER drain (all 4 fetches completed). const afterFirstDrain = fake.calls.length; - // Now re-subscribe — refcount goes to 2, but since NVDA is already first-demand - // and the cache is warm, no new queue entries should appear. + // Now re-subscribe — refcount goes to 2, and queueIfNeeded re-queues stale/missing kinds. const rowsBefore = db.prepare("SELECT key FROM adapter_queue WHERE status='pending'").all(); await cache.subscribe("NVDA", "equity"); + // After drain, quote/candle/symbol_meta are considered fresh (written by first drain). + // queueIfNeeded should NOT re-queue anything since all kinds are present. const rowsAfter = db.prepare("SELECT key FROM adapter_queue WHERE status='pending'").all(); assert.deepEqual( rowsAfter.map((r) => r.key).sort(), rowsBefore.map((r) => r.key).sort(), - "re-subscribe should not add new pending rows", + "re-subscribe should not add new pending rows after warm drain", ); // Drain again: no new work, so fake.calls.length must stay the same. diff --git a/app/server/src/admin/admin.ts b/app/server/src/admin/admin.ts index a134f9b..c33a607 100644 --- a/app/server/src/admin/admin.ts +++ b/app/server/src/admin/admin.ts @@ -213,6 +213,99 @@ export function queueHealth(db: DatabaseSync): QueueHealthRow[] { .all() as unknown as QueueHealthRow[]; } +/** Latest run of every registered alert producer, from producer_run_log. */ +export interface ProducerRunStatus { + producer: string; + frequency: string; + lastRunAt: string | null; + lastEventsFired: number; + lastDurationMs: number; + lastError: string | null; +} + +export function producerRunStatus(db: DatabaseSync): ProducerRunStatus[] { + try { + return db + .prepare( + `SELECT producer, frequency, + MAX(run_at) AS last_run_at, + events_fired AS last_events_fired, + duration_ms AS last_duration_ms, + error AS last_error + FROM producer_run_log + WHERE run_at = (SELECT MAX(r2.run_at) FROM producer_run_log r2 WHERE r2.producer = producer_run_log.producer) + GROUP BY producer`, + ) + .all() as unknown as ProducerRunStatus[]; + } catch { + return []; + } +} + +/** + * Unified alert-system observability: producer cadence/history plus the data + * age of the sources each producer reads. This ties "queue health" to "alert + * output" so operators no longer diagnose the two separately. + */ +export interface AlertStatusSnapshot { + producers: ProducerRunStatus[]; + subscriptionsByType: Array<{ alertType: string; enabledCount: number }>; + freshness: { + vixLevelAgeSec: number | null; + latestScFilingAt: string | null; + latestInsiderAt: string | null; + oldestQuoteAgeSec: number | null; + }; +} + +export function alertStatus(db: DatabaseSync): AlertStatusSnapshot { + const producers = producerRunStatus(db); + + let subscriptionsByType: Array<{ alertType: string; enabledCount: number }> = []; + try { + subscriptionsByType = db.prepare( + `SELECT alert_type, COUNT(*) AS enabledCount + FROM alerts WHERE enabled = 1 + GROUP BY alert_type ORDER BY alert_type ASC`, + ).all() as unknown as Array<{ alertType: string; enabledCount: number }>; + } catch { + /* alerts may not exist in minimal contexts */ + } + + // How stale is each source's newest data? (null when the source has nothing.) + const vixRow = db.prepare( + `SELECT observed_at FROM quotes WHERE symbol = '^VIX'`, + ).get() as { observed_at: string } | undefined; + const scRow = db.prepare( + `SELECT MAX(filed_at) AS filed_at FROM institution_filings + WHERE form LIKE '13D%' OR form LIKE '13G%' OR form IN ('13D','13G')`, + ).get() as { filed_at: string | null } | undefined; + const insiderRow = db.prepare( + `SELECT MAX(filed_at) AS filed_at FROM insider_transactions`, + ).get() as { filed_at: string | null } | undefined; + const oldestQuoteRow = db.prepare( + `SELECT observed_at FROM quotes ORDER BY observed_at DESC LIMIT 1`, + ).get() as { observed_at: string } | undefined; + + const ageSec = (iso: string | null | undefined): number | null => { + if (!iso) return null; + const t = Date.parse(iso); + if (!Number.isFinite(t)) return null; + return Math.max(0, Math.floor((Date.now() - t) / 1000)); + }; + + return { + producers, + subscriptionsByType, + freshness: { + vixLevelAgeSec: ageSec(vixRow?.observed_at), + latestScFilingAt: scRow?.filed_at ?? null, + latestInsiderAt: insiderRow?.filed_at ?? null, + oldestQuoteAgeSec: ageSec(oldestQuoteRow?.observed_at), + }, + }; +} + /** List active sessions for a user (read-only). Never returns pw_hash. */ export interface SessionRecord { id: string; diff --git a/app/server/src/admin/cli.ts b/app/server/src/admin/cli.ts index 5e83918..88e62e5 100644 --- a/app/server/src/admin/cli.ts +++ b/app/server/src/admin/cli.ts @@ -33,7 +33,7 @@ async function readHidden(prompt: string): Promise { } function printGdprFooter() { - stdout.write('\nEducational analysis, not investment advice. Verify the underlying data; you are responsible for your own decisions.\n'); + stdout.write('\nEducational observation only.\n'); } async function main() { diff --git a/app/server/src/alerts/AlertEngine.ts b/app/server/src/alerts/AlertEngine.ts index 56f5fc8..a4ca717 100644 --- a/app/server/src/alerts/AlertEngine.ts +++ b/app/server/src/alerts/AlertEngine.ts @@ -20,7 +20,11 @@ export type AlertType = | 'thesis_weakening' | 'cluster_breach' | 'drawdown_halt' - | 'asymmetry_warning'; + | 'asymmetry_warning' + | 'fund_capture' + | 'fund_13f' + | 'mirror_diff' + | 'vix_level'; // ─── Alert Severity ────────────────────────────────────────────────────────── @@ -77,6 +81,10 @@ export function defaultSeverity(type: AlertType): AlertSeverity { case 'regime_shift': case 'conviction_unlock': case 'thesis_weakening': + case 'fund_capture': + case 'fund_13f': + case 'mirror_diff': + case 'vix_level': return 'info'; } } @@ -111,6 +119,14 @@ export function alertTitle(type: AlertType, symbol?: string): string { return `Drawdown tolerance breached`; case 'asymmetry_warning': return `Portfolio asymmetry below threshold`; + case 'fund_capture': + return `New position update from tracked fund${symbolTag(symbol)}`; + case 'fund_13f': + return `New 13F from tracked fund`; + case 'mirror_diff': + return `Mirror target changed${symbolTag(symbol)}`; + case 'vix_level': + return `Volatility index level changed`; } } @@ -122,7 +138,7 @@ export function alertDescription(type: AlertType, details?: string, symbol?: str case 'informed_sell': return `A company insider sold shares${symbolTag(symbol)}. This filing was not part of a 10b5-1 trading plan.`; case 'new_13da': - return `An institutional investor reported a new position${symbolTag(symbol)} in a 13F filing.`; + return `An institutional investor reported a new position${symbolTag(symbol)}.`; case 'rotation_incipient': return `The sector rotation detector identified an incipient rotation signal. Capital may be moving between sectors.`; case 'regime_shift': @@ -139,6 +155,14 @@ export function alertDescription(type: AlertType, details?: string, symbol?: str return `Your portfolio drawdown has exceeded the tolerance threshold. The circuit breaker has paused new entries for 24 hours. Existing positions continue unaffected.`; case 'asymmetry_warning': return `Your portfolio's reward-to-risk ratio has fallen below 1.0, meaning risk outweighs expected reward across your positions.`; + case 'fund_capture': + return `A tracked fund posted a position update${symbolTag(symbol)}. This is a disclosure, not advice.`; + case 'fund_13f': + return `A tracked fund filed a new 13F. This is a disclosure, not advice.`; + case 'mirror_diff': + return `The mirror target changed${symbolTag(symbol)}. Showing the arithmetic delta; it is not advice.`; + case 'vix_level': + return `The VIX, a market-wide measure of expected near-term volatility, has moved into a new level that historically mattered to market participants.`; } })(); @@ -280,4 +304,8 @@ export const ALERT_THROTTLE: Record = { cluster_breach: { maxPerHour: 2 }, drawdown_halt: { maxPerHour: 1 }, asymmetry_warning: { maxPerHour: 2 }, + fund_capture: { maxPerHour: 3 }, + fund_13f: { maxPerHour: 3 }, + mirror_diff: { maxPerHour: 3 }, + vix_level: { maxPerHour: 1 }, }; diff --git a/app/server/src/alerts/__tests__/AlertEngine.test.ts b/app/server/src/alerts/__tests__/AlertEngine.test.ts index c161e6e..625323f 100644 --- a/app/server/src/alerts/__tests__/AlertEngine.test.ts +++ b/app/server/src/alerts/__tests__/AlertEngine.test.ts @@ -43,7 +43,7 @@ test('alertDescription never contains trade verbs', () => { const types: AlertType[] = [ 'informed_buy', 'informed_sell', 'new_13da', 'rotation_incipient', 'regime_shift', 'conviction_unlock', 'thesis_broken', 'thesis_weakening', - 'cluster_breach', 'drawdown_halt', 'asymmetry_warning', + 'cluster_breach', 'drawdown_halt', 'asymmetry_warning', 'vix_level', ]; const forbidden = ['buy ', 'sell ', 'cut ', 'trim ', 'buy.', 'sell.', 'cut.', 'trim.']; for (const type of types) { @@ -229,7 +229,7 @@ test('ALERT_THROTTLE defines limits for all alert types', () => { const types: AlertType[] = [ 'informed_buy', 'informed_sell', 'new_13da', 'rotation_incipient', 'regime_shift', 'conviction_unlock', 'thesis_broken', 'thesis_weakening', - 'cluster_breach', 'drawdown_halt', 'asymmetry_warning', + 'cluster_breach', 'drawdown_halt', 'asymmetry_warning', 'vix_level', ]; for (const type of types) { assert.ok(ALERT_THROTTLE[type], `Throttle config exists for ${type}`); diff --git a/app/server/src/alerts/__tests__/mirrorAlertProducers.test.ts b/app/server/src/alerts/__tests__/mirrorAlertProducers.test.ts new file mode 100644 index 0000000..b76b99f --- /dev/null +++ b/app/server/src/alerts/__tests__/mirrorAlertProducers.test.ts @@ -0,0 +1,78 @@ +// Mirror Portfolio (M21) — alert producers tests (sub-task 6). +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { fundCaptureProducer, fund13FRecordProducer, mirrorDiffProducer } from '../producers/mirrorAlertProducers.ts'; +import { upsertTrackedFund, insertPositionRecord } from '../../db/fundRepository.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', '..', 'db', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +function subscribe(db: DatabaseSync, userId: string, type: string, symbol: string | null) { + db.prepare( + `INSERT OR IGNORE INTO users (id, email, pw_hash, created_at) + VALUES (?, ?, '', ?)`, + ).run(userId, `${userId}@t.local`, new Date().toISOString()); + db.prepare( + `INSERT INTO alerts (id, owner_id, symbol, alert_type, enabled, params, created_at) + VALUES (?, ?, ?, ?, 1, '{}', ?)`, + ).run(`sub-${userId}-${type}-${symbol ?? 'x'}`, userId, symbol, type, new Date().toISOString()); +} + +const hasAdvice = (s: string) => /you should\b|^\s*(buy|sell)\s/i.test(s); + +test('fund_capture: fires once per subscribed user, dedupes, mechanical framing', async () => { + const db = freshDb(); + const fund = upsertTrackedFund(db, { ci_key: '0002096493', fund_name: 'Alpine Fox Capital', manager_name: 'M' }); + subscribe(db, 'u1', 'fund_capture', 'IREN'); + subscribe(db, 'u2', 'fund_capture', 'IREN'); + + insertPositionRecord(db, { + fund_id: fund.id, symbol: 'IREN', shares: 750000, cost_basis: 22.4, + as_of: '2026-02-10', source: 'capture', evidence_url: 'https://x.com/mikealfred/status/1', + }); + + const first = await fundCaptureProducer.check(db); + assert.equal(first.length, 2); + for (const a of first) assert.ok(!hasAdvice(a.description), a.description); + assert.match(first[0].description, /shares/); + + const second = await fundCaptureProducer.check(db); + assert.equal(second.length, 0); // no new post → no re-fire +}); + +test('mirror_diff: fires on a book change, mechanical framing, dedupes', async () => { + const db = freshDb(); + const fund = upsertTrackedFund(db, { ci_key: '0002096493', fund_name: 'Fund X', manager_name: 'M' }); + subscribe(db, 'u1', 'mirror_diff', 'IREN'); + insertPositionRecord(db, { fund_id: fund.id, symbol: 'IREN', shares: 1000, as_of: '2026-02-01', source: 'capture', evidence_url: 'p1' }); + const out = await mirrorDiffProducer.check(db); + assert.equal(out.length, 1); + assert.match(out[0].description, /delta/i); + assert.ok(!hasAdvice(out[0].description), out[0].description); + assert.equal((await mirrorDiffProducer.check(db)).length, 0); +}); + +test('fund_13f: fires on a new quarter for the tracked fund CIK', async () => { + const db = freshDb(); + upsertTrackedFund(db, { ci_key: '0002096493', fund_name: 'Fund Coin', manager_name: 'A' }); + subscribe(db, 'u1', 'fund_13f', 'IREN'); + db.prepare( + `INSERT INTO institution_filings (filer_cik, filer_name, symbol, form, shares, reported_quarter, filed_at, fetched_at, accession) + VALUES ('0002096493','Fund Coin','IREN','13F-HR',100,'2026-Q1','2026-02-15','2026-02-15','ACC-1')`, + ).run(); + const out = await fund13FRecordProducer.check(db); + assert.equal(out.length, 1); + assert.match(out[0].description, /13F/i); + assert.equal((await fund13FRecordProducer.check(db)).length, 0); +}); \ No newline at end of file diff --git a/app/server/src/alerts/__tests__/new13daProducer.test.ts b/app/server/src/alerts/__tests__/new13daProducer.test.ts new file mode 100644 index 0000000..0847815 --- /dev/null +++ b/app/server/src/alerts/__tests__/new13daProducer.test.ts @@ -0,0 +1,112 @@ +// new_13da producer regression tests — institutional 13D/13G filing alerts. +// +// Guards the "never broadcast to non-subscribers" rule: a new 13D/13G filing for a +// symbol must only alert users who explicitly subscribed (ticker-level or global). +// Regression for: alert fired for CDNS/HOOD with zero subscriptions because the +// producer fell back to "all users" when getSubscribedUsers returned empty. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { new13daProducer } from '../producers/new13daProducer.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', '..', 'db', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +function addUser(db: DatabaseSync, id: string) { + db.prepare( + `INSERT OR IGNORE INTO users (id, email, pw_hash, created_at) + VALUES (?, ?, '', ?)`, + ).run(id, `${id}@t.local`, new Date().toISOString()); +} + +function subscribe(db: DatabaseSync, userId: string, symbol: string | null) { + db.prepare( + `INSERT INTO alerts (id, owner_id, symbol, alert_type, enabled, params, created_at) + VALUES (?, ?, ?, 'new_13da', 1, '{}', ?)`, + ).run(`sub-${userId}-${symbol ?? 'x'}`, userId, symbol, new Date().toISOString()); +} + +function seedFiling(db: DatabaseSync, symbol: string, accession: string, quarter: string, filedAt = new Date().toISOString().slice(0, 10)) { + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, accession, fetched_at) + VALUES (?, ?, ?, '13D', NULL, NULL, ?, ?, ?, ?)`, + ).run('0009999999', 'The Vanguard Group', symbol, quarter, filedAt, accession, filedAt); +} + +test('new_13da: zero subscriptions → NO alert (no broadcast to all users)', async () => { + const db = freshDb(); + addUser(db, 'u1'); + addUser(db, 'u2'); + seedFiling(db, 'CDNS', 'acc-1', '2026-Q1'); + + const out = await new13daProducer.check(db); + assert.equal(out.length, 0, 'no subscribers → no alerts, regardless of users present'); + const events = db.prepare('SELECT COUNT(*) AS n FROM alert_events WHERE type = ?').get('new_13da') as { n: number }; + assert.equal(events.n, 0); +}); + +test('new_13da: ticker-level subscriber gets the alert, other users do not', async () => { + const db = freshDb(); + addUser(db, 'u1'); + addUser(db, 'u2'); + subscribe(db, 'u1', 'CDNS'); + seedFiling(db, 'CDNS', 'acc-1', '2026-Q1'); + + const out = await new13daProducer.check(db); + assert.equal(out.length, 1); + assert.equal(out[0].userId, 'u1'); + assert.equal(out[0].symbol, 'CDNS'); + assert.match(out[0].description, /2026-Q1/); + + const recipients = db.prepare('SELECT DISTINCT user_id FROM alert_events').all() as Array<{ user_id: string }>; + assert.deepEqual(recipients.map((r) => r.user_id).sort(), ['u1']); +}); + +test('new_13da: global subscriber (symbol IS NULL) gets alerts for any symbol', async () => { + const db = freshDb(); + addUser(db, 'u1'); + addUser(db, 'u2'); + subscribe(db, 'u1', null); // global subscription + seedFiling(db, 'HOOD', 'acc-2', '2026-Q1'); + + const out = await new13daProducer.check(db); + assert.equal(out.length, 1); + assert.equal(out[0].userId, 'u1'); +}); + +test('new_13da: same accession never re-alerts (comparison state + dedup)', async () => { + const db = freshDb(); + addUser(db, 'u1'); + subscribe(db, 'u1', 'CDNS'); + seedFiling(db, 'CDNS', 'acc-1', '2026-Q1'); + + const first = await new13daProducer.check(db); + assert.equal(first.length, 1); + const second = await new13daProducer.check(db); + assert.equal(second.length, 0, 'same latest accession → no re-fire'); +}); + +test('new_13da: second filing with newer accession alerts once each', async () => { + const db = freshDb(); + addUser(db, 'u1'); + subscribe(db, 'u1', 'IREN'); + const daysAgo = (n: number) => new Date(Date.now() - n * 86400000).toISOString().slice(0, 10); + seedFiling(db, 'IREN', 'acc-a', '2026-Q2', daysAgo(3)); + const first = await new13daProducer.check(db); + assert.equal(first.length, 1); + + seedFiling(db, 'IREN', 'acc-b', '2026-Q2', daysAgo(1)); + const second = await new13daProducer.check(db); + assert.equal(second.length, 1); + assert.match(second[0].description, /acc-b|The Vanguard Group/); +}); diff --git a/app/server/src/alerts/__tests__/producerSuite.test.ts b/app/server/src/alerts/__tests__/producerSuite.test.ts new file mode 100644 index 0000000..0f4fc5f --- /dev/null +++ b/app/server/src/alerts/__tests__/producerSuite.test.ts @@ -0,0 +1,290 @@ +// Producer tests for the Phase-4.5 alert producers: +// rotation_incipient / regime_shift / conviction_unlock / thesis_broken / +// thesis_weakening / cluster_breach / drawdown_halt / asymmetry_warning +// +// Guards the core contracts: +// - Watermarks prevent re-firing on repeated checks of the same state. +// - First observation of a baseline is silent (regime, unlock, thesis) or +// fires once and never again (rotation incipient signals). +// - Owners of an unlock/thesis/portfolio are always notified of their own +// state even without an explicit subscription. +// - Portfolio risk alerts fire on the breach transition, not continuously. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { rotationIncipientProducer, regimeShiftProducer } from '../producers/rotationProducer.ts'; +import { convictionUnlockProducer } from '../producers/unlockProducer.ts'; +import { thesisBrokenProducer, thesisWeakeningProducer } from '../producers/thesisProducer.ts'; +import { clusterBreachProducer, drawdownHaltProducer, asymmetryWarningProducer } from '../producers/portfolioRiskProducer.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', '..', 'db', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +function addUser(db: DatabaseSync, id: string, opts: { complexity?: string; drawdownTolerance?: number } = {}) { + db.prepare( + `INSERT OR IGNORE INTO users (id, email, pw_hash, complexity, drawdown_tolerance, created_at) + VALUES (?, ?, '', ?, ?, ?)`, + ).run(id, `${id}@t.local`, opts.complexity ?? 'beginner', opts.drawdownTolerance ?? 20, new Date().toISOString()); +} + +function subscribeGlobal(db: DatabaseSync, userId: string, alertType: string) { + db.prepare( + `INSERT INTO alerts (id, owner_id, symbol, alert_type, enabled, params, created_at) + VALUES (?, ?, NULL, ?, 1, '{}', ?)`, + ).run(`sub-${userId}-${alertType}`, userId, alertType, new Date().toISOString()); +} + +function alertCount(db: DatabaseSync, type: string): number { + const r = db.prepare('SELECT COUNT(*) AS n FROM alert_events WHERE type = ?').get(type) as { n: number }; + return r.n; +} + +function comparisonState(db: DatabaseSync, symbol: string, alertType: string): Record { + const r = db.prepare( + 'SELECT state FROM alert_comparison_state WHERE symbol = ? AND alert_type = ?', + ).get(symbol, alertType) as { state: string } | undefined; + return r ? JSON.parse(r.state) : {}; +} + +// ─── rotation_incipient ────────────────────────────────────────────────────── + +function seedRotation(db: DatabaseSync, sector: string, phase: string, detectedAt: string, confidence = 0.8) { + db.prepare( + `INSERT INTO rotation_signals (id, sector, detected_at, phase, confidence, price_resolved, inst_resolved) + VALUES (?, ?, ?, ?, ?, 0, 0)`, + ).run(`${sector}-${detectedAt}`, sector, detectedAt, phase, confidence); +} + +test('rotation_incipient: fires for a new accumulation signal, watermarks so it never re-fires', async () => { + const db = freshDb(); + addUser(db, 'u1'); + subscribeGlobal(db, 'u1', 'rotation_incipient'); + seedRotation(db, 'XLK', 'accumulation', '2026-04-01T10:00:00Z', 0.85); + + const out1 = await rotationIncipientProducer.check(db); + assert.equal(out1.length, 1); + assert.equal(out1[0].type, 'rotation_incipient'); + assert.equal(out1[0].symbol, 'XLK'); + assert.equal(alertCount(db, 'rotation_incipient'), 1); + + // Second check: same signal → silent. + const out2 = await rotationIncipientProducer.check(db); + assert.equal(out2.length, 0); + assert.equal(alertCount(db, 'rotation_incipient'), 1); + + // New signal on the same sector → fires again. + seedRotation(db, 'XLK', 'expansion', '2026-04-02T10:00:00Z'); + const out3 = await rotationIncipientProducer.check(db); + assert.equal(out3.length, 1); + assert.equal(alertCount(db, 'rotation_incipient'), 2); +}); + +test('rotation_incipient: distribution phase is watermarked but never fires', async () => { + const db = freshDb(); + addUser(db, 'u1'); + subscribeGlobal(db, 'u1', 'rotation_incipient'); + seedRotation(db, 'XLV', 'distribution', '2026-04-01T10:00:00Z'); + + const out = await rotationIncipientProducer.check(db); + assert.equal(out.length, 0); + assert.equal(alertCount(db, 'rotation_incipient'), 0); + assert.ok(comparisonState(db, 'XLV', 'rotation_incipient').lastDetectedAt); +}); + +// ─── regime_shift ──────────────────────────────────────────────────────────── + +function seedRegime(db: DatabaseSync, asOf: string, regime: string) { + db.prepare( + `INSERT INTO macro_regimes (as_of, regime, signals) VALUES (?, ?, '{}')`, + ).run(asOf, regime); +} + +test('regime_shift: first observation is baseline (silent), change fires once', async () => { + const db = freshDb(); + addUser(db, 'u1'); + subscribeGlobal(db, 'u1', 'regime_shift'); + seedRegime(db, '2026-04-01', 'trending_up'); + + const out1 = await regimeShiftProducer.check(db); + assert.equal(out1.length, 0, 'baseline observation must not page anyone'); + assert.equal(alertCount(db, 'regime_shift'), 0); + + // Same regime, new as_of → still silent. + seedRegime(db, '2026-04-02', 'trending_up'); + const out2 = await regimeShiftProducer.check(db); + assert.equal(out2.length, 0); + + // Regime change → fires. + seedRegime(db, '2026-04-03', 'range_bound'); + const out3 = await regimeShiftProducer.check(db); + assert.equal(out3.length, 1); + assert.equal(out3[0].type, 'regime_shift'); + assert.equal(alertCount(db, 'regime_shift'), 1); + + // Repeated check → no re-fire. + const out4 = await regimeShiftProducer.check(db); + assert.equal(out4.length, 0); +}); + +// ─── conviction_unlock ─────────────────────────────────────────────────────── + +function seedUnlock(db: DatabaseSync, userId: string, unlock: string, earnedAt: string) { + db.prepare( + `INSERT INTO sizing_unlocks (user_id, unlock, earned_at, evidence) + VALUES (?, ?, ?, '{"winRate":0.62}')`, + ).run(userId, unlock, earnedAt); +} + +test('conviction_unlock: baseline silent, new unlock notifies the earning user directly', async () => { + const db = freshDb(); + addUser(db, 'u1'); + seedUnlock(db, 'u1', 'tier_a', '2026-04-01T00:00:00Z'); + + const out1 = await convictionUnlockProducer.check(db); + assert.equal(out1.length, 0, 'existing unlock at boot must not page anyone'); + assert.equal(alertCount(db, 'conviction_unlock'), 0); + + // Update earned_at (re-earned) → fires, recipient includes the owner even + // though u1 never subscribed. + seedUnlock(db, 'u1', 'tier_a_star', '2026-04-02T00:00:00Z'); + const out2 = await convictionUnlockProducer.check(db); + assert.equal(out2.length, 1); + assert.equal(out2[0].userId, 'u1'); + assert.equal(out2[0].payload?.unlock, 'tier_a_star'); + + const out3 = await convictionUnlockProducer.check(db); + assert.equal(out3.length, 0, 'same unlock must not re-fire'); +}); + +// ─── thesis_broken / thesis_weakening ──────────────────────────────────────── + +function seedThesis(db: DatabaseSync, id: string, userId: string, symbol: string, status = 'intact') { + db.prepare( + `INSERT INTO theses (id, user_id, symbol, statement, invalidation_criteria, status, created_at, updated_at) + VALUES (?, ?, ?, 't', '[]', ?, ?, ?)`, + ).run(id, userId, symbol, status, new Date().toISOString(), new Date().toISOString()); +} + +function setThesisStatus(db: DatabaseSync, id: string, status: string, updatedAt: string) { + db.prepare('UPDATE theses SET status = ?, updated_at = ? WHERE id = ?').run(status, updatedAt, id); +} + +test('thesis_broken: baseline silent, status transition fires for the owner', async () => { + const db = freshDb(); + addUser(db, 'u1'); + seedThesis(db, 'th1', 'u1', 'AAPL', 'intact'); + + // Baseline: intact → nothing. + const out0 = await thesisBrokenProducer.check(db); + assert.equal(out0.length, 0); + + // Transition intact → broken. + setThesisStatus(db, 'th1', 'broken', '2026-04-05T00:00:00Z'); + const out1 = await thesisBrokenProducer.check(db); + assert.equal(out1.length, 1); + assert.equal(out1[0].type, 'thesis_broken'); + assert.equal(out1[0].symbol, 'AAPL'); + assert.equal(out1[0].userId, 'u1', 'owner notified without an explicit subscription'); + assert.equal(alertCount(db, 'thesis_broken'), 1); + + const out2 = await thesisBrokenProducer.check(db); + assert.equal(out2.length, 0, 'persisted broken status must not re-fire'); +}); + +test('thesis_weakening: fires on transition to weakening, distinct from broken', async () => { + const db = freshDb(); + addUser(db, 'u1'); + seedThesis(db, 'th2', 'u1', 'MSFT', 'intact'); + await thesisWeakeningProducer.check(db); // baseline + + setThesisStatus(db, 'th2', 'weakening', '2026-04-06T00:00:00Z'); + const out1 = await thesisWeakeningProducer.check(db); + assert.equal(out1.length, 1); + assert.equal(out1[0].type, 'thesis_weakening'); + assert.equal(alertCount(db, 'thesis_weakening'), 1); + assert.equal(alertCount(db, 'thesis_broken'), 0, 'weakening must not fire broken'); +}); + +// ─── cluster_breach / drawdown_halt / asymmetry_warning ───────────────────── + +function seedHolding(db: DatabaseSync, id: string, owner: string, symbol: string, qty: number, avgCost: number) { + db.prepare( + `INSERT INTO portfolio_holdings (id, owner_id, symbol, qty, avg_cost, acquired_at, status) + VALUES (?, ?, ?, ?, ?, ?, 'open')`, + ).run(id, owner, symbol, qty, avgCost, new Date().toISOString()); +} + +function seedTrade(db: DatabaseSync, owner: string, symbol: string, stopLoss: number, targets: Array<{ price: number }>) { + db.prepare( + `INSERT INTO trades (id, owner_id, trade_id, symbol, date, status, tier, stop_loss, targets) + VALUES (?, ?, ?, ?, ?, 'active', 'B', ?, ?)`, + ).run(`t-${owner}-${symbol}`, owner, `tid-${owner}-${symbol}`, symbol, new Date().toISOString().slice(0, 10), stopLoss, JSON.stringify(targets)); +} + +test('cluster_breach: fires when uncategorized exposure exceeds the beginner cap', async () => { + const db = freshDb(); + addUser(db, 'u1', { complexity: 'beginner', drawdownTolerance: 20 }); + // $10,000 equity, all in one symbol → 100% in "uncategorized" cluster. + seedHolding(db, 'h1', 'u1', 'AAPL', 100, 100); + + const out = await clusterBreachProducer.check(db); + assert.equal(out.length, 1, '100% of equity in one cluster exceeds the 25% beginner cap'); + assert.equal(out[0].type, 'cluster_breach'); + assert.equal(out[0].userId, 'u1'); + assert.equal(alertCount(db, 'cluster_breach'), 1); + + // Repeated check with same exposure → no re-fire (watermarked). + const out2 = await clusterBreachProducer.check(db); + assert.equal(out2.length, 0); + assert.equal(alertCount(db, 'cluster_breach'), 1); +}); + +test('drawdown_halt: fires on the breach transition only', async () => { + const db = freshDb(); + addUser(db, 'u1', { complexity: 'beginner', drawdownTolerance: 20 }); + seedHolding(db, 'h1', 'u1', 'AAPL', 100, 100); + + // First check: baseline peak equity = current equity → no breach. + const out1 = await drawdownHaltProducer.check(db); + assert.equal(out1.length, 0); + + // Mark drops to $50 → equity $5,000, 50% drawdown vs 20% tolerance → breach. + db.prepare( + `INSERT INTO quotes (symbol, price, observed_at) VALUES ('AAPL', 50, ?)`, + ).run(new Date().toISOString()); + const out2 = await drawdownHaltProducer.check(db); + assert.equal(out2.length, 1); + assert.equal(out2[0].type, 'drawdown_halt'); + assert.equal(alertCount(db, 'drawdown_halt'), 1); + + // Still breached → silent (already alerted). + const out3 = await drawdownHaltProducer.check(db); + assert.equal(out3.length, 0); + assert.equal(alertCount(db, 'drawdown_halt'), 1); +}); + +test('asymmetry_warning: fires when reward:risk is < 1 and watermarks', async () => { + const db = freshDb(); + addUser(db, 'u1', { complexity: 'beginner', drawdownTolerance: 20 }); + seedHolding(db, 'h1', 'u1', 'AAPL', 100, 100); + // Reward 5/share (105 target) vs risk 50/share (50 stop) → asymmetry 0.1. + seedTrade(db, 'u1', 'AAPL', 50, [{ price: 105 }]); + + const out = await asymmetryWarningProducer.check(db); + assert.equal(out.length, 1); + assert.equal(out[0].type, 'asymmetry_warning'); + assert.ok((out[0].payload?.asymmetry as number) < 1, 'payload carries the aggregate asymmetry ratio'); + assert.equal(alertCount(db, 'asymmetry_warning'), 1); + + const out2 = await asymmetryWarningProducer.check(db); + assert.equal(out2.length, 0, 'same asymmetry bucket must not re-fire'); +}); diff --git a/app/server/src/alerts/__tests__/vixLevelProducer.test.ts b/app/server/src/alerts/__tests__/vixLevelProducer.test.ts new file mode 100644 index 0000000..e3efe0f --- /dev/null +++ b/app/server/src/alerts/__tests__/vixLevelProducer.test.ts @@ -0,0 +1,99 @@ +// VIX level producer tests. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { vixLevelProducer, vixZoneFor, VIX_SYMBOL } from '../producers/vixLevelProducer.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', '..', 'db', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +function subscribe(db: DatabaseSync, userId: string, type: string, symbol: string | null) { + db.prepare( + `INSERT OR IGNORE INTO users (id, email, pw_hash, created_at) + VALUES (?, ?, '', ?)`, + ).run(userId, `${userId}@t.local`, new Date().toISOString()); + db.prepare( + `INSERT INTO alerts (id, owner_id, symbol, alert_type, enabled, params, created_at) + VALUES (?, ?, ?, ?, 1, '{}', ?)`, + ).run(`sub-${userId}-${type}-${symbol ?? 'x'}`, userId, symbol, type, new Date().toISOString()); +} + +function setVix(db: DatabaseSync, price: number) { + db.prepare( + `INSERT OR REPLACE INTO quotes (symbol, price, observed_at) + VALUES (?, ?, ?)`, + ).run(VIX_SYMBOL, price, new Date().toISOString()); +} + +test('vixZoneFor maps levels to named bands', () => { + assert.equal(vixZoneFor(62).zone, 'extreme'); + assert.equal(vixZoneFor(50).zone, 'stress'); + assert.equal(vixZoneFor(38).zone, 'elevated'); + assert.equal(vixZoneFor(20).zone, 'watch'); + assert.equal(vixZoneFor(12).zone, 'calm'); + assert.equal(vixZoneFor(80).zone, 'extreme'); +}); + +test('vix_level: first observation seeds watermark without firing', async () => { + const db = freshDb(); + subscribe(db, 'u1', 'vix_level', null); + setVix(db, 20); + const out = await vixLevelProducer.check(db); + assert.equal(out.length, 0); +}); + +test('vix_level: fires once when the band changes, dedups within band', async () => { + const db = freshDb(); + subscribe(db, 'u1', 'vix_level', null); + subscribe(db, 'u2', 'vix_level', null); + + setVix(db, 20); + await vixLevelProducer.check(db); + + setVix(db, 40); + const first = await vixLevelProducer.check(db); + assert.equal(first.length, 2); + for (const a of first) { + assert.equal(a.type, 'vix_level'); + assert.ok(!/you should|^\s*(buy|sell)\b/i.test(a.description), a.description); + assert.match(a.description, /VIX is at 40.0/); + assert.match(a.description, /elevated/); + } + + // Same band, higher price — no re-fire. + setVix(db, 42); + const same = await vixLevelProducer.check(db); + assert.equal(same.length, 0); +}); + +test('vix_level: only subscribed users receive the alert', async () => { + const db = freshDb(); + subscribe(db, 'u1', 'vix_level', null); + subscribe(db, 'u2', 'some_other_type', null); + + setVix(db, 20); + await vixLevelProducer.check(db); + setVix(db, 65); + const out = await vixLevelProducer.check(db); + + assert.equal(out.length, 1); + assert.equal(out[0].userId, 'u1'); + assert.equal(out[0].payload.zone, 'extreme'); +}); + +test('vix_level: no quote means no alert', async () => { + const db = freshDb(); + subscribe(db, 'u1', 'vix_level', null); + const out = await vixLevelProducer.check(db); + assert.equal(out.length, 0); +}); \ No newline at end of file diff --git a/app/server/src/alerts/producers/index.ts b/app/server/src/alerts/producers/index.ts index 40a896e..1e64834 100644 --- a/app/server/src/alerts/producers/index.ts +++ b/app/server/src/alerts/producers/index.ts @@ -11,6 +11,15 @@ export function registerProducer(p: AlertProducer): void { producers.push(p); } +// M21 Mirror Portfolio producers (registered lazily on first run). +export function registerMirrorProducers(): void { + import('./mirrorAlertProducers.ts').then((m) => { + registerProducer(m.fundCaptureProducer); + registerProducer(m.fund13FRecordProducer); + registerProducer(m.mirrorDiffProducer); + }).catch(() => { /* offline */ }); +} + /** Run all producers for a given frequency tier. */ export async function runProducers( db: DatabaseSync, @@ -20,17 +29,43 @@ export async function runProducers( const tier = producers.filter((p) => p.frequency === frequency); for (const producer of tier) { + const started = Date.now(); + let alerts: Alert[] = []; + let error: string | null = null; try { - const alerts = await producer.check(db); + alerts = await producer.check(db); results.push(...alerts); } catch (err) { + error = err instanceof Error ? err.message : String(err); console.error(`[alert:${producer.alertType}] check failed:`, err); } + writeProducerRun(db, producer.alertType, frequency, { + eventsFired: alerts.length, + durationMs: Date.now() - started, + error, + }); } return results; } +function writeProducerRun( + db: DatabaseSync, + producer: string, + frequency: ProducerFrequency, + run: { eventsFired: number; durationMs: number; error: string | null }, +): void { + try { + db.prepare( + `INSERT INTO producer_run_log + (id, producer, frequency, events_fired, inspected, duration_ms, error, run_at) + VALUES (?, ?, ?, ?, 0, ?, ?, ?)`, + ).run(crypto.randomUUID(), producer, frequency, run.eventsFired, run.durationMs, run.error, new Date().toISOString()); + } catch { + /* producer_run_log may not exist until migration runs on old DBs */ + } +} + /** Get all registered producer types. */ export function registeredAlertTypes(): string[] { return producers.map((p) => p.alertType); diff --git a/app/server/src/alerts/producers/mirrorAlertProducers.ts b/app/server/src/alerts/producers/mirrorAlertProducers.ts new file mode 100644 index 0000000..d84246a --- /dev/null +++ b/app/server/src/alerts/producers/mirrorAlertProducers.ts @@ -0,0 +1,143 @@ +// Investor Flow — Mirror Portfolio (M21) alert producers. +// +// Three batched producers wired into the existing alert system (10-min batch, +// subscriptions keyed by alert type + symbol, dedup): +// 1. fund_capture — a new capture/claim was recorded for a tracked fund. +// 2. fund_13f — a new 13F quarter landed from a tracked fund. +// 3. mirror_diff — the live book behind the user's mirror changed materially. +// +// ADR-0010: every payload string is MECHANICAL ("to match your mirror target, the +// delta is ..."). No advice-shaped sentence. Lint-enforced in sub-task 9. + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { readComparisonState, writeComparisonState, getSubscribedUsers, persistAlert, isDuplicate } from './types.ts'; +import { liveBook } from '../../db/fundRepository.ts'; + +const rnd = (n: number) => Math.round(n * 100) / 100; + +// ─── 1. fund_capture — new capture/claim recorded ─────────────────────────── +export const fundCaptureProducer: AlertProducer = { + alertType: 'fund_capture', + frequency: 'batched', + async check(db: DatabaseSync): Promise { + const alerts: import('../AlertEngine.ts').Alert[] = []; + const funds = db.prepare('SELECT id, fund_name FROM tracked_funds WHERE enabled = 1').all() as + { id: string; fund_name: string }[]; + for (const fund of funds) { + const book = liveBook(db, fund.id); + for (const pos of book) { + if (pos.source !== 'capture' && pos.source !== 'claim') continue; + const key = `fund_capture:${fund.id}:${pos.symbol}`; + const prev = readComparisonState(db, key, 'fund_capture'); + const prevId = prev ? (JSON.parse(prev.state).latestKey as string | undefined) : null; + // Use evidence_url as the stable id (post id) — each new post is a new capture. + const latestKey = pos.evidence_url ?? `${pos.as_of}:${pos.shares}`; + if (latestKey === prevId) continue; + const users = getSubscribedUsers(db, 'fund_capture', pos.symbol); + for (const userId of users) { + const dedup = `fund_capture:${userId}:${fund.id}:${latestKey}`; + if (isDuplicate(db, dedup)) continue; + const shares = pos.shares != null ? `${rnd(pos.shares).toLocaleString('en-US')} shares` : 'a disclosed position'; + const alert = createAlert( + crypto.randomUUID(), userId, 'fund_capture', pos.symbol, + `${fund.fund_name} posted a new position update: ${pos.symbol} ${shares} as of ${pos.as_of}.`, + dedup, { fund: fund.id, symbol: pos.symbol, shares: pos.shares, as_of: pos.as_of, source: pos.source }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + writeComparisonState(db, key, 'fund_capture', { latestKey }); + } + } + return alerts; + }, +}; + +// ─── 2. fund_13f — new 13F quarter landed ─────────────────────────────────── +export const fund13FRecordProducer: AlertProducer = { + alertType: 'fund_13f', + frequency: 'batched', + async check(db): Promise { + const alerts: Alert[] = []; + const funds = db.prepare('SELECT id, ci_key, fund_name FROM tracked_funds WHERE enabled = 1').all() as + { id: string; ci_key: string; fund_name: string }[]; + for (const f of funds) { + // Newest 13F-HR accession for the fund's CIK. + const latest = db.prepare( + `SELECT accession, reported_quarter FROM institution_filings + WHERE filer_cik = ? AND form = '13F-HR' + ORDER BY reported_quarter DESC LIMIT 1`, + ).get(f.ci_key) as { accession?: string; reported_quarter?: string } | undefined; + if (!latest?.accession) continue; + const key = `fund_13f:${f.id}`; + const prev = readComparisonState(db, key, 'fund_13f'); + const prevAcc = prev ? (JSON.parse(prev.state).latestAccession as string | undefined) : undefined; + if (latest.accession === prevAcc) continue; + // The fund's disclosed symbols — subscriptions are symbol-cued. + const symbols = db.prepare( + `SELECT DISTINCT symbol FROM institution_filings WHERE filer_cik = ? AND form = '13F-HR'`, + ).all(f.ci_key) as { symbol: string }[]; + const symbolSet = symbols.map((s) => s.symbol); + const userSet = new Map(); + for (const sym of symbolSet) { + for (const u of getSubscribedUsers(db, 'fund_13f', sym)) userSet.set(u, sym); + } + for (const [userId, sym] of userSet) { + const dedup = `fund_13f:${userId}:${f.id}:${latest.accession}`; + if (isDuplicate(db, dedup)) continue; + const alert = createAlert( + crypto.randomUUID(), userId, 'fund_13f', sym, + `${f.fund_name} filed its latest 13F for ${latest.reported_quarter}.`, + dedup, { fund: f.fund_name, quarter: latest.reported_quarter }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + writeComparisonState(db, key, 'fund_13f', { latestAccession: latest.accession }); + } + return alerts; + }, +}; + +// ─── 3. mirror_diff — target changed materially (mechanical) ──────────────── +export const mirrorDiffProducer: AlertProducer = { + alertType: 'mirror_diff', + frequency: 'batched', + async check(db): Promise { + const alerts: Alert[] = []; + const funds = db.prepare('SELECT id, fund_name FROM tracked_funds WHERE enabled = 1').all() as + { id: string; fund_name: string }[]; + for (const fund of funds) { + const book = liveBook(db, fund.id); + for (const pos of book) { + const userIds = getSubscribedUsers(db, 'mirror_diff', pos.symbol); + if (userIds.length === 0) continue; + // Signal: disclosed shares + as-of for this position. + const sig = pos.shares != null ? `${pos.as_of}:${pos.shares}` : `${pos.as_of}`; + const key = `mirror_diff:${fund.id}:${pos.symbol}`; + const prev = readComparisonState(db, key, 'mirror_diff'); + const prevSig = prev ? (JSON.parse(prev.state).sig as string | undefined) : undefined; + if (sig === prevSig) continue; + for (const userId of userIds) { + const dedup = `mirror_diff:${userId}:${fund.id}:${pos.symbol}:${sig}`; + if (isDuplicate(db, dedup)) continue; + const delta = pos.shares != null + ? `To match your mirror target for ${pos.symbol}, the delta is ${rnd(pos.shares).toLocaleString('en-US')} shares as disclosed ${pos.as_of}.` + : `The ${pos.symbol} position in the mirror target updated as of ${pos.as_of}.`; + const alert = createAlert( + crypto.randomUUID(), userId, 'mirror_diff', pos.symbol, + `${fund.fund_name} updated its ${pos.symbol} book (${pos.as_of}). ${delta}`, + dedup, { fund: fund.fund_name, symbol: pos.symbol, as_of: pos.as_of, source: pos.source }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + writeComparisonState(db, key, 'mirror_diff', { sig }); + } + } + return alerts; + }, +}; \ No newline at end of file diff --git a/app/server/src/alerts/producers/new13daProducer.ts b/app/server/src/alerts/producers/new13daProducer.ts index 5cecfd0..f9a270e 100644 --- a/app/server/src/alerts/producers/new13daProducer.ts +++ b/app/server/src/alerts/producers/new13daProducer.ts @@ -2,59 +2,280 @@ import type { DatabaseSync } from 'node:sqlite'; import type { Alert } from '../AlertEngine.ts'; import { createAlert } from '../AlertEngine.ts'; import type { AlertProducer } from './types.ts'; -import { readComparisonState, writeComparisonState, getSubscribedUsers, persistAlert, isDuplicate } from './types.ts'; +import { + readComparisonState, + writeComparisonState, + getSubscribedUsers, + persistAlert, + isDuplicate, +} from './types.ts'; +import { seedAdminDefaultAlertSubscriptions } from '../../db/alertSubscriptionRepository.ts'; +/** + * Alert on new 13D / 13G (and amendments) as they land. + * + * Watermark: set of seen accessions per symbol (not "one latest by quarter"), + * so multiple same-quarter filings each fire once. Ordered by filed_at so + * the newest SEC file date is what drives discovery. + * + * Also seeds a global default subscription (all catalog types, enabled) for + * admin users (is_admin=1) so operators receive everything by default. + * Regular users opt in via the per-type toggle list on the alerts page. + */ export const new13daProducer: AlertProducer = { alertType: 'new_13da', frequency: 'batched', async check(db: DatabaseSync): Promise { + seedAdminDefaultAlertSubscriptions(db); + const alerts: Alert[] = []; const symbols = db.prepare( - `SELECT DISTINCT symbol FROM institution_filings WHERE form LIKE '13D%' OR form LIKE '13G%'`, + `SELECT DISTINCT symbol FROM institution_filings + WHERE form LIKE '13D%' OR form LIKE '13G%' OR form IN ('13D','13G')`, ).all() as { symbol: string }[]; for (const { symbol } of symbols) { const prev = readComparisonState(db, symbol, 'new_13da'); - const prevState = prev ? JSON.parse(prev.state) : { latestAccession: null }; - - const row = db.prepare( - `SELECT accession, reported_quarter, filer_name, put_call - FROM institution_filings - WHERE symbol = ? AND (form LIKE '13D%' OR form LIKE '13G%') - ORDER BY reported_quarter DESC - LIMIT 1`, - ).get(symbol) as { accession: string; reported_quarter: string; filer_name: string; put_call: string | null } | undefined; - - if (!row) continue; - - if (row.accession === prevState.latestAccession) { - writeComparisonState(db, symbol, 'new_13da', { latestAccession: row.accession }); - continue; + let prevState: { latestAccession?: string | null; seenAccessions?: string[] } = {}; + try { + prevState = prev ? JSON.parse(prev.state) : {}; + } catch { + prevState = {}; } + const seen = new Set(prevState.seenAccessions ?? []); + // Migrate legacy single-accession watermark into the set + if (prevState.latestAccession) seen.add(prevState.latestAccession); + + const rows = db.prepare( + `SELECT accession, reported_quarter, filer_name, put_call, form, filed_at + FROM institution_filings + WHERE symbol = ? + AND (form LIKE '13D%' OR form LIKE '13G%' OR form IN ('13D','13G')) + AND accession IS NOT NULL AND accession != '' + ORDER BY filed_at DESC + LIMIT 50`, + ).all(symbol) as Array<{ + accession: string; + reported_quarter: string; + filer_name: string; + put_call: string | null; + form: string; + filed_at: string; + }>; + + if (rows.length === 0) continue; const users = getSubscribedUsers(db, 'new_13da', symbol); - for (const userId of users) { - const dedupKey = `new_13da:${userId}:${row.accession}`; - if (isDuplicate(db, dedupKey)) continue; + const isFirstObservation = seen.size === 0; + let newestAccession = prevState.latestAccession ?? rows[0].accession; + let firstRunAlerts = 0; - const alert = createAlert( - crypto.randomUUID(), - userId, - 'new_13da', - symbol, - `${row.filer_name} reported a new position${row.put_call ? ` (${row.put_call})` : ''} for ${row.reported_quarter}.`, - row.accession, - { filer: row.filer_name, quarter: row.reported_quarter, putCall: row.put_call }, - ); - persistAlert(db, alert); - alerts.push(alert); + for (const row of rows) { + if (seen.has(row.accession)) continue; + + // Skip stale filings (>90 days). Historical bulk ingest should not page users. + const filedDate = new Date(row.filed_at); + const ageDays = (Date.now() - filedDate.getTime()) / (1000 * 60 * 60 * 24); + if (!Number.isFinite(ageDays) || ageDays > 90) { + seen.add(row.accession); + continue; + } + + // First-ever watermark: only alert on very fresh filings (≤7d), cap 5, + // so bulk seed does not page users with a full SC history. + if (isFirstObservation) { + if (ageDays > 7 || firstRunAlerts >= 5) { + seen.add(row.accession); + continue; + } + } + + for (const userId of users) { + const dedupKey = `new_13da:${userId}:${row.accession}`; + if (isDuplicate(db, dedupKey)) continue; + + const accessionNoDashes = row.accession.replace(/-/g, ''); + const filerCik = padAccessionCik(row.accession); + const filingUrl = `https://www.sec.gov/Archives/edgar/data/${filerCik}/${accessionNoDashes}/`; + + const alert = createAlert( + crypto.randomUUID(), + userId, + 'new_13da', + symbol, + `${row.filer_name} reported a new position (${row.form})${row.put_call ? ` (${row.put_call})` : ''} for ${row.reported_quarter} (filed ${String(row.filed_at).slice(0, 10)}).`, + row.accession, + { + filer: row.filer_name, + quarter: row.reported_quarter, + form: row.form, + putCall: row.put_call, + filedAt: row.filed_at, + url: filingUrl, + }, + ); + persistAlert(db, alert); + alerts.push(alert); + firstRunAlerts += 1; + } + + seen.add(row.accession); + if ( + !newestAccession || + row.filed_at >= (rows.find((r) => r.accession === newestAccession)?.filed_at ?? '') + ) { + newestAccession = row.accession; + } } - writeComparisonState(db, symbol, 'new_13da', { latestAccession: row.accession }); + // On first observation still record every accession as seen (including skipped history). + if (isFirstObservation) { + for (const r of rows) seen.add(r.accession); + } + + writeComparisonState(db, symbol, 'new_13da', { + latestAccession: newestAccession, + seenAccessions: [...seen].slice(0, 200), + }); } return alerts; }, }; + +/** + * New 13F-HR accessions for demand-set equities (the "as they come in" feed). + * Shares new_13da subscriptions so one institutional opt-in covers SC + 13F. + */ +export const new13fFilingProducer: AlertProducer = { + alertType: 'new_13da', // same subscription type / UX label "institutional filing" + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + seedAdminDefaultAlertSubscriptions(db); + const alerts: Alert[] = []; + + // Only symbols users actually follow (demand set), not entire filings table. + let symbols: string[] = []; + try { + symbols = ( + db.prepare( + `SELECT DISTINCT symbol FROM symbol_demand + WHERE COALESCE(ticker_kind,'equity') NOT IN ('crypto','fx','index') + ORDER BY symbol`, + ).all() as Array<{ symbol: string }> + ).map((r) => r.symbol); + } catch { + symbols = ( + db.prepare( + `SELECT DISTINCT symbol FROM institution_filings WHERE form = '13F-HR' LIMIT 80`, + ).all() as Array<{ symbol: string }> + ).map((r) => r.symbol); + } + + for (const symbol of symbols) { + const prev = readComparisonState(db, symbol, 'new_13f_filing'); + let prevState: { seenAccessions?: string[] } = {}; + try { + prevState = prev ? JSON.parse(prev.state) : {}; + } catch { + prevState = {}; + } + const seen = new Set(prevState.seenAccessions ?? []); + + const rows = db.prepare( + `SELECT accession, reported_quarter, filer_name, shares, filed_at + FROM institution_filings + WHERE symbol = ? AND form = '13F-HR' + AND accession IS NOT NULL AND accession != '' + ORDER BY filed_at DESC + LIMIT 40`, + ).all(symbol) as Array<{ + accession: string; + reported_quarter: string; + filer_name: string; + shares: number | null; + filed_at: string; + }>; + + if (rows.length === 0) continue; + + const users = getSubscribedUsers(db, 'new_13da', symbol); + const isFirstObservation = seen.size === 0; + let firstRunAlerts = 0; + + for (const row of rows) { + if (seen.has(row.accession)) continue; + + const filedDate = new Date(row.filed_at); + const ageDays = (Date.now() - filedDate.getTime()) / (1000 * 60 * 60 * 24); + // 13F wave is ~45 days after quarter end; older = backfill noise + if (!Number.isFinite(ageDays) || ageDays > 45) { + seen.add(row.accession); + continue; + } + + // First watermark: only page for very fresh filings (≤3d), cap 10 per symbol. + if (isFirstObservation) { + if (ageDays > 3 || firstRunAlerts >= 10) { + seen.add(row.accession); + continue; + } + } + + for (const userId of users) { + const dedupKey = `new_13f:${userId}:${row.accession}`; + if (isDuplicate(db, dedupKey)) continue; + + const accessionNoDashes = row.accession.replace(/-/g, ''); + const filerCik = padAccessionCik(row.accession); + const filingUrl = `https://www.sec.gov/Archives/edgar/data/${filerCik}/${accessionNoDashes}/`; + const sh = + row.shares != null && Number.isFinite(row.shares) + ? ` (${Math.round(row.shares).toLocaleString()} sh)` + : ''; + + const alert = createAlert( + crypto.randomUUID(), + userId, + 'new_13da', + symbol, + `${row.filer_name} filed 13F-HR for ${symbol}${sh} — ${row.reported_quarter} (filed ${String(row.filed_at).slice(0, 10)}).`, + row.accession, + { + filer: row.filer_name, + quarter: row.reported_quarter, + form: '13F-HR', + filedAt: row.filed_at, + shares: row.shares, + url: filingUrl, + kind: '13f_holder', + }, + ); + persistAlert(db, alert); + alerts.push(alert); + firstRunAlerts += 1; + } + + seen.add(row.accession); + } + + if (isFirstObservation) { + for (const r of rows) seen.add(r.accession); + } + + writeComparisonState(db, symbol, 'new_13f_filing', { + seenAccessions: [...seen].slice(0, 300), + }); + } + + return alerts; + }, +}; + +function padAccessionCik(accession: string): string { + const digits = accession.replace(/\D/g, '').slice(0, 10); + return digits.padStart(10, '0'); +} diff --git a/app/server/src/alerts/producers/portfolioRiskProducer.ts b/app/server/src/alerts/producers/portfolioRiskProducer.ts new file mode 100644 index 0000000..276c955 --- /dev/null +++ b/app/server/src/alerts/producers/portfolioRiskProducer.ts @@ -0,0 +1,333 @@ +// Investor Flow — Portfolio risk alert producers. +// +// cluster_breach / drawdown_halt / asymmetry_warning fire when the RiskEngine +// posture for a user crosses a threshold: +// - cluster_breach: a correlated cluster's exposure exceeds its cap +// - drawdown_halt: portfolio drawdown breaches the user's tolerance +// - asymmetry_warning: aggregate reward:risk ratio < 1.0 +// +// Watermarks keep repeated checks silent (fires on state transitions, not on +// every 10-minute tick). ADR-0007: observations and math, never directives. +// Portfolio-level alerts go to global subscribers of each type (no ticker +// makes sense for a whole-portfolio observation). + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert, buildDedupKey } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { readComparisonState, writeComparisonState, persistAlert, isDuplicate } from './types.ts'; +import { assessRisk, type RiskEngineInput, type RiskPosture } from '../../risk/RiskEngine.ts'; +import { listHoldings } from '../../db/portfolioRepository.ts'; + +/** Global-only subscribers of a portfolio-level alert type. */ +function globalSubscribers(db: DatabaseSync, alertType: string): string[] { + const rows = db.prepare( + `SELECT owner_id FROM alerts + WHERE alert_type = ? AND symbol IS NULL AND watchlist_id IS NULL AND enabled = 1`, + ).all(alertType) as Array<{ owner_id: string }>; + return rows.map((r) => r.owner_id); +} + +/** Global-only subscribers (portfolio-level alerts have no ticker), plus the owner. */ +function recipientsFor(db: DatabaseSync, alertType: string, ownerId: string): string[] { + const users = new Set(globalSubscribers(db, alertType)); + users.add(ownerId); + return Array.from(users); +} + +interface UserProfile { + complexity: 'beginner' | 'intermediate' | 'advanced'; + drawdownTolerancePct: number; +} + +function profileFor(db: DatabaseSync, userId: string): UserProfile { + const row = db.prepare( + 'SELECT complexity, drawdown_tolerance FROM users WHERE id=?', + ).get(userId) as { complexity: string | null; drawdown_tolerance: number | null } | undefined; + const complexity = (row?.complexity ?? 'beginner') as UserProfile['complexity']; + const drawdownTolerancePct = Math.abs(row?.drawdown_tolerance ?? 20); + return { complexity, drawdownTolerancePct }; +} + +/** Current marks for a symbol set: live quote price, else cost basis at call site. */ +function marksFor(db: DatabaseSync, symbols: string[]): Map { + const marks = new Map(); + for (let i = 0; i < symbols.length; i += 50) { + const chunk = symbols.slice(i, i + 50); + const placeholders = chunk.map(() => '?').join(','); + const rows = db.prepare( + `SELECT symbol, price FROM quotes WHERE symbol IN (${placeholders})`, + ).all(...chunk) as Array<{ symbol: string; price: number | null }>; + for (const r of rows) { + if (typeof r.price === 'number' && Number.isFinite(r.price)) marks.set(r.symbol, r.price); + } + } + return marks; +} + +interface BuiltPosture { + posture: RiskPosture; + equity: number; + peakEquity: number; + marks: Map; +} + +/** + * Reward/stop intent per symbol from the user's active trades: stop_loss + + * first target. Gives aggregate asymmetry a real basis (downside vs upside) + * instead of defaulting to cost basis on both sides. + */ +function tradeIntents(db: DatabaseSync, userId: string, symbols: string[]): Map { + const intents = new Map(); + for (let i = 0; i < symbols.length; i += 50) { + const chunk = symbols.slice(i, i + 50); + const placeholders = chunk.map(() => '?').join(','); + const rows = db.prepare( + `SELECT t.symbol, t.stop_loss, t.targets + FROM trades t + JOIN (SELECT symbol, MAX(date) AS max_date + FROM trades + WHERE owner_id = ? AND status = 'active' + AND symbol IN (${placeholders}) + GROUP BY symbol) m + ON t.symbol = m.symbol AND t.date = m.max_date + WHERE t.owner_id = ? AND t.status = 'active' + AND t.symbol IN (${placeholders})`, + ).all(userId, ...chunk, userId, ...chunk) as Array<{ symbol: string; stop_loss: number | null; targets: string | null }>; + for (const r of rows) { + const intent: { rewardTarget?: number; stopPrice?: number } = {}; + if (typeof r.stop_loss === 'number' && Number.isFinite(r.stop_loss)) intent.stopPrice = r.stop_loss; + if (r.targets) { + try { + const targets = JSON.parse(r.targets) as Array<{ price: number }>; + const first = targets.find((t) => typeof t?.price === 'number'); + if (first) intent.rewardTarget = first.price; + } catch { /* malformed targets JSON — treat as none */ } + } + intents.set(r.symbol, intent); + } + } + return intents; +} + +function buildPosture(db: DatabaseSync, userId: string): BuiltPosture | null { + const holdings = listHoldings(db, userId); + if (holdings.length === 0) return null; + + const { complexity, drawdownTolerancePct } = profileFor(db, userId); + const marks = marksFor(db, holdings.map((h) => h.symbol)); + const intents = tradeIntents(db, userId, holdings.map((h) => h.symbol)); + + const portfolio = holdings.map((h) => { + const intent = intents.get(h.symbol); + return { + symbol: h.symbol, + shares: h.shares, + avgCost: h.avg_cost, + cluster: 'uncategorized' as const, + rewardTarget: intent?.rewardTarget, + stopPrice: intent?.stopPrice, + }; + }); + const equity = portfolio.reduce((sum, p) => { + const mark = marks.get(p.symbol) ?? p.avgCost; + return sum + p.shares * mark; + }, 0); + if (!Number.isFinite(equity) || equity <= 0) return null; + + const prev = readComparisonState(db, userId, 'drawdown_halt'); + const prevState = prev ? JSON.parse(prev.state) : { peakEquity: equity, halted: false }; + const peakEquity = Math.max(Number(prevState.peakEquity) || equity, equity); + + const defaultCap = complexity === 'beginner' ? equity * 0.25 + : complexity === 'intermediate' ? equity * 0.40 + : null; + const clusterCaps = defaultCap !== null ? { uncategorized: defaultCap } : null; + + const input: RiskEngineInput = { + portfolio, + account: { equity, drawdownTolerancePct, complexity }, + peakEquity, + sizingContext: { clusterCaps }, + }; + const posture = assessRisk(input); + return { posture, equity, peakEquity, marks }; +} + +// ─── cluster_breach ────────────────────────────────────────────────────────── + +export const clusterBreachProducer: AlertProducer = { + alertType: 'cluster_breach', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + const users = db.prepare( + `SELECT DISTINCT owner_id FROM portfolio_holdings WHERE status='open'`, + ).all() as Array<{ owner_id: string }>; + + for (const { owner_id: userId } of users) { + const built = buildPosture(db, userId); + if (!built) continue; + const { posture } = built; + + const prev = readComparisonState(db, userId, 'cluster_breach'); + const prevState = prev ? JSON.parse(prev.state) : { breaches: {} as Record }; + + for (const [cluster, exposure] of Object.entries(posture.clusterExposure)) { + const cap = posture.clusterCaps?.[cluster]; + if (cap === undefined || exposure <= cap) { + if (prevState.breaches[cluster] !== undefined) delete prevState.breaches[cluster]; + continue; + } + const breachPct = ((exposure - cap) / cap) * 100; + const prevPct = prevState.breaches[cluster]; + if (typeof prevPct === 'number' && prevPct >= breachPct - 5) continue; + + for (const uid of recipientsFor(db, 'cluster_breach', userId)) { + const eventId = `${userId}:${cluster}:${Math.round(exposure)}`; + const dedupKey = buildDedupKey('cluster_breach', undefined, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const description = + `Exposure in the "${cluster}" cluster is $${Math.round(exposure)} against a cap of ` + + `$${Math.round(cap)} (${breachPct.toFixed(1)}% over). Correlated positions share this driver.`; + + const alert = createAlert( + crypto.randomUUID(), + uid, + 'cluster_breach', + undefined, + description, + eventId, + { userId, cluster, exposureUsd: Math.round(exposure), capUsd: Math.round(cap), breachPct: Number(breachPct.toFixed(1)) }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + prevState.breaches[cluster] = breachPct; + } + + writeComparisonState(db, userId, 'cluster_breach', prevState); + } + + return alerts; + }, +}; + +// ─── drawdown_halt ─────────────────────────────────────────────────────────── + +export const drawdownHaltProducer: AlertProducer = { + alertType: 'drawdown_halt', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + const users = db.prepare( + `SELECT DISTINCT owner_id FROM portfolio_holdings WHERE status='open'`, + ).all() as Array<{ owner_id: string }>; + + for (const { owner_id: userId } of users) { + const built = buildPosture(db, userId); + if (!built) continue; + const { posture, peakEquity } = built; + const { status, currentDrawdownPct, tolerancePct } = posture.drawdownVsTolerance; + + const prev = readComparisonState(db, userId, 'drawdown_halt'); + const prevState = prev ? JSON.parse(prev.state) : { peakEquity, halted: false }; + const wasHalted = Boolean(prevState.halted); + const nowHalted = status === 1; + + if (nowHalted && !wasHalted) { + for (const uid of recipientsFor(db, 'drawdown_halt', userId)) { + const eventId = `${userId}:halt:${Math.round(currentDrawdownPct * 10)}`; + const dedupKey = buildDedupKey('drawdown_halt', undefined, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const description = + `Portfolio drawdown of ${currentDrawdownPct.toFixed(1)}% has breached the ` + + `${tolerancePct}% tolerance. The gentle-halt circuit breaker is now active for this account ` + + `(no new entries for 24 hours; existing positions unaffected).`; + + const alert = createAlert( + crypto.randomUUID(), + uid, + 'drawdown_halt', + undefined, + description, + eventId, + { userId, drawdownPct: Number(currentDrawdownPct.toFixed(1)), tolerancePct, peakEquity: Math.round(peakEquity) }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + } + + writeComparisonState(db, userId, 'drawdown_halt', { + peakEquity, + halted: nowHalted, + drawdownPct: Number(currentDrawdownPct.toFixed(1)), + }); + } + + return alerts; + }, +}; + +// ─── asymmetry_warning ─────────────────────────────────────────────────────── + +export const asymmetryWarningProducer: AlertProducer = { + alertType: 'asymmetry_warning', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + const users = db.prepare( + `SELECT DISTINCT owner_id FROM portfolio_holdings WHERE status='open'`, + ).all() as Array<{ owner_id: string }>; + + for (const { owner_id: userId } of users) { + const built = buildPosture(db, userId); + if (!built) continue; + const { posture } = built; + const asymmetry = posture.asymmetry; + if (!(asymmetry < 1) || !Number.isFinite(asymmetry)) continue; + + // Watermark: bucket the ratio so 0.85 fires once, 0.72 re-fires when it + // gets meaningfully worse. + const bucket = Math.round(asymmetry * 20) / 20; // 0.05 steps + const prev = readComparisonState(db, userId, 'asymmetry_warning'); + const prevState = prev ? JSON.parse(prev.state) : { bucket: null as number | null }; + if (prevState.bucket !== null && prevState.bucket <= bucket) continue; + + for (const uid of recipientsFor(db, 'asymmetry_warning', userId)) { + const eventId = `${userId}:asym:${bucket}`; + const dedupKey = buildDedupKey('asymmetry_warning', undefined, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const description = + `Aggregate reward:risk asymmetry is ${asymmetry.toFixed(2)} (< 1.0), so expected downside ` + + `outweighs expected upside across this portfolio.`; + + const alert = createAlert( + crypto.randomUUID(), + uid, + 'asymmetry_warning', + undefined, + description, + eventId, + { userId, asymmetry: Number(asymmetry.toFixed(2)) }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + + writeComparisonState(db, userId, 'asymmetry_warning', { bucket }); + } + + return alerts; + }, +}; diff --git a/app/server/src/alerts/producers/rotationProducer.ts b/app/server/src/alerts/producers/rotationProducer.ts new file mode 100644 index 0000000..6da4ff9 --- /dev/null +++ b/app/server/src/alerts/producers/rotationProducer.ts @@ -0,0 +1,157 @@ +// Investor Flow — Sector rotation + regime shift alert producers. +// +// rotation_incipient: fires when the rotation detector records a new signal +// (accumulation/expansion phase) for a sector. Watermark = last signal per +// sector, so repeat observations of the same signal stay silent. +// regime_shift: fires when the market regime classification changes. Watermark +// = last as_of + regime, so a persisted regime never re-fires. +// +// ADR-0007: observations, never directives. A rotation signal or regime change +// is a state change, not a call to action. + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert, buildDedupKey } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { readComparisonState, writeComparisonState, getSubscribedUsers, persistAlert, isDuplicate } from './types.ts'; + +// ─── rotation_incipient ────────────────────────────────────────────────────── + +/** Phases that count as "incipient" (capital moving in). */ +const INCIPIENT_PHASES = new Set(['accumulation', 'expansion']); + +export const rotationIncipientProducer: AlertProducer = { + alertType: 'rotation_incipient', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + // Newest signal per sector (a sector may appear once per detected event). + const rows = db.prepare( + `SELECT r.sector, r.detected_at, r.phase, r.confidence + FROM rotation_signals r + JOIN (SELECT sector, MAX(detected_at) AS max_detected + FROM rotation_signals + GROUP BY sector) m + ON r.sector = m.sector AND r.detected_at = m.max_detected + ORDER BY r.detected_at DESC`, + ).all() as Array<{ sector: string; detected_at: string; phase: string; confidence: number | null }>; + + for (const row of rows) { + const sector = row.sector; + const prev = readComparisonState(db, sector, 'rotation_incipient'); + const prevState = prev ? JSON.parse(prev.state) : { lastDetectedAt: null, lastPhase: null }; + + // Already seen this exact signal. + if (prevState.lastDetectedAt === row.detected_at && prevState.lastPhase === row.phase) { + continue; + } + + // Fire only for incipient phases. Still watermark distribution/markdown + // observations so they don't re-fire later. + if (INCIPIENT_PHASES.has(row.phase)) { + const users = getSubscribedUsers(db, 'rotation_incipient', sector); + for (const userId of users) { + const eventId = `${sector}:${row.detected_at}:${row.phase}`; + const dedupKey = buildDedupKey('rotation_incipient', sector, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const conf = row.confidence != null ? Math.round(row.confidence * 100) : null; + const description = + `The rotation detector flagged ${sector} in a ${row.phase} phase` + + (conf != null ? ` with ${conf}% confidence` : '') + + ` (detected ${row.detected_at}). Capital may be rotating into this sector.`; + + const alert = createAlert( + crypto.randomUUID(), + userId, + 'rotation_incipient', + sector, + description, + eventId, + { sector, phase: row.phase, confidence: row.confidence, detectedAt: row.detected_at }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + } + + writeComparisonState(db, sector, 'rotation_incipient', { + lastDetectedAt: row.detected_at, + lastPhase: row.phase, + }); + } + + return alerts; + }, +}; + +// ─── regime_shift ──────────────────────────────────────────────────────────── + +export const regimeShiftProducer: AlertProducer = { + alertType: 'regime_shift', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + // Most recent regime classification. + const row = db.prepare( + 'SELECT as_of, regime FROM macro_regimes ORDER BY as_of DESC LIMIT 1', + ).get() as { as_of: string; regime: string } | undefined; + if (!row) return alerts; + + const prev = readComparisonState(db, 'MARKET', 'regime_shift'); + const prevState = prev ? JSON.parse(prev.state) : { lastAsOf: null, lastRegime: null }; + + // Same classification persisted (same regime, even if as_of refreshed) → silent. + if (prevState.lastRegime === row.regime) { + writeComparisonState(db, 'MARKET', 'regime_shift', { + lastAsOf: row.as_of, + lastRegime: row.regime, + }); + return alerts; + } + + // First observation records the watermark without firing (no baseline to + // compare against — backfill of the current regime shouldn't page anyone). + if (prevState.lastRegime === null) { + writeComparisonState(db, 'MARKET', 'regime_shift', { + lastAsOf: row.as_of, + lastRegime: row.regime, + }); + return alerts; + } + + // Regime actually changed → fire for global subscribers. + const users = getSubscribedUsers(db, 'regime_shift', 'MARKET'); + for (const userId of users) { + const eventId = `${prevState.lastRegime}->${row.regime}:${row.as_of}`; + const dedupKey = buildDedupKey('regime_shift', undefined, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const description = + `The market regime changed from "${prevState.lastRegime}" to "${row.regime}" ` + + `(classified ${row.as_of}). Portfolio-level risk assessments now use the new regime.`; + + const alert = createAlert( + crypto.randomUUID(), + userId, + 'regime_shift', + undefined, + description, + eventId, + { from: prevState.lastRegime, to: row.regime, asOf: row.as_of }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + + writeComparisonState(db, 'MARKET', 'regime_shift', { + lastAsOf: row.as_of, + lastRegime: row.regime, + }); + return alerts; + }, +}; diff --git a/app/server/src/alerts/producers/thesisProducer.ts b/app/server/src/alerts/producers/thesisProducer.ts new file mode 100644 index 0000000..71c6da8 --- /dev/null +++ b/app/server/src/alerts/producers/thesisProducer.ts @@ -0,0 +1,109 @@ +// Investor Flow — Thesis status alert producers. +// +// thesis_broken / thesis_weakening: fire when a stored thesis transitions into +// the broken or weakening state. The producer tracks the last seen status for +// EVERY thesis (not just target-status ones), so intact → broken is a real +// transition, not a "first observation". A thesis that is already broken at +// boot watermarks silently (baseline). +// +// ADR-0007: "criteria met — consider reviewing" never "sell". The description +// frames the observation and invites the user to review the thesis. + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert, buildDedupKey } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { readComparisonState, writeComparisonState, getSubscribedUsers, persistAlert, isDuplicate } from './types.ts'; + +interface ThesisRow { + id: string; + user_id: string; + symbol: string; + status: string; + updated_at: string; +} + +export const thesisBrokenProducer: AlertProducer = { + alertType: 'thesis_broken', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + return checkThesisTransitions(db, 'broken', 'thesis_broken'); + }, +}; + +export const thesisWeakeningProducer: AlertProducer = { + alertType: 'thesis_weakening', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + return checkThesisTransitions(db, 'weakening', 'thesis_weakening'); + }, +}; + +async function checkThesisTransitions( + db: DatabaseSync, + targetStatus: string, + alertType: 'thesis_broken' | 'thesis_weakening', +): Promise { + const alerts: Alert[] = []; + + // All theses, all statuses — the watermark must see intact states too, or + // an intact → broken transition would be misread as a baseline. + const rows = db.prepare( + `SELECT id, user_id, symbol, status, updated_at FROM theses`, + ).all() as unknown as ThesisRow[]; + + for (const row of rows) { + const prev = readComparisonState(db, row.id, alertType); + const prevState = prev ? JSON.parse(prev.state) : { lastStatus: null, lastUpdatedAt: null }; + + // First observation of this thesis: watermark, never fire. Existing + // broken/weakening theses at boot must not page anyone. + if (prevState.lastStatus === null) { + writeComparisonState(db, row.id, alertType, { + lastStatus: row.status, + lastUpdatedAt: row.updated_at, + }); + continue; + } + + // Fire only on the transition INTO the target status (from any other). + const transitionedInto = row.status === targetStatus && row.status !== prevState.lastStatus; + + // Always refresh the watermark to the current state so later transitions + // are detected against the right baseline. + writeComparisonState(db, row.id, alertType, { + lastStatus: row.status, + lastUpdatedAt: row.updated_at, + }); + + if (!transitionedInto) continue; + + const users = new Set(getSubscribedUsers(db, alertType, row.symbol)); + users.add(row.user_id); + for (const userId of users) { + const eventId = `${row.id}:${row.status}:${row.updated_at}`; + const dedupKey = buildDedupKey(alertType, row.symbol, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const description = alertType === 'thesis_broken' + ? `The invalidation criteria for your thesis on ${row.symbol} have been met (updated ${row.updated_at}). Consider reviewing your thesis.` + : `Some signals suggest your thesis on ${row.symbol} may be weakening (updated ${row.updated_at}), but invalidation criteria are not yet met.`; + + const alert = createAlert( + crypto.randomUUID(), + userId, + alertType, + row.symbol, + description, + eventId, + { thesisId: row.id, status: row.status, updatedAt: row.updated_at }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + } + + return alerts; +} diff --git a/app/server/src/alerts/producers/types.ts b/app/server/src/alerts/producers/types.ts index b64a6fb..d218bef 100644 --- a/app/server/src/alerts/producers/types.ts +++ b/app/server/src/alerts/producers/types.ts @@ -1,7 +1,7 @@ import type { DatabaseSync } from 'node:sqlite'; import type { Alert } from '../AlertEngine.ts'; -export type ProducerFrequency = 'per-fetch' | 'batched'; +export type ProducerFrequency = 'realtime' | 'per-fetch' | 'batched'; export interface AlertProducer { readonly alertType: string; diff --git a/app/server/src/alerts/producers/unlockProducer.ts b/app/server/src/alerts/producers/unlockProducer.ts new file mode 100644 index 0000000..3996465 --- /dev/null +++ b/app/server/src/alerts/producers/unlockProducer.ts @@ -0,0 +1,96 @@ +// Investor Flow — Conviction tier unlock alert producer. +// +// conviction_unlock: fires when a user earns a new sizing unlock (tier_a, +// tier_a_star, etc.) from accumulated trading history. Watermark = last +// earned_at per user, so re-reading the same unlock stays silent. +// +// ADR-0007: an unlock is an earned capability, not a buy signal. + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert, buildDedupKey } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { readComparisonState, writeComparisonState, getSubscribedUsers, persistAlert, isDuplicate } from './types.ts'; + +/** Human-readable tier names for the payload/unlock display. */ +const UNLOCK_LABELS: Record = { + tier_a: 'A tier', + tier_a_star: 'A* tier', + covered_income: 'covered income strategies', + uncovered_income: 'uncovered income strategies', +}; + +export const convictionUnlockProducer: AlertProducer = { + alertType: 'conviction_unlock', + frequency: 'batched', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + // Newest unlock per user. + const rows = db.prepare( + `SELECT s.user_id, s.unlock, s.earned_at, s.evidence + FROM sizing_unlocks s + JOIN (SELECT user_id, MAX(earned_at) AS max_earned + FROM sizing_unlocks + GROUP BY user_id) m + ON s.user_id = m.user_id AND s.earned_at = m.max_earned + ORDER BY s.earned_at DESC`, + ).all() as Array<{ user_id: string; unlock: string; earned_at: string; evidence: string | null }>; + + for (const row of rows) { + const userId = row.user_id; + const prev = readComparisonState(db, userId, 'conviction_unlock'); + const prevState = prev ? JSON.parse(prev.state) : { lastEarnedAt: null, lastUnlock: null }; + + if (prevState.lastEarnedAt === row.earned_at && prevState.lastUnlock === row.unlock) { + continue; + } + + // First observation records the watermark without firing (existing + // unlocks at boot shouldn't page anyone). + if (prevState.lastUnlock === null) { + writeComparisonState(db, userId, 'conviction_unlock', { + lastEarnedAt: row.earned_at, + lastUnlock: row.unlock, + }); + continue; + } + + // A genuinely new unlock → notify the earning user first (they are the + // primary subscriber; a global subscription is honored too). + const users = new Set(getSubscribedUsers(db, 'conviction_unlock', userId)); + users.add(userId); + for (const uid of users) { + const eventId = `${userId}:${row.unlock}:${row.earned_at}`; + const dedupKey = buildDedupKey('conviction_unlock', undefined, eventId); + if (isDuplicate(db, dedupKey)) continue; + + const label = UNLOCK_LABELS[row.unlock] ?? row.unlock; + const description = + `Your trading history now qualifies for the ${label} conviction tier` + + (row.evidence ? `. Evidence: ${row.evidence}` : '') + + `. Sizing at this tier uses the higher conviction multiplier.`; + + const alert = createAlert( + crypto.randomUUID(), + uid, + 'conviction_unlock', + undefined, + description, + eventId, + { unlock: row.unlock, earnedAt: row.earned_at, evidence: row.evidence }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + + writeComparisonState(db, userId, 'conviction_unlock', { + lastEarnedAt: row.earned_at, + lastUnlock: row.unlock, + }); + } + + return alerts; + }, +}; diff --git a/app/server/src/alerts/producers/vixLevelProducer.ts b/app/server/src/alerts/producers/vixLevelProducer.ts new file mode 100644 index 0000000..d6f7c07 --- /dev/null +++ b/app/server/src/alerts/producers/vixLevelProducer.ts @@ -0,0 +1,119 @@ +// Investor Flow — VIX level alert producer. +// +// Fires a `vix_level` alert when the VIX shifts into a new historical band. +// Bands come from the "buy when it's scary / evaluate when it's calm" reading +// of the index (calm < 15, elevated ≥ 35, stress ≥ 45, extreme ≥ 60). +// +// ADR-0007: alert text says "the level changed", never "you should act". +// No trade verbs. A zone crossing is a state observation, not a directive. + +import type { DatabaseSync } from 'node:sqlite'; +import type { Alert } from '../AlertEngine.ts'; +import { createAlert } from '../AlertEngine.ts'; +import type { AlertProducer } from './types.ts'; +import { + readComparisonState, + writeComparisonState, + getSubscribedUsers, + persistAlert, + isDuplicate, +} from './types.ts'; + +export const VIX_SYMBOL = '^VIX'; + +/** VIX bands (inclusive lower bound). */ +export const VIX_BANDS: Array<{ min: number; zone: string; label: string; note: string }> = [ + { min: 60, zone: 'extreme', label: 'extreme', note: 'an extreme reading; historically a rare, high-fear zone.' }, + { min: 45, zone: 'stress', label: 'stress', note: 'a high-fear zone that historically has been brief.' }, + { min: 35, zone: 'elevated', label: 'elevated', note: 'a fear-elevated zone.' }, + { min: 15, zone: 'watch', label: 'watch', note: 'above the calm zone but below the elevated fear zone.' }, + { min: 0, zone: 'calm', label: 'calm', note: 'a low-volatility zone.' }, +]; + +export type VixBand = { min: number; zone: string; label: string; note: string }; + +export function vixZoneFor(level: number): VixBand { + for (const b of VIX_BANDS) { + if (level >= b.min) return b; + } + return VIX_BANDS[VIX_BANDS.length - 1]; +} + +/** + * Alert on VIX zone transitions. + * + * Watermark: last known zone. Fires only when the VIX moves from one named + * band to another, so a quiet drift within a band produces silence. The first + * observation records the watermark without firing, so a bulk backfill never + * pages subscribers for a level they are already sitting at. + */ +export const vixLevelProducer: AlertProducer = { + alertType: 'vix_level', + frequency: 'realtime', + + async check(db: DatabaseSync): Promise { + const alerts: Alert[] = []; + + let price: number | null = null; + try { + const row = db.prepare('SELECT price FROM quotes WHERE symbol = ?').get(VIX_SYMBOL) as + { price: number | null } | undefined; + price = row?.price ?? null; + } catch { /* quotes table may not exist in some schema contexts */ } + + if (price == null || !Number.isFinite(price)) { + // No cached VIX yet — nothing to compare, stay silent. + return alerts; + } + + const band = vixZoneFor(price); + + const prev = readComparisonState(db, VIX_SYMBOL, 'vix_level'); + let prevZone: string | null = null; + try { + prevZone = prev ? ((JSON.parse(prev.state).zone as string | undefined) ?? null) : null; + } catch { + prevZone = null; + } + + // First observation: seed the watermark, do not page. + if (prevZone === null) { + writeComparisonState(db, VIX_SYMBOL, 'vix_level', { zone: band.zone, price }); + return alerts; + } + + if (prevZone === band.zone) { + // Still in the same band — silence. + writeComparisonState(db, VIX_SYMBOL, 'vix_level', { zone: band.zone, price }); + return alerts; + } + + const movedUp = band.min >= 35; // entering elevated/stress/extreme + const direction = band.min >= 45 ? 'surged into' : band.min >= 35 ? 'climbed into' : band.min >= 15 ? 'moved into' : 'fallen to'; + const why = movedUp + ? 'Fear, as priced by the VIX, is elevated. This is an observation of a state change, not a directive.' + : 'Fear, as priced by the VIX, is low. This is an observation of a state change, not a directive.'; + + const users = getSubscribedUsers(db, 'vix_level', VIX_SYMBOL); + + for (const userId of users) { + const dedupKey = `vix_level:${userId}:${band.zone}`; + if (isDuplicate(db, dedupKey)) continue; + + const alert = createAlert( + crypto.randomUUID(), + userId, + 'vix_level', + VIX_SYMBOL, + `The VIX is at ${price.toFixed(1)}, having ${direction} the ${band.label} band. ${band.note}\n\n${why}.`, + band.zone, + { level: price, zone: band.zone, label: band.label }, + ); + persistAlert(db, alert); + alerts.push(alert); + } + + writeComparisonState(db, VIX_SYMBOL, 'vix_level', { zone: band.zone, price }); + return alerts; + }, +}; \ No newline at end of file diff --git a/app/server/src/analysis/__fixtures__/dealer-surface-synthetic.json b/app/server/src/analysis/__fixtures__/dealer-surface-synthetic.json new file mode 100644 index 0000000..c024c86 --- /dev/null +++ b/app/server/src/analysis/__fixtures__/dealer-surface-synthetic.json @@ -0,0 +1,33 @@ +{ + "symbol": "TEST", + "spot": 100, + "asOf": "2026-08-06T15:00:00.000Z", + "providerId": "synthetic", + "delaySeconds": 900, + "oiFreshness": "prior_session", + "provenance": { + "fetchedAt": "2026-08-06T15:00:00.000Z", + "sourceKind": "yfinance", + "rawSourceId": "fixture:TEST" + }, + "rows": [ + { "strike": 95, "expiry": "2026-08-08", "right": "call", "openInterest": 2000, "volume": 100, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.03, "vanna": 0.1, "delta": 0.6 } }, + { "strike": 95, "expiry": "2026-08-08", "right": "put", "openInterest": 1500, "volume": 80, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.03, "vanna": 0.05, "delta": -0.4 } }, + { "strike": 100, "expiry": "2026-08-08", "right": "call", "openInterest": 5000, "volume": 200, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.28, "greeks": { "gamma": 0.04, "vanna": 0.1, "delta": 0.5 } }, + { "strike": 100, "expiry": "2026-08-08", "right": "put", "openInterest": 4000, "volume": 150, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.28, "greeks": { "gamma": 0.04, "vanna": 0.08, "delta": -0.5 } }, + { "strike": 105, "expiry": "2026-08-08", "right": "call", "openInterest": 8000, "volume": 300, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.27, "greeks": { "gamma": 0.05, "vanna": 0.1, "delta": 0.4 } }, + { "strike": 105, "expiry": "2026-08-08", "right": "put", "openInterest": 1000, "volume": 50, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.27, "greeks": { "gamma": 0.02, "vanna": 0.05, "delta": -0.3 } }, + { "strike": 95, "expiry": "2026-08-15", "right": "call", "openInterest": 1800, "volume": 90, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.32, "greeks": { "gamma": 0.025, "vanna": 0.1, "delta": 0.55 } }, + { "strike": 95, "expiry": "2026-08-15", "right": "put", "openInterest": 1600, "volume": 70, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.32, "greeks": { "gamma": 0.025, "vanna": 0.05, "delta": -0.45 } }, + { "strike": 100, "expiry": "2026-08-15", "right": "call", "openInterest": 4500, "volume": 180, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.035, "vanna": 0.1, "delta": 0.5 } }, + { "strike": 100, "expiry": "2026-08-15", "right": "put", "openInterest": 3500, "volume": 140, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.035, "vanna": 0.08, "delta": -0.5 } }, + { "strike": 105, "expiry": "2026-08-15", "right": "call", "openInterest": 7000, "volume": 250, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.29, "greeks": { "gamma": 0.045, "vanna": 0.1, "delta": 0.4 } }, + { "strike": 105, "expiry": "2026-08-15", "right": "put", "openInterest": 900, "volume": 40, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.29, "greeks": { "gamma": 0.018, "vanna": 0.05, "delta": -0.3 } }, + { "strike": 95, "expiry": "2026-08-22", "right": "call", "openInterest": 1200, "volume": 60, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.33, "greeks": { "gamma": 0.02, "vanna": 0.1, "delta": 0.55 } }, + { "strike": 95, "expiry": "2026-08-22", "right": "put", "openInterest": 1100, "volume": 50, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.33, "greeks": { "gamma": 0.02, "vanna": 0.05, "delta": -0.45 } }, + { "strike": 100, "expiry": "2026-08-22", "right": "call", "openInterest": 3000, "volume": 120, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.31, "greeks": { "gamma": 0.03, "vanna": 0.1, "delta": 0.5 } }, + { "strike": 100, "expiry": "2026-08-22", "right": "put", "openInterest": 2800, "volume": 100, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.31, "greeks": { "gamma": 0.03, "vanna": 0.08, "delta": -0.5 } }, + { "strike": 105, "expiry": "2026-08-22", "right": "call", "openInterest": 6000, "volume": 200, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.04, "vanna": 0.1, "delta": 0.4 } }, + { "strike": 105, "expiry": "2026-08-22", "right": "put", "openInterest": 800, "volume": 30, "bid": 1, "ask": 1.1, "mid": 1.05, "iv": 0.3, "greeks": { "gamma": 0.015, "vanna": 0.05, "delta": -0.3 } } + ] +} diff --git a/app/server/src/analysis/__tests__/dashboardRollup.test.ts b/app/server/src/analysis/__tests__/dashboardRollup.test.ts index 2049dfd..4104ff2 100644 --- a/app/server/src/analysis/__tests__/dashboardRollup.test.ts +++ b/app/server/src/analysis/__tests__/dashboardRollup.test.ts @@ -19,6 +19,7 @@ import { computeConvictionDelta, aggregateFlowDirection, detectClassRoll, + matchedBookShareTotals, DashboardRollupEngine, generateDashboardRollupSummary, ADR0007_FOOTER, @@ -195,10 +196,14 @@ test('DashboardRollupEngine: increasing conviction (institutional only)', async const wlIns = db.prepare('INSERT INTO watchlists (id, owner_id, name, symbols, created_at, sort_order) VALUES (?, ?, ?, ?, ?, ?)'); wlIns.run('wl1', 'user-inc', 'default', JSON.stringify(['NVDA']), '2026-01-01', 0); - // Seed institution_filings: current quarter has more shares than previous. + // Same filer both quarters so matched-book path is exercised. seedInstitutionFilings(db, 'NVDA', [ { filer_cik: 'cik1', filer_sic: '60', shares: 10000, reported_quarter: '2025-Q1' }, - { filer_cik: 'cik2', filer_sic: '60', shares: 15000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik1', filer_sic: '60', shares: 15000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 8000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 12000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 5000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 9000, reported_quarter: '2025-Q2' }, ]); const edgar = createNoOpEdgar(); @@ -219,7 +224,11 @@ test('DashboardRollupEngine: reducing conviction (institutional only)', async () seedInstitutionFilings(db, 'TSLA', [ { filer_cik: 'cik1', filer_sic: '60', shares: 20000, reported_quarter: '2025-Q1' }, - { filer_cik: 'cik2', filer_sic: '60', shares: 5000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik1', filer_sic: '60', shares: 5000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 18000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 4000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 10000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 2000, reported_quarter: '2025-Q2' }, ]); const edgar = createNoOpEdgar(); @@ -232,16 +241,67 @@ test('DashboardRollupEngine: reducing conviction (institutional only)', async () assert.equal(rows[0].convictionDelta, 'reducing'); }); +test('matchedBookShareTotals: incomplete season does not look like selling', () => { + const db = createTestDb(); + // Q1 full book; Q2 only a few filers who actually added — raw sum falls, matched rises. + seedInstitutionFilings(db, 'IREN', [ + { filer_cik: 'big1', filer_sic: '60', shares: 50_000_000, reported_quarter: '2026-Q1' }, + { filer_cik: 'big2', filer_sic: '60', shares: 40_000_000, reported_quarter: '2026-Q1' }, + { filer_cik: 'big3', filer_sic: '60', shares: 30_000_000, reported_quarter: '2026-Q1' }, + { filer_cik: 'br', filer_sic: '60', shares: 4_000_000, reported_quarter: '2026-Q1' }, + { filer_cik: 'br', filer_sic: '60', shares: 13_000_000, reported_quarter: '2026-Q2' }, + { filer_cik: 'ss', filer_sic: '60', shares: 3_000_000, reported_quarter: '2026-Q1' }, + { filer_cik: 'ss', filer_sic: '60', shares: 6_000_000, reported_quarter: '2026-Q2' }, + { filer_cik: 'tiny', filer_sic: '60', shares: 1000, reported_quarter: '2026-Q1' }, + { filer_cik: 'tiny', filer_sic: '60', shares: 2000, reported_quarter: '2026-Q2' }, + ]); + const rawQ1 = 50e6 + 40e6 + 30e6 + 4e6 + 3e6 + 1000; + const rawQ2 = 13e6 + 6e6 + 2000; + assert.ok(rawQ2 < rawQ1 * 0.9, 'raw totals falsely lighten'); + const m = matchedBookShareTotals(db, 'IREN', '2026-Q2', '2026-Q1'); + assert.ok(m); + assert.equal(m!.filerCount, 3); + assert.ok(m!.currShares > m!.prevShares * 1.1, 'matched book shows adding'); +}); + +test('DashboardRollupEngine: incomplete latest quarter uses matched book not raw drop', async () => { + const db = createTestDb(); + seedUser(db, 'user-partial'); + const wlIns = db.prepare('INSERT INTO watchlists (id, owner_id, name, symbols, created_at, sort_order) VALUES (?, ?, ?, ?, ?, ?)'); + wlIns.run('wl1', 'user-partial', 'default', JSON.stringify(['IREN']), '2026-01-01', 0); + + // Many Q1 filers; few Q2 filers who increased — would be "reducing" on raw SUM. + const rows: Array<{ filer_cik: string; filer_sic: string; shares: number; reported_quarter: string }> = []; + for (let i = 0; i < 20; i++) { + rows.push({ filer_cik: `f${i}`, filer_sic: '60', shares: 1_000_000, reported_quarter: '2026-Q1' }); + } + // 5 of them file Q2 with more shares; rest missing (incomplete season) + for (let i = 0; i < 5; i++) { + rows.push({ filer_cik: `f${i}`, filer_sic: '60', shares: 2_500_000, reported_quarter: '2026-Q2' }); + } + seedInstitutionFilings(db, 'IREN', rows); + + const edgar = createNoOpEdgar(); + const flowEngine = new InstitutionFlowEngine(edgar); + const engine = new DashboardRollupEngine(db, flowEngine); + const out = await engine.computeRollup('user-partial'); + assert.equal(out[0].convictionDelta, 'increasing'); +}); + test('DashboardRollupEngine: mixed conviction (flow vs insider disagree)', async () => { const db = createTestDb(); seedUser(db, 'user-mix'); const wlIns = db.prepare('INSERT INTO watchlists (id, owner_id, name, symbols, created_at, sort_order) VALUES (?, ?, ?, ?, ?, ?)'); wlIns.run('wl1', 'user-mix', 'default', JSON.stringify(['MSFT']), '2026-01-01', 0); - // Institutional: increasing. + // Institutional: increasing (matched book). seedInstitutionFilings(db, 'MSFT', [ { filer_cik: 'cik1', filer_sic: '60', shares: 10000, reported_quarter: '2025-Q1' }, - { filer_cik: 'cik2', filer_sic: '60', shares: 15000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik1', filer_sic: '60', shares: 15000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 8000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik2', filer_sic: '60', shares: 14000, reported_quarter: '2025-Q2' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 5000, reported_quarter: '2025-Q1' }, + { filer_cik: 'cik3', filer_sic: '60', shares: 9000, reported_quarter: '2025-Q2' }, ]); // Insider: reducing (more sell than buy). diff --git a/app/server/src/analysis/__tests__/dealerExposureEngine.test.ts b/app/server/src/analysis/__tests__/dealerExposureEngine.test.ts new file mode 100644 index 0000000..0893728 --- /dev/null +++ b/app/server/src/analysis/__tests__/dealerExposureEngine.test.ts @@ -0,0 +1,420 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + buildDealerMap, + contractGex, + isDegradedDealerMap, + isHealthyDealerMap, + isSaneIv, + layer0Educator, + resolveGreeks, + type DealerMap, +} from '../dealerExposureEngine.ts'; +import type { NormalizedOptionSurface, NormalizedOptionRow } from '../../options/types.ts'; + +function row(partial: Partial & Pick): NormalizedOptionRow { + return { + openInterest: 1000, + volume: 100, + bid: 1, + ask: 1.1, + mid: 1.05, + iv: 0.25, + greeks: { gamma: 0.02, vanna: 0.1, delta: 0.5 }, + ...partial, + }; +} + +function surface(rows: NormalizedOptionRow[], spot = 100): NormalizedOptionSurface { + return { + symbol: 'TEST', + spot, + asOf: '2026-08-06T15:00:00.000Z', + providerId: 'synthetic', + delaySeconds: 900, + oiFreshness: 'prior_session', + rows, + provenance: { + fetchedAt: '2026-08-06T15:00:00.000Z', + sourceKind: 'yfinance', + rawSourceId: 'synthetic:TEST', + }, + }; +} + +test('contractGex: call positive, put negative', () => { + const call = contractGex(0.01, 100, 100, 'call'); + const put = contractGex(0.01, 100, 100, 'put'); + assert.ok(call > 0); + assert.ok(put < 0); + assert.equal(call, -put); +}); + +test('buildDealerMap: expiries sorted nearest-left and capped', () => { + const rows: NormalizedOptionRow[] = []; + for (const exp of ['2026-08-08', '2026-08-15', '2026-08-22', '2026-09-19', '2026-10-17', '2026-11-21', '2026-12-19', '2027-01-16']) { + rows.push(row({ strike: 100, expiry: exp, right: 'call', openInterest: 500 })); + rows.push(row({ strike: 100, expiry: exp, right: 'put', openInterest: 400, greeks: { gamma: 0.015, vanna: -0.05 } })); + } + const map = buildDealerMap(surface(rows), { maxExpiries: 4 }); + assert.equal(map.expiries.length, 4); + assert.equal(map.expiries[0], '2026-08-08'); + assert.ok(map.expiries.every((e, i) => i === 0 || e >= map.expiries[i - 1])); + assert.equal(map.providerId, 'synthetic'); + assert.equal(map.convention, 'classic_call_pos_put_neg'); + assert.match(map.delayNote, /prior session/i); +}); + +test('buildDealerMap: king is max absolute GEX cell', () => { + const map = buildDealerMap( + surface([ + row({ strike: 95, expiry: '2026-08-15', right: 'put', openInterest: 2000, greeks: { gamma: 0.03, vanna: 0.05 } }), + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 500, greeks: { gamma: 0.02, vanna: 0.05 } }), + row({ strike: 105, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.05 } }), + ]), + ); + assert.ok(map.levels.king); + assert.equal(map.levels.king!.strike, 105); + assert.ok(map.cells.some((c) => c.isKing && c.strike === 105)); +}); + +test('buildDealerMap: skips rows without OI', () => { + const map = buildDealerMap( + surface([ + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: null }), + row({ strike: 100, expiry: '2026-08-15', right: 'put', openInterest: 0 }), + row({ strike: 101, expiry: '2026-08-15', right: 'call', openInterest: 1000 }), + ]), + ); + assert.ok(map.byStrike.some((s) => s.strike === 101)); + assert.ok(!map.byStrike.some((s) => s.strike === 100)); + assert.ok(map.coveragePct < 100); +}); + +test('buildDealerMap: BSM fill when gamma missing but IV present', () => { + // Asymmetric OI so call/put GEX does not cancel to zero at the strike. + const map = buildDealerMap( + surface([ + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 5000, + iv: 0.3, + greeks: null, + }), + row({ + strike: 100, + expiry: '2026-09-15', + right: 'put', + openInterest: 1000, + iv: 0.3, + greeks: null, + }), + ]), + ); + assert.ok(map.cells.length >= 1); + assert.ok(map.bsmFillPct > 0); + assert.ok(map.byStrike.some((s) => s.absGex > 0), `byStrike=${JSON.stringify(map.byStrike)}`); +}); + +test('withExposureConvention: dealer_inventory negates classic GEX and keeps king strike', async () => { + const { withExposureConvention } = await import('../dealerExposureEngine.ts'); + const classic = buildDealerMap( + surface([ + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.05 } }), + row({ strike: 95, expiry: '2026-08-15', right: 'put', openInterest: 2000, greeks: { gamma: 0.03, vanna: 0.05 } }), + ], 100), + ); + assert.equal(classic.convention, 'classic_call_pos_put_neg'); + const dealer = withExposureConvention(classic, 'dealer_inventory'); + assert.equal(dealer.convention, 'dealer_inventory'); + assert.ok(classic.levels.king && dealer.levels.king); + assert.equal(classic.levels.king!.strike, dealer.levels.king!.strike); + assert.ok( + Math.abs(classic.levels.king!.gex + dealer.levels.king!.gex) < 1e-6, + 'king gex should flip sign', + ); + for (const c of classic.cells) { + const d = dealer.cells.find((x) => x.strike === c.strike && x.expiry === c.expiry); + assert.ok(d); + assert.ok(Math.abs(c.gex + d!.gex) < 1e-6, `cell ${c.strike} gex should flip`); + } + // Round-trip + const back = withExposureConvention(dealer, 'classic_call_pos_put_neg'); + assert.equal(back.convention, 'classic_call_pos_put_neg'); + assert.ok(Math.abs(back.levels.king!.gex - classic.levels.king!.gex) < 1e-6); +}); + +test('buildDealerMap: flip between +GEX and −GEX strikes', () => { + // Strong + below and − above (or vice versa) to create a sign change + const map = buildDealerMap( + surface([ + row({ strike: 90, expiry: '2026-08-15', right: 'call', openInterest: 8000, greeks: { gamma: 0.03, vanna: 0.1 } }), + row({ strike: 95, expiry: '2026-08-15', right: 'call', openInterest: 6000, greeks: { gamma: 0.03, vanna: 0.1 } }), + row({ strike: 105, expiry: '2026-08-15', right: 'put', openInterest: 8000, greeks: { gamma: 0.03, vanna: 0.1 } }), + row({ strike: 110, expiry: '2026-08-15', right: 'put', openInterest: 6000, greeks: { gamma: 0.03, vanna: 0.1 } }), + ], 100), + ); + // puts contribute negative GEX, calls positive → flip exists between clusters + assert.ok(map.levels.flip != null, 'expected a polarity flip'); +}); + +test('buildDealerMap: detects rug_setup pattern', () => { + // +GEX above spot, large −GEX pivot below, more −GEX stack + const map = buildDealerMap( + surface([ + row({ strike: 110, expiry: '2026-08-15', right: 'call', openInterest: 10000, greeks: { gamma: 0.05, vanna: 0.1 } }), + row({ strike: 95, expiry: '2026-08-15', right: 'put', openInterest: 12000, greeks: { gamma: 0.05, vanna: 0.1 } }), + row({ strike: 90, expiry: '2026-08-15', right: 'put', openInterest: 9000, greeks: { gamma: 0.04, vanna: 0.1 } }), + row({ strike: 85, expiry: '2026-08-15', right: 'put', openInterest: 8000, greeks: { gamma: 0.04, vanna: 0.1 } }), + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 500, greeks: { gamma: 0.01, vanna: 0.05 } }), + ], 100), + ); + assert.ok( + map.patterns.some((p) => p.id === 'rug_setup'), + `patterns=${JSON.stringify(map.patterns)}`, + ); +}); + +test('buildDealerMap: detects whipsaw with air pocket between edges', () => { + // Strong nodes at 90 and 110, near-zero middle + const rows: NormalizedOptionRow[] = [ + row({ strike: 90, expiry: '2026-08-15', right: 'call', openInterest: 15000, greeks: { gamma: 0.05, vanna: 0.1 } }), + row({ strike: 110, expiry: '2026-08-15', right: 'call', openInterest: 15000, greeks: { gamma: 0.05, vanna: 0.1 } }), + ]; + for (const k of [95, 98, 100, 102, 105]) { + rows.push( + row({ + strike: k, + expiry: '2026-08-15', + right: 'call', + openInterest: 10, + greeks: { gamma: 0.0001, vanna: 0.001 }, + }), + ); + } + const map = buildDealerMap(surface(rows, 100)); + assert.ok(map.levels.airPockets.length >= 1, 'expected air pocket'); + assert.ok( + map.patterns.some((p) => p.id === 'whipsaw'), + `patterns=${JSON.stringify(map.patterns)} scenario=${map.scenario}`, + ); +}); + +test('buildDealerMap: velocity growing near air pocket', () => { + const curr = buildDealerMap( + surface([ + row({ strike: 90, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.1 } }), + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 50, greeks: { gamma: 0.001, vanna: 0.01 } }), + row({ strike: 110, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.1 } }), + ]), + ); + const prior: Pick = { + asOf: '2026-08-05T15:00:00.000Z', + byStrike: curr.byStrike.map((s) => + s.strike === 90 ? { ...s, absGex: s.absGex * 0.3, gex: s.gex * 0.3 } : { ...s, absGex: s.absGex * 0.9, gex: s.gex * 0.9 }, + ), + levels: curr.levels, + }; + const withVel = buildDealerMap( + surface([ + row({ strike: 90, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.1 } }), + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 50, greeks: { gamma: 0.001, vanna: 0.01 } }), + row({ strike: 110, expiry: '2026-08-15', right: 'call', openInterest: 5000, greeks: { gamma: 0.04, vanna: 0.1 } }), + ]), + { prior }, + ); + assert.ok(withVel.velocity); + assert.ok(withVel.velocity!.nodes.some((n) => n.growing)); +}); + +test('buildDealerMap: OPEX Friday banner within 5 days', () => { + // 2026-08-07 is a Friday + const map = buildDealerMap( + surface([ + row({ strike: 100, expiry: '2026-08-07', right: 'call', openInterest: 1000 }), + ]), + ); + assert.equal(map.opexHints.isNearOpex, true); + assert.match(map.opexHints.note, /OPEX|Near-term/i); +}); + +test('layer0Educator: no buy/sell imperatives', () => { + const map = buildDealerMap( + surface([ + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 3000 }), + row({ strike: 95, expiry: '2026-08-15', right: 'put', openInterest: 3000 }), + ]), + ); + const copy = layer0Educator(map); + const blob = JSON.stringify(copy).toLowerCase(); + assert.equal(copy.footer, 'Educational observation only.'); + assert.ok(!/\bbuy\b/.test(blob) || blob.includes('buying pressure') === false); + // Primary rule: no "you should buy/sell" + assert.ok(!/you should (buy|sell)/i.test(blob)); + assert.ok(!/\b(buy|sell) (calls|puts|shares|stock)\b/i.test(blob)); + assert.ok(copy.questions.length >= 3); + assert.ok(copy.caveats.some((c) => /prior session|delay/i.test(c))); +}); + +test('buildDealerMap: engine has no vendor coupling (synthetic only path)', () => { + // Sanity: providerId synthetic preserved; computation local_engine + const map = buildDealerMap( + surface([row({ strike: 50, expiry: '2026-08-20', right: 'call', openInterest: 100 })]), + ); + assert.equal(map.computation, 'local_engine'); + assert.equal(map.unit, 'usd_delta_adj_gamma_notional'); + assert.ok(map.quality); + assert.ok(['ok', 'degraded', 'poor'].includes(map.quality.greekQuality)); +}); + +test('isSaneIv: rejects premarket junk and absurd IV', () => { + assert.equal(isSaneIv(1e-5), false); + assert.equal(isSaneIv(0.02), false); + assert.equal(isSaneIv(0.25), true); + assert.equal(isSaneIv(4.0), false); + assert.equal(isSaneIv(null), false); +}); + +test('resolveGreeks: skips junk IV without vendor gamma', () => { + const r = resolveGreeks( + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 1000, + iv: 1e-5, + greeks: null, + }), + 100, + '2026-08-07T13:14:00.000Z', + 0.04, + 0, + ); + assert.equal(r.skippedBadIv, true); + assert.equal(r.gamma, 0); +}); + +test('resolveGreeks: uses fallback IV hint when current IV is junk', () => { + const r = resolveGreeks( + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 5000, + iv: 1e-5, + greeks: null, + }), + 100, + '2026-08-07T13:14:00.000Z', + 0.04, + 0, + { fallbackIv: 0.35 }, + ); + assert.equal(r.skippedBadIv, false); + assert.ok(r.gamma > 0); + assert.equal(r.usedFallbackIv, true); + assert.equal(r.usedBsm, true); +}); + +test('buildDealerMap: junk IV rows skipped; quality poor when most OI unusable', () => { + // No sane IV on the surface → no ATM proxy → skip + const map = buildDealerMap( + surface([ + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 9000, + iv: 1e-5, + greeks: null, + }), + row({ + strike: 100, + expiry: '2026-09-15', + right: 'put', + openInterest: 8000, + iv: 0.01, + greeks: null, + }), + row({ + strike: 105, + expiry: '2026-09-15', + right: 'call', + openInterest: 100, + iv: 0.02, + greeks: null, + }), + ]), + ); + assert.ok(map.quality.skippedBadIv >= 2, `skipped=${map.quality.skippedBadIv}`); + assert.equal(map.quality.greekQuality, 'poor'); + assert.ok(isDegradedDealerMap(map)); + assert.equal(isHealthyDealerMap(map), false); +}); + +test('buildDealerMap: ATM IV proxy fills junk rows when some sane IV exists', () => { + const map = buildDealerMap( + surface([ + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 5000, + iv: 1e-5, + greeks: null, + }), + row({ + strike: 100, + expiry: '2026-09-15', + right: 'put', + openInterest: 1000, + iv: 1e-5, + greeks: null, + }), + row({ + strike: 101, + expiry: '2026-09-15', + right: 'call', + openInterest: 200, + iv: 0.28, + greeks: null, + }), + ]), + ); + assert.ok(map.byStrike.some((s) => s.absGex > 0)); + assert.ok(map.quality.fallbackIvUses >= 1 || map.quality.included >= 1); +}); + +test('buildDealerMap: ivHints restore structure when live IV is junk', () => { + const map = buildDealerMap( + surface([ + row({ + strike: 100, + expiry: '2026-09-15', + right: 'call', + openInterest: 5000, + iv: 1e-5, + greeks: null, + }), + row({ + strike: 100, + expiry: '2026-09-15', + right: 'put', + openInterest: 1000, + iv: 1e-5, + greeks: null, + }), + ]), + { + ivHints: { + '2026-09-15|100|call': 0.3, + '2026-09-15|100|put': 0.3, + }, + }, + ); + assert.ok(map.byStrike.some((s) => s.absGex > 0)); + assert.ok(map.quality.fallbackIvUses >= 1); + assert.ok(map.quality.greekQuality !== 'poor' || map.cells.length > 0); +}); diff --git a/app/server/src/analysis/__tests__/dealerMapGolden.test.ts b/app/server/src/analysis/__tests__/dealerMapGolden.test.ts new file mode 100644 index 0000000..8f3a66b --- /dev/null +++ b/app/server/src/analysis/__tests__/dealerMapGolden.test.ts @@ -0,0 +1,63 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { buildDealerMap } from '../dealerExposureEngine.ts'; +import { auditDealerMap } from '../dealerMapIntegrity.ts'; +import type { NormalizedOptionSurface } from '../../options/types.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const fixturePath = join(__dirname, '../__fixtures__/dealer-surface-synthetic.json'); + +function loadFixture(): NormalizedOptionSurface { + return JSON.parse(readFileSync(fixturePath, 'utf8')) as NormalizedOptionSurface; +} + +/** Small fixture uses looser strike-count thresholds; production defaults stay strict. */ +const FIXTURE_THRESHOLDS = { + minOiStrikesPerExpiry: 3, + minNearSpotOiStrikes: 3, + minCells: 6, + minByStrike: 3, +}; + +test('golden fixture: map is complete and king is max |gex| call cluster', () => { + const surface = loadFixture(); + const map = buildDealerMap(surface, { maxExpiries: 3 }); + const integrity = auditDealerMap(map, surface, { thresholds: FIXTURE_THRESHOLDS }); + + assert.equal(integrity.status, 'complete', JSON.stringify(integrity.hardFailed)); + assert.ok(map.levels.king); + // Heaviest call OI + gamma at 105 08-08 + assert.equal(map.levels.king!.strike, 105); + assert.equal(map.levels.king!.expiry, '2026-08-08'); + assert.ok(map.cells.length >= 6); + assert.ok(map.quality.greekQuality === 'ok' || map.quality.greekQuality === 'degraded'); + + // Determinism + const map2 = buildDealerMap(surface, { maxExpiries: 3 }); + assert.equal(map2.levels.king!.gex, map.levels.king!.gex); + assert.equal(map2.cells.length, map.cells.length); +}); + +test('golden fixture: zero IV across surface → incomplete integrity', () => { + const surface = loadFixture(); + surface.rows = surface.rows.map((r) => ({ + ...r, + iv: 1e-5, + greeks: null, + })); + const map = buildDealerMap(surface, { maxExpiries: 3 }); + const integrity = auditDealerMap(map, surface, { thresholds: FIXTURE_THRESHOLDS }); + assert.equal(integrity.status, 'incomplete'); +}); + +test('golden fixture: drop to one expiry → incomplete', () => { + const surface = loadFixture(); + surface.rows = surface.rows.filter((r) => r.expiry === '2026-08-08'); + const map = buildDealerMap(surface, { maxExpiries: 3 }); + const integrity = auditDealerMap(map, surface, { thresholds: FIXTURE_THRESHOLDS }); + assert.equal(integrity.status, 'incomplete'); + assert.ok(integrity.hardFailed.includes('expiries')); +}); diff --git a/app/server/src/analysis/__tests__/dealerMapIntegrity.test.ts b/app/server/src/analysis/__tests__/dealerMapIntegrity.test.ts new file mode 100644 index 0000000..1e8bc14 --- /dev/null +++ b/app/server/src/analysis/__tests__/dealerMapIntegrity.test.ts @@ -0,0 +1,130 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + auditDealerMap, + auditDealerSurface, + integritySummary, +} from '../dealerMapIntegrity.ts'; +import { buildDealerMap } from '../dealerExposureEngine.ts'; +import type { NormalizedOptionRow, NormalizedOptionSurface } from '../../options/types.ts'; + +function row( + partial: Partial & Pick, +): NormalizedOptionRow { + return { + openInterest: 1000, + volume: 100, + bid: 1, + ask: 1.1, + mid: 1.05, + iv: 0.3, + greeks: { gamma: 0.02, vanna: 0.1, delta: 0.5 }, + ...partial, + }; +} + +function fatSurface(): NormalizedOptionSurface { + const rows: NormalizedOptionRow[] = []; + const exps = ['2026-08-08', '2026-08-15', '2026-08-22', '2026-08-29']; + for (const exp of exps) { + for (let k = 90; k <= 110; k += 1) { + rows.push(row({ strike: k, expiry: exp, right: 'call', openInterest: 500 })); + rows.push( + row({ + strike: k, + expiry: exp, + right: 'put', + openInterest: 400, + greeks: { gamma: 0.015, vanna: -0.05 }, + }), + ); + } + } + return { + symbol: 'TEST', + spot: 100, + asOf: '2026-08-06T15:00:00.000Z', + providerId: 'synthetic', + delaySeconds: 900, + oiFreshness: 'prior_session', + rows, + provenance: { + fetchedAt: '2026-08-06T15:00:00.000Z', + sourceKind: 'yfinance', + rawSourceId: 'synthetic:TEST', + }, + }; +} + +test('auditDealerSurface: null surface incomplete', () => { + const r = auditDealerSurface(null); + assert.equal(r.status, 'incomplete'); + assert.ok(r.hardFailed.includes('surface')); +}); + +test('auditDealerMap: complete fat surface + map', () => { + const s = fatSurface(); + const map = buildDealerMap(s, { maxExpiries: 4 }); + const r = auditDealerMap(map, s); + assert.equal(r.status, 'complete', JSON.stringify(r.hardFailed)); + assert.ok(r.metrics.cells >= 8); + assert.match(integritySummary(r), /complete/i); +}); + +test('auditDealerSurface: delay alone does not hard-fail', () => { + const s = fatSurface(); + s.delaySeconds = 900; + const map = buildDealerMap(s); + const r = auditDealerMap(map, s); + assert.notEqual(r.status, 'incomplete'); + assert.ok(r.checks.some((c) => c.id === 'delay' && c.ok)); +}); + +test('auditDealerSurface: missing spot hard-fails', () => { + const s = fatSurface(); + s.spot = 0; + const r = auditDealerSurface(s); + assert.equal(r.status, 'incomplete'); + assert.ok(r.hardFailed.includes('spot')); +}); + +test('auditDealerSurface: too few expiries incomplete', () => { + const s = fatSurface(); + s.rows = s.rows.filter((r) => r.expiry === '2026-08-08'); + const r = auditDealerSurface(s); + assert.equal(r.status, 'incomplete'); + assert.ok(r.hardFailed.includes('expiries')); +}); + +test('auditDealerSurface: junk IV only incomplete on usable greeks', () => { + const s = fatSurface(); + s.rows = s.rows.map((r) => ({ + ...r, + iv: 1e-5, + greeks: null, + })); + const map = buildDealerMap(s); + const r = auditDealerMap(map, s); + assert.equal(r.status, 'incomplete'); + assert.ok( + r.hardFailed.some((id) => id === 'usable_greeks' || id === 'cells' || id === 'greek_quality'), + JSON.stringify(r.hardFailed), + ); +}); + +test('auditDealerMap: thin cell grid hard-fails cells', () => { + const s: NormalizedOptionSurface = { + ...fatSurface(), + rows: [ + row({ strike: 100, expiry: '2026-08-08', right: 'call', openInterest: 1000 }), + row({ strike: 100, expiry: '2026-08-15', right: 'call', openInterest: 1000 }), + row({ strike: 100, expiry: '2026-08-22', right: 'call', openInterest: 1000 }), + ], + }; + const map = buildDealerMap(s); + // Force tiny map metrics path via direct audit with map that has few cells + const tiny = { ...map, cells: map.cells.slice(0, 2), byStrike: map.byStrike.slice(0, 1) }; + const r = auditDealerMap(tiny as typeof map, s); + assert.equal(r.status, 'incomplete'); + assert.ok(r.hardFailed.includes('cells') || r.hardFailed.includes('by_strike')); +}); diff --git a/app/server/src/analysis/__tests__/dealerMapService.test.ts b/app/server/src/analysis/__tests__/dealerMapService.test.ts new file mode 100644 index 0000000..ac7d7da --- /dev/null +++ b/app/server/src/analysis/__tests__/dealerMapService.test.ts @@ -0,0 +1,168 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { + getDealerMap, + recomputeDealerMapFromCache, + scheduleDealerMapInputs, + writeDealerMapCache, + dealerMapCacheKey, +} from '../dealerMapService.ts'; +import { buildDealerMap } from '../dealerExposureEngine.ts'; +import type { NormalizedOptionSurface } from '../../options/types.ts'; + +function memDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE quotes (symbol TEXT PRIMARY KEY, price REAL, bid REAL, ask REAL, change REAL, change_pct REAL, iv REAL, session TEXT, regular_price REAL, observed_at TEXT); + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + CREATE TABLE options_chains ( + symbol TEXT, expiry TEXT, strike REAL, type TEXT, + bid REAL, ask REAL, iv REAL, delta REAL, gamma REAL, theta REAL, vega REAL, rho REAL, + open_interest INTEGER, volume INTEGER, ts TEXT, + PRIMARY KEY (symbol, expiry, strike, type, ts) + ); + CREATE TABLE dealer_map_snapshots ( + symbol TEXT NOT NULL, + as_of_date TEXT NOT NULL, + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (symbol, as_of_date) + ); + CREATE TABLE symbol_demand ( + symbol TEXT PRIMARY KEY, + refcount INTEGER NOT NULL DEFAULT 0, + ticker_kind TEXT, + system_pin INTEGER NOT NULL DEFAULT 0 + ); + `); + return db; +} + +function seedChain(db: DatabaseSync): void { + db.prepare( + 'INSERT INTO quotes (symbol,price,observed_at) VALUES (?,?,?)', + ).run('SPY', 500, '2026-08-06T14:00:00.000Z'); + const exps = ['2026-08-08', '2026-08-15', '2026-08-22']; + db.prepare( + 'INSERT INTO kv_cache (key,value,observed_at) VALUES (?,?,?)', + ).run('options_expiry:SPY', JSON.stringify(exps), '2026-08-06T14:00:00.000Z'); + const ins = db.prepare( + `INSERT INTO options_chains (symbol,expiry,strike,type,bid,ask,iv,delta,gamma,theta,vega,open_interest,volume,ts) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, + ); + const ts = '2026-08-06T14:00:00.000Z'; + for (const exp of exps) { + for (let k = 480; k <= 520; k += 5) { + ins.run('SPY', exp, k, 'call', 1, 1.1, 0.2, 0.5, 0.04, -0.1, 0.2, 5000, 100, ts); + ins.run('SPY', exp, k, 'put', 1, 1.1, 0.2, -0.4, 0.03, -0.1, 0.2, 4000, 100, ts); + } + } +} + +test('recomputeDealerMapFromCache builds map without vendor', () => { + const db = memDb(); + seedChain(db); + const map = recomputeDealerMapFromCache(db, 'SPY'); + assert.ok(map); + assert.equal(map!.symbol, 'SPY'); + assert.ok(map!.levels.king); + assert.equal(map!.providerId, 'yfinance'); + // Complete maps are written to kv; incomplete recomputes are not. + const row = db.prepare('SELECT value FROM kv_cache WHERE key=?').get(dealerMapCacheKey('SPY')) as + | { value: string } + | undefined; + // Smoke: recompute path runs; cache write depends on integrity completeness. + assert.ok(map!.cells.length > 0); + if (row?.value) assert.ok(row.value.includes('SPY')); +}); + +test('getDealerMap never requires queue for cached recompute', async () => { + const db = memDb(); + seedChain(db); + const result = await getDealerMap({ db, cache: null, queue: null }, 'SPY'); + assert.ok(result.map); + assert.equal(result.scheduled, false); + assert.ok(result.educator); + assert.ok(result.hanScript); + assert.ok(result.hanScript!.script.length > 10); + assert.match(result.hanScript!.script, /educational observation only/i); + assert.ok( + result.status === 'ok' || result.status === 'stale' || result.status === 'incomplete', + `status=${result.status}`, + ); + assert.ok(result.integrity); +}); + +test('getDealerMap schedules when empty and queue present', async () => { + const db = memDb(); + const queued: string[] = []; + const queue = { + queue: async (key: string) => { queued.push(key); }, + } as any; + const cache = { + ensureInDemand: async () => {}, + } as any; + + const result = await getDealerMap({ db, cache, queue }, 'AAPL'); + assert.equal(result.map, null); + assert.equal(result.status, 'pending'); + assert.equal(result.scheduled, true); + assert.ok(queued.some((k) => k.includes('quote:AAPL'))); + assert.ok(queued.some((k) => k.includes('expiry_dates:AAPL'))); +}); + +test('scheduleDealerMapInputs caps chain jobs to maxExpiries', async () => { + const db = memDb(); + db.prepare( + 'INSERT INTO kv_cache (key,value,observed_at) VALUES (?,?,?)', + ).run( + 'options_expiry:QQQ', + JSON.stringify(['2026-08-08', '2026-08-15', '2026-08-22', '2026-08-29', '2026-09-05', '2026-09-12', '2026-09-19', '2026-10-17']), + '2026-08-06T14:00:00.000Z', + ); + const queued: string[] = []; + const queue = { queue: async (key: string) => { queued.push(key); } } as any; + await scheduleDealerMapInputs({ db, queue, cache: { ensureInDemand: async () => {} } as any }, 'QQQ', 4); + const chains = queued.filter((k) => k.startsWith('yfinance:chain:QQQ:')); + assert.equal(chains.length, 4); +}); + +test('getDealerMap returns stale cached map when chains missing', async () => { + const db = memDb(); + const synthetic: NormalizedOptionSurface = { + symbol: 'MSFT', + spot: 400, + asOf: '2026-08-01T15:00:00.000Z', + providerId: 'synthetic', + delaySeconds: 900, + oiFreshness: 'prior_session', + rows: [ + { + strike: 400, + expiry: '2026-08-15', + right: 'call', + openInterest: 1000, + volume: 10, + bid: 1, + ask: 1.1, + mid: 1.05, + iv: 0.25, + greeks: { gamma: 0.02, vanna: 0.05 }, + }, + ], + provenance: { fetchedAt: '2026-08-01T15:00:00.000Z', sourceKind: 'yfinance' }, + }; + const map = buildDealerMap(synthetic); + writeDealerMapCache(db, map); + + const queued: string[] = []; + const result = await getDealerMap( + { db, queue: { queue: async (k: string) => { queued.push(k); } } as any, cache: { ensureInDemand: async () => {} } as any }, + 'MSFT', + ); + assert.ok(result.map); + // Cached thin synthetic map is incomplete by integrity gates, but still served as last map. + assert.ok(result.status === 'stale' || result.status === 'incomplete', result.status); + assert.ok(queued.length > 0); +}); diff --git a/app/server/src/analysis/__tests__/dealerStudyEngine.test.ts b/app/server/src/analysis/__tests__/dealerStudyEngine.test.ts new file mode 100644 index 0000000..a10c0d8 --- /dev/null +++ b/app/server/src/analysis/__tests__/dealerStudyEngine.test.ts @@ -0,0 +1,141 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { + gradeStudySetup, + proposeStudySetups, + type CandleBar, +} from '../dealerStudyEngine.ts'; +import type { DealerMap } from '../dealerExposureEngine.ts'; + +function baseMap(over: Partial = {}): DealerMap { + return { + symbol: 'TEST', + spot: 100, + asOf: '2026-08-01T00:00:00.000Z', + delayNote: 'delayed', + delaySeconds: 900, + oiFreshness: 'prior_session', + providerId: 'yfinance', + convention: 'classic_retail', + unit: 'usd', + computation: 'settled_oi', + expiries: ['2026-08-15'], + cells: [], + byStrike: [], + netVexByStrike: [], + levels: { + king: { strike: 105, expiry: '2026-08-15', gex: 1e6 }, + gatekeepers: [], + flip: 98, + callWall: 108, + putWall: 95, + airPockets: [], + clusters: [], + }, + regimeAtSpot: 'positive', + scenario: 'gex_dominates', + patterns: [], + opexHints: { isNearOpex: false, note: '' }, + provenance: { source: 'test' } as DealerMap['provenance'], + coveragePct: 90, + bsmFillPct: 80, + ...over, + }; +} + +describe('proposeStudySetups', () => { + it('proposes cushion pin when positive regime and king away from spot', () => { + const props = proposeStudySetups(baseMap()); + assert.ok(props.some((p) => p.hypothesisId === 'cushion_pin_toward_king')); + assert.ok(props.every((p) => p.disclaimer === 'Educational observation only.')); + }); + + it('proposes spring setup when negative regime', () => { + const props = proposeStudySetups(baseMap({ regimeAtSpot: 'negative' })); + assert.ok(props.some((p) => p.hypothesisId === 'spring_accelerate_from_spot')); + }); +}); + +describe('gradeStudySetup', () => { + const loggedAt = '2026-08-01T00:00:00.000Z'; + + it('grades long win when target before invalidation', () => { + const bars: CandleBar[] = [ + { ts: '2026-08-02', o: 100, h: 101, l: 99, c: 100.5 }, + { ts: '2026-08-03', o: 100.5, h: 106, l: 100, c: 105 }, + ]; + const r = gradeStudySetup( + { + bias: 'educational_long', + entry: 100, + invalidation: 95, + target: 105, + horizonDays: 5, + loggedAt, + }, + bars, + new Date('2026-08-10'), + ); + assert.equal(r.grade, 'win'); + assert.ok(r.score >= 55); + }); + + it('grades long loss when invalidation first', () => { + const bars: CandleBar[] = [ + { ts: '2026-08-02', o: 100, h: 100.5, l: 94, c: 95 }, + { ts: '2026-08-03', o: 95, h: 110, l: 94, c: 108 }, + ]; + const r = gradeStudySetup( + { + bias: 'educational_long', + entry: 100, + invalidation: 95, + target: 105, + horizonDays: 5, + loggedAt, + }, + bars, + new Date('2026-08-10'), + ); + assert.equal(r.grade, 'loss'); + }); + + it('grades range win if held inside edges', () => { + const bars: CandleBar[] = [ + { ts: '2026-08-02', o: 100, h: 102, l: 98, c: 100 }, + { ts: '2026-08-03', o: 100, h: 103, l: 97, c: 101 }, + { ts: '2026-08-04', o: 101, h: 102, l: 99, c: 100 }, + { ts: '2026-08-05', o: 100, h: 101, l: 99, c: 100 }, + { ts: '2026-08-06', o: 100, h: 102, l: 98, c: 100 }, + ]; + const r = gradeStudySetup( + { + bias: 'educational_range', + entry: 100, + invalidation: 95, + target: 105, + horizonDays: 5, + loggedAt, + }, + bars, + new Date('2026-08-10'), + ); + assert.equal(r.grade, 'win'); + }); + + it('returns incomplete when no bars', () => { + const r = gradeStudySetup( + { + bias: 'educational_long', + entry: 100, + invalidation: 95, + target: 105, + horizonDays: 5, + loggedAt, + }, + [], + new Date('2026-08-10'), + ); + assert.equal(r.grade, 'incomplete'); + }); +}); diff --git a/app/server/src/analysis/__tests__/hanStyleLevels.test.ts b/app/server/src/analysis/__tests__/hanStyleLevels.test.ts new file mode 100644 index 0000000..559d90f --- /dev/null +++ b/app/server/src/analysis/__tests__/hanStyleLevels.test.ts @@ -0,0 +1,205 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { extractHanStyleLevels } from '../hanStyleLevels.ts'; +import { + buildDealerMap, + withExposureConvention, + type DealerMap, +} from '../dealerExposureEngine.ts'; +import type { NormalizedOptionSurface, NormalizedOptionRow } from '../../options/types.ts'; + +function row( + partial: Partial & Pick, +): NormalizedOptionRow { + return { + openInterest: 1000, + volume: 100, + bid: 1, + ask: 1.1, + mid: 1.05, + iv: 0.25, + greeks: { gamma: 0.02, vanna: 0.1, delta: 0.5 }, + ...partial, + }; +} + +function surface(rows: NormalizedOptionRow[], spot = 40): NormalizedOptionSurface { + return { + symbol: 'IREN', + spot, + asOf: '2026-08-07T15:00:00.000Z', + providerId: 'synthetic', + delaySeconds: 900, + oiFreshness: 'prior_session', + rows, + provenance: { + fetchedAt: '2026-08-07T15:00:00.000Z', + sourceKind: 'yfinance', + rawSourceId: 'synthetic:IREN', + }, + }; +} + +/** Han-like stack: long support below, short king lower, resist above. */ +function irenLikeMap(spot = 41.2): DealerMap { + const expNear = '2026-08-15'; + const expMid = '2026-08-22'; + const expFar = '2026-09-19'; + return buildDealerMap( + surface( + [ + // Strong +GEX at 38 (long support below spot) + row({ + strike: 38, + expiry: expNear, + right: 'call', + openInterest: 12_000, + greeks: { gamma: 0.05, vanna: 0.1, delta: 0.4 }, + }), + row({ + strike: 38, + expiry: expNear, + right: 'put', + openInterest: 2_000, + greeks: { gamma: 0.02, vanna: 0.05, delta: -0.3 }, + }), + // Short / put heavy at 34 + row({ + strike: 34, + expiry: expNear, + right: 'put', + openInterest: 15_000, + greeks: { gamma: 0.04, vanna: 0.08, delta: -0.55 }, + }), + row({ + strike: 34, + expiry: expNear, + right: 'call', + openInterest: 1_000, + greeks: { gamma: 0.01, vanna: 0.05, delta: 0.2 }, + }), + // Resistance pile at 45 (heavy calls above) + row({ + strike: 45, + expiry: expNear, + right: 'call', + openInterest: 10_000, + greeks: { gamma: 0.04, vanna: 0.1, delta: 0.35 }, + }), + row({ + strike: 45, + expiry: expNear, + right: 'put', + openInterest: 1_500, + greeks: { gamma: 0.015, vanna: 0.05, delta: -0.25 }, + }), + // ATM mixed + row({ + strike: 41, + expiry: expNear, + right: 'call', + openInterest: 4_000, + greeks: { gamma: 0.03, vanna: 0.1, delta: 0.5 }, + }), + row({ + strike: 41, + expiry: expNear, + right: 'put', + openInterest: 3_500, + greeks: { gamma: 0.03, vanna: 0.08, delta: -0.5 }, + }), + // Far OPEX structure lower (35 / 30) + row({ + strike: 35, + expiry: expFar, + right: 'put', + openInterest: 8_000, + greeks: { gamma: 0.025, vanna: 0.06, delta: -0.5 }, + }), + row({ + strike: 30, + expiry: expFar, + right: 'put', + openInterest: 7_000, + greeks: { gamma: 0.02, vanna: 0.05, delta: -0.6 }, + }), + row({ + strike: 38, + expiry: expMid, + right: 'call', + openInterest: 5_000, + greeks: { gamma: 0.03, vanna: 0.08, delta: 0.4 }, + }), + ], + spot, + ), + { maxExpiries: 6 }, + ); +} + +test('extractHanStyleLevels: long support / resist / short vocabulary', () => { + const map = irenLikeMap(41.2); + const han = extractHanStyleLevels(map); + + assert.equal(han.symbol, 'IREN'); + assert.equal(han.convention, 'classic_call_pos_put_neg'); + assert.ok(han.longSupport != null, 'expects long support below spot'); + assert.ok(han.longSupport! <= map.spot + 1e-6); + assert.ok(han.resistance != null, 'expects resistance above or at tough level'); + assert.ok(han.shortKing != null, 'expects a short king somewhere'); + + const blob = `${han.script}\n${han.chips.join('\n')}`.toLowerCase(); + assert.match(blob, /long/); + assert.ok( + /support|resist|short|opex/.test(blob), + `expected Han vocabulary in chips/script, got: ${blob}`, + ); + assert.match(han.script, /educational observation only/i); + assert.ok(han.chips.length >= 1); +}); + +test('extractHanStyleLevels: dealer convention flips signs and updates script', () => { + const classic = irenLikeMap(41.2); + const dealer = withExposureConvention(classic, 'dealer_inventory'); + const hClassic = extractHanStyleLevels(classic); + const hDealer = extractHanStyleLevels(dealer); + + assert.equal(hDealer.convention, 'dealer_inventory'); + assert.match(hDealer.script, /dealer\/hs/i); + assert.match(hClassic.script, /classic/i); + + // Full sign flip → long/short labels at a strike can swap + if (hClassic.longSupport != null && hDealer.shortKing != null) { + // Not always equal strikes, but regime or king sign must differ somewhere + const sameSupport = + hClassic.longSupport === hDealer.longSupport && + hClassic.shortKing === hDealer.shortKing && + hClassic.regimeLabel === hDealer.regimeLabel; + assert.equal(sameSupport, false, 'dealer flip should change Han levels or regime'); + } +}); + +test('extractHanStyleLevels: empty-ish map still returns safe script', () => { + const thin = buildDealerMap( + surface([ + row({ + strike: 40, + expiry: '2026-08-15', + right: 'call', + openInterest: 50, + greeks: { gamma: 0.01, vanna: 0.02, delta: 0.5 }, + }), + ], 40), + ); + const han = extractHanStyleLevels(thin); + assert.equal(han.symbol, 'IREN'); + assert.ok(han.script.length > 20); + assert.match(han.script, /educational observation only/i); + assert.ok(['long_gamma', 'short_gamma', 'mixed'].includes(han.regimeLabel)); +}); + +test('extractHanStyleLevels: no buy/sell imperatives', () => { + const han = extractHanStyleLevels(irenLikeMap(41.2)); + const bad = /\b(buy|sell|short it|long it|enter|exit|target)\b/i; + assert.ok(!bad.test(han.script), `imperative found: ${han.script}`); +}); diff --git a/app/server/src/analysis/__tests__/marketRotationRs.test.ts b/app/server/src/analysis/__tests__/marketRotationRs.test.ts index 65ab34f..d53b742 100644 --- a/app/server/src/analysis/__tests__/marketRotationRs.test.ts +++ b/app/server/src/analysis/__tests__/marketRotationRs.test.ts @@ -2,6 +2,8 @@ import { test } from 'node:test'; import { strict as assert } from 'node:assert'; import { totalReturnPct, + dayChangePct, + returnsForHorizons, relativeStrength, buildSectorRsMap, summarizeRotation, @@ -25,6 +27,27 @@ test('totalReturnPct: computes window return', () => { assert.ok(Math.abs((r as number) - 10) < 1.5); }); +test('dayChangePct: last close vs previous close', () => { + assert.equal(dayChangePct(candles([100, 110])), 10); + assert.equal(dayChangePct(candles([100, 90])), -10); + assert.equal(dayChangePct(candles([100])), null); +}); + +test('returnsForHorizons oneDay: not zero when 24h-ago target falls on a gap (13:30Z bars, Monday)', () => { + // yfinance daily bars carry 13:30Z timestamps. Latest = Monday 08-03, prev = Friday 07-31. + // A window-based oneDay (latestTs − 24h) lands on Sunday — closer to the latest candle + // than to Friday — which used to produce exactly 0. + const ts = (d: string) => new Date(`${d}T13:30:00.000Z`).toISOString(); + const bars: CandlePoint[] = [ + { ts: ts('2026-07-30'), c: 100, v: 100 }, + { ts: ts('2026-07-31'), c: 102, v: 100 }, + { ts: ts('2026-08-03'), c: 104, v: 100 }, // Monday + ]; + const r = returnsForHorizons(bars); + assert.notEqual(r.oneDay, 0); + assert.ok(Math.abs((r.oneDay as number) - (104 / 102 - 1) * 100) < 1e-9); +}); + test('relativeStrength: sector minus bench', () => { assert.equal(relativeStrength(12, 5), 7); assert.equal(relativeStrength(null, 5), null); @@ -72,5 +95,5 @@ test('summarizeRotation: produces educational summary without trade verbs', () = const s = summarizeRotation(rows); assert.ok(s.leadershipSpread > 0 || s.strength === 'none' || s.leadingCount >= 0); assert.ok(!/you should|buy |sell /i.test(s.summary)); - assert.ok(/outperform|underperform|leadership|educational|broad market/i.test(s.summary)); + assert.ok(/outperform|underperform|leadership|relative|broad market/i.test(s.summary)); }); diff --git a/app/server/src/analysis/dashboardRollup.ts b/app/server/src/analysis/dashboardRollup.ts index 77dcae8..730f797 100644 --- a/app/server/src/analysis/dashboardRollup.ts +++ b/app/server/src/analysis/dashboardRollup.ts @@ -164,6 +164,51 @@ function classifyHolder(filers: Array<{ cik: string; sic: string | null }>): str // Core rollup engine // --------------------------------------------------------------------------- +/** + * Sum shares only for filers present in both quarters (matched book). + * Avoids incomplete 13F season false "lightening" when raw quarter totals drop + * because managers simply have not filed yet. + */ +export function matchedBookShareTotals( + db: DatabaseSync, + symbol: string, + currQuarter: string, + prevQuarter: string, +): { currShares: number; prevShares: number; filerCount: number } | null { + try { + const row = db + .prepare( + `SELECT + SUM(curr.shares) AS curr_shares, + SUM(prev.shares) AS prev_shares, + COUNT(*) AS filer_count + FROM ( + SELECT filer_cik, MAX(COALESCE(shares, 0)) AS shares + FROM institution_filings + WHERE symbol = ? AND reported_quarter = ? + GROUP BY filer_cik + ) curr + INNER JOIN ( + SELECT filer_cik, MAX(COALESCE(shares, 0)) AS shares + FROM institution_filings + WHERE symbol = ? AND reported_quarter = ? + GROUP BY filer_cik + ) prev ON prev.filer_cik = curr.filer_cik`, + ) + .get(symbol, currQuarter, symbol, prevQuarter) as + | { curr_shares: number | null; prev_shares: number | null; filer_count: number } + | undefined; + if (!row || !row.filer_count) return null; + return { + currShares: Number(row.curr_shares) || 0, + prevShares: Number(row.prev_shares) || 0, + filerCount: Number(row.filer_count) || 0, + }; + } catch { + return null; + } +} + /** * DashboardRollup — pure aggregation module. * @@ -284,18 +329,40 @@ export class DashboardRollupEngine { let flowDirection: 'increasing' | 'reducing' | 'flat' = 'flat'; - // Find two complete (non-current) quarters to compare. Skip the most recent - // quarter if it has far fewer filers than the one before it — that means the - // 13F window (45 days post-quarter-end) hasn't closed yet. + // 13F season: the latest quarter is often incomplete (many managers have not + // filed yet). Comparing raw SUM(shares) then falsely looks like "lightening" + // because missing filers drop out of the total — not because holders sold. + // + // Rules: + // 1. If latest quarter has <75% of the prior quarter's filer count, skip it + // and compare the two previous quarters (stricter than the old 50% cut). + // 2. Prefer matched-book QoQ: only filers present in BOTH quarters. That + // measures real position changes (e.g. BlackRock +8.9M) instead of + // coverage gaps. let cmpIdx = 0; - if (quarters.length >= 3 && quarters[0].num_filings < quarters[1].num_filings * 0.5) { + if ( + quarters.length >= 3 && + quarters[0].num_filings < quarters[1].num_filings * 0.75 + ) { cmpIdx = 1; } if (quarters.length >= cmpIdx + 2) { - const currQ = quarters[cmpIdx]; - const prevQ = quarters[cmpIdx + 1]; - if (currQ.total_shares > prevQ.total_shares * 1.1) flowDirection = 'increasing'; - else if (currQ.total_shares < prevQ.total_shares * 0.9) flowDirection = 'reducing'; + const currLabel = quarters[cmpIdx].reported_quarter; + const prevLabel = quarters[cmpIdx + 1].reported_quarter; + const matched = matchedBookShareTotals(d, symbol, currLabel, prevLabel); + if (matched && matched.filerCount >= 3) { + const { currShares, prevShares } = matched; + if (prevShares > 0 && currShares > prevShares * 1.1) flowDirection = 'increasing'; + else if (prevShares > 0 && currShares < prevShares * 0.9) flowDirection = 'reducing'; + else if (prevShares === 0 && currShares > 10000) flowDirection = 'increasing'; + } else { + // Fallback: full-quarter sums only when we did not skip for incompleteness + // and matched set is too thin. + const currQ = quarters[cmpIdx]; + const prevQ = quarters[cmpIdx + 1]; + if (currQ.total_shares > prevQ.total_shares * 1.1) flowDirection = 'increasing'; + else if (currQ.total_shares < prevQ.total_shares * 0.9) flowDirection = 'reducing'; + } } else if (quarters.length > cmpIdx) { if (quarters[cmpIdx].total_shares > 10000) flowDirection = 'increasing'; } diff --git a/app/server/src/analysis/dealerExposureEngine.ts b/app/server/src/analysis/dealerExposureEngine.ts new file mode 100644 index 0000000..4da2a58 --- /dev/null +++ b/app/server/src/analysis/dealerExposureEngine.ts @@ -0,0 +1,1168 @@ +// Pure dealer exposure map: GEX / VEX / levels / scenarios / named patterns. +// Vendor-free: only consumes NormalizedOptionSurface (+ optional prior snapshot for velocity). + +import { bsmGreeks, yearsToExpiry } from '../options/bsm.ts'; +import type { + ExposureComputation, + ExposureConvention, + ExposureUnit, + NormalizedOptionRow, + NormalizedOptionSurface, + OptionsProviderId, + OiFreshness, +} from '../options/types.ts'; +import type { Provenance } from '../cache/CacheRepository.ts'; +import { DEALER_FLOW_EXPLAIN_NOTES } from './dealerFlowExplainNotes.ts'; + +// ----- Public types ----- + +export type Polarity = 'pos' | 'neg' | 'near_zero'; + +export type ScenarioCode = + | 'pos_gex_pos_vex' + | 'neg_gex_neg_vex' + | 'gex_dominates' + | 'vex_dominates' + | 'mixed' + | 'insufficient_data'; + +export type PatternHit = + | { + id: 'whipsaw'; + edges: [number, number]; + desert: { from: number; to: number }; + confidence: number; + } + | { + id: 'rainbow_road'; + reason: string; + confidence: number; + } + | { + id: 'rug_setup'; + longAbove: number; + shortPivot: number; + shortStackBelow: number[]; + confidence: number; + }; + +export interface DealerMapCell { + strike: number; + expiry: string; + gex: number; + vex: number; + oi: number; + volume: number; + polarity: Polarity; + isKing: boolean; +} + +export interface DealerMapLevels { + king: { strike: number; expiry: string; gex: number } | null; + gatekeepers: Array<{ strike: number; absGex: number }>; + flip: number | null; + callWall: number | null; + putWall: number | null; + airPockets: Array<{ from: number; to: number }>; + clusters: Array<{ strikes: number[]; absGexSum: number }>; +} + +export interface VelocityNode { + strike: number; + prevAbsGex: number; + currAbsGex: number; + deltaAbsGex: number; + growing: boolean; +} + +export interface VelocitySummary { + asOfPrev: string; + nodes: VelocityNode[]; + growingNearAirPocket: boolean; +} + +export interface DealerMap { + symbol: string; + spot: number; + asOf: string; + delayNote: string; + delaySeconds: number | null; + oiFreshness: OiFreshness; + providerId: OptionsProviderId; + convention: ExposureConvention; + unit: ExposureUnit; + computation: ExposureComputation; + /** Left → right = nearest → farthest expiry. */ + expiries: string[]; + cells: DealerMapCell[]; + byStrike: Array<{ strike: number; gex: number; vex: number; absGex: number }>; + netVexByStrike: Array<{ strike: number; vex: number }>; + levels: DealerMapLevels; + regimeAtSpot: 'positive' | 'negative' | 'mixed'; + scenario: ScenarioCode; + patterns: PatternHit[]; + velocity?: VelocitySummary; + opexHints: { isNearOpex: boolean; note: string }; + provenance: Provenance; + coveragePct: number; + /** Fraction of included contracts that used BSM fill for gamma. */ + bsmFillPct: number; + /** Greek / IV quality for free-path hygiene. */ + quality: DealerMapQuality; +} + +/** How trustworthy BSM/vendor greeks were for this build. */ +export type GreekQuality = 'ok' | 'degraded' | 'poor'; + +export interface DealerMapQuality { + greekQuality: GreekQuality; + /** Contracts with OI that entered cell aggregation. */ + oiEligible: number; + /** Of OI-eligible, how many contributed gamma. */ + included: number; + /** OI-eligible skipped for junk/missing IV and no gamma. */ + skippedBadIv: number; + /** Included contracts that used BSM (not vendor gamma). */ + bsmFills: number; + /** Included contracts that used last-good / ATM proxy IV. */ + fallbackIvUses: number; + /** 0–100. */ + bsmFillPct: number; + /** 0–100 of OI-eligible skipped for bad IV. */ + badIvSkipPct: number; + /** 0–100 of included that used fallback IV. */ + fallbackIvPct: number; + notes: string[]; +} + +export interface BuildDealerMapOpts { + /** Cap columns (nearest N by DTE). Default 6. */ + maxExpiries?: number; + riskFreeRate?: number; + dividendYield?: number; + /** |GEX| below this fraction of max is “near zero” for polarity / deserts. Default 0.05. */ + nearZeroFrac?: number; + /** Optional prior map for velocity (same symbol). */ + prior?: Pick | null; + now?: Date; + /** + * Prior sane IVs keyed `expiry|strike|call|put` (from recent chain history). + * Used when current row IV is junk and vendor gamma is missing. + */ + ivHints?: Map | Record; + /** + * Sign convention. Default classic (call +, put −). + * `dealer_inventory` is classic negated (Heatseeker-style dealer view). + */ + convention?: ExposureConvention; +} + +// ----- Internals ----- + +const MULTIPLIER = 100; +const PCT_MOVE = 0.01; + +/** Reject premarket junk and absurd IV before BSM. */ +export const IV_SANE_MIN = 0.05; // 5% +export const IV_SANE_MAX = 3.0; // 300% + +/** Prefer last healthy map over a worse recompute for this long. */ +export const HEALTHY_MAP_MAX_AGE_MS = 6 * 60 * 60_000; + +export function isSaneIv(iv: number | null | undefined): iv is number { + return iv != null && Number.isFinite(iv) && iv >= IV_SANE_MIN && iv <= IV_SANE_MAX; +} + +export function ivHintKey(expiry: string, strike: number, right: 'call' | 'put'): string { + return `${expiry}|${strike}|${right}`; +} + +/** + * Classic retail: call +, put −. + * Dealer inventory: opposite (call −, put +) ≡ full negation of classic GEX/VEX. + */ +export function signForRight( + right: 'call' | 'put', + convention: ExposureConvention = 'classic_call_pos_put_neg', +): number { + const classic = right === 'call' ? 1 : -1; + return convention === 'dealer_inventory' ? -classic : classic; +} + +function lookupIvHint( + hints: BuildDealerMapOpts['ivHints'] | undefined, + expiry: string, + strike: number, + right: 'call' | 'put', +): number | null { + if (!hints) return null; + const k = ivHintKey(expiry, strike, right); + const v = hints instanceof Map ? hints.get(k) : hints[k]; + return isSaneIv(v) ? v : null; +} + +/** + * Resolve gamma/vanna for one row. + * Prefer vendor gamma; else BSM with sane IV, then last-good hint, then expiry ATM proxy. + */ +export function resolveGreeks( + row: NormalizedOptionRow, + spot: number, + asOf: string, + r: number, + q: number, + opts?: { + fallbackIv?: number | null; + atmIv?: number | null; + }, +): { + gamma: number; + vanna: number; + usedBsm: boolean; + usedFallbackIv: boolean; + skippedBadIv: boolean; +} { + const g = row.greeks; + const hasGamma = g?.gamma != null && Number.isFinite(g.gamma) && (g.gamma as number) > 0; + const hasVanna = g?.vanna != null && Number.isFinite(g.vanna as number); + + if (hasGamma && hasVanna) { + return { + gamma: g!.gamma as number, + vanna: g!.vanna as number, + usedBsm: false, + usedFallbackIv: false, + skippedBadIv: false, + }; + } + + if (hasGamma && !hasVanna) { + // Vendor gamma OK; try BSM only for vanna if IV sane + let usedFallbackIv = false; + let sigma: number | null = isSaneIv(row.iv) ? row.iv : null; + if (sigma == null && isSaneIv(opts?.fallbackIv)) { + sigma = opts!.fallbackIv!; + usedFallbackIv = true; + } + if (sigma == null && isSaneIv(opts?.atmIv)) { + sigma = opts!.atmIv!; + usedFallbackIv = true; + } + if (sigma != null) { + const T = yearsToExpiry(asOf, row.expiry); + const bs = bsmGreeks({ + S: spot, + K: row.strike, + T, + sigma, + r, + q, + right: row.right, + }); + return { + gamma: g!.gamma as number, + vanna: bs.vanna, + usedBsm: true, + usedFallbackIv, + skippedBadIv: false, + }; + } + return { + gamma: g!.gamma as number, + vanna: 0, + usedBsm: false, + usedFallbackIv: false, + skippedBadIv: false, + }; + } + + // Need BSM for gamma + let usedFallbackIv = false; + let sigma: number | null = isSaneIv(row.iv) ? row.iv : null; + if (sigma == null && isSaneIv(opts?.fallbackIv)) { + sigma = opts!.fallbackIv!; + usedFallbackIv = true; + } + if (sigma == null && isSaneIv(opts?.atmIv)) { + sigma = opts!.atmIv!; + usedFallbackIv = true; + } + if (sigma == null) { + return { + gamma: 0, + vanna: 0, + usedBsm: false, + usedFallbackIv: false, + skippedBadIv: true, + }; + } + + const T = yearsToExpiry(asOf, row.expiry); + const bs = bsmGreeks({ + S: spot, + K: row.strike, + T, + sigma, + r, + q, + right: row.right, + }); + + return { + gamma: bs.gamma, + vanna: hasVanna ? (g!.vanna as number) : bs.vanna, + usedBsm: true, + usedFallbackIv, + skippedBadIv: false, + }; +} + +/** Score greek quality from counters. */ +export function scoreGreekQuality(q: { + oiEligible: number; + included: number; + skippedBadIv: number; + bsmFills: number; + fallbackIvUses: number; +}): Omit & { notes: string[] } { + const oiEligible = q.oiEligible; + const included = q.included; + const skippedBadIv = q.skippedBadIv; + const bsmFills = q.bsmFills; + const fallbackIvUses = q.fallbackIvUses; + const bsmFillPct = + included === 0 ? 0 : Math.round((bsmFills / included) * 1000) / 10; + const badIvSkipPct = + oiEligible === 0 ? 0 : Math.round((skippedBadIv / oiEligible) * 1000) / 10; + const fallbackIvPct = + included === 0 ? 0 : Math.round((fallbackIvUses / included) * 1000) / 10; + const includePct = oiEligible === 0 ? 0 : included / oiEligible; + + const notes: string[] = []; + let greekQuality: GreekQuality = 'ok'; + + if (oiEligible === 0 || included === 0) { + greekQuality = 'poor'; + notes.push('No contracts with usable greeks - map structure is empty or unreliable.'); + } else if (badIvSkipPct >= 40 || includePct < 0.35) { + greekQuality = 'poor'; + notes.push( + `Greeks degraded: ${badIvSkipPct.toFixed(0)}% of OI contracts skipped for bad/missing IV; only ${(includePct * 100).toFixed(0)}% contributed.`, + ); + } else if (badIvSkipPct >= 15 || fallbackIvPct >= 25 || bsmFillPct >= 90) { + greekQuality = 'degraded'; + if (badIvSkipPct >= 15) { + notes.push(`Some IV junk filtered (${badIvSkipPct.toFixed(0)}% of OI contracts skipped).`); + } + if (fallbackIvPct >= 25) { + notes.push(`Using prior/ATM IV for ${fallbackIvPct.toFixed(0)}% of filled contracts.`); + } + if (bsmFillPct >= 90) { + notes.push('Gamma mostly BSM-filled (vendor gamma unavailable).'); + } + } + + if (greekQuality === 'ok' && bsmFillPct >= 50) { + notes.push('Gamma partly BSM-filled from implied vol.'); + } + + return { + greekQuality, + oiEligible, + included, + skippedBadIv, + bsmFills, + fallbackIvUses, + bsmFillPct, + badIvSkipPct, + fallbackIvPct, + notes, + }; +} + +/** True when this map is good enough to prefer over a worse recompute. */ +export function isHealthyDealerMap(map: Pick): boolean { + if (!map.cells.length) return false; + if (!map.quality) return map.cells.length >= 4; + return map.quality.greekQuality === 'ok' || map.quality.greekQuality === 'degraded'; +} + +/** True when a recompute should not replace a healthier cached map. */ +export function isDegradedDealerMap(map: Pick): boolean { + if (!map.cells.length) return true; + if (!map.quality) return false; + return map.quality.greekQuality === 'poor'; +} + +/** Dollar GEX for one contract lot (OI contracts). */ +export function contractGex( + gamma: number, + oi: number, + spot: number, + right: 'call' | 'put', + convention: ExposureConvention = 'classic_call_pos_put_neg', +): number { + return gamma * oi * MULTIPLIER * spot * spot * PCT_MOVE * signForRight(right, convention); +} + +/** Dollar VEX approximation for one contract lot. */ +export function contractVex( + vanna: number, + oi: number, + spot: number, + right: 'call' | 'put', + convention: ExposureConvention = 'classic_call_pos_put_neg', +): number { + // Scale vanna similarly: notional delta change for 1% vol move × spot weight. + return vanna * 0.01 * oi * MULTIPLIER * spot * signForRight(right, convention); +} + +function polarityOf(gex: number, nearZero: number): Polarity { + if (Math.abs(gex) < nearZero) return 'near_zero'; + return gex > 0 ? 'pos' : 'neg'; +} + +function uniqueSortedExpiries(rows: NormalizedOptionRow[], asOf: string, max: number): string[] { + const set = new Set(rows.map((r) => r.expiry).filter(Boolean)); + const list = [...set].sort((a, b) => a.localeCompare(b)); + // Drop expired relative to asOf date (string compare works for ISO dates). + const asOfDay = asOf.slice(0, 10); + const live = list.filter((e) => e >= asOfDay); + return live.slice(0, max); +} + +function isFridayYmd(ymd: string): boolean { + const d = new Date(`${ymd}T12:00:00.000Z`); + return d.getUTCDay() === 5; +} + +function findAirPockets( + byStrike: Array<{ strike: number; absGex: number }>, + nearZero: number, +): Array<{ from: number; to: number }> { + if (byStrike.length < 2) return []; + const pockets: Array<{ from: number; to: number }> = []; + let runStart: number | null = null; + let prevStrike: number | null = null; + + for (const row of byStrike) { + if (row.absGex < nearZero) { + if (runStart == null) runStart = row.strike; + prevStrike = row.strike; + } else if (runStart != null && prevStrike != null && prevStrike > runStart) { + pockets.push({ from: runStart, to: prevStrike }); + runStart = null; + prevStrike = null; + } else { + runStart = null; + prevStrike = null; + } + } + if (runStart != null && prevStrike != null && prevStrike > runStart) { + pockets.push({ from: runStart, to: prevStrike }); + } + return pockets; +} + +function findClusters( + byStrike: Array<{ strike: number; absGex: number }>, + nearZero: number, +): Array<{ strikes: number[]; absGexSum: number }> { + // Adjacent strikes both above nearZero with small relative spacing. + if (byStrike.length === 0) return []; + const strong = byStrike.filter((s) => s.absGex >= nearZero); + if (strong.length === 0) return []; + + const clusters: Array<{ strikes: number[]; absGexSum: number }> = []; + let cur: number[] = [strong[0].strike]; + let sum = strong[0].absGex; + + for (let i = 1; i < strong.length; i++) { + const prev = strong[i - 1].strike; + const strike = strong[i].strike; + const gap = strike - prev; + const step = Math.max(Math.abs(prev) * 0.005, 0.5); // ~0.5% or $0.50 + if (gap <= step * 2.5) { + cur.push(strike); + sum += strong[i].absGex; + } else { + if (cur.length >= 2) clusters.push({ strikes: cur, absGexSum: sum }); + cur = [strike]; + sum = strong[i].absGex; + } + } + if (cur.length >= 2) clusters.push({ strikes: cur, absGexSum: sum }); + return clusters.sort((a, b) => b.absGexSum - a.absGexSum).slice(0, 5); +} + +function findFlip(byStrike: Array<{ strike: number; gex: number }>, spot: number): number | null { + if (byStrike.length < 2) return null; + let best: { strike: number; dist: number } | null = null; + for (let i = 0; i < byStrike.length - 1; i++) { + const a = byStrike[i]; + const b = byStrike[i + 1]; + if (a.gex === 0 || b.gex === 0) continue; + if ((a.gex > 0 && b.gex < 0) || (a.gex < 0 && b.gex > 0)) { + // Interpolate zero crossing. + const t = Math.abs(a.gex) / (Math.abs(a.gex) + Math.abs(b.gex)); + const strike = a.strike + t * (b.strike - a.strike); + const dist = Math.abs(strike - spot); + if (!best || dist < best.dist) best = { strike, dist }; + } + } + return best ? Math.round(best.strike * 100) / 100 : null; +} + +function classifyScenario( + netGexNear: number, + netVexNear: number, + maxAbsGex: number, + maxAbsVex: number, +): ScenarioCode { + if (maxAbsGex <= 0 && maxAbsVex <= 0) return 'insufficient_data'; + const gexStrong = Math.abs(netGexNear) >= 0.15 * maxAbsGex; + const vexStrong = Math.abs(netVexNear) >= 0.15 * maxAbsVex || maxAbsVex === 0; + + if (netGexNear > 0 && netVexNear > 0 && gexStrong) return 'pos_gex_pos_vex'; + if (netGexNear < 0 && netVexNear < 0 && gexStrong) return 'neg_gex_neg_vex'; + + if (Math.abs(netGexNear) >= Math.abs(netVexNear) * 2 && gexStrong) return 'gex_dominates'; + if (Math.abs(netVexNear) > Math.abs(netGexNear) * 1.5 && vexStrong && maxAbsVex > 0) { + return 'vex_dominates'; + } + return 'mixed'; +} + +function detectPatterns( + byStrike: Array<{ strike: number; gex: number; absGex: number }>, + spot: number, + nearZero: number, + maxAbs: number, + airPockets: Array<{ from: number; to: number }>, +): PatternHit[] { + const hits: PatternHit[] = []; + if (maxAbs <= 0 || byStrike.length < 3) return hits; + + const threshold = 0.25 * maxAbs; + const strong = byStrike.filter((s) => s.absGex >= threshold); + + // Whipsaw: two strong nodes with desert between + if (strong.length >= 2 && airPockets.length > 0) { + const low = strong[0]; + const high = strong[strong.length - 1]; + if (high.strike > low.strike) { + const desert = airPockets.find( + (p) => p.from >= low.strike && p.to <= high.strike, + ) ?? airPockets[0]; + const spanOk = high.strike - low.strike > spot * 0.01; + if (spanOk && desert) { + hits.push({ + id: 'whipsaw', + edges: [low.strike, high.strike], + desert, + confidence: Math.min(0.95, 0.45 + (low.absGex + high.absGex) / (2 * maxAbs) * 0.5), + }); + } + } + } + + // Rainbow road: no dominant king, mixed signs, wide span + const king = [...byStrike].sort((a, b) => b.absGex - a.absGex)[0]; + const second = [...byStrike].sort((a, b) => b.absGex - a.absGex)[1]; + const kingRatio = second && second.absGex > 0 ? king.absGex / second.absGex : Infinity; + const signs = new Set(byStrike.filter((s) => s.absGex >= nearZero).map((s) => Math.sign(s.gex))); + const span = byStrike[byStrike.length - 1].strike - byStrike[0].strike; + if (kingRatio < 1.35 && signs.has(1) && signs.has(-1) && span > spot * 0.04) { + hits.push({ + id: 'rainbow_road', + reason: 'Mixed-sign nodes across a wide range without a dominant king', + confidence: Math.min(0.9, 0.4 + (1.35 - kingRatio) * 0.4), + }); + } + + // Rug setup: +GEX above spot-ish, large −GEX below it, more −GEX stack further below + const posAbove = byStrike.filter((s) => s.gex > threshold && s.strike >= spot * 0.98); + const neg = byStrike.filter((s) => s.gex < -threshold); + if (posAbove.length > 0 && neg.length >= 2) { + const longAbove = posAbove.sort((a, b) => b.absGex - a.absGex)[0]; + const shortPivot = neg + .filter((s) => s.strike < longAbove.strike) + .sort((a, b) => b.absGex - a.absGex)[0]; + if (shortPivot) { + const shortStackBelow = neg + .filter((s) => s.strike <= shortPivot.strike) + .sort((a, b) => b.strike - a.strike) + .map((s) => s.strike); + if (shortStackBelow.length >= 2) { + hits.push({ + id: 'rug_setup', + longAbove: longAbove.strike, + shortPivot: shortPivot.strike, + shortStackBelow, + confidence: Math.min( + 0.92, + 0.4 + (longAbove.absGex + shortPivot.absGex) / (2 * maxAbs) * 0.5, + ), + }); + } + } + } + + return hits; +} + +function buildVelocity( + prior: NonNullable, + byStrike: Array<{ strike: number; absGex: number }>, + airPockets: Array<{ from: number; to: number }>, +): VelocitySummary { + const prevMap = new Map(prior.byStrike.map((s) => [s.strike, s.absGex])); + const nodes: VelocityNode[] = []; + for (const cur of byStrike) { + const prevAbs = prevMap.get(cur.strike) ?? 0; + const delta = cur.absGex - prevAbs; + if (Math.abs(delta) < 1e-9 && prevAbs === 0) continue; + nodes.push({ + strike: cur.strike, + prevAbsGex: prevAbs, + currAbsGex: cur.absGex, + deltaAbsGex: delta, + growing: delta > 0, + }); + } + nodes.sort((a, b) => Math.abs(b.deltaAbsGex) - Math.abs(a.deltaAbsGex)); + + const growingNearAirPocket = nodes.some((n) => { + if (!n.growing) return false; + return airPockets.some( + (p) => n.strike >= p.from - (p.to - p.from) * 0.25 && n.strike <= p.to + (p.to - p.from) * 0.25, + ); + }); + + return { asOfPrev: prior.asOf, nodes: nodes.slice(0, 20), growingNearAirPocket }; +} + +function delayNote(delaySeconds: number | null, oi: OiFreshness): string { + const parts: string[] = []; + if (delaySeconds == null) parts.push('Delay unknown'); + else if (delaySeconds <= 0) parts.push('Quotes real-time'); + else parts.push(`Quotes delayed ~${Math.round(delaySeconds / 60)}m`); + if (oi === 'prior_session') parts.push('Open interest typically from the prior session'); + else if (oi === 'intraday') parts.push('Open interest may update intraday'); + else parts.push('Open interest freshness unknown'); + return parts.join('. ') + '.'; +} + +/** + * Build a full DealerMap from a normalized surface. + * Pure: no I/O, no vendor imports. + */ +export function buildDealerMap( + surface: NormalizedOptionSurface, + opts: BuildDealerMapOpts = {}, +): DealerMap { + const maxExpiries = opts.maxExpiries ?? 6; + const r = opts.riskFreeRate ?? 0.04; + const q = opts.dividendYield ?? 0; + const nearZeroFrac = opts.nearZeroFrac ?? 0.05; + const convention: ExposureConvention = opts.convention ?? 'classic_call_pos_put_neg'; + + const expiries = uniqueSortedExpiries(surface.rows, surface.asOf, maxExpiries); + const expirySet = new Set(expiries); + + // Per-expiry ATM IV proxy from sane IVs near spot (for junk-IV fallback). + const atmIvByExpiry = new Map(); + { + const byExp = new Map>(); + for (const row of surface.rows) { + if (!expirySet.has(row.expiry) || !isSaneIv(row.iv)) continue; + const dist = Math.abs(row.strike - surface.spot); + const list = byExp.get(row.expiry) ?? []; + list.push({ dist, iv: row.iv }); + byExp.set(row.expiry, list); + } + for (const [exp, list] of byExp) { + list.sort((a, b) => a.dist - b.dist); + const top = list.slice(0, Math.min(6, list.length)); + const avg = top.reduce((s, x) => s + x.iv, 0) / top.length; + if (isSaneIv(avg)) atmIvByExpiry.set(exp, avg); + } + } + + let totalContracts = 0; + let oiEligible = 0; + let includedContracts = 0; + let bsmFills = 0; + let skippedBadIv = 0; + let fallbackIvUses = 0; + + // Aggregate per (strike, expiry) + const cellMap = new Map(); + + for (const row of surface.rows) { + totalContracts += 1; + if (!expirySet.has(row.expiry)) continue; + const oi = row.openInterest; + if (oi == null || !Number.isFinite(oi) || oi <= 0) continue; + oiEligible += 1; + + const hint = lookupIvHint(opts.ivHints, row.expiry, row.strike, row.right); + const { gamma, vanna, usedBsm, usedFallbackIv, skippedBadIv: bad } = resolveGreeks( + row, + surface.spot, + surface.asOf, + r, + q, + { + fallbackIv: hint, + atmIv: atmIvByExpiry.get(row.expiry) ?? null, + }, + ); + + if (bad && gamma <= 0) { + skippedBadIv += 1; + continue; + } + if (gamma <= 0 && vanna === 0) { + skippedBadIv += 1; + continue; + } + + includedContracts += 1; + if (usedBsm) bsmFills += 1; + if (usedFallbackIv) fallbackIvUses += 1; + + const gex = contractGex(gamma, oi, surface.spot, row.right, convention); + const vex = contractVex(vanna, oi, surface.spot, row.right, convention); + const key = `${row.expiry}|${row.strike}`; + const prev = cellMap.get(key) ?? { + strike: row.strike, + expiry: row.expiry, + gex: 0, + vex: 0, + oi: 0, + volume: 0, + }; + prev.gex += gex; + prev.vex += vex; + prev.oi += oi; + prev.volume += row.volume ?? 0; + cellMap.set(key, prev); + } + + const rawCells = [...cellMap.values()]; + let maxAbsCell = 0; + for (const c of rawCells) maxAbsCell = Math.max(maxAbsCell, Math.abs(c.gex)); + const nearZeroCell = maxAbsCell * nearZeroFrac; + + // King = max |gex| cell on surface + let kingCell: { strike: number; expiry: string; gex: number } | null = null; + for (const c of rawCells) { + if (!kingCell || Math.abs(c.gex) > Math.abs(kingCell.gex)) { + kingCell = { strike: c.strike, expiry: c.expiry, gex: c.gex }; + } + } + + const cells: DealerMapCell[] = rawCells + .map((c) => ({ + strike: c.strike, + expiry: c.expiry, + gex: c.gex, + vex: c.vex, + oi: c.oi, + volume: c.volume, + polarity: polarityOf(c.gex, nearZeroCell), + isKing: + kingCell != null && + c.strike === kingCell.strike && + c.expiry === kingCell.expiry, + })) + .sort((a, b) => a.strike - b.strike || a.expiry.localeCompare(b.expiry)); + + // Aggregate by strike (sum across selected expiries) + const strikeAgg = new Map(); + for (const c of rawCells) { + const prev = strikeAgg.get(c.strike) ?? { gex: 0, vex: 0 }; + prev.gex += c.gex; + prev.vex += c.vex; + strikeAgg.set(c.strike, prev); + } + const byStrike = [...strikeAgg.entries()] + .map(([strike, v]) => ({ + strike, + gex: v.gex, + vex: v.vex, + absGex: Math.abs(v.gex), + })) + .sort((a, b) => a.strike - b.strike); + + const netVexByStrike = byStrike.map((s) => ({ strike: s.strike, vex: s.vex })); + + let maxAbsStrike = 0; + let maxAbsVex = 0; + for (const s of byStrike) { + maxAbsStrike = Math.max(maxAbsStrike, s.absGex); + maxAbsVex = Math.max(maxAbsVex, Math.abs(s.vex)); + } + const nearZeroStrike = maxAbsStrike * nearZeroFrac; + + const gatekeepers = [...byStrike] + .filter((s) => !kingCell || s.strike !== kingCell.strike) + .sort((a, b) => b.absGex - a.absGex) + .slice(0, 4) + .map((s) => ({ strike: s.strike, absGex: s.absGex })); + + const above = byStrike.filter((s) => s.strike >= surface.spot && s.gex > 0); + const below = byStrike.filter((s) => s.strike <= surface.spot && s.gex < 0); + const callWall = above.length + ? above.sort((a, b) => b.gex - a.gex)[0].strike + : null; + const putWall = below.length + ? below.sort((a, b) => a.gex - b.gex)[0].strike + : null; + + const airPockets = findAirPockets(byStrike, nearZeroStrike); + const clusters = findClusters(byStrike, nearZeroStrike); + const flip = findFlip(byStrike, surface.spot); + + // Regime / scenario: use nearest expiry column if available, else aggregate + const nearest = expiries[0]; + let netGexNear = 0; + let netVexNear = 0; + if (nearest) { + for (const c of rawCells) { + if (c.expiry === nearest) { + netGexNear += c.gex; + netVexNear += c.vex; + } + } + } else { + for (const s of byStrike) { + netGexNear += s.gex; + netVexNear += s.vex; + } + } + + // Spot regime: sign of GEX at nearest strike to spot (aggregate) + let regimeAtSpot: DealerMap['regimeAtSpot'] = 'mixed'; + if (byStrike.length > 0) { + let nearestStrike = byStrike[0]; + for (const s of byStrike) { + if (Math.abs(s.strike - surface.spot) < Math.abs(nearestStrike.strike - surface.spot)) { + nearestStrike = s; + } + } + if (nearestStrike.absGex < nearZeroStrike) regimeAtSpot = 'mixed'; + else regimeAtSpot = nearestStrike.gex > 0 ? 'positive' : 'negative'; + } + + const scenario = classifyScenario(netGexNear, netVexNear, maxAbsStrike, maxAbsVex); + const patterns = detectPatterns(byStrike, surface.spot, nearZeroStrike, maxAbsStrike, airPockets); + + const levels: DealerMapLevels = { + king: kingCell, + gatekeepers, + flip, + callWall, + putWall, + airPockets, + clusters, + }; + + let velocity: VelocitySummary | undefined; + if (opts.prior && opts.prior.byStrike?.length) { + velocity = buildVelocity(opts.prior, byStrike, airPockets); + } + + // OPEX: Friday within selected expiries and within 5 calendar days of asOf + const asOfDay = surface.asOf.slice(0, 10); + const asOfMs = Date.parse(`${asOfDay}T12:00:00.000Z`); + const nearFriday = expiries.some((e) => { + if (!isFridayYmd(e)) return false; + const expMs = Date.parse(`${e}T12:00:00.000Z`); + const days = (expMs - asOfMs) / (24 * 3600 * 1000); + return days >= 0 && days <= 5; + }); + const opexHints = nearFriday + ? { + isNearOpex: true, + note: + 'Near-term expiry/OPEX window: nearest columns dominate. Long-gamma structure often pins harder; short-gamma structure can accelerate.', + } + : { isNearOpex: false, note: '' }; + + const coveragePct = + totalContracts === 0 ? 0 : Math.round((includedContracts / totalContracts) * 1000) / 10; + const quality = scoreGreekQuality({ + oiEligible, + included: includedContracts, + skippedBadIv, + bsmFills, + fallbackIvUses, + }); + const bsmFillPct = quality.bsmFillPct; + + return { + symbol: surface.symbol, + spot: surface.spot, + asOf: surface.asOf, + delayNote: delayNote(surface.delaySeconds, surface.oiFreshness), + delaySeconds: surface.delaySeconds, + oiFreshness: surface.oiFreshness, + providerId: surface.providerId, + convention, + unit: 'usd_delta_adj_gamma_notional', + computation: 'local_engine', + expiries, + cells, + byStrike, + netVexByStrike, + levels, + regimeAtSpot, + scenario, + patterns, + velocity, + opexHints, + provenance: surface.provenance, + coveragePct, + bsmFillPct, + quality, + }; +} + +/** + * Re-express a classic map under another convention without rebuilding from chains. + * `dealer_inventory` = full GEX/VEX sign flip (dealers short customer long options). + * Levels / regime / scenario are recomputed so walls and flip stay consistent. + */ +export function withExposureConvention( + map: DealerMap, + convention: ExposureConvention, + opts?: { nearZeroFrac?: number }, +): DealerMap { + if (map.convention === convention) return map; + // Only classic ↔ dealer_inventory are supported (each is the negation of the other). + const nearZeroFrac = opts?.nearZeroFrac ?? 0.05; + + const flipSign = (n: number) => -n; + const rawCells = map.cells.map((c) => ({ + strike: c.strike, + expiry: c.expiry, + gex: flipSign(c.gex), + vex: flipSign(c.vex), + oi: c.oi, + volume: c.volume, + })); + + let maxAbsCell = 0; + for (const c of rawCells) maxAbsCell = Math.max(maxAbsCell, Math.abs(c.gex)); + const nearZeroCell = maxAbsCell * nearZeroFrac; + + let kingCell: { strike: number; expiry: string; gex: number } | null = null; + for (const c of rawCells) { + if (!kingCell || Math.abs(c.gex) > Math.abs(kingCell.gex)) { + kingCell = { strike: c.strike, expiry: c.expiry, gex: c.gex }; + } + } + + const cells: DealerMapCell[] = rawCells + .map((c) => ({ + strike: c.strike, + expiry: c.expiry, + gex: c.gex, + vex: c.vex, + oi: c.oi, + volume: c.volume, + polarity: polarityOf(c.gex, nearZeroCell), + isKing: + kingCell != null && c.strike === kingCell.strike && c.expiry === kingCell.expiry, + })) + .sort((a, b) => a.strike - b.strike || a.expiry.localeCompare(b.expiry)); + + const strikeAgg = new Map(); + for (const c of rawCells) { + const prev = strikeAgg.get(c.strike) ?? { gex: 0, vex: 0 }; + prev.gex += c.gex; + prev.vex += c.vex; + strikeAgg.set(c.strike, prev); + } + const byStrike = [...strikeAgg.entries()] + .map(([strike, v]) => ({ + strike, + gex: v.gex, + vex: v.vex, + absGex: Math.abs(v.gex), + })) + .sort((a, b) => a.strike - b.strike); + + const netVexByStrike = byStrike.map((s) => ({ strike: s.strike, vex: s.vex })); + + let maxAbsStrike = 0; + let maxAbsVex = 0; + for (const s of byStrike) { + maxAbsStrike = Math.max(maxAbsStrike, s.absGex); + maxAbsVex = Math.max(maxAbsVex, Math.abs(s.vex)); + } + const nearZeroStrike = maxAbsStrike * nearZeroFrac; + + const gatekeepers = [...byStrike] + .filter((s) => !kingCell || s.strike !== kingCell.strike) + .sort((a, b) => b.absGex - a.absGex) + .slice(0, 4) + .map((s) => ({ strike: s.strike, absGex: s.absGex })); + + const above = byStrike.filter((s) => s.strike >= map.spot && s.gex > 0); + const below = byStrike.filter((s) => s.strike <= map.spot && s.gex < 0); + const callWall = above.length ? above.sort((a, b) => b.gex - a.gex)[0].strike : null; + const putWall = below.length ? below.sort((a, b) => a.gex - b.gex)[0].strike : null; + + const airPockets = findAirPockets(byStrike, nearZeroStrike); + const clusters = findClusters(byStrike, nearZeroStrike); + const flip = findFlip(byStrike, map.spot); + + const nearest = map.expiries[0]; + let netGexNear = 0; + let netVexNear = 0; + if (nearest) { + for (const c of rawCells) { + if (c.expiry === nearest) { + netGexNear += c.gex; + netVexNear += c.vex; + } + } + } else { + for (const s of byStrike) { + netGexNear += s.gex; + netVexNear += s.vex; + } + } + + let regimeAtSpot: DealerMap['regimeAtSpot'] = 'mixed'; + if (byStrike.length > 0) { + let nearestStrike = byStrike[0]; + for (const s of byStrike) { + if (Math.abs(s.strike - map.spot) < Math.abs(nearestStrike.strike - map.spot)) { + nearestStrike = s; + } + } + if (nearestStrike.absGex < nearZeroStrike) regimeAtSpot = 'mixed'; + else regimeAtSpot = nearestStrike.gex > 0 ? 'positive' : 'negative'; + } + + const scenario = classifyScenario(netGexNear, netVexNear, maxAbsStrike, maxAbsVex); + const patterns = detectPatterns(byStrike, map.spot, nearZeroStrike, maxAbsStrike, airPockets); + + let velocity = map.velocity; + if (velocity) { + // abs deltas unchanged; growing flags still valid + velocity = { ...velocity }; + } + + return { + ...map, + convention, + cells, + byStrike, + netVexByStrike, + levels: { + king: kingCell, + gatekeepers, + flip, + callWall, + putWall, + airPockets, + clusters, + }, + regimeAtSpot, + scenario, + patterns, + velocity, + }; +} + +// ----- Layer-0 educator (no LLM) ----- + +export interface EducatorCopy { + headline: string; + regime: string; + levels: string[]; + scenario: string; + patterns: string[]; + questions: string[]; + caveats: string[]; + footer: string; +} + +export function layer0Educator(map: DealerMap): EducatorCopy { + // Plain-English voice from in-app notes only (never Obsidian at runtime). + const n = DEALER_FLOW_EXPLAIN_NOTES; + // Lazy import avoided - keep pure cycle free: han script is optional add-on in service layer. + // (hanStyleLevels imported dynamically by dealerMapService for payload.hanScript) + + const regime = + map.regimeAtSpot === 'positive' + ? `Near today's price: ${n.plainMetaphors.cushion} (+GEX / long gamma in this convention).` + : map.regimeAtSpot === 'negative' + ? `Near today's price: ${n.plainMetaphors.spring} (−GEX / short gamma in this convention).` + : `Near today's price: ${n.plainMetaphors.mixed}`; + + const levels: string[] = []; + if (map.levels.king) { + const sign = + map.levels.king.gex > 0 ? 'long gamma (cushion)' : 'short gamma (accelerate)'; + levels.push( + `King ★ at ${map.levels.king.strike} (${map.levels.king.expiry}): ${n.levels.king} Reads as ${sign}.`, + ); + } + if (map.levels.flip != null) { + levels.push(`Flip near ${map.levels.flip}: ${n.levels.flip}`); + } + if (map.levels.callWall != null) { + levels.push(`Call wall near ${map.levels.callWall}: ${n.levels.callWall}`); + } + if (map.levels.putWall != null) { + levels.push(`Put wall near ${map.levels.putWall}: ${n.levels.putWall}`); + } + for (const p of map.levels.airPockets.slice(0, 2)) { + levels.push( + `Thin zone roughly ${p.from}–${p.to}: ${n.plainMetaphors.thinZone}`, + ); + } + + const patternText = map.patterns.map((p) => { + const conf = `How clearly the shape matches: about ${(p.confidence * 100).toFixed(0)}% (rough score, not a forecast).`; + const base = + n.patterns[p.id] ?? + 'A named shape on the map.'; + if (p.id === 'whipsaw' && 'edges' in p) { + return `${base} Magnets near ${p.edges[0]} and ${p.edges[1]}. ${conf}`; + } + if (p.id === 'rug_setup' && 'longAbove' in p) { + return `${base} Watch around ${p.shortPivot} and ${p.longAbove}. ${conf}`; + } + return `${base} ${conf}`; + }); + + return { + headline: `${map.symbol} DealerFlow - ${n.whatMapIs}`, + regime: `${n.whatMapIs} ${regime}`, + levels, + scenario: n.scenarios[map.scenario] ?? n.scenarios.mixed, + patterns: patternText, + questions: [ + 'What does the price chart show first - does this map agree or disagree?', + 'Is the nearest expiry column (left) telling the same story as farther dates?', + 'How close is price to the king, and has that level already been tested?', + 'What would change this picture (expiry, new open interest, a vol shock)?', + ], + caveats: [ + map.delayNote, + map.convention === 'dealer_inventory' + ? 'Sign convention: dealer inventory (Heatseeker-style) - opposite of classic call+/put−.' + : 'Sign convention: classic OI GEX (calls +, puts −).', + ...(map.quality?.notes ?? []), + ...n.caveats, + ], + footer: 'Educational observation only.', + }; +} diff --git a/app/server/src/analysis/dealerFlowExplainNotes.ts b/app/server/src/analysis/dealerFlowExplainNotes.ts new file mode 100644 index 0000000..f540904 --- /dev/null +++ b/app/server/src/analysis/dealerFlowExplainNotes.ts @@ -0,0 +1,137 @@ +/** + * In-app plain-language notes for Dealer Flow explains. + * Runtime reads THIS file only - never Obsidian. + * + * Regenerated by: `node --experimental-strip-types scripts/dealer-flow-distill.ts` + * (distill also writes a personal copy into your Obsidian vault for offline reading). + */ + +export interface DealerFlowExplainNotes { + /** Bump when regenerating so L1 cache invalidates. */ + version: string; + updatedAt: string; + /** One sentence: what the picture is. */ + whatMapIs: string; + plainMetaphors: { + cushion: string; + spring: string; + mixed: string; + magnet: string; + floor: string; + ceiling: string; + thinZone: string; + }; + levels: { + king: string; + flip: string; + callWall: string; + putWall: string; + airPocket: string; + }; + scenarios: Record; + patterns: Record; + caveats: string[]; + forbidden: string[]; + /** Optional snippets distilled from harvested sources (plain English only). */ + sourceSnippets: string[]; +} + +export const DEALER_FLOW_EXPLAIN_NOTES: DealerFlowExplainNotes = { + version: "2026-08-06.20", + updatedAt: "2026-08-06T23:42:33.647Z", + whatMapIs: + 'This picture shows modeled hedging pressure from options open interest across prices and dates - not a forecast of where the stock must go, and not a live list of any firm\'s inventory.', + plainMetaphors: { + cushion: + 'Cushion: near this area, modeled hedges tend to lean against moves, so big swings may get pushed back toward the middle.', + spring: + 'Spring: near this area, modeled hedges tend to lean with moves, so once price starts moving it can keep going.', + mixed: 'Mixed or thin: neither a clear cushion nor a clear spring near today\'s price.', + magnet: 'Magnet: a loud strike the map cares about more than quieter levels nearby.', + floor: 'Floor people watch: heavy modeled pressure below the market - not a guarantee price bounces.', + ceiling: 'Ceiling people watch: heavy modeled pressure above the market - not a guarantee price stops.', + thinZone: 'Thin zone: little modeled pressure - price may travel through with less mechanical friction in the model.', + }, + levels: { + king: 'King is the loudest strike on this map (largest absolute modeled pressure).', + flip: 'Flip is where the map\'s personality can switch between cushion and spring territory.', + callWall: 'Call wall is a heavy modeled pile above spot - a ceiling people watch.', + putWall: 'Put wall is a heavy modeled pile below spot - a floor people watch.', + airPocket: 'Air pocket is a thin band of pressure between stronger levels.', + }, + scenarios: { + pos_gex_pos_vex: + 'Reads relatively calm: both price-linked and vol-linked pieces tend to absorb shocks in the model.', + neg_gex_neg_vex: + 'Reads jumpy: price and vol can reinforce each other in the model - swings may feel larger.', + gex_dominates: 'Price-level magnets matter more than volatility for this snapshot - start with king and walls.', + vex_dominates: 'Volatility shifts may matter more than pin levels for this snapshot.', + mixed: 'Signals disagree - treat the map as context, not one clean story.', + insufficient_data: 'Not enough open-interest data yet to label a clear scenario.', + }, + patterns: { + whipsaw: + 'Whipsaw shape: two strong magnets with a thinner middle - price can bounce between the poles.', + rainbow_road: + 'Rainbow-road shape: pressure spread across a wide range without one dominant magnet - more wandering.', + rug_setup: + 'Rug-like stack: support-looking structure that can fail and speed moves lower in the model if the soft floor breaks.', + }, + caveats: [ + 'Quotes may be delayed; open interest is often from the prior session.', + 'Numbers are modeled dollar hedging pressure, not reported dealer inventory.', + 'This explains a snapshot, not a price prediction.', + ], + forbidden: [ + 'Do not say buy, sell, short, or long as instructions.', + 'Do not give position size, entries, stops, or targets as recommendations.', + 'Do not claim the map knows where price must go.', + ], + sourceSnippets: [ + "DealerFlow for Beginners: How Nodes, Gamma, Vanna, and Dealer Positioning Actually Work: A thread: https://t.co/1rqWy0cIuU", + "🟡 Yellow = calm zone for dealers 🟣 Purple = No control zone for dealers Nodes are: • Concentrations of the hedging demand and they’re acting as attractors or accelerants.", + "These work as rebounds or rejections too, but with higher volatility on the opposite direction.", + "This behavior dampens volatility and promotes mean reversion.", + "This behavior amplifies volatility and promotes momentum.", + "The larger the absolute value of a node, the greater the gravitational influence it has on the price through dealer hedging activity.", + "The ★ marks the node with the largest absolute exposure across the surface.", + "5) In DealerFlow we also have Vanna available: the NetVex column -> Vanna is the sensitivity of delta to implied volatility.", + "When IV falls and vanna is positive at a strike, dealer hedges add to upward pressure.", + "Meaning yellow nodes bring buying pressure from dealers.", + "VEX explains whether volatility will fuel or dampen it.", + "Note what’s above and below: 🟣 If nearest is big purple below = fast downside magnet.", + "🟡 If nearest big yellow above = pin & resistance + reverse if swapped.", + "Also, check cluster nodes: Multiple stacked nodes at adjacent strikes multiply the magnet pull and have stronger gravity.", + "A possible mean reversion toward the center of gravity could happen as well.", + "Proximity Bias : Price is naturally biased toward the larger nearby node.", + "Although if there’s only one huge king gamma that’s far from price action, but no intermediate nodes and nothing in between price and gamma king, then move is clear, even if far from current price.", + "As price is attracted, it will fall lower to reach the Yellow king node.", + "Key characteristics : Two or more high absolute value nodes define the edges.", + "•Very few significant nodes in the middle which make price oscillate violently between edges." +], +}; + +/** Compact block for LLM system prompts. */ +export function explainNotesForPrompt(notes: DealerFlowExplainNotes = DEALER_FLOW_EXPLAIN_NOTES): string { + const lines = [ + `version: ${notes.version}`, + `whatMapIs: ${notes.whatMapIs}`, + `metaphors: ${Object.values(notes.plainMetaphors).join(' | ')}`, + `levels: ${Object.values(notes.levels).join(' | ')}`, + `scenarios: ${Object.entries(notes.scenarios) + .map(([k, v]) => `${k}=${v}`) + .join(' | ')}`, + `patterns: ${Object.entries(notes.patterns) + .map(([k, v]) => `${k}=${v}`) + .join(' | ')}`, + `caveats: ${notes.caveats.join(' | ')}`, + `forbidden: ${notes.forbidden.join(' | ')}`, + ]; + if (notes.sourceSnippets.length > 0) { + lines.push( + 'extra plain-English snippets from distilled sources:', + ...notes.sourceSnippets.slice(0, 12).map((s) => `- ${s}`), + ); + } + return lines.join('\n'); +} diff --git a/app/server/src/analysis/dealerMapExplain.ts b/app/server/src/analysis/dealerMapExplain.ts new file mode 100644 index 0000000..e3a3b01 --- /dev/null +++ b/app/server/src/analysis/dealerMapExplain.ts @@ -0,0 +1,241 @@ +// L1 explain for Dealer Flow via per-user OpenAI-compatible endpoint. +// Educational only (ADR-0005 / ADR-0007). Compact payload only. + +import { createHash } from 'node:crypto'; +import type { DatabaseSync } from 'node:sqlite'; +import type { DealerMap } from './dealerExposureEngine.ts'; +import { layer0Educator } from './dealerExposureEngine.ts'; +import { DEALER_FLOW_EXPLAIN_NOTES, explainNotesForPrompt } from './dealerFlowExplainNotes.ts'; +import { chatCompletions } from '../llm/openaiCompatible.ts'; +import { loadUserLlmSecret } from '../llm/userLlmEndpoint.ts'; + +const EXPLAIN_FEATURE = 'dealer_map_explain'; +const MAX_EXPLAINS_PER_HOUR = 10; + +export interface ExplainResult { + ok: boolean; + source: 'llm' | 'layer0_fallback' | 'rate_limited' | 'no_endpoint' | 'error'; + headline: string; + sections: { + whatMapShows: string; + keyLevels: string[]; + scenarioInPlainEnglish: string; + questionsToAsk: string[]; + caveats: string[]; + dataFreshness: string; + }; + model?: string; + cached?: boolean; + error?: string; +} + +function compactMap(map: DealerMap) { + return { + symbol: map.symbol, + spot: map.spot, + asOf: map.asOf, + delayNote: map.delayNote, + delaySeconds: map.delaySeconds, + oiFreshness: map.oiFreshness, + providerId: map.providerId, + regimeAtSpot: map.regimeAtSpot, + scenario: map.scenario, + levels: map.levels, + patterns: map.patterns, + topByStrike: [...map.byStrike].sort((a, b) => b.absGex - a.absGex).slice(0, 8), + nearestExpiry: map.expiries[0] ?? null, + velocity: map.velocity + ? { + asOfPrev: map.velocity.asOfPrev, + growingNearAirPocket: map.velocity.growingNearAirPocket, + topDeltas: map.velocity.nodes.slice(0, 5), + } + : null, + opex: map.opexHints, + coveragePct: map.coveragePct, + }; +} + +function systemPreamble(): string { + return [ + 'You explain DealerFlow map snapshots in plain English for someone who does not know options, GEX, or VEX.', + 'Goal: make THIS snapshot easy to understand - not teach a course.', + 'PRIMARY RULE: education only, never investment advice.', + 'Forbidden: buy, sell, short, long as trade directives; "you should"; position sizing; price targets as recommendations.', + 'Allowed: plain description of what the modeled map shows, everyday metaphors, what the numbers do not mean, short process questions if helpful.', + 'Always mention delay / prior-session open interest when the payload says so.', + 'Trust the map JSON for all numbers and levels. Never invent strikes or values.', + 'Use the in-app plain-language notes below for wording only.', + 'Respond with JSON only matching the schema in the user message.', + '', + 'In-app plain-language notes:', + explainNotesForPrompt(DEALER_FLOW_EXPLAIN_NOTES), + ].join('\n'); +} + +function promptHash(baseUrl: string, model: string, compact: unknown): string { + const h = createHash('sha256'); + h.update(EXPLAIN_FEATURE); + h.update('|'); + h.update(baseUrl); + h.update('|'); + h.update(model); + h.update('|'); + h.update(JSON.stringify(compact)); + return h.digest('hex'); +} + +function readCache(db: DatabaseSync, hash: string): ExplainResult | null { + try { + const row = db.prepare( + 'SELECT output FROM llm_summaries WHERE hash=? AND feature=?', + ).get(hash, EXPLAIN_FEATURE) as { output: string } | undefined; + if (!row) return null; + return JSON.parse(row.output) as ExplainResult; + } catch { + return null; + } +} + +function writeCache(db: DatabaseSync, hash: string, result: ExplainResult): void { + try { + db.prepare( + `INSERT OR REPLACE INTO llm_summaries (hash, feature, complexity, output, cited_sources, created_at) + VALUES (?,?,?,?,?,?)`, + ).run(hash, EXPLAIN_FEATURE, 'standard', JSON.stringify(result), null, new Date().toISOString()); + } catch { /* ignore */ } +} + +function rateLimitOk(db: DatabaseSync, userId: string): boolean { + try { + db.exec(` + CREATE TABLE IF NOT EXISTS user_llm_rate ( + user_id TEXT NOT NULL, + feature TEXT NOT NULL, + ts TEXT NOT NULL + ); + `); + const since = new Date(Date.now() - 60 * 60_000).toISOString(); + const row = db.prepare( + 'SELECT COUNT(*) AS n FROM user_llm_rate WHERE user_id=? AND feature=? AND ts>=?', + ).get(userId, EXPLAIN_FEATURE, since) as { n: number }; + if ((row?.n ?? 0) >= MAX_EXPLAINS_PER_HOUR) return false; + db.prepare( + 'INSERT INTO user_llm_rate (user_id, feature, ts) VALUES (?,?,?)', + ).run(userId, EXPLAIN_FEATURE, new Date().toISOString()); + return true; + } catch { + return true; + } +} + +function fromLayer0(map: DealerMap, source: ExplainResult['source'], error?: string): ExplainResult { + const e = layer0Educator(map); + return { + ok: source === 'layer0_fallback' || source === 'llm', + source, + headline: e.headline, + sections: { + whatMapShows: e.regime, + keyLevels: e.levels, + scenarioInPlainEnglish: e.scenario, + questionsToAsk: e.questions, + caveats: e.caveats, + dataFreshness: map.delayNote, + }, + error, + }; +} + +function parseLlmJson(text: string): Partial & { headline?: string } | null { + const start = text.indexOf('{'); + const end = text.lastIndexOf('}'); + if (start < 0 || end <= start) return null; + try { + return JSON.parse(text.slice(start, end + 1)) as Partial & { headline?: string }; + } catch { + return null; + } +} + +export async function explainDealerMap( + db: DatabaseSync, + userId: string, + map: DealerMap, +): Promise { + const secret = loadUserLlmSecret(db, userId); + if (!secret) { + return fromLayer0(map, 'no_endpoint', 'Add an OpenAI-compatible base URL and API key in Settings for guided interpretation.'); + } + + const compact = compactMap(map); + // Notes version invalidates cache when in-app wording is regenerated. + const hash = promptHash(secret.baseUrl, secret.model, { + compact, + notesVersion: DEALER_FLOW_EXPLAIN_NOTES.version, + }); + const cached = readCache(db, hash); + if (cached) return { ...cached, cached: true }; + + if (!rateLimitOk(db, userId)) { + return fromLayer0(map, 'rate_limited', 'Explain rate limit reached (10/hour). Showing built-in education copy.'); + } + + const userMsg = [ + 'Explain this DealerFlow map for someone who does not know options jargon.', + 'Cover in plain English: what the picture is, data freshness, near-price feel, key levels on THIS symbol, scenario, any shapes, what it is not.', + 'Keep it concrete and short. Return JSON only:', + JSON.stringify({ + headline: 'string', + whatMapShows: 'string', + keyLevels: ['string'], + scenarioInPlainEnglish: 'string', + questionsToAsk: ['string'], + caveats: ['string'], + dataFreshness: 'string', + }), + 'Map payload:', + JSON.stringify(compact), + ].join('\n'); + + try { + const r = await chatCompletions({ + baseUrl: secret.baseUrl, + apiKey: secret.apiKey, + model: secret.model, + messages: [ + { role: 'system', content: systemPreamble() }, + { role: 'user', content: userMsg }, + ], + temperature: 0.25, + // Reasoning models (Ornith) burn tokens on reasoning_content before JSON. + maxTokens: 4096, + timeoutMs: 180_000, + }); + + const parsed = parseLlmJson(r.content); + if (!parsed) { + return fromLayer0(map, 'error', 'LLM returned unparseable content; showing built-in education copy.'); + } + + const result: ExplainResult = { + ok: true, + source: 'llm', + headline: parsed.headline ?? layer0Educator(map).headline, + sections: { + whatMapShows: parsed.whatMapShows ?? layer0Educator(map).regime, + keyLevels: parsed.keyLevels ?? layer0Educator(map).levels, + scenarioInPlainEnglish: parsed.scenarioInPlainEnglish ?? layer0Educator(map).scenario, + questionsToAsk: parsed.questionsToAsk ?? layer0Educator(map).questions, + caveats: parsed.caveats ?? layer0Educator(map).caveats, + dataFreshness: parsed.dataFreshness ?? map.delayNote, + }, + model: r.model, + cached: false, + }; + writeCache(db, hash, result); + return result; + } catch (e) { + return fromLayer0(map, 'error', e instanceof Error ? e.message : String(e)); + } +} diff --git a/app/server/src/analysis/dealerMapIntegrity.ts b/app/server/src/analysis/dealerMapIntegrity.ts new file mode 100644 index 0000000..280420f --- /dev/null +++ b/app/server/src/analysis/dealerMapIntegrity.ts @@ -0,0 +1,412 @@ +/** + * GEX/VEX map integrity: delayed data is OK; missing/unusable pieces are not. + * Pure audits - no I/O. + */ + +import type { NormalizedOptionSurface } from '../options/types.ts'; +import type { DealerMap, DealerMapQuality } from './dealerExposureEngine.ts'; +import { isSaneIv } from './dealerExposureEngine.ts'; + +export type IntegrityStatus = 'complete' | 'degraded' | 'incomplete'; + +export interface IntegrityCheck { + id: string; + level: 'hard' | 'soft'; + ok: boolean; + detail: string; +} + +export interface DealerIntegrityMetrics { + expiries: number; + oiEligible: number; + included: number; + cells: number; + byStrike: number; + nearSpotOiStrikes: number; + minOiStrikesPerExpiry: number; + includePct: number; + badIvSkipPct: number; + bsmFillPct: number; +} + +export interface DealerIntegrityReport { + status: IntegrityStatus; + checks: IntegrityCheck[]; + hardFailed: string[]; + softFailed: string[]; + metrics: DealerIntegrityMetrics; +} + +export interface DealerIntegrityThresholds { + minExpiries: number; + minOiStrikesPerExpiry: number; + nearSpotPct: number; + minNearSpotOiStrikes: number; + minIncludePctOfOiEligible: number; + minCells: number; + minByStrike: number; + /** Soft: BSM fill above this → degraded */ + softBsmFillPct: number; + /** Soft: bad IV skip above this → degraded (hard uses map quality / include) */ + softBadIvSkipPct: number; +} + +export const DEALER_INTEGRITY_DEFAULTS: DealerIntegrityThresholds = { + minExpiries: 3, + minOiStrikesPerExpiry: 10, + nearSpotPct: 0.15, + minNearSpotOiStrikes: 5, + minIncludePctOfOiEligible: 0.5, + minCells: 8, + minByStrike: 5, + softBsmFillPct: 90, + softBadIvSkipPct: 15, +}; + +function check( + id: string, + level: 'hard' | 'soft', + ok: boolean, + detail: string, +): IntegrityCheck { + return { id, level, ok, detail }; +} + +/** + * Audit the option surface before/without a full map build. + * `included` / quality metrics optional when map already built. + */ +export function auditDealerSurface( + surface: NormalizedOptionSurface | null | undefined, + opts: { + thresholds?: Partial; + /** From map.quality when available */ + quality?: DealerMapQuality | null; + selectedExpiries?: string[]; + } = {}, +): DealerIntegrityReport { + const t = { ...DEALER_INTEGRITY_DEFAULTS, ...opts.thresholds }; + const checks: IntegrityCheck[] = []; + + if (!surface) { + return finalize( + [ + check('surface', 'hard', false, 'No option surface assembled (missing quote or chains).'), + ], + emptyMetrics(), + ); + } + + const spotOk = Number.isFinite(surface.spot) && surface.spot > 0; + checks.push( + check( + 'spot', + 'hard', + spotOk, + spotOk ? `Spot ${surface.spot}` : 'Spot missing or non-positive', + ), + ); + + const asOfMs = Date.parse(surface.asOf); + const asOfOk = Number.isFinite(asOfMs); + checks.push( + check( + 'asOf', + 'hard', + asOfOk, + asOfOk ? `asOf ${surface.asOf}` : 'asOf not parseable', + ), + ); + + const asOfDay = (surface.asOf || '').slice(0, 10); + // Expiries that have at least one OI>0 contract (ignore empty / zero-OI columns). + const oiAll = surface.rows.filter( + (r) => + r.openInterest != null && + Number.isFinite(r.openInterest) && + (r.openInterest as number) > 0, + ); + const allOiExpiries = [...new Set(oiAll.map((r) => r.expiry).filter(Boolean))].sort(); + const preferred = ( + opts.selectedExpiries ?? + allOiExpiries.filter((e) => !asOfDay || e >= asOfDay) + ).filter((e) => allOiExpiries.includes(e)); + const liveExpiries = (preferred.length > 0 ? preferred : allOiExpiries).slice(0, 6); + + // Fail closed: need minExpiries with OI>0 chain data. + const expOk = liveExpiries.length >= t.minExpiries; + checks.push( + check( + 'expiries', + 'hard', + expOk, + expOk + ? `${liveExpiries.length} live expiries with OI` + : `Need ≥${t.minExpiries} live expiries with OI, have ${liveExpiries.length}`, + ), + ); + + const expSet = new Set(liveExpiries); + const oiRows = oiAll.filter((r) => expSet.has(r.expiry)); + const oiEligible = oiRows.length; + + // Per-expiry OI strike count (unique strikes with any OI) — only expiries that have rows + let minOiStrikes = Infinity; + for (const exp of liveExpiries) { + const strikes = new Set( + oiRows.filter((r) => r.expiry === exp).map((r) => r.strike), + ); + minOiStrikes = Math.min(minOiStrikes, strikes.size); + const ok = strikes.size >= t.minOiStrikesPerExpiry; + checks.push( + check( + `expiry_oi_${exp}`, + 'hard', + ok, + ok + ? `${exp}: ${strikes.size} OI strikes` + : `${exp}: only ${strikes.size} OI strikes (need ≥${t.minOiStrikesPerExpiry})`, + ), + ); + } + if (liveExpiries.length === 0) minOiStrikes = 0; + + // Near-spot OI coverage + let nearSpotOiStrikes = 0; + if (spotOk) { + const lo = surface.spot * (1 - t.nearSpotPct); + const hi = surface.spot * (1 + t.nearSpotPct); + nearSpotOiStrikes = new Set( + oiRows.filter((r) => r.strike >= lo && r.strike <= hi).map((r) => r.strike), + ).size; + } + const nearOk = !spotOk || nearSpotOiStrikes >= t.minNearSpotOiStrikes; + checks.push( + check( + 'near_spot_oi', + 'hard', + nearOk, + nearOk + ? `${nearSpotOiStrikes} OI strikes within ±${(t.nearSpotPct * 100).toFixed(0)}% of spot` + : `Only ${nearSpotOiStrikes} OI strikes near spot (need ≥${t.minNearSpotOiStrikes})`, + ), + ); + + // Usable greeks: prefer map.quality; else estimate from surface (sane IV or gamma) + let included = opts.quality?.included ?? 0; + let badIvSkipPct = opts.quality?.badIvSkipPct ?? 0; + let bsmFillPct = opts.quality?.bsmFillPct ?? 0; + if (!opts.quality) { + let usable = 0; + for (const r of oiRows) { + const hasG = + r.greeks?.gamma != null && + Number.isFinite(r.greeks.gamma) && + (r.greeks.gamma as number) > 0; + if (hasG || isSaneIv(r.iv)) usable += 1; + } + included = usable; + badIvSkipPct = + oiEligible === 0 ? 100 : Math.round(((oiEligible - usable) / oiEligible) * 1000) / 10; + } + const includePct = oiEligible === 0 ? 0 : included / oiEligible; + const includeOk = oiEligible === 0 ? false : includePct >= t.minIncludePctOfOiEligible; + checks.push( + check( + 'usable_greeks', + 'hard', + includeOk, + includeOk + ? `${(includePct * 100).toFixed(0)}% of OI contracts have usable greeks/IV` + : `Only ${(includePct * 100).toFixed(0)}% of OI contracts usable (need ≥${(t.minIncludePctOfOiEligible * 100).toFixed(0)}%)`, + ), + ); + + // Soft: BSM-heavy / moderate bad IV + if (opts.quality) { + const softBsm = opts.quality.bsmFillPct < t.softBsmFillPct; + checks.push( + check( + 'bsm_fill', + 'soft', + softBsm || opts.quality.bsmFillPct === 0, + `BSM fill ${opts.quality.bsmFillPct.toFixed(0)}%`, + ), + ); + const softSkip = opts.quality.badIvSkipPct < t.softBadIvSkipPct; + checks.push( + check( + 'bad_iv_skip', + 'soft', + softSkip, + `Bad-IV skip ${opts.quality.badIvSkipPct.toFixed(0)}%`, + ), + ); + if (opts.quality.greekQuality === 'poor') { + checks.push( + check('greek_quality', 'hard', false, 'Map greek quality is poor'), + ); + } else if (opts.quality.greekQuality === 'degraded') { + checks.push( + check('greek_quality', 'soft', false, 'Map greek quality is degraded'), + ); + } + } + + // Delay is soft informational only (always ok for hard) + if (surface.delaySeconds != null && surface.delaySeconds > 0) { + checks.push( + check( + 'delay', + 'soft', + true, + `Quotes delayed ~${Math.round(surface.delaySeconds / 60)}m (labeled, not a hard fail)`, + ), + ); + } + if (surface.oiFreshness === 'prior_session') { + checks.push( + check( + 'oi_freshness', + 'soft', + true, + 'OI typically prior session (labeled, not a hard fail)', + ), + ); + } + + const metrics: DealerIntegrityMetrics = { + expiries: liveExpiries.length, + oiEligible, + included, + cells: 0, + byStrike: 0, + nearSpotOiStrikes, + minOiStrikesPerExpiry: Number.isFinite(minOiStrikes) ? minOiStrikes : 0, + includePct: Math.round(includePct * 1000) / 10, + badIvSkipPct, + bsmFillPct: opts.quality?.bsmFillPct ?? bsmFillPct, + }; + + return finalize(checks, metrics); +} + +/** Audit a built map (+ optional surface for feed checks). */ +export function auditDealerMap( + map: DealerMap | null | undefined, + surface?: NormalizedOptionSurface | null, + opts: { thresholds?: Partial } = {}, +): DealerIntegrityReport { + const t = { ...DEALER_INTEGRITY_DEFAULTS, ...opts.thresholds }; + const base = auditDealerSurface(surface ?? null, { + thresholds: opts.thresholds, + quality: map?.quality ?? null, + selectedExpiries: map?.expiries, + }); + + const checks = [...base.checks]; + const metrics = { ...base.metrics }; + + if (!map) { + checks.push(check('map', 'hard', false, 'No dealer map built')); + return finalize(checks, metrics); + } + + metrics.cells = map.cells.length; + metrics.byStrike = map.byStrike.length; + metrics.included = map.quality?.included ?? metrics.included; + metrics.oiEligible = map.quality?.oiEligible ?? metrics.oiEligible; + metrics.bsmFillPct = map.quality?.bsmFillPct ?? metrics.bsmFillPct; + metrics.badIvSkipPct = map.quality?.badIvSkipPct ?? metrics.badIvSkipPct; + if (map.quality && map.quality.oiEligible > 0) { + metrics.includePct = + Math.round((map.quality.included / map.quality.oiEligible) * 1000) / 10; + } + + const cellsOk = map.cells.length >= t.minCells; + checks.push( + check( + 'cells', + 'hard', + cellsOk, + cellsOk + ? `${map.cells.length} map cells` + : `Only ${map.cells.length} cells (need ≥${t.minCells})`, + ), + ); + + const byOk = map.byStrike.length >= t.minByStrike; + checks.push( + check( + 'by_strike', + 'hard', + byOk, + byOk + ? `${map.byStrike.length} strikes aggregated` + : `Only ${map.byStrike.length} strikes (need ≥${t.minByStrike})`, + ), + ); + + if (map.quality?.greekQuality === 'ok' || map.quality?.greekQuality === 'degraded') { + const kingOk = map.levels.king != null; + checks.push( + check( + 'king', + 'soft', + kingOk, + kingOk + ? `King ${map.levels.king!.strike} ${map.levels.king!.expiry}` + : 'No king node (soft)', + ), + ); + } + + // Spot on map + const spotOk = Number.isFinite(map.spot) && map.spot > 0; + if (!checks.some((c) => c.id === 'spot')) { + checks.push( + check('spot', 'hard', spotOk, spotOk ? `Spot ${map.spot}` : 'Map spot invalid'), + ); + } + + return finalize(checks, metrics); +} + +function emptyMetrics(): DealerIntegrityMetrics { + return { + expiries: 0, + oiEligible: 0, + included: 0, + cells: 0, + byStrike: 0, + nearSpotOiStrikes: 0, + minOiStrikesPerExpiry: 0, + includePct: 0, + badIvSkipPct: 0, + bsmFillPct: 0, + }; +} + +function finalize( + checks: IntegrityCheck[], + metrics: DealerIntegrityMetrics, +): DealerIntegrityReport { + const hardFailed = checks.filter((c) => c.level === 'hard' && !c.ok).map((c) => c.id); + const softFailed = checks.filter((c) => c.level === 'soft' && !c.ok).map((c) => c.id); + let status: IntegrityStatus = 'complete'; + if (hardFailed.length > 0) status = 'incomplete'; + else if (softFailed.length > 0) status = 'degraded'; + return { status, checks, hardFailed, softFailed, metrics }; +} + +export function integritySummary(report: DealerIntegrityReport): string { + if (report.status === 'complete') return 'Map data complete (delay labeled separately).'; + if (report.status === 'degraded') { + return `Map degraded: ${report.softFailed.join(', ') || 'soft checks'}.`; + } + const details = report.checks + .filter((c) => c.level === 'hard' && !c.ok) + .map((c) => c.detail) + .slice(0, 4); + return `Map incomplete: ${details.join(' · ') || report.hardFailed.join(', ')}`; +} diff --git a/app/server/src/analysis/dealerMapReplay.ts b/app/server/src/analysis/dealerMapReplay.ts new file mode 100644 index 0000000..66460a9 --- /dev/null +++ b/app/server/src/analysis/dealerMapReplay.ts @@ -0,0 +1,320 @@ +/** + * Historical GEX/VEX replay for integrity backtests. + * Assembles surface as-of chain timestamps (ts ≤ asOf), rebuilds map, audits, diffs snapshots. + */ + +import type { DatabaseSync } from 'node:sqlite'; +import type { + NormalizedOptionRow, + NormalizedOptionSurface, + OptionsProviderId, +} from '../options/types.ts'; +import { providerMeta, resolveOptionsProviderId } from '../options/OptionsChainRouter.ts'; +import { buildDealerMap, type DealerMap } from './dealerExposureEngine.ts'; +import { + auditDealerMap, + integritySummary, + type DealerIntegrityReport, +} from './dealerMapIntegrity.ts'; +import { loadSaneIvHints } from './dealerMapService.ts'; + +export type SpotSource = 'quote_table' | 'snapshot' | 'unknown'; + +export interface ReplayResult { + symbol: string; + asOf: string; + spot: number | null; + spotSource: SpotSource; + map: DealerMap | null; + integrity: DealerIntegrityReport; + surfaceRowCount: number; + snapshotCompare: SnapshotCompare | null; + ok: boolean; + summary: string; +} + +export interface SnapshotCompare { + hasSnapshot: boolean; + asOfDate: string | null; + kingMatch: boolean | null; + cellCountDelta: number | null; + topGexOverlap: number | null; + detail: string; +} + +function num(v: unknown): number | null { + return typeof v === 'number' && Number.isFinite(v) ? v : null; +} +function intOrNull(v: unknown): number | null { + if (typeof v === 'number' && Number.isFinite(v)) return Math.trunc(v); + return null; +} + +/** + * Build surface from chains with ts ≤ asOf (per expiry MAX ts), for backtest. + */ +export function assembleSurfaceAsOf( + db: DatabaseSync, + symbolRaw: string, + asOfIso: string, + opts: { maxExpiries?: number; providerId?: OptionsProviderId } = {}, +): { surface: NormalizedOptionSurface | null; spotSource: SpotSource } { + const symbol = symbolRaw.toUpperCase(); + const maxExpiries = opts.maxExpiries ?? 6; + const providerId = opts.providerId ?? resolveOptionsProviderId(); + const meta = providerMeta(providerId === 'synthetic' ? 'yfinance' : providerId); + + let spot: number | null = null; + let spotSource: SpotSource = 'unknown'; + const quote = db.prepare('SELECT price, observed_at FROM quotes WHERE symbol=?').get(symbol) as + | { price: number; observed_at: string } + | undefined; + if (quote && Number.isFinite(quote.price) && quote.price > 0) { + spot = quote.price; + spotSource = 'quote_table'; + } + + // Prefer same-day snapshot spot if quote is much later / missing + const day = asOfIso.slice(0, 10); + try { + const snap = db.prepare( + `SELECT payload_json FROM dealer_map_snapshots WHERE symbol=? AND as_of_date=?`, + ).get(symbol, day) as { payload_json: string } | undefined; + if (snap?.payload_json) { + const m = JSON.parse(snap.payload_json) as DealerMap; + if (Number.isFinite(m.spot) && m.spot > 0) { + if (spot == null || Date.parse(quote?.observed_at ?? '') > Date.parse(asOfIso) + 3600_000) { + spot = m.spot; + spotSource = 'snapshot'; + } + } + } + } catch { + /* ignore */ + } + + if (spot == null || spot <= 0) { + return { surface: null, spotSource }; + } + + let expiries: string[] = []; + const expRow = db.prepare('SELECT value FROM kv_cache WHERE key=?').get(`options_expiry:${symbol}`) as + | { value: string } + | undefined; + if (expRow?.value) { + try { + const parsed = JSON.parse(expRow.value) as unknown; + if (Array.isArray(parsed)) expiries = parsed.map(String).sort(); + } catch { + /* ignore */ + } + } + if (expiries.length === 0) { + const fromChains = db.prepare( + 'SELECT DISTINCT expiry FROM options_chains WHERE symbol=? ORDER BY expiry ASC', + ).all(symbol) as Array<{ expiry: string }>; + expiries = fromChains.map((r) => r.expiry); + } + + const asOfDay = asOfIso.slice(0, 10); + const liveExpiries = expiries.filter((e) => e >= asOfDay).slice(0, maxExpiries); + if (liveExpiries.length === 0) return { surface: null, spotSource }; + + const rows: NormalizedOptionRow[] = []; + let latestTs = asOfIso; + + for (const expiry of liveExpiries) { + const maxTsRow = db.prepare( + `SELECT MAX(ts) AS ts FROM options_chains WHERE symbol=? AND expiry=? AND ts<=?`, + ).get(symbol, expiry, asOfIso) as { ts: string | null } | undefined; + const maxTs = maxTsRow?.ts; + if (!maxTs) continue; + + const chainRows = db.prepare( + `SELECT strike, type, bid, ask, iv, delta, gamma, theta, vega, open_interest, volume, ts + FROM options_chains WHERE symbol=? AND expiry=? AND ts=? + ORDER BY strike ASC, type ASC`, + ).all(symbol, expiry, maxTs) as Array>; + + for (const r of chainRows) { + const bid = num(r.bid); + const ask = num(r.ask); + const mid = + bid != null && ask != null && bid > 0 && ask > 0 ? (bid + ask) / 2 : bid ?? ask; + const gamma = num(r.gamma); + const delta = num(r.delta); + const theta = num(r.theta); + const vega = num(r.vega); + const hasGreeks = gamma != null || delta != null || theta != null || vega != null; + rows.push({ + strike: Number(r.strike), + expiry, + right: r.type === 'put' ? 'put' : 'call', + openInterest: intOrNull(r.open_interest), + volume: intOrNull(r.volume), + bid, + ask, + mid, + iv: num(r.iv), + greeks: hasGreeks ? { delta, gamma, theta, vega, vanna: null } : null, + }); + if (typeof r.ts === 'string' && r.ts > latestTs) latestTs = r.ts; + } + } + + if (rows.length === 0) return { surface: null, spotSource }; + + const surface: NormalizedOptionSurface = { + symbol, + spot, + asOf: asOfIso, + providerId: providerId === 'synthetic' ? 'synthetic' : providerId, + delaySeconds: meta.delaySeconds, + oiFreshness: meta.oiFreshness, + rows, + provenance: { + fetchedAt: latestTs, + sourceKind: 'yfinance', + rawSourceId: `replay:${symbol}:${asOfIso}`, + }, + }; + return { surface, spotSource }; +} + +function compareToSnapshot( + db: DatabaseSync, + symbol: string, + asOfIso: string, + map: DealerMap | null, +): SnapshotCompare { + const day = asOfIso.slice(0, 10); + try { + const snap = db.prepare( + `SELECT payload_json FROM dealer_map_snapshots WHERE symbol=? AND as_of_date=?`, + ).get(symbol, day) as { payload_json: string } | undefined; + if (!snap?.payload_json || !map) { + return { + hasSnapshot: false, + asOfDate: day, + kingMatch: null, + cellCountDelta: null, + topGexOverlap: null, + detail: 'No same-day snapshot to compare', + }; + } + const prev = JSON.parse(snap.payload_json) as DealerMap; + const kingMatch = + (prev.levels.king?.strike ?? null) === (map.levels.king?.strike ?? null) && + (prev.levels.king?.expiry ?? null) === (map.levels.king?.expiry ?? null); + const cellCountDelta = map.cells.length - (prev.cells?.length ?? 0); + const topN = (m: DealerMap) => + [...(m.cells ?? [])] + .sort((a, b) => Math.abs(b.gex) - Math.abs(a.gex)) + .slice(0, 5) + .map((c) => `${c.expiry}|${c.strike}`); + const a = new Set(topN(map)); + const b = new Set(topN(prev)); + let overlap = 0; + for (const k of a) if (b.has(k)) overlap += 1; + const topGexOverlap = a.size === 0 ? 0 : overlap / Math.max(a.size, 1); + return { + hasSnapshot: true, + asOfDate: day, + kingMatch, + cellCountDelta, + topGexOverlap, + detail: kingMatch + ? `King match; top-5 |GEX| overlap ${(topGexOverlap * 100).toFixed(0)}%; cell Δ ${cellCountDelta}` + : `King mismatch (replay ${map.levels.king?.strike ?? '—'} vs snap ${prev.levels.king?.strike ?? '—'}); top overlap ${(topGexOverlap * 100).toFixed(0)}%`, + }; + } catch (e) { + return { + hasSnapshot: false, + asOfDate: day, + kingMatch: null, + cellCountDelta: null, + topGexOverlap: null, + detail: e instanceof Error ? e.message : 'snapshot compare failed', + }; + } +} + +export function replayDealerMapAt( + db: DatabaseSync, + symbolRaw: string, + asOfIso: string, + opts: { maxExpiries?: number } = {}, +): ReplayResult { + const symbol = symbolRaw.toUpperCase(); + const { surface, spotSource } = assembleSurfaceAsOf(db, symbol, asOfIso, opts); + if (!surface) { + const integrity = auditDealerMap(null, null); + return { + symbol, + asOf: asOfIso, + spot: null, + spotSource, + map: null, + integrity, + surfaceRowCount: 0, + snapshotCompare: null, + ok: false, + summary: 'Could not assemble surface as-of (no spot or no chains ≤ asOf)', + }; + } + + const ivHints = loadSaneIvHints(db, symbol); + const map = buildDealerMap(surface, { + maxExpiries: opts.maxExpiries ?? 6, + ivHints, + }); + const integrity = auditDealerMap(map, surface); + const snapshotCompare = compareToSnapshot(db, symbol, asOfIso, map); + const driftBad = + snapshotCompare.hasSnapshot && + snapshotCompare.kingMatch === false && + (snapshotCompare.topGexOverlap ?? 1) < 0.4; + const ok = integrity.status !== 'incomplete' && !driftBad; + + return { + symbol, + asOf: asOfIso, + spot: surface.spot, + spotSource, + map, + integrity, + surfaceRowCount: surface.rows.length, + snapshotCompare, + ok, + summary: [ + integritySummary(integrity), + `spot=${surface.spot} (${spotSource}) rows=${surface.rows.length}`, + snapshotCompare.detail, + ].join(' | '), + }; +} + +export function replayFromSnapshots( + db: DatabaseSync, + symbolRaw: string, + days: number, +): ReplayResult[] { + const symbol = symbolRaw.toUpperCase(); + const rows = db.prepare( + `SELECT as_of_date, payload_json FROM dealer_map_snapshots + WHERE symbol=? ORDER BY as_of_date DESC LIMIT ?`, + ).all(symbol, days) as Array<{ as_of_date: string; payload_json: string }>; + + const out: ReplayResult[] = []; + for (const r of rows) { + let asOf = `${r.as_of_date}T20:00:00.000Z`; + try { + const m = JSON.parse(r.payload_json) as DealerMap; + if (m.asOf) asOf = m.asOf; + } catch { + /* use eod */ + } + out.push(replayDealerMapAt(db, symbol, asOf)); + } + return out; +} diff --git a/app/server/src/analysis/dealerMapService.ts b/app/server/src/analysis/dealerMapService.ts new file mode 100644 index 0000000..c8c2d84 --- /dev/null +++ b/app/server/src/analysis/dealerMapService.ts @@ -0,0 +1,528 @@ +// Dealer map service: cache-only reads, schedule-on-miss, pure recompute from SQLite chains. +// ADR-0009: never call Yahoo/options vendors on the request path. + +import type { DatabaseSync } from 'node:sqlite'; +import type { CacheRepository } from '../cache/CacheRepository.ts'; +import type { AdapterQueue } from '../queue/AdapterQueue.ts'; +import { + assembleSurfaceFromCache, + resolveOptionsProviderId, + type AssembleSurfaceOpts, +} from '../options/OptionsChainRouter.ts'; +import { + buildDealerMap, + layer0Educator, + isDegradedDealerMap, + isHealthyDealerMap, + isSaneIv, + ivHintKey, + HEALTHY_MAP_MAX_AGE_MS, + type DealerMap, + type EducatorCopy, +} from './dealerExposureEngine.ts'; +import { + auditDealerMap, + integritySummary, + type DealerIntegrityReport, +} from './dealerMapIntegrity.ts'; + +export const DEALER_MAP_TTL_MS = 15 * 60_000; +export const DEFAULT_MAX_EXPIRIES = 6; + +export function dealerMapCacheKey(symbol: string): string { + return `dealer_map:${symbol.toUpperCase()}`; +} + +export interface DealerMapGetResult { + map: DealerMap | null; + educator: EducatorCopy | null; + /** Han-style support / resist / short king / OPEX path under active convention. */ + hanScript: import('./hanStyleLevels.ts').HanStyleLevels | null; + status: 'ok' | 'stale' | 'pending' | 'empty' | 'incomplete'; + isStale: boolean; + scheduled: boolean; + providerId: string; + message?: string; + /** Completeness audit (delayed is OK; missing pieces are not). */ + integrity?: DealerIntegrityReport; +} + +export interface DealerMapServiceDeps { + db: DatabaseSync; + cache?: CacheRepository | null; + queue?: AdapterQueue | null; +} + +function readCachedMap(db: DatabaseSync, symbol: string): { map: DealerMap; observedAt: string } | null { + const key = dealerMapCacheKey(symbol); + const row = db.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(key) as + | { value: string; observed_at: string } + | undefined; + if (!row) return null; + try { + const map = JSON.parse(row.value) as DealerMap; + if (!map || typeof map !== 'object' || map.symbol !== symbol.toUpperCase()) return null; + return { map, observedAt: row.observed_at }; + } catch { + return null; + } +} + +export function writeDealerMapCache(db: DatabaseSync, map: DealerMap): void { + const key = dealerMapCacheKey(map.symbol); + try { + db.prepare( + 'INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)', + ).run(key, JSON.stringify(map), map.asOf); + } catch (e) { + // Never let cache write failure (busy/readonly) crash the request path. + console.warn('[dealerMap] write cache failed:', e instanceof Error ? e.message : e); + } +} + +function isStale(observedAt: string, now = Date.now()): boolean { + const t = Date.parse(observedAt); + if (Number.isNaN(t)) return true; + return now - t > DEALER_MAP_TTL_MS; +} + +/** Load prior day snapshot payload for velocity (if any). */ +export function loadPriorSnapshot( + db: DatabaseSync, + symbol: string, + beforeAsOf: string, +): BuildDealerMapOpts['prior'] { + try { + const row = db.prepare( + `SELECT as_of_date, payload_json FROM dealer_map_snapshots + WHERE symbol=? AND as_of_date < ? + ORDER BY as_of_date DESC LIMIT 1`, + ).get(symbol.toUpperCase(), beforeAsOf.slice(0, 10)) as + | { as_of_date: string; payload_json: string } + | undefined; + if (!row) return null; + const parsed = JSON.parse(row.payload_json) as DealerMap; + return { + asOf: parsed.asOf ?? `${row.as_of_date}T20:00:00.000Z`, + byStrike: parsed.byStrike ?? [], + levels: parsed.levels, + }; + } catch { + return null; + } +} + +/** Persist EOD-ish snapshot once per calendar day (idempotent replace). */ +export function upsertDailySnapshot(db: DatabaseSync, map: DealerMap): void { + const day = map.asOf.slice(0, 10); + try { + db.prepare( + `INSERT INTO dealer_map_snapshots (symbol, as_of_date, payload_json, created_at) + VALUES (?,?,?,?) + ON CONFLICT(symbol, as_of_date) DO UPDATE SET + payload_json=excluded.payload_json, + created_at=excluded.created_at`, + ).run(map.symbol, day, JSON.stringify(map), new Date().toISOString()); + } catch { + /* table may not exist until migration */ + } +} + +/** + * Load last-known sane IVs from recent chain history (not only latest junk premarket row). + * Key: expiry|strike|call|put + */ +export function loadSaneIvHints(db: DatabaseSync, symbolRaw: string, limitRows = 8000): Map { + const symbol = symbolRaw.toUpperCase(); + const out = new Map(); + try { + // Newest rows first; keep first sane IV per key. + const rows = db.prepare( + `SELECT expiry, strike, type, iv, ts FROM options_chains + WHERE symbol=? AND iv IS NOT NULL + ORDER BY ts DESC + LIMIT ?`, + ).all(symbol, limitRows) as Array<{ + expiry: string; + strike: number; + type: string; + iv: number; + ts: string; + }>; + for (const r of rows) { + if (!isSaneIv(r.iv)) continue; + const right = r.type === 'put' ? 'put' : 'call'; + const k = ivHintKey(r.expiry, r.strike, right); + if (!out.has(k)) out.set(k, r.iv); + } + } catch { + /* table may differ in tests */ + } + return out; +} + +export interface RecomputeDealerMapResult { + map: DealerMap | null; + surface: import('../options/types.ts').NormalizedOptionSurface | null; + /** True when we kept a healthier cached map instead of writing a poor recompute. */ + keptPriorMap: boolean; + message?: string; + integrity?: DealerIntegrityReport; +} + +/** Persist latest integrity row for ops / admin. */ +export function writeDealerMapDataQuality( + db: DatabaseSync, + symbol: string, + integrity: DealerIntegrityReport, +): void { + try { + const missing = integrity.hardFailed.length; + db.prepare( + `INSERT INTO data_quality ( + symbol, kind, last_checked_at, stored_count, discovered_count, + missing_count, stale, status, detail + ) VALUES (?,?,?,?,?,?,?,?,?) + ON CONFLICT(symbol, kind) DO UPDATE SET + last_checked_at=excluded.last_checked_at, + stored_count=excluded.stored_count, + discovered_count=excluded.discovered_count, + missing_count=excluded.missing_count, + stale=excluded.stale, + status=excluded.status, + detail=excluded.detail`, + ).run( + symbol.toUpperCase(), + 'dealer_map', + new Date().toISOString(), + integrity.metrics.included, + integrity.metrics.oiEligible, + missing, + integrity.status === 'incomplete' ? 1 : 0, + integrity.status, + JSON.stringify({ + summary: integritySummary(integrity), + hardFailed: integrity.hardFailed, + softFailed: integrity.softFailed, + metrics: integrity.metrics, + checks: integrity.checks, + }), + ); + } catch { + /* table may not exist in unit tests */ + } +} + +/** + * Pure recompute from cached chains + quote. No vendor I/O. + * Returns null if surface cannot be assembled. + * Will not overwrite a healthier recent map with a poor-IV recompute. + */ +export function recomputeDealerMapFromCache( + db: DatabaseSync, + symbolRaw: string, + opts: AssembleSurfaceOpts & { write?: boolean } = {}, +): DealerMap | null { + return recomputeDealerMapFromCacheDetailed(db, symbolRaw, opts).map; +} + +/** + * Same as recomputeDealerMapFromCache but exposes keep-prior metadata for the request path. + */ +export function recomputeDealerMapFromCacheDetailed( + db: DatabaseSync, + symbolRaw: string, + opts: AssembleSurfaceOpts & { write?: boolean } = {}, +): RecomputeDealerMapResult { + const symbol = symbolRaw.toUpperCase(); + const surface = assembleSurfaceFromCache(db, symbol, { + maxExpiries: opts.maxExpiries ?? DEFAULT_MAX_EXPIRIES, + providerId: opts.providerId, + asOf: opts.asOf, + }); + if (!surface) { + return { map: null, surface: null, keptPriorMap: false }; + } + + const prior = loadPriorSnapshot(db, symbol, surface.asOf); + const ivHints = loadSaneIvHints(db, symbol); + const map = buildDealerMap(surface, { + maxExpiries: opts.maxExpiries ?? DEFAULT_MAX_EXPIRIES, + prior, + ivHints, + }); + + const integrity = auditDealerMap(map, surface); + const incomplete = integrity.status === 'incomplete'; + const greekPoor = isDegradedDealerMap(map); + + const cached = readCachedMap(db, symbol); + const cachedAgeMs = cached + ? Date.now() - Date.parse(cached.observedAt) + : Number.POSITIVE_INFINITY; + const cachedHealthy = + cached?.map && + isHealthyDealerMap(cached.map) && + Number.isFinite(cachedAgeMs) && + cachedAgeMs >= 0 && + cachedAgeMs <= HEALTHY_MAP_MAX_AGE_MS; + + // Prefer last healthy map when recompute is incomplete or greek-poor. + if (opts.write !== false && (incomplete || greekPoor) && cachedHealthy && cached) { + const priorIntegrity = auditDealerMap(cached.map, surface); + writeDealerMapDataQuality(db, symbol, integrity); + return { + map: cached.map, + surface, + keptPriorMap: true, + integrity: priorIntegrity.status === 'incomplete' ? integrity : priorIntegrity, + message: incomplete + ? `Serving last good map - latest recompute incomplete. ${integritySummary(integrity)}` + : 'Serving last good map - latest chain IV looks unreliable (common premarket on free data).', + }; + } + + if (opts.write !== false) { + // Do not cache incomplete maps as the canonical "good" snapshot. + if (!incomplete) { + writeDealerMapCache(db, map); + upsertDailySnapshot(db, map); + } + writeDealerMapDataQuality(db, symbol, integrity); + } + return { map, surface, keptPriorMap: false, integrity }; +} + +/** + * Schedule options data needed for a map - never more than nearest N expiries. + * Uses ensureInDemand + queue only (no live fetch). + */ +export async function scheduleDealerMapInputs( + deps: DealerMapServiceDeps, + symbolRaw: string, + maxExpiries = DEFAULT_MAX_EXPIRIES, +): Promise { + const symbol = symbolRaw.toUpperCase(); + const { cache, queue, db } = deps; + if (!queue) return false; + + try { + await cache?.ensureInDemand(symbol, 'equity'); + } catch { /* ignore */ } + + // Always want a quote for spot + try { + await queue.queue(`yfinance:quote:${symbol}`); + } catch { /* ignore */ } + + try { + await queue.queue(`yfinance:expiry_dates:${symbol}`); + } catch { /* ignore */ } + + // Resolve expiries from cache if present + let expiries: string[] = []; + const expRow = db.prepare('SELECT value FROM kv_cache WHERE key=?').get(`options_expiry:${symbol}`) as + | { value: string } + | undefined; + if (expRow?.value) { + try { + const parsed = JSON.parse(expRow.value) as unknown; + if (Array.isArray(parsed)) expiries = parsed.map(String).sort(); + } catch { /* ignore */ } + } + if (expiries.length === 0) { + const fromChains = db.prepare( + 'SELECT DISTINCT expiry FROM options_chains WHERE symbol=? ORDER BY expiry ASC', + ).all(symbol) as Array<{ expiry: string }>; + expiries = fromChains.map((r) => r.expiry); + } + + const today = new Date().toISOString().slice(0, 10); + const targets = expiries.filter((e) => e >= today).slice(0, maxExpiries); + + // If we have no expiry list yet, only expiry_dates job is scheduled; chains follow on next get/schedule. + for (const expiry of targets) { + try { + await queue.queue(`yfinance:chain:${symbol}:${expiry}`); + } catch { /* ignore */ } + } + return true; +} + +/** + * Request-path entry: read cache, optionally recompute from chains, schedule if missing. + * Never calls vendor HTTP. + */ +export async function getDealerMap( + deps: DealerMapServiceDeps, + symbolRaw: string, + opts: { + forceSchedule?: boolean; + maxExpiries?: number; + /** classic (default) or dealer_inventory (Heatseeker-style sign flip). */ + convention?: import('../options/types.ts').ExposureConvention; + } = {}, +): Promise { + const symbol = symbolRaw.toUpperCase(); + const providerId = resolveOptionsProviderId(); + const maxExpiries = opts.maxExpiries ?? DEFAULT_MAX_EXPIRIES; + const convention = opts.convention ?? 'classic_call_pos_put_neg'; + const { withExposureConvention } = await import('./dealerExposureEngine.ts'); + + try { + const cached = readCachedMap(deps.db, symbol); + const stale = cached ? isStale(cached.observedAt) : true; + + // Prefer pure recompute when chains exist (keeps map aligned with latest options_snapshot). + let recomputed: DealerMap | null = null; + let recomputeSurface: import('../options/types.ts').NormalizedOptionSurface | null = null; + let keptPrior = false; + let recomputeMsg: string | undefined; + let recomputeIntegrity: DealerIntegrityReport | undefined; + try { + const detail = recomputeDealerMapFromCacheDetailed(deps.db, symbol, { + maxExpiries, + write: true, + }); + recomputed = detail.map; + recomputeSurface = detail.surface; + keptPrior = detail.keptPriorMap; + recomputeMsg = detail.message; + recomputeIntegrity = detail.integrity; + } catch (e) { + console.warn('[dealerMap] recompute failed:', e instanceof Error ? e.message : e); + recomputed = null; + } + + if (recomputed) { + // Cache/store always classic; apply view convention on the way out. + const viewMap = withExposureConvention(recomputed, convention); + const educator = layer0Educator(viewMap); + const { extractHanStyleLevels } = await import('./hanStyleLevels.ts'); + const han = extractHanStyleLevels(viewMap); + // Surface Han vocabulary on the educator card as well as hanScript. + if (han.chips.length > 0) { + educator.levels = [ + ...han.chips.map((c) => `Day script: ${c}`), + ...educator.levels, + ]; + } + let integrity = + recomputeIntegrity ?? + auditDealerMap(recomputed, recomputeSurface); + const incomplete = integrity.status === 'incomplete'; + const qualityPoor = isDegradedDealerMap(recomputed); + const needSchedule = + stale || + opts.forceSchedule || + keptPrior || + qualityPoor || + incomplete || + integrity.status === 'degraded'; + let scheduled = false; + if (needSchedule) { + scheduled = await scheduleDealerMapInputs(deps, symbol, maxExpiries); + } + const qualityNotes = recomputed.quality?.notes?.join(' ') ?? ''; + const message = [ + recomputeMsg, + incomplete && !keptPrior ? integritySummary(integrity) : null, + qualityPoor && !incomplete + ? qualityNotes || 'Greek quality poor - treat levels cautiously.' + : null, + recomputed.quality?.greekQuality === 'degraded' && integrity.status !== 'incomplete' + ? qualityNotes + : null, + ] + .filter(Boolean) + .join(' ') || undefined; + + // Incomplete recompute with no usable prior: do not present as a normal map. + if (incomplete && !keptPrior) { + writeDealerMapDataQuality(deps.db, symbol, integrity); + return { + map: viewMap.cells.length > 0 ? viewMap : null, + educator: viewMap.cells.length > 0 ? educator : null, + hanScript: viewMap.cells.length > 0 ? han : null, + status: 'incomplete', + isStale: true, + scheduled, + providerId: viewMap.providerId, + message: message ?? integritySummary(integrity), + integrity, + }; + } + + let status: DealerMapGetResult['status'] = 'ok'; + if (keptPrior || stale || qualityPoor || integrity.status === 'degraded') status = 'stale'; + if (integrity.status === 'incomplete') status = 'incomplete'; + + return { + map: viewMap, + educator, + hanScript: han, + status, + isStale: keptPrior || qualityPoor || incomplete || (stale && !opts.forceSchedule), + scheduled, + providerId: viewMap.providerId, + message, + integrity, + }; + } + + if (cached?.map) { + const scheduled = await scheduleDealerMapInputs(deps, symbol, maxExpiries); + const integrity = auditDealerMap(cached.map, null); + writeDealerMapDataQuality(deps.db, symbol, integrity); + const viewMap = withExposureConvention(cached.map, convention); + const { extractHanStyleLevels } = await import('./hanStyleLevels.ts'); + const han = extractHanStyleLevels(viewMap); + const educator = layer0Educator(viewMap); + if (han.chips.length > 0) { + educator.levels = [ + ...han.chips.map((c) => `Day script: ${c}`), + ...educator.levels, + ]; + } + return { + map: viewMap, + educator, + hanScript: han, + status: integrity.status === 'incomplete' ? 'incomplete' : 'stale', + isStale: true, + scheduled, + providerId: viewMap.providerId ?? providerId, + message: 'Serving last map while options chains refresh.', + integrity, + }; + } + + const scheduled = await scheduleDealerMapInputs(deps, symbol, maxExpiries); + const emptyIntegrity = auditDealerMap(null, null); + return { + map: null, + educator: null, + hanScript: null, + status: scheduled ? 'pending' : 'empty', + isStale: true, + scheduled, + providerId, + message: scheduled + ? 'No map yet. Options chain refresh scheduled (delayed data).' + : 'No map and no queue available to schedule refresh.', + integrity: emptyIntegrity, + }; + } catch (e) { + console.error('[dealerMap] getDealerMap failed:', e); + return { + map: null, + educator: null, + hanScript: null, + status: 'empty', + isStale: true, + scheduled: false, + providerId, + message: e instanceof Error ? e.message : 'Failed to load dealer map', + }; + } +} diff --git a/app/server/src/analysis/dealerStudyEngine.ts b/app/server/src/analysis/dealerStudyEngine.ts new file mode 100644 index 0000000..90549bc --- /dev/null +++ b/app/server/src/analysis/dealerStudyEngine.ts @@ -0,0 +1,368 @@ +/** + * Dealer Flow Study Desk - practice setups from a map snapshot. + * Grades ask: "did price path match the mechanical hypothesis?" not "did you profit?" + */ + +import type { DealerMap } from './dealerExposureEngine.ts'; + +export type StudyBias = 'educational_long' | 'educational_short' | 'educational_range'; + +export type StudyHypothesisId = + | 'cushion_pin_toward_king' + | 'spring_accelerate_from_spot' + | 'bounce_from_put_wall' + | 'reject_at_call_wall' + | 'whipsaw_range_edges' + | 'soft_floor_watch'; + +export type StudyGrade = + | 'win' // path matched hypothesis (target before invalidation) + | 'loss' // invalidation first + | 'expired' // neither hit by horizon + | 'incomplete'; // not enough price data yet + +export interface StudySetupProposal { + hypothesisId: StudyHypothesisId; + title: string; + bias: StudyBias; + /** Why this structure appears on the map (plain English). */ + mapReading: string; + /** What a student would watch (not "you should trade"). */ + studyQuestion: string; + entry: number; + invalidation: number; + target: number; + horizonDays: number; + confidence: number; // 0-1 rough structure fit + disclaimer: string; +} + +export interface StudyGradeResult { + grade: StudyGrade; + score: number; // 0-100 + summary: string; + pathNotes: string[]; + gradedAt: string; + barsUsed: number; +} + +export interface CandleBar { + ts: string; + o: number; + h: number; + l: number; + c: number; +} + +const DISCLAIMER = 'Educational observation only.'; + +function round2(n: number): number { + return Math.round(n * 100) / 100; +} + +/** Propose 1–3 educational study setups from the current map. */ +export function proposeStudySetups(map: DealerMap): StudySetupProposal[] { + const out: StudySetupProposal[] = []; + const spot = map.spot; + const king = map.levels.king?.strike ?? null; + const flip = map.levels.flip; + const callWall = map.levels.callWall; + const putWall = map.levels.putWall; + const whipsaw = map.patterns.find((p) => p.id === 'whipsaw'); + const rug = map.patterns.find((p) => p.id === 'rug_setup'); + + // 1) Cushion / pin toward king + if (map.regimeAtSpot === 'positive' && king != null && Math.abs(king - spot) / spot > 0.005) { + const towardUp = king > spot; + out.push({ + hypothesisId: 'cushion_pin_toward_king', + title: towardUp ? 'Cushion pull toward king above' : 'Cushion pull toward king below', + bias: towardUp ? 'educational_long' : 'educational_short', + mapReading: `Near spot this looks like a cushion. King ★ at ${king} is the strongest pull on the map.`, + studyQuestion: towardUp + ? `Does price move toward ${king} before breaking the floor (flip / put wall)?` + : `Does price move toward ${king} before breaking the ceiling (flip / call wall)?`, + entry: round2(spot), + invalidation: round2( + towardUp + ? (putWall ?? flip ?? spot * 0.97) + : (callWall ?? flip ?? spot * 1.03), + ), + target: round2(king), + horizonDays: 5, + confidence: 0.55 + Math.min(0.25, Math.abs(king - spot) / spot), + disclaimer: DISCLAIMER, + }); + } + + // 2) Spring amplify + if (map.regimeAtSpot === 'negative') { + const down = true; // spring often studied as continuation once moving; default educational short-side watch if put wall below + const inv = callWall ?? flip ?? spot * 1.03; + const tgt = putWall ?? (king != null && king < spot ? king : spot * 0.95); + out.push({ + hypothesisId: 'spring_accelerate_from_spot', + title: 'Spring near spot', + bias: down ? 'educational_short' : 'educational_long', + mapReading: 'Near spot this looks like a spring - once price moves, it can speed up.', + studyQuestion: `Does price reach ${round2(tgt)} before reclaiming ${round2(inv)}?`, + entry: round2(spot), + invalidation: round2(inv), + target: round2(tgt), + horizonDays: 3, + confidence: 0.5, + disclaimer: DISCLAIMER, + }); + } + + // 3) Put wall bounce (educational long structure) + if (putWall != null && putWall < spot && spot - putWall < spot * 0.08) { + out.push({ + hypothesisId: 'bounce_from_put_wall', + title: 'Reaction at put wall', + bias: 'educational_long', + mapReading: `Put wall near ${putWall} is a heavy pile below price - a floor people watch.`, + studyQuestion: `Does price hold above ${round2(putWall * 0.99)} and move toward ${round2(king ?? spot * 1.02)}?`, + entry: round2(Math.max(putWall, spot * 0.995)), + invalidation: round2(putWall * 0.985), + target: round2(king != null && king > putWall ? king : spot * 1.03), + horizonDays: 5, + confidence: 0.52, + disclaimer: DISCLAIMER, + }); + } + + // 4) Call wall reject (educational short structure) + if (callWall != null && callWall > spot && callWall - spot < spot * 0.08) { + out.push({ + hypothesisId: 'reject_at_call_wall', + title: 'Reaction at call wall', + bias: 'educational_short', + mapReading: `Call wall near ${callWall} is a heavy pile above price - a ceiling people watch.`, + studyQuestion: `Does price fail near ${round2(callWall)} and move toward ${round2(king != null && king < callWall ? king : spot * 0.97)} before clearing ${round2(callWall * 1.01)}?`, + entry: round2(Math.min(callWall, spot * 1.005)), + invalidation: round2(callWall * 1.015), + target: round2(king != null && king < callWall ? king : spot * 0.97), + horizonDays: 5, + confidence: 0.52, + disclaimer: DISCLAIMER, + }); + } + + // 5) Whipsaw range + if (whipsaw && 'edges' in whipsaw) { + const [lo, hi] = whipsaw.edges[0] < whipsaw.edges[1] ? whipsaw.edges : [whipsaw.edges[1], whipsaw.edges[0]]; + out.push({ + hypothesisId: 'whipsaw_range_edges', + title: 'Whipsaw between edges', + bias: 'educational_range', + mapReading: `Whipsaw shape: strong levels near ${lo} and ${hi}, thinner middle.`, + studyQuestion: `Does price stay between ${lo} and ${hi} for the full window, or break out?`, + entry: round2(spot), + invalidation: round2(lo), // break of low edge = range fail for grade + target: round2(hi), + horizonDays: 5, + confidence: whipsaw.confidence, + disclaimer: DISCLAIMER, + }); + } + + // 6) Rug soft floor watch + if (rug && 'shortPivot' in rug) { + out.push({ + hypothesisId: 'soft_floor_watch', + title: 'Soft floor / stack', + bias: 'educational_short', + mapReading: `Stack structure: short pivot near ${rug.shortPivot}, long-side near ${rug.longAbove}. Soft floors can fail and moves can speed up.`, + studyQuestion: `If ${round2(rug.shortPivot)} breaks, does price move lower before reclaiming ${round2(rug.longAbove)}?`, + entry: round2(spot), + invalidation: round2(rug.longAbove), + target: round2( + rug.shortStackBelow.length + ? Math.min(...rug.shortStackBelow) + : rug.shortPivot * 0.97, + ), + horizonDays: 5, + confidence: rug.confidence, + disclaimer: DISCLAIMER, + }); + } + + // Deduplicate by hypothesisId, cap 3, sort by confidence + const byId = new Map(); + for (const p of out) { + const prev = byId.get(p.hypothesisId); + if (!prev || p.confidence > prev.confidence) byId.set(p.hypothesisId, p); + } + return [...byId.values()].sort((a, b) => b.confidence - a.confidence).slice(0, 3); +} + +/** + * Auto-grade using daily (or any) OHLC bars after entry time. + * Long: win if high>=target before low<=invalidation. + * Short: win if low<=target before high>=invalidation. + * Range: win if all closes stay within [invalidation, target] edges (lo/hi). + */ +export function gradeStudySetup( + setup: { + bias: StudyBias; + entry: number; + invalidation: number; + target: number; + horizonDays: number; + loggedAt: string; + }, + bars: CandleBar[], + now: Date = new Date(), +): StudyGradeResult { + const logged = Date.parse(setup.loggedAt); + const horizonMs = setup.horizonDays * 86_400_000; + const endMs = logged + horizonMs; + const future = bars + .filter((b) => { + const t = Date.parse(b.ts); + return Number.isFinite(t) && t >= logged && t <= Math.max(endMs, now.getTime()); + }) + .sort((a, b) => Date.parse(a.ts) - Date.parse(b.ts)); + + if (future.length === 0) { + return { + grade: 'incomplete', + score: 0, + summary: 'Not enough price data yet. Grade again later.', + pathNotes: [], + gradedAt: now.toISOString(), + barsUsed: 0, + }; + } + + const pathNotes: string[] = []; + const pastHorizon = now.getTime() >= endMs; + + if (setup.bias === 'educational_range') { + const lo = Math.min(setup.invalidation, setup.target); + const hi = Math.max(setup.invalidation, setup.target); + let broke = false; + for (const b of future) { + if (b.l < lo || b.h > hi) { + broke = true; + pathNotes.push(`Range broken on ${b.ts.slice(0, 10)} (L=${b.l} H=${b.h} vs ${lo}-${hi}).`); + break; + } + } + if (broke) { + return { + grade: 'loss', + score: 25, + summary: 'Price left the range before the window ended.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + if (pastHorizon) { + pathNotes.push(`Held inside ${lo}-${hi} for ${setup.horizonDays}d study window.`); + return { + grade: 'win', + score: 80, + summary: 'Price stayed inside the range for the full window.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + return { + grade: 'incomplete', + score: 50, + summary: 'Still inside the range. Grade again when the window ends.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + + const isLong = setup.bias === 'educational_long'; + for (const b of future) { + if (isLong) { + if (b.l <= setup.invalidation) { + pathNotes.push(`Invalidation ${setup.invalidation} touched on ${b.ts.slice(0, 10)} (low ${b.l}).`); + return { + grade: 'loss', + score: 20, + summary: 'Wrong-side level hit first. Setup idea failed.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + if (b.h >= setup.target) { + pathNotes.push(`Target ${setup.target} touched on ${b.ts.slice(0, 10)} (high ${b.h}).`); + const r = Math.abs(setup.target - setup.entry) / Math.max(1e-9, Math.abs(setup.entry - setup.invalidation)); + return { + grade: 'win', + score: Math.min(100, Math.round(55 + r * 20)), + summary: 'Target hit first. Setup idea held.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + } else { + if (b.h >= setup.invalidation) { + pathNotes.push(`Invalidation ${setup.invalidation} touched on ${b.ts.slice(0, 10)} (high ${b.h}).`); + return { + grade: 'loss', + score: 20, + summary: 'Wrong-side level hit first. Setup idea failed.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + if (b.l <= setup.target) { + pathNotes.push(`Target ${setup.target} touched on ${b.ts.slice(0, 10)} (low ${b.l}).`); + const r = Math.abs(setup.entry - setup.target) / Math.max(1e-9, Math.abs(setup.invalidation - setup.entry)); + return { + grade: 'win', + score: Math.min(100, Math.round(55 + r * 20)), + summary: 'Target hit first. Setup idea held.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + } + } + + if (pastHorizon) { + pathNotes.push('Horizon ended without target or invalidation.'); + return { + grade: 'expired', + score: 40, + summary: 'Neither target nor wrong-side level hit in time. Inconclusive.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; + } + + return { + grade: 'incomplete', + score: 50, + summary: 'Study window still open - re-grade after more sessions or when horizon ends.', + pathNotes, + gradedAt: now.toISOString(), + barsUsed: future.length, + }; +} + +export interface StudyScorecardRow { + hypothesisId: string; + n: number; + wins: number; + losses: number; + expired: number; + incomplete: number; + winRate: number | null; // among win+loss only + avgScore: number | null; +} diff --git a/app/server/src/analysis/etfHoldingsFallback.ts b/app/server/src/analysis/etfHoldingsFallback.ts index b5328c3..859997c 100644 --- a/app/server/src/analysis/etfHoldingsFallback.ts +++ b/app/server/src/analysis/etfHoldingsFallback.ts @@ -66,6 +66,15 @@ export const ETF_TOP_HOLDINGS_FALLBACK: Record = { { symbol: "FROG", holdingName: "JFrog Ltd Ordinary Shares", holdingPercent: 0.0030702 }, { symbol: "RIOT", holdingName: "Riot Platforms Inc", holdingPercent: 0.0030254 }, ], + "MAGS": [ + { symbol: "AAPL", holdingName: "Apple Inc (via swap)", holdingPercent: 0.143 }, + { symbol: "AMZN", holdingName: "Amazon.com Inc (via swap)", holdingPercent: 0.143 }, + { symbol: "GOOGL", holdingName: "Alphabet Inc (via swap)", holdingPercent: 0.143 }, + { symbol: "META", holdingName: "Meta Platforms Inc (via swap)", holdingPercent: 0.143 }, + { symbol: "MSFT", holdingName: "Microsoft Corp (via swap)", holdingPercent: 0.143 }, + { symbol: "NVDA", holdingName: "NVIDIA Corp (via swap)", holdingPercent: 0.143 }, + { symbol: "TSLA", holdingName: "Tesla Inc (via swap)", holdingPercent: 0.142 }, + ], "SMH": [ { symbol: "NVDA", holdingName: "NVIDIA Corp", holdingPercent: 0.177539 }, { symbol: "TSM", holdingName: "Taiwan Semiconductor Manufacturing Co Ltd ADR", holdingPercent: 0.0919043 }, diff --git a/app/server/src/analysis/hanStyleLevels.ts b/app/server/src/analysis/hanStyleLevels.ts new file mode 100644 index 0000000..5c1c3d8 --- /dev/null +++ b/app/server/src/analysis/hanStyleLevels.ts @@ -0,0 +1,225 @@ +/** + * Han-style gamma script from a DealerMap. + * + * Vocabulary aligned with public gamma educators (e.g. Han_Akamatsu): + * long gamma = +GEX in the *active* convention (cushion / support) + * short gamma = −GEX (accelerate / magnet / short king) + * resistance = tough level above spot + * opex path = large far-dated structure still below cash + * + * Pure: no I/O. Call after withExposureConvention so "Dealer (HS)" mode + * matches Heatseeker colors when that toggle is on. + */ + +import type { ExposureConvention } from '../options/types.ts'; +import type { DealerMap } from './dealerExposureEngine.ts'; + +export interface HanStyleLevels { + symbol: string; + spot: number; + asOf: string; + convention: ExposureConvention; + /** One-paragraph day script (Han-like). */ + script: string; + /** Short chip lines for UI. */ + chips: string[]; + longSupport: number | null; + shortKing: number | null; + resistance: number | null; + /** Far-dated strikes still "below" that matter if support fails. */ + opexPath: number[]; + /** Near-spot regime in Han words. */ + regimeLabel: 'long_gamma' | 'short_gamma' | 'mixed'; +} + +function money(n: number): string { + const a = Math.abs(n); + if (a >= 1e6) return `$${(a / 1e6).toFixed(1)}M`; + if (a >= 1e3) return `$${(a / 1e3).toFixed(0)}K`; + return `$${a.toFixed(0)}`; +} + +function fmtStrike(s: number): string { + return Number.isInteger(s) ? String(s) : s.toFixed(1); +} + +function daysBetween(asOfIso: string, expiryYmd: string): number { + const a = Date.parse(`${asOfIso.slice(0, 10)}T12:00:00.000Z`); + const b = Date.parse(`${expiryYmd}T12:00:00.000Z`); + if (!Number.isFinite(a) || !Number.isFinite(b)) return 999; + return (b - a) / (24 * 3600_000); +} + +/** + * Extract a Han-style levels script from the map under its *current* convention. + */ +export function extractHanStyleLevels(map: DealerMap): HanStyleLevels { + const spot = map.spot; + const band = Math.max(spot * 0.2, 5); // day-relevant band + const byStrike = map.byStrike ?? []; + const nearZero = + byStrike.length === 0 + ? 0 + : Math.max(...byStrike.map((s) => s.absGex), 0) * 0.05; + + const inBand = byStrike.filter( + (s) => s.strike >= spot - band && s.strike <= spot + band && s.absGex >= nearZero, + ); + const below = inBand.filter((s) => s.strike <= spot + 1e-9); + const above = inBand.filter((s) => s.strike >= spot - 1e-9); + + // Long support: strongest +GEX at or below spot (Han: "$38 long gamma support") + const longsBelow = below + .filter((s) => s.gex > nearZero) + .sort((a, b) => b.gex - a.gex || Math.abs(a.strike - spot) - Math.abs(b.strike - spot)); + const longSupport = longsBelow[0]?.strike ?? null; + + // Short king: most negative GEX in band, else global most negative + const shortsBand = inBand.filter((s) => s.gex < -nearZero).sort((a, b) => a.gex - b.gex); + const shortsAll = byStrike.filter((s) => s.gex < -nearZero).sort((a, b) => a.gex - b.gex); + let shortKing: number | null = shortsBand[0]?.strike ?? shortsAll[0]?.strike ?? null; + if ( + map.levels.king && + map.levels.king.gex < -nearZero && + (shortKing == null || + Math.abs(map.levels.king.gex) >= Math.abs(shortsBand[0]?.gex ?? shortsAll[0]?.gex ?? 0)) + ) { + shortKing = map.levels.king.strike; + } + + // Resistance: toughest above spot — prefer short (−GEX) ceiling, else large +GEX pile + const shortsAbove = above.filter((s) => s.gex < -nearZero).sort((a, b) => a.gex - b.gex); + const longsAbove = above.filter((s) => s.gex > nearZero).sort((a, b) => b.gex - a.gex); + const resistance = + shortsAbove[0]?.strike ?? + longsAbove[0]?.strike ?? + map.levels.callWall ?? + null; + + // OPEX path: far expiries (≥10 DTE or 3rd+ column), large |GEX| strikes well below spot + const farExps = (map.expiries ?? []).filter( + (e, i) => i >= 2 || daysBetween(map.asOf, e) >= 10, + ); + const farBelow = new Map(); // strike → absGex sum + for (const c of map.cells ?? []) { + if (!farExps.includes(c.expiry)) continue; + if (c.strike > spot - Math.max(spot * 0.03, 1)) continue; + const abs = Math.abs(c.gex); + if (abs < nearZero) continue; + farBelow.set(c.strike, (farBelow.get(c.strike) ?? 0) + abs); + } + const opexPath = [...farBelow.entries()] + .sort((a, b) => b[1] - a[1]) + .slice(0, 3) + .map(([s]) => s) + .sort((a, b) => b - a); + + // Near-spot regime from nearest strike to cash + let regimeLabel: HanStyleLevels['regimeLabel'] = 'mixed'; + if (byStrike.length > 0) { + let nearest = byStrike[0]; + for (const s of byStrike) { + if (Math.abs(s.strike - spot) < Math.abs(nearest.strike - spot)) nearest = s; + } + if (nearest.absGex >= nearZero) { + regimeLabel = nearest.gex > 0 ? 'long_gamma' : 'short_gamma'; + } + } + + const chips: string[] = []; + if (longSupport != null) { + chips.push(`Long support $${fmtStrike(longSupport)}`); + } + if (resistance != null && resistance !== longSupport) { + const kind = + shortsAbove.some((s) => s.strike === resistance) ? 'short resist' : 'resist'; + chips.push(`${kind} $${fmtStrike(resistance)}`); + } + if (shortKing != null) { + chips.push(`Short king $${fmtStrike(shortKing)}`); + } + if (opexPath.length > 0) { + chips.push( + `OPEX path $${opexPath.map(fmtStrike).join(' / ')}`, + ); + } + + const script = buildScript({ + symbol: map.symbol, + spot, + longSupport, + resistance, + shortKing, + opexPath, + regimeLabel, + convention: map.convention, + king: map.levels.king, + }); + + return { + symbol: map.symbol, + spot, + asOf: map.asOf, + convention: map.convention, + script, + chips, + longSupport, + shortKing, + resistance, + opexPath, + regimeLabel, + }; +} + +function buildScript(p: { + symbol: string; + spot: number; + longSupport: number | null; + resistance: number | null; + shortKing: number | null; + opexPath: number[]; + regimeLabel: HanStyleLevels['regimeLabel']; + convention: ExposureConvention; + king: DealerMap['levels']['king']; +}): string { + const parts: string[] = []; + const conv = + p.convention === 'dealer_inventory' + ? 'dealer/HS signs' + : 'classic OI signs'; + + if (p.longSupport != null) { + parts.push( + `As long as ${p.symbol} stays above $${fmtStrike(p.longSupport)} long gamma, the near-term setup holds under ${conv}.`, + ); + } else if (p.regimeLabel === 'short_gamma') { + parts.push( + `${p.symbol} is sitting in short gamma near $${fmtStrike(p.spot)} - moves can extend once they start (${conv}).`, + ); + } else { + parts.push( + `${p.symbol} near $${fmtStrike(p.spot)}: mixed gamma - treat levels as context only (${conv}).`, + ); + } + + if (p.resistance != null && (p.longSupport == null || p.resistance !== p.longSupport)) { + parts.push(`Resistance around $${fmtStrike(p.resistance)} is the toughest overhead level on this map.`); + } + + if (p.shortKing != null && p.shortKing !== p.resistance) { + parts.push( + `Short king still loud near $${fmtStrike(p.shortKing)}${ + p.king && Math.abs(p.king.gex) > 0 ? ` (${money(p.king.gex)} on ${p.king.expiry})` : '' + }.`, + ); + } + + if (p.opexPath.length > 0) { + parts.push( + `If near support fails, farther-dated exposure still sits lower toward $${p.opexPath.map(fmtStrike).join('/')} into OPEX week.`, + ); + } + + parts.push('Educational observation only - not a forecast.'); + return parts.join(' '); +} diff --git a/app/server/src/analysis/marketRotationRs.ts b/app/server/src/analysis/marketRotationRs.ts index 4062965..e182fc7 100644 --- a/app/server/src/analysis/marketRotationRs.ts +++ b/app/server/src/analysis/marketRotationRs.ts @@ -2,11 +2,12 @@ // ADR-0007: describes leadership / lag, never "capital is flowing into X, allocate there." // Benchmark is typically SPY. RS = sector total return − benchmark total return over the same window. -export type Horizon = 'oneWeek' | 'oneMonth' | 'threeMonth' | 'sixMonth' | 'oneYear' | 'threeYear' | 'fiveYear'; +export type Horizon = 'oneDay' | 'oneWeek' | 'oneMonth' | 'threeMonth' | 'sixMonth' | 'oneYear' | 'threeYear' | 'fiveYear'; -export const HORIZONS: Horizon[] = ['oneWeek', 'oneMonth', 'threeMonth', 'sixMonth', 'oneYear', 'threeYear', 'fiveYear']; +export const HORIZONS: Horizon[] = ['oneDay', 'oneWeek', 'oneMonth', 'threeMonth', 'sixMonth', 'oneYear', 'threeYear', 'fiveYear']; export const HORIZON_MS: Record = { + oneDay: 86_400_000, oneWeek: 7 * 86_400_000, oneMonth: 30 * 86_400_000, threeMonth: 90 * 86_400_000, @@ -27,7 +28,7 @@ export interface SectorDef { name: string; /** GICS-ish group for UI chips. */ group: string; - kind: 'sector' | 'style' | 'region' | 'thematic'; + kind: 'sector' | 'style' | 'region' | 'thematic' | 'custom' | 'auto'; } /** Core GICS + style/region set used for market-first rotation map. */ @@ -44,6 +45,7 @@ export const MARKET_ROTATION_UNIVERSE: SectorDef[] = [ { symbol: 'XLC', name: 'Communication Services', group: 'Comm. Services', kind: 'sector' }, { symbol: 'XLB', name: 'Materials', group: 'Materials', kind: 'sector' }, { symbol: 'SMH', name: 'Semiconductors', group: 'Semiconductors', kind: 'thematic' }, + { symbol: 'MAGS', name: 'Magnificent Seven', group: 'Mega Cap Tech', kind: 'thematic' }, { symbol: 'XBI', name: 'Biotech', group: 'Biotech', kind: 'thematic' }, { symbol: 'IWF', name: 'Growth', group: 'Style: Growth', kind: 'style' }, { symbol: 'IWD', name: 'Value', group: 'Style: Value', kind: 'style' }, @@ -76,12 +78,30 @@ export function totalReturnPct(candles: CandlePoint[], windowMs: number): number return ((latest.c - closest.c) / closest.c) * 100; } +/** + * Day-over-day change: latest close vs previous candle close. + * Used for the oneDay horizon — a window-based lookup (latestTs − 24h) is unreliable + * because daily candles carry intraday timestamps (13:30Z from yfinance), so the + * target can land on a weekend/holiday and the closest candle becomes the latest + * itself, yielding 0. Previous-close is both robust and the semantically correct + * "today" number. + */ +export function dayChangePct(candles: CandlePoint[]): number | null { + if (candles.length < 2) return null; + const latest = candles[candles.length - 1]; + const prev = candles[candles.length - 2]; + if (latest.c <= 0 || prev.c <= 0) return null; + return ((latest.c - prev.c) / prev.c) * 100; +} + export function returnsForHorizons( candles: CandlePoint[], ): Record { const out = {} as Record; for (const h of HORIZONS) { - out[h] = totalReturnPct(candles, HORIZON_MS[h]); + out[h] = h === 'oneDay' + ? dayChangePct(candles) + : totalReturnPct(candles, HORIZON_MS[h]); } return out; } @@ -305,7 +325,7 @@ export function summarizeRotation(rows: SectorRsRow[]): RotationSummary { (earlyWatchCount > 0 ? `${earlyWatchCount} name(s) show early improvement on a one-week relative basis. ` : '') + - 'Educational context for relative leadership — not an allocation instruction.'; + 'Relative leadership context only.'; return { strength, diff --git a/app/server/src/analysis/mentorCallExtract.ts b/app/server/src/analysis/mentorCallExtract.ts new file mode 100644 index 0000000..9cb8c7f --- /dev/null +++ b/app/server/src/analysis/mentorCallExtract.ts @@ -0,0 +1,178 @@ +/** + * Rule-based extraction of mentor map-style calls from harvested post text. + * Local-only; no LLM. Privacy: run on gitignored raw harvest files. + */ + +export type MentorBias = 'educational_long' | 'educational_short' | 'educational_range'; + +export type MentorClaimType = + | 'pin_to_king' + | 'wall_bounce' + | 'wall_reject' + | 'spring' + | 'range' + | 'other'; + +export interface MentorCallDraft { + handle: string; + postId: string; + postedAt: string | null; + symbol: string | null; + bias: MentorBias | 'unclear'; + claimType: MentorClaimType; + entry: number | null; + invalidation: number | null; + target: number | null; + horizonDays: number; + excerpt: string; + extractConfidence: number; + rawPath?: string; +} + +/** Map Study Desk hypothesis ids ↔ mentor claim types for score blending later. */ +export const HYPOTHESIS_TO_CLAIM: Record = { + cushion_pin_toward_king: 'pin_to_king', + bounce_from_put_wall: 'wall_bounce', + reject_at_call_wall: 'wall_reject', + spring_accelerate_from_spot: 'spring', + whipsaw_range_edges: 'range', + soft_floor_watch: 'spring', +}; + +function stripFrontmatter(md: string): { meta: Record; body: string } { + const meta: Record = {}; + if (!md.startsWith('---')) return { meta, body: md }; + const end = md.indexOf('\n---', 3); + if (end < 0) return { meta, body: md }; + const head = md.slice(3, end); + for (const line of head.split('\n')) { + const m = line.match(/^(\w+):\s*"?(.*?)"?\s*$/); + if (m) meta[m[1]] = m[2]; + } + return { meta, body: md.slice(end + 4) }; +} + +function parsePrices(text: string): number[] { + const out: number[] = []; + // $32 or 32$ or near 105 + const re = /\$\s*(\d+(?:\.\d+)?)\b|\b(\d+(?:\.\d+)?)\s*\$/g; + let m: RegExpExecArray | null; + while ((m = re.exec(text)) !== null) { + const n = Number(m[1] ?? m[2]); + if (Number.isFinite(n) && n > 0 && n < 1_000_000) out.push(n); + } + return out; +} + +function cashtags(text: string): string[] { + const tags = new Set(); + for (const m of text.matchAll(/\$([A-Z]{1,5})\b/g)) { + tags.add(m[1]); + } + return [...tags]; +} + +export function extractMentorCallFromMarkdown( + md: string, + opts: { handle: string; rawPath?: string }, +): MentorCallDraft | null { + const { meta, body } = stripFrontmatter(md); + const text = body.replace(/\s+/g, ' ').trim(); + if (text.length < 40) return null; + + const prices = parsePrices(text); + const symbols = cashtags(text); + // Need at least one price level to be gradeable + if (prices.length === 0) return null; + + const lower = text.toLowerCase(); + let bias: MentorCallDraft['bias'] = 'unclear'; + let claimType: MentorClaimType = 'other'; + let conf = 0.35; + + if (/\b(king|pin|magnet|mean reversion|yellow)\b/i.test(text)) { + claimType = 'pin_to_king'; + conf += 0.15; + } + if (/\b(put wall|support|bounce|floor)\b/i.test(text)) { + claimType = 'wall_bounce'; + conf += 0.1; + } + if (/\b(call wall|resistance|reject|ceiling)\b/i.test(text)) { + claimType = 'wall_reject'; + conf += 0.1; + } + if (/\b(purple|short gamma|amplif|momentum|spring)\b/i.test(text)) { + claimType = 'spring'; + conf += 0.1; + } + if (/\b(whipsaw|range|chop|between)\b/i.test(text)) { + claimType = 'range'; + conf += 0.1; + } + + if (/\b(long|upside|support|bounce|bid)\b/i.test(lower) && !/\bshort\b/i.test(lower)) { + bias = 'educational_long'; + conf += 0.1; + } else if (/\b(short|downside|reject|puts)\b/i.test(lower)) { + bias = 'educational_short'; + conf += 0.1; + } else if (claimType === 'range') { + bias = 'educational_range'; + } + + // Heuristic levels: smallest = support-ish, largest = resistance-ish + const sorted = [...prices].sort((a, b) => a - b); + let entry: number | null = sorted[Math.floor(sorted.length / 2)] ?? null; + let invalidation: number | null = null; + let target: number | null = null; + + if (bias === 'educational_long' && sorted.length >= 2) { + invalidation = sorted[0]; + target = sorted[sorted.length - 1]; + entry = sorted[Math.min(1, sorted.length - 1)]; + } else if (bias === 'educational_short' && sorted.length >= 2) { + invalidation = sorted[sorted.length - 1]; + target = sorted[0]; + entry = sorted[Math.max(0, sorted.length - 2)]; + } else if (sorted.length >= 2) { + bias = 'educational_range'; + claimType = claimType === 'other' ? 'range' : claimType; + invalidation = sorted[0]; + target = sorted[sorted.length - 1]; + entry = (invalidation + target) / 2; + } else { + // single price - treat as target with soft invalidation + target = sorted[0]; + entry = sorted[0]; + invalidation = sorted[0] * 0.97; + conf -= 0.1; + } + + if (bias === 'unclear' || entry == null || invalidation == null || target == null) { + return null; + } + + let horizonDays = 5; + if (/\bopex\b/i.test(text)) horizonDays = 5; + if (/\b(week|weekly)\b/i.test(text)) horizonDays = 5; + if (/\b(day|today|intraday)\b/i.test(text)) horizonDays = 2; + + conf = Math.max(0.15, Math.min(0.9, conf + (prices.length >= 2 ? 0.1 : 0))); + + return { + handle: opts.handle || meta.handle || 'unknown', + postId: meta.post_id || opts.rawPath || String(Date.now()), + postedAt: meta.date || null, + symbol: symbols[0] ?? null, + bias, + claimType, + entry, + invalidation, + target, + horizonDays, + excerpt: text.slice(0, 400), + extractConfidence: conf, + rawPath: opts.rawPath, + }; +} diff --git a/app/server/src/cache/CacheRepository.ts b/app/server/src/cache/CacheRepository.ts index 7889e79..ed97134 100644 --- a/app/server/src/cache/CacheRepository.ts +++ b/app/server/src/cache/CacheRepository.ts @@ -5,19 +5,37 @@ // the injected scheduler (SourceAdapter/AdapterQueue satisfy `CacheScheduler`). import { DatabaseSync } from 'node:sqlite'; import { db as defaultDb } from '../db/client.ts'; +import { + CANDLE_FRESH_MS, + quoteTtlMs, + SYMBOL_META_INCOMPLETE_TTL_MS, +} from '../queue/sourceRatePolicy.ts'; +import { KvReadCache } from './LruCache.ts'; -export type SourceKind = 'yfinance' | 'nasdaq' | 'finra-bulk' | 'sec' | 'sec-fetch' | 'reddit' | 'x' | 'macro' | 'llm' | 'sec-lint-holders' | 'sec-lint-insiders' | 'fred'; +export type SourceKind = 'yfinance' | 'nasdaq' | 'finra-bulk' | 'finra-si' | 'sec' | 'sec-fetch' | 'sec-sc-fetch' | 'sec-tickers' | 'reddit' | 'x' | 'macro' | 'llm' | 'sec-lint-holders' | 'sec-lint-insiders' | 'fred'; export type TickerKind = 'equity' | 'crypto' | 'etf' | 'index'; export type CacheKey = string; // `${SourceKind}:${kind}:${id}` e.g. 'yfinance:quote:NVDA', 'yfinance:candles:NVDA:1d' export type TtlClass = | 'live_quote' | 'intraday' | 'daily_permanent' | 'options_snapshot' | 'filing_immutable' | 'quarterly_immutable' | 'thread_7d' | 'macro_event' | 'regime_classification' | 'llm_summary' | 'symbol_meta' - | 'short_interest'; + | 'short_interest' | 'dividend_fundamentals' | 'fred_macro'; export interface Provenance { fetchedAt: string; sourceKind: SourceKind; rawSourceId?: string; } -export interface Quote { symbol: string; price: number; bid?: number | null; ask?: number | null; change?: number | null; changePercent?: number | null; iv?: number | null; } +export interface Quote { + symbol: string; + price: number; + bid?: number | null; + ask?: number | null; + change?: number | null; + changePercent?: number | null; + iv?: number | null; + /** Which session the `price` print came from (pre/post when Yahoo has extended data). */ + session?: 'regular' | 'pre' | 'post' | null; + /** RTH last (close during extended); useful when `price` is pre/post. */ + regularPrice?: number | null; +} export interface PriceCandle { ts: string; o: number; h: number; l: number; c: number; v: number; adjClose?: number | null; } export interface SymbolMeta { symbol: string; name?: string | null; sector?: string | null; industry?: string | null; exchange?: string | null; tickerKind: TickerKind; peers?: string[] | null; description?: string | null; website?: string | null; marketCap?: number | null; currency?: string | null; employees?: number | null; country?: string | null; city?: string | null; } export interface PriceAdjustment { symbol: string; exDate: string; type: "split" | "dividend"; ratio: number } @@ -27,10 +45,18 @@ export interface CacheScheduler { queue(key: CacheKey): Promise; } export interface CacheEntry { value: T | null; provenance: Provenance | null; isStale: boolean; } +// Module-level LRU cache for kv_cache reads — shared across all handler instances +let kvReadCache = new KvReadCache({ max: 2000, ttl: 5 * 60 * 1000 }); + +/** Set the LRU cache instance (for testing or runtime configuration) */ +export function setKvReadCache(cache: KvReadCache): void { + kvReadCache = cache; +} + export const TTL_MS: Record = { - live_quote: 60_000, // 1min (mkt hrs); after-hours 15min refined in adapter slice + live_quote: 60_000, // 1min (mkt hrs); after-hours uses quoteTtlMs() intraday: 5 * 60_000, - daily_permanent: Number.POSITIVE_INFINITY, // immutable once fetched; stale only when absent + daily_permanent: Number.POSITIVE_INFINITY, // full history kept; EOD refresh is schedule-driven options_snapshot: 15 * 60_000, filing_immutable: Number.POSITIVE_INFINITY, quarterly_immutable: Number.POSITIVE_INFINITY, @@ -40,6 +66,8 @@ export const TTL_MS: Record = { llm_summary: Number.POSITIVE_INFINITY, // cached forever by prompt-hash symbol_meta: 7 * 24 * 60 * 60_000, // weekly (slow-changing sector/industry) short_interest: 24 * 60 * 60_000, // refreshed twice/month per source + dividend_fundamentals: 7 * 24 * 60 * 60_000, // weekly (yield/payout change slowly) + fred_macro: 24 * 60 * 60_000, // daily (rates move daily; other series slower) }; /** Parse 'source:kind:id...' into { source, kind, id } (id may contain colons). */ @@ -73,19 +101,48 @@ function tsAgeMs(ts: string | null, now: number): number { const quoteHandler: KindHandler = { ttlClass: 'live_quote', read(d, symbol) { - const r = d.prepare('SELECT symbol,price,bid,ask,change,change_pct,iv,observed_at FROM quotes WHERE symbol=?').get(symbol) as Record | undefined; + const r = d.prepare( + 'SELECT symbol,price,bid,ask,change,change_pct,iv,session,regular_price,observed_at FROM quotes WHERE symbol=?', + ).get(symbol) as Record | undefined; if (!r) return null; + const sessionRaw = r.session == null ? null : String(r.session); + const session = + sessionRaw === 'regular' || sessionRaw === 'pre' || sessionRaw === 'post' + ? sessionRaw + : null; return { - value: { symbol: r.symbol, price: r.price, bid: r.bid, ask: r.ask, change: r.change, changePercent: r.change_pct, iv: r.iv } as Quote, + value: { + symbol: r.symbol, + price: r.price, + bid: r.bid, + ask: r.ask, + change: r.change, + changePercent: r.change_pct, + iv: r.iv, + session, + regularPrice: r.regular_price as number | null | undefined, + } as Quote, stalenessTs: r.observed_at as string, }; }, write(d, symbol, value, provenance) { const q = value as Quote; - d.prepare('INSERT OR REPLACE INTO quotes (symbol,price,bid,ask,change,change_pct,iv,observed_at) VALUES (?,?,?,?,?,?,?,?)') - .run(symbol, q.price, q.bid ?? null, q.ask ?? null, q.change ?? null, q.changePercent ?? null, q.iv ?? null, provenance.fetchedAt); + d.prepare( + 'INSERT OR REPLACE INTO quotes (symbol,price,bid,ask,change,change_pct,iv,session,regular_price,observed_at) VALUES (?,?,?,?,?,?,?,?,?,?)', + ).run( + symbol, + q.price, + q.bid ?? null, + q.ask ?? null, + q.change ?? null, + q.changePercent ?? null, + q.iv ?? null, + q.session ?? null, + q.regularPrice ?? null, + provenance.fetchedAt, + ); }, - isStale(ts, now) { return tsAgeMs(ts, now) > TTL_MS.live_quote; }, + isStale(ts, now) { return tsAgeMs(ts, now) > quoteTtlMs(new Date(now)); }, }; const candlesHandler: KindHandler = { @@ -96,16 +153,53 @@ const candlesHandler: KindHandler = { const rows = d.prepare('SELECT ts,o,h,l,c,v,adj_close,observed_at FROM price_candles WHERE symbol=? AND timeframe=? ORDER BY ts ASC').all(symbol, timeframe) as Array>; if (!rows.length) return null; const value: PriceCandle[] = rows.map((r) => ({ ts: r.ts as string, o: r.o as number, h: r.h as number, l: r.l as number, c: r.c as number, v: r.v as number, adjClose: r.adj_close as number | null })); - return { value, stalenessTs: rows[rows.length - 1].observed_at as string }; + // Staleness for SWR uses last *bar* ts (not observed_at) so schedule + get agree on lag. + return { value, stalenessTs: rows[rows.length - 1].ts as string }; }, write(d, id, value, provenance) { const [symbol, timeframe] = id.split(':'); const ins = d.prepare('INSERT OR REPLACE INTO price_candles (symbol,timeframe,ts,o,h,l,c,v,adj_close,observed_at) VALUES (?,?,?,?,?,?,?,?,?,?)'); for (const c of value as PriceCandle[]) ins.run(symbol, timeframe, c.ts, c.o, c.h, c.l, c.c, c.v, c.adjClose ?? null, provenance.fetchedAt); }, - isStale(ts) { return ts === null; }, // permanent: stale only when absent + // Do not thrash on every read: schedule owns EOD refresh. Stale only when absent. + isStale(ts) { return ts === null; }, }; +/** True when daily candles are missing or last bar is older than CANDLE_FRESH_MS. */ +export function needsCandleRefresh(d: DatabaseSync, symbol: string, timeframe = '1d', now = Date.now()): boolean { + const row = d.prepare( + 'SELECT MAX(ts) AS last_ts FROM price_candles WHERE symbol=? AND timeframe=?', + ).get(symbol, timeframe) as { last_ts: string | null } | undefined; + if (!row?.last_ts) return true; + return tsAgeMs(row.last_ts, now) > CANDLE_FRESH_MS; +} + +/** Last daily candle bar timestamp, or null if none. */ +export function lastCandleTs(d: DatabaseSync, symbol: string, timeframe = '1d'): string | null { + const row = d.prepare( + 'SELECT MAX(ts) AS last_ts FROM price_candles WHERE symbol=? AND timeframe=?', + ).get(symbol, timeframe) as { last_ts: string | null } | undefined; + return row?.last_ts ?? null; +} + +/** True when quote is missing or past session-aware TTL. */ +export function needsQuoteRefresh(d: DatabaseSync, symbol: string, now = Date.now()): boolean { + const row = d.prepare('SELECT observed_at FROM quotes WHERE symbol=?').get(symbol) as { observed_at: string } | undefined; + if (!row?.observed_at) return true; + return tsAgeMs(row.observed_at, now) > quoteTtlMs(new Date(now)); +} + +/** True when symbol meta missing, incomplete (no name), or past weekly TTL. */ +export function needsSymbolMetaRefresh(d: DatabaseSync, symbol: string, now = Date.now()): boolean { + const row = d.prepare('SELECT name, sector, updated_at FROM symbols WHERE symbol=?').get(symbol) as + | { name: string | null; sector: string | null; updated_at: string | null } + | undefined; + if (!row?.updated_at) return true; + const incomplete = !row.name; + const ttl = incomplete ? SYMBOL_META_INCOMPLETE_TTL_MS : TTL_MS.symbol_meta; + return tsAgeMs(row.updated_at, now) > ttl; +} + const adjustmentsHandler: KindHandler = { ttlClass: 'daily_permanent', read(d, symbol) { @@ -138,7 +232,10 @@ const symbolHandler: KindHandler = { d.prepare('INSERT OR REPLACE INTO symbols (symbol,name,sector,industry,exchange,ticker_kind,peers,updated_at) VALUES (?,?,?,?,?,?,?,?)') .run(symbol, s.name ?? null, s.sector ?? null, s.industry ?? null, s.exchange ?? null, s.tickerKind, s.peers ? JSON.stringify(s.peers) : null, provenance.fetchedAt); }, - isStale(ts, now) { return tsAgeMs(ts, now) > TTL_MS.symbol_meta; }, + isStale(ts, now) { + // Age-only here; incomplete rows (null name) are handled in needsSymbolMetaRefresh + get override. + return tsAgeMs(ts, now) > TTL_MS.symbol_meta; + }, }; // ----- Options handlers (slice 15) ----- @@ -178,10 +275,15 @@ const optionsChainHandler: KindHandler = { for (const r of rows) { const strike = typeof r.strike === 'number' ? r.strike : 0; const right = r.right === 'put' ? 'put' : 'call'; + // OptionChainRow nests greeks; accept flat fields too for older writers. + const g = (r.greeks && typeof r.greeks === 'object') + ? (r.greeks as Record) + : null; ins.run( symbol, expiry, strike, right, numOrNull(r.bid), numOrNull(r.ask), numOrNull(r.impliedVolatility), - numOrNull(r.delta), numOrNull(r.gamma), numOrNull(r.theta), numOrNull(r.vega), + numOrNull(g?.delta ?? r.delta), numOrNull(g?.gamma ?? r.gamma), + numOrNull(g?.theta ?? r.theta), numOrNull(g?.vega ?? r.vega), numOrNull(r.openInterest), numOrNull(r.volume), provenance.fetchedAt ); @@ -190,6 +292,33 @@ const optionsChainHandler: KindHandler = { isStale(ts, now) { return tsAgeMs(ts, now) > TTL_MS.options_snapshot; }, }; +// Helper: LRU cache-aware read wrapper for handlers that use kv_cache +function withKvCacheRead( + handler: KindHandler, + getCacheKey: (id: string) => string | null, +): KindHandler { + return { + ...handler, + read(d, id) { + const cacheKey = getCacheKey(id); + if (!cacheKey) return handler.read(d, id); + + // Check LRU cache first + const cached = kvReadCache.get(cacheKey); + if (cached) return cached as { value: unknown; stalenessTs: string } | null; + + const result = handler.read(d, id); + + // Populate LRU cache on hit + if (result && !cacheKey.includes(':*')) { + kvReadCache.set(cacheKey, result.value as unknown, result.stalenessTs); + } + + return result; + }, + }; +} + const optionsExpiryDatesHandler: KindHandler = { ttlClass: 'intraday', read(d, symbol) { @@ -261,9 +390,13 @@ function parseGreeksId(id: string): { symbol: string; expiry: string; strike: st const fetchHandler: KindHandler = { ttlClass: 'daily_permanent', read(d, id) { - const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(`sec-fetch:${id}`) as { value: string; observed_at: string } | undefined; + const cacheKey = `sec-fetch:${id}`; + const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(cacheKey) as { value: string; observed_at: string } | undefined; if (!r) return null; - try { return { value: JSON.parse(r.value), stalenessTs: r.observed_at }; } catch { return null; } + try { + const value = JSON.parse(r.value); + return { value, stalenessTs: r.observed_at }; + } catch { return null; } }, write(d, id, value, provenance) { d.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)').run(`sec-fetch:${id}`, JSON.stringify(value), provenance.fetchedAt); @@ -292,9 +425,13 @@ const lintInsidersHandler: KindHandler = { const shortInterestHandler: KindHandler = { ttlClass: 'short_interest', read(d, id) { - const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(`yfinance:shortinterest:${id}`) as { value: string; observed_at: string } | undefined; + const cacheKey = `yfinance:shortinterest:${id}`; + const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(cacheKey) as { value: string; observed_at: string } | undefined; if (!r) return null; - try { return { value: JSON.parse(r.value), stalenessTs: r.observed_at }; } catch { return null; } + try { + const value = JSON.parse(r.value); + return { value, stalenessTs: r.observed_at }; + } catch { return null; } }, write(d, id, value, provenance) { d.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)').run(`yfinance:shortinterest:${id}`, JSON.stringify(value), provenance.fetchedAt); @@ -305,9 +442,13 @@ const shortInterestHandler: KindHandler = { const nasdaqShortinterestHandler: KindHandler = { ttlClass: 'short_interest', read(d, id) { - const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(`nasdaq:nasdaqShortinterest:${id}`) as { value: string; observed_at: string } | undefined; + const cacheKey = `nasdaq:nasdaqShortinterest:${id}`; + const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(cacheKey) as { value: string; observed_at: string } | undefined; if (!r) return null; - try { return { value: JSON.parse(r.value), stalenessTs: r.observed_at }; } catch { return null; } + try { + const value = JSON.parse(r.value); + return { value, stalenessTs: r.observed_at }; + } catch { return null; } }, write(d, id, value, provenance) { d.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)').run(`nasdaq:nasdaqShortinterest:${id}`, JSON.stringify(value), provenance.fetchedAt); @@ -346,32 +487,143 @@ const finraShortInterestHandler: KindHandler = { }, }; +const finraSiBiweeklyHandler: KindHandler = { + ttlClass: 'short_interest', + read(d, symbol) { + const r = d.prepare( + `SELECT current_short_position, previous_short_position, avg_daily_volume, days_to_cover, change_percent, settlement_date, ingested_at + FROM finra_short_interest_biweekly WHERE symbol=? ORDER BY settlement_date DESC LIMIT 1` + ).get(symbol) as Record | undefined; + if (!r) return null; + return { + value: { + currentShortPosition: r.current_short_position as number, + previousShortPosition: r.previous_short_position as number | null, + avgDailyVolume: r.avg_daily_volume as number | null, + daysToCover: r.days_to_cover as number | null, + changePercent: r.change_percent as number | null, + settlementDate: r.settlement_date as string, + }, + stalenessTs: r.ingested_at as string, + }; + }, + write(_d, _symbol, _value, _provenance) {}, + isStale(ts, now) { + if (ts === null) return false; + return tsAgeMs(ts, now) > TTL_MS.short_interest; + }, +}; + +const dividendFundamentalsHandler: KindHandler = { + ttlClass: 'dividend_fundamentals', + read(d, symbol) { + const r = d.prepare( + `SELECT dividend_yield, payout_ratio, dividend_rate, ex_dividend_date, trailing_eps, forward_pe, growth_streak_years, observed_at FROM dividend_fundamentals WHERE symbol=?` + ).get(symbol) as Record | undefined; + if (!r) return null; + return { value: { symbol, dividendYield: r.dividend_yield, payoutRatio: r.payout_ratio, dividendRate: r.dividend_rate, exDividendDate: r.ex_dividend_date, trailingEps: r.trailing_eps, forwardPE: r.forward_pe, growthStreakYears: r.growth_streak_years }, stalenessTs: r.observed_at as string }; + }, + write(d, symbol, value, provenance) { + d.prepare('INSERT OR REPLACE INTO dividend_fundamentals (symbol, dividend_yield, payout_ratio, dividend_rate, ex_dividend_date, trailing_eps, forward_pe, growth_streak_years, observed_at) VALUES (?,?,?,?,?,?,?,?)').run(symbol, (value as any).dividendYield ?? null, (value as any).payoutRatio ?? null, (value as any).dividendRate ?? null, (value as any).exDividendDate ?? null, (value as any).trailingEps ?? null, (value as any).forwardPE ?? null, (value as any).growthStreakYears ?? null, provenance.fetchedAt); + }, + isStale(ts, now) { return ts ? tsAgeMs(ts, now) > 7 * 86400000 : true; }, +}; + +/** ETF composition — same kv key as market.sectorHoldings for cache sharing. */ +const topHoldingsHandler: KindHandler = { + ttlClass: 'symbol_meta', + read(d, symbol) { + const key = `etf:topHoldings:${symbol.toUpperCase()}`; + const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(key) as + | { value: string; observed_at: string } + | undefined; + if (!r?.value) return null; + try { + const parsed = JSON.parse(r.value) as { holdings?: unknown }; + return { value: parsed, stalenessTs: r.observed_at }; + } catch { + return null; + } + }, + write(d, symbol, value, provenance) { + const key = `etf:topHoldings:${symbol.toUpperCase()}`; + const payload = typeof value === 'string' ? value : JSON.stringify(value); + d.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)') + .run(key, payload, provenance.fetchedAt); + }, + isStale(ts, now) { return tsAgeMs(ts, now) > TTL_MS.symbol_meta; }, +}; + +// Wrap kv_cache-reading handlers with LRU cache +const wrappedOptionsExpiryDates = withKvCacheRead(optionsExpiryDatesHandler, (id) => `options_expiry:${id}`); +const wrappedFetch = withKvCacheRead(fetchHandler, (id) => `sec-fetch:${id}`); +const wrappedShortInterest = withKvCacheRead(shortInterestHandler, (id) => `yfinance:shortinterest:${id}`); +const wrappedNasdaqShortInterest = withKvCacheRead(nasdaqShortinterestHandler, (id) => `nasdaq:nasdaqShortinterest:${id}`); +const wrappedTopHoldings = withKvCacheRead(topHoldingsHandler, (symbol) => `etf:topHoldings:${(symbol as string).toUpperCase()}`); + +/** FRED macro series — kv_cache keyed `fred:series:{seriesId}` (warmed by the queue, off request path). */ +const seriesHandler: KindHandler = { + ttlClass: 'fred_macro', + read(d, seriesId) { + const cacheKey = `fred:series:${seriesId}`; + const r = d.prepare('SELECT value, observed_at FROM kv_cache WHERE key=?').get(cacheKey) as + | { value: string; observed_at: string } + | undefined; + if (!r?.value) return null; + try { + return { value: JSON.parse(r.value), stalenessTs: r.observed_at }; + } catch { return null; } + }, + write(d, seriesId, value, provenance) { + d.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)') + .run(`fred:series:${seriesId}`, JSON.stringify(value), provenance.fetchedAt); + }, + isStale(ts, now) { return tsAgeMs(ts, now) > TTL_MS.fred_macro; }, +}; + const HANDLERS = new Map([ ['quote', quoteHandler], ['candles', candlesHandler], ['symbol', symbolHandler], ['adjustments', adjustmentsHandler], ['chain', optionsChainHandler], - ['expiry_dates', optionsExpiryDatesHandler], + ['expiry_dates', wrappedOptionsExpiryDates], ['greeks', greeksHandler], - ['fetch', fetchHandler], + ['fetch', wrappedFetch], ['holders', lintHoldersHandler], ['insiders', lintInsidersHandler], - ['shortinterest', shortInterestHandler], - ['nasdaqShortinterest', nasdaqShortinterestHandler], + ['shortinterest', wrappedShortInterest], + ['nasdaqShortinterest', wrappedNasdaqShortInterest], ['finraShortinterest', finraShortInterestHandler], + ['finraSiBiweekly', finraSiBiweeklyHandler], + ['dividendFundamentals', dividendFundamentalsHandler], + ['topHoldings', wrappedTopHoldings], + ['series', seriesHandler], ]); export interface CacheRepository { get(key: CacheKey): Promise>; set(key: CacheKey, value: T, ttlClass: TtlClass, provenance: Provenance): Promise; stale(key: CacheKey): boolean; + /** + * Watchlist/portfolio ownership: bumps refcount and seeds cache on first demand. + * Do NOT call from page-view paths (rotation, seasonality, condition) — use ensureInDemand. + */ subscribe(symbol: string, tickerKind: TickerKind): Promise; unsubscribe(symbol: string): Promise; + /** + * Request-path pin: mark symbol in demand + queue missing/stale data without inflating refcount. + * Safe to call on every Market Outlook / ticker context load. + */ + ensureInDemand(symbol: string, tickerKind: TickerKind): Promise; + /** Permanent system pin (rotation universe, SPY, VIX) — survives unsubscribe. */ + pinSystemSymbol(symbol: string, tickerKind: TickerKind): Promise; demandSet(): Promise; getMany(keys: CacheKey[]): Promise>; /** Delete a cache entry by key (or, for wildcard keys ending in `:*`, all matching entries). */ del(key: CacheKey): Promise; + /** Underlying DB for schedule TTL checks (queue only). */ + readonly db: DatabaseSync; } export class CacheRepositoryImpl implements CacheRepository { @@ -381,6 +633,7 @@ export class CacheRepositoryImpl implements CacheRepository { this._db = opts.db; this._scheduler = opts.scheduler; } + get db(): DatabaseSync { return this._db; } private handler(kind: string): KindHandler { const h = HANDLERS.get(kind); if (!h) throw new Error(`unknown cache kind: ${kind}`); @@ -391,7 +644,14 @@ export class CacheRepositoryImpl implements CacheRepository { const h = this.handler(kind); const row = h.read(this._db, id); const now = Date.now(); - const stale = h.isStale(row ? row.stalenessTs : null, now); + let stale = h.isStale(row ? row.stalenessTs : null, now); + // Incomplete symbol meta (null name) is always treated as stale for SWR re-fetch. + if (kind === 'symbol' && row) { + const meta = row.value as SymbolMeta; + if (!meta?.name && tsAgeMs(row.stalenessTs, now) > SYMBOL_META_INCOMPLETE_TTL_MS) { + stale = true; + } + } if (stale) { try { await this._scheduler.queue(key); } catch { /* background refresh; never block readers */ } } @@ -406,33 +666,94 @@ export class CacheRepositoryImpl implements CacheRepository { const h = this.handler(kind); if (h.ttlClass !== ttlClass) throw new Error(`ttlClass mismatch for kind '${kind}': expected ${h.ttlClass}, got ${ttlClass}`); h.write(this._db, id, value, provenance); + // Touch demand last_refreshed when a quote lands. + if (kind === 'quote') { + try { + this._db.prepare('UPDATE symbol_demand SET last_refreshed_at=? WHERE symbol=?').run(provenance.fetchedAt, id); + } catch { /* ignore */ } + } } stale(key: CacheKey): boolean { const { kind, id } = parseCacheKey(key); const h = this.handler(kind); const row = h.read(this._db, id); + if (kind === 'symbol' && row) { + const meta = row.value as SymbolMeta; + if (!meta?.name) return true; + } return h.isStale(row ? row.stalenessTs : null, Date.now()); } + + private ensureDemandRow(symbol: string, tickerKind: TickerKind): void { + this._db.prepare( + 'INSERT OR IGNORE INTO symbol_demand (symbol,refcount,ticker_kind,in_demand,last_refreshed_at,system_pin) VALUES (?,?,?,?,?,?)', + ).run(symbol, 0, tickerKind, 1, null, 0); + } + + /** Queue only what is missing or past TTL — no refcount change. */ + private async queueIfNeeded(symbol: string): Promise { + const d = this._db; + if (needsQuoteRefresh(d, symbol)) { + try { await this._scheduler.queue(`yfinance:quote:${symbol}`); } catch { /* ignore */ } + } + if (needsCandleRefresh(d, symbol)) { + try { await this._scheduler.queue(`yfinance:candles:${symbol}:1d`); } catch { /* ignore */ } + } + if (needsSymbolMetaRefresh(d, symbol)) { + try { await this._scheduler.queue(`yfinance:symbol:${symbol}`); } catch { /* ignore */ } + } + } + async subscribe(symbol: string, tickerKind: TickerKind): Promise { const d = this._db; d.prepare('INSERT OR IGNORE INTO symbol_demand (symbol,refcount,ticker_kind,in_demand,last_refreshed_at) VALUES (?,?,?,?,?)').run(symbol, 0, tickerKind, 1, null); const prev = d.prepare('SELECT refcount FROM symbol_demand WHERE symbol=?').get(symbol) as { refcount: number } | undefined; const before = prev?.refcount ?? 0; - d.prepare('UPDATE symbol_demand SET refcount = refcount + 1, in_demand = 1 WHERE symbol=?').run(symbol); + d.prepare('UPDATE symbol_demand SET refcount = refcount + 1, in_demand = 1, ticker_kind=? WHERE symbol=?').run(tickerKind, symbol); if (before === 0) { - // First demand: schedule initial cache population (slice 1: yfinance quote + symbol meta + candles + adjustments) - for (const k of [`yfinance:quote:${symbol}`, `yfinance:symbol:${symbol}`, `yfinance:candles:${symbol}:1d`, `yfinance:adjustments:${symbol}`, `yfinance:shortinterest:${symbol}`, `nasdaq:nasdaqShortinterest:${symbol}`]) { + // First user demand: full seed once (not every schedule tick). + for (const k of [ + `yfinance:quote:${symbol}`, + `yfinance:symbol:${symbol}`, + `yfinance:candles:${symbol}:1d`, + `yfinance:adjustments:${symbol}`, + `yfinance:shortinterest:${symbol}`, + `nasdaq:nasdaqShortinterest:${symbol}`, + ]) { try { await this._scheduler.queue(k); } catch { /* ignore */ } } + } else { + // Subsequent demand: re-check staleness and queue missing kinds (no refcount change). + await this.queueIfNeeded(symbol); } } + + async ensureInDemand(symbol: string, tickerKind: TickerKind): Promise { + this.ensureDemandRow(symbol, tickerKind); + this._db.prepare('UPDATE symbol_demand SET in_demand = 1, ticker_kind=COALESCE(ticker_kind, ?) WHERE symbol=?').run(tickerKind, symbol); + await this.queueIfNeeded(symbol); + } + + async pinSystemSymbol(symbol: string, tickerKind: TickerKind): Promise { + this.ensureDemandRow(symbol, tickerKind); + this._db.prepare( + 'UPDATE symbol_demand SET in_demand = 1, system_pin = 1, ticker_kind=? WHERE symbol=?', + ).run(tickerKind, symbol); + await this.queueIfNeeded(symbol); + } + async unsubscribe(symbol: string): Promise { const d = this._db; d.prepare('UPDATE symbol_demand SET refcount = MAX(refcount - 1, 0) WHERE symbol=?').run(symbol); - d.prepare('UPDATE symbol_demand SET in_demand = 0 WHERE symbol=? AND refcount = 0').run(symbol); + // Keep system pins in demand even when no user owns them. + d.prepare( + 'UPDATE symbol_demand SET in_demand = 0 WHERE symbol=? AND refcount = 0 AND COALESCE(system_pin, 0) = 0', + ).run(symbol); } async demandSet(): Promise { - return (this._db.prepare('SELECT symbol FROM symbol_demand WHERE refcount > 0 ORDER BY symbol').all() as Array<{ symbol: string }>).map((r) => r.symbol); + return (this._db.prepare( + 'SELECT symbol FROM symbol_demand WHERE in_demand = 1 OR COALESCE(system_pin, 0) = 1 ORDER BY symbol', + ).all() as Array<{ symbol: string }>).map((r) => r.symbol); } async getMany(keys: CacheKey[]): Promise> { return Promise.all(keys.map(async (key) => { @@ -440,6 +761,14 @@ export class CacheRepositoryImpl implements CacheRepository { return { key, value: e.value, isStale: e.isStale }; })); } + private clearKvCachePrefix(prefix: string): void { + // Invalidate all LRU cache entries matching the prefix + for (const key of kvReadCache.keys()) { + if (key.startsWith(prefix)) { + kvReadCache.delete(key); + } + } + } async del(key: CacheKey): Promise { const { source, kind, id } = parseCacheKey(key); const d = this._db; @@ -464,22 +793,41 @@ export class CacheRepositoryImpl implements CacheRepository { d.prepare('DELETE FROM options_chains WHERE symbol=? AND expiry=?').run(symbol, expiry); break; } - case 'expiry_dates': - d.prepare('DELETE FROM kv_cache WHERE key=?').run(`options_expiry:${id}`); + case 'expiry_dates': { + const cacheKey = `options_expiry:${id}`; + d.prepare('DELETE FROM kv_cache WHERE key=?').run(cacheKey); + kvReadCache.delete(cacheKey); break; - case 'shortinterest': - d.prepare('DELETE FROM kv_cache WHERE key=?').run(`yfinance:shortinterest:${id}`); + } + case 'shortinterest': { + const cacheKey = `yfinance:shortinterest:${id}`; + d.prepare('DELETE FROM kv_cache WHERE key=?').run(cacheKey); + kvReadCache.delete(cacheKey); break; - case 'nasdaqShortinterest': - d.prepare('DELETE FROM kv_cache WHERE key=?').run(`nasdaq:nasdaqShortinterest:${id}`); + } + case 'nasdaqShortinterest': { + const cacheKey = `nasdaq:nasdaqShortinterest:${id}`; + d.prepare('DELETE FROM kv_cache WHERE key=?').run(cacheKey); + kvReadCache.delete(cacheKey); break; + } case 'finraShortinterest': d.prepare('DELETE FROM finra_short_interest WHERE symbol=?').run(id); break; + case 'finraSiBiweekly': + d.prepare('DELETE FROM finra_short_interest_biweekly WHERE symbol=?').run(id); + break; + case 'dividendFundamentals': + d.prepare('DELETE FROM dividend_fundamentals WHERE symbol=?').run(id); + break; default: { // Unknown/wildcard kind (e.g. `x:cashtag:*`): best-effort delete from kv_cache via LIKE. const like = key.endsWith(':*') ? `${key.slice(0, -1)}%` : key; d.prepare('DELETE FROM kv_cache WHERE key LIKE ?').run(like); + // Invalidate LRU cache for this prefix + if (key.endsWith(':*')) { + this.clearKvCachePrefix(key.slice(0, -1)); + } } } void source; diff --git a/app/server/src/cache/LruCache.ts b/app/server/src/cache/LruCache.ts new file mode 100644 index 0000000..5b647cd --- /dev/null +++ b/app/server/src/cache/LruCache.ts @@ -0,0 +1,83 @@ +// Investor Flow — In-memory LRU cache for kv_cache reads. +// Reduces SQLite read overhead for frequently accessed keys (e.g., short interest, options expiry). + +export interface CacheEntry { + value: unknown; + stalenessTs: string; +} + +export interface LruCacheOptions { + /** Maximum number of entries to keep in memory */ + max?: number; + /** Time-to-live per entry in milliseconds. Default: 5 minutes */ + ttl?: number; +} + +/** + * Simple LRU cache implementation without external dependencies. + * Thread-safe for single-threaded Node.js, TTL-aware, and evicts least-recently-used entries when full. + */ +export class KvReadCache { + private readonly cache: Map; + private readonly maxSize: number; + private readonly ttlMs: number; + + constructor(opts: LruCacheOptions = {}) { + this.cache = new Map(); + this.maxSize = opts.max ?? 1000; + this.ttlMs = opts.ttl ?? 5 * 60 * 1000; // 5 minutes default + } + + /** Check cache for a key. Returns undefined if not found or expired. */ + get(key: string): CacheEntry | undefined { + const entry = this.cache.get(key); + if (!entry) return undefined; + + // Check TTL + if (Date.now() - entry.timestamp > this.ttlMs) { + this.cache.delete(key); + return undefined; + } + + // Move to end (most recently used) + this.cache.delete(key); + this.cache.set(key, entry); + return entry; + } + + /** Store a value in the cache. */ + set(key: string, value: unknown, stalenessTs: string): void { + // If key exists, delete it first to update position + if (this.cache.has(key)) { + this.cache.delete(key); + } else if (this.cache.size >= this.maxSize) { + // Evict least recently used (first entry in Map) + const firstKey = this.cache.keys().next().value; + if (firstKey !== undefined) { + this.cache.delete(firstKey); + } + } + + this.cache.set(key, { value, stalenessTs, timestamp: Date.now() }); + } + + /** Delete a specific key from the cache. */ + delete(key: string): boolean { + return this.cache.delete(key); + } + + /** Snapshot of all keys currently held (for prefix invalidation). */ + keys(): string[] { + return [...this.cache.keys()]; + } + + /** Clear all entries. */ + clear(): void { + this.cache.clear(); + } + + /** Get current size (number of entries). */ + get size(): number { + return this.cache.size; + } +} diff --git a/app/server/src/cache/__tests__/CacheRepository.test.ts b/app/server/src/cache/__tests__/CacheRepository.test.ts index 972ac81..b0e8886 100644 --- a/app/server/src/cache/__tests__/CacheRepository.test.ts +++ b/app/server/src/cache/__tests__/CacheRepository.test.ts @@ -31,7 +31,8 @@ test('get returns fresh cached value and does NOT schedule a refresh', async () test('get returns stale value (stale-while-revalidate) AND schedules refresh', async () => { const { repo, scheduler } = setup(); - await repo.set('yfinance:quote:NVDA', { symbol: 'NVDA', price: 131.1 } as Quote, 'live_quote', { fetchedAt: iso(-120_000), sourceKind: 'yfinance' }); + // 20 min old exceeds both session (1m) and after-hours (15m) quote TTL + await repo.set('yfinance:quote:NVDA', { symbol: 'NVDA', price: 131.1 } as Quote, 'live_quote', { fetchedAt: iso(-20 * 60_000), sourceKind: 'yfinance' }); scheduler.reset(); const e = await repo.get('yfinance:quote:NVDA'); assert.equal(e.isStale, true); @@ -47,7 +48,7 @@ test('get on never-cached key returns null + schedules refresh', async () => { assert.ok(scheduler.queued.includes('yfinance:quote:NVDA')); }); -test('subscribe bumps refcount and schedules fetches on FIRST demand only', async () => { +test('subscribe bumps refcount and schedules full seed on FIRST demand', async () => { const { repo, scheduler, db } = setup(); await repo.subscribe('NVDA', 'equity'); let r = db.prepare('SELECT refcount,in_demand FROM symbol_demand WHERE symbol=?').get('NVDA') as { refcount: number; in_demand: number }; @@ -56,10 +57,22 @@ test('subscribe bumps refcount and schedules fetches on FIRST demand only', asyn assert.ok(scheduler.queued.includes('yfinance:quote:NVDA')); assert.ok(scheduler.queued.includes('yfinance:symbol:NVDA')); scheduler.reset(); - await repo.subscribe('NVDA', 'equity'); // second subscribe: no new initial fetches + // Second subscribe: TTL-aware requeue of missing/stale kinds only (no adjustments seed). + await repo.subscribe('NVDA', 'equity'); r = db.prepare('SELECT refcount FROM symbol_demand WHERE symbol=?').get('NVDA') as { refcount: number; in_demand: number }; assert.equal(r.refcount, 2); - assert.equal(scheduler.queued.length, 0); + assert.ok(scheduler.queued.includes('yfinance:quote:NVDA')); + assert.ok(scheduler.queued.includes('yfinance:candles:NVDA:1d')); + assert.ok(scheduler.queued.includes('yfinance:symbol:NVDA')); // meta still missing + assert.ok(!scheduler.queued.includes('yfinance:adjustments:NVDA')); + // Once a quote is present, subsequent subscribe does not re-queue quote. + scheduler.reset(); + await repo.set('yfinance:quote:NVDA', { + symbol: 'NVDA', price: 100, bid: null, ask: null, change: null, changePercent: null, iv: null, + }, 'live_quote', { fetchedAt: new Date().toISOString(), sourceKind: 'yfinance' }); + await repo.subscribe('NVDA', 'equity'); + assert.ok(!scheduler.queued.includes('yfinance:quote:NVDA')); + assert.ok(scheduler.queued.includes('yfinance:candles:NVDA:1d')); // still no candles }); test('unsubscribe decrements; refcount->0 sets in_demand=0 (halts refresh)', async () => { @@ -76,13 +89,41 @@ test('unsubscribe decrements; refcount->0 sets in_demand=0 (halts refresh)', asy assert.equal(r.in_demand, 0); }); -test('demandSet lists only symbols with refcount>0', async () => { - const { repo } = setup(); +test('demandSet lists in_demand symbols; ensureInDemand does not inflate refcount', async () => { + const { repo, db, scheduler } = setup(); await repo.subscribe('NVDA', 'equity'); await repo.subscribe('AAPL', 'equity'); await repo.unsubscribe('AAPL'); // ->0 - const set = await repo.demandSet(); + let set = await repo.demandSet(); assert.deepEqual(set, ['NVDA']); + + scheduler.reset(); + await repo.ensureInDemand('SPY', 'etf'); + await repo.ensureInDemand('SPY', 'etf'); + await repo.ensureInDemand('SPY', 'etf'); + const r = db.prepare('SELECT refcount, in_demand FROM symbol_demand WHERE symbol=?').get('SPY') as { + refcount: number; in_demand: number; + }; + assert.equal(r.refcount, 0, 'ensureInDemand must not bump refcount'); + assert.equal(r.in_demand, 1); + set = await repo.demandSet(); + assert.ok(set.includes('SPY')); + assert.ok(set.includes('NVDA')); +}); + +test('pinSystemSymbol survives unsubscribe', async () => { + const { repo, db } = setup(); + await repo.pinSystemSymbol('XLK', 'etf'); + await repo.subscribe('XLK', 'etf'); + await repo.unsubscribe('XLK'); // refcount -> 0 + const r = db.prepare('SELECT refcount, in_demand, system_pin FROM symbol_demand WHERE symbol=?').get('XLK') as { + refcount: number; in_demand: number; system_pin: number; + }; + assert.equal(r.refcount, 0); + assert.equal(r.system_pin, 1); + assert.equal(r.in_demand, 1); + const set = await repo.demandSet(); + assert.ok(set.includes('XLK')); }); test('candles are permanent: not stale once present, stale when absent', async () => { @@ -117,7 +158,7 @@ test('symbol_meta stale after 7d, fresh within', async () => { test('stale() checks staleness WITHOUT scheduling a refresh', async () => { const { repo, scheduler } = setup(); - await repo.set('yfinance:quote:NVDA', { symbol: 'NVDA', price: 131 } as Quote, 'live_quote', { fetchedAt: iso(-120_000), sourceKind: 'yfinance' }); + await repo.set('yfinance:quote:NVDA', { symbol: 'NVDA', price: 131 } as Quote, 'live_quote', { fetchedAt: iso(-20 * 60_000), sourceKind: 'yfinance' }); scheduler.reset(); assert.equal(repo.stale('yfinance:quote:NVDA'), true); assert.equal(scheduler.queued.length, 0); diff --git a/app/server/src/db/__tests__/fund13fSync.test.ts b/app/server/src/db/__tests__/fund13fSync.test.ts new file mode 100644 index 0000000..21f15d9 --- /dev/null +++ b/app/server/src/db/__tests__/fund13fSync.test.ts @@ -0,0 +1,44 @@ +// per-fund-13f-ingest (sub-task 3) — 13F → Live-Book records sync. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { upsertTrackedFund, sync13FIntoRecords, liveBook, quarterEnd } from '../fundRepository.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +test('sync13FIntoRecords: idempotent; maps quarter → record, removes 13F-only quarter-end', () => { + const db = freshDb(); + const fund = upsertTrackedFund(db, { ci_key: '0002096493', fund_name: 'Alpine Fox Capital', manager_name: 'M' }); + db.prepare( + `INSERT INTO institution_filings (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at, accession) + VALUES ('0002096493','AFC','IREN','13F-HR',750000,30000000,'2026-Q1','2026-02-15','2026-02-15','A1'), + ('0002096493','AFC','CIFR','13F-HR',100000,1287000,'2026-Q1','2026-02-15','2026-02-15','A2')`, + ).run(); + + assert.equal(sync13FIntoRecords(db, fund.id), 2); + assert.equal(sync13FIntoRecords(db, fund.id), 0); // idempotent + + const book = liveBook(db, fund.id); + assert.equal(book.length, 2); + const irene = book.find((p) => p.symbol === 'IREN')!; + assert.equal(irene.source, '13f'); + assert.equal(irene.as_of, '2026-03-31'); // quarter-end + assert.equal(irene.shares, 750000); +}); + +test('quarterEnd helper', () => { + assert.equal(quarterEnd('2025-Q4'), '2025-12-31'); + assert.equal(quarterEnd('2026-Q2'), '2026-06-30'); + assert.equal(quarterEnd('2026-Q1'), '2026-03-31'); +}); \ No newline at end of file diff --git a/app/server/src/db/__tests__/fundRepository.test.ts b/app/server/src/db/__tests__/fundRepository.test.ts new file mode 100644 index 0000000..565cfa8 --- /dev/null +++ b/app/server/src/db/__tests__/fundRepository.test.ts @@ -0,0 +1,108 @@ +// Mirror Portfolio (M21) — fund registry + position-record / Live Book tests. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { + listTrackedFunds, getTrackedFund, upsertTrackedFund, + setFundEnabled, deleteTrackedFund, + insertPositionRecord, liveBook, listRecords, +} from '../fundRepository.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const SCHEMA_SQL = readFileSync(join(__dirname, '..', 'schema.sql'), 'utf8'); + +function freshDb(): DatabaseSync { + const db = new DatabaseSync(':memory:', { enableForeignKeyConstraints: true }); + db.exec(SCHEMA_SQL); + return db; +} + +function seedFund(db: DatabaseSync): string { + const f = upsertTrackedFund(db, { + ci_key: '0002096493', fund_name: 'Alpine Fox Capital LLC', + manager_name: 'Mike Alfred', x_handle: 'mikealfred', paywall_status: 'paywalled', + }); + return f.id; +} + +// ─── Registry (sub-task 1) ──────────────────────────────────────────────── +test('registry: upsert creates and is idempotent by CIK, then lists enabled first', () => { + const db = freshDb(); + const id1 = seedFund(db); + const id2 = seedFund(db); // same ci_key → same id (idempotent re-seed) + assert.equal(id1, id2); + + const funds = listTrackedFunds(db); + assert.equal(funds.length, 1); + assert.equal(funds[0].fund_name, 'Alpine Fox Capital LLC'); + assert.equal(funds[0].paywall_status, 'paywalled'); + + // Disabled funds are hidden by default. + const second = upsertTrackedFund(db, { + ci_key: '0000000000', fund_name: 'Second Fund', manager_name: 'Jane', enabled: false, + }); + assert.equal(listTrackedFunds(db).length, 1); + assert.equal(listTrackedFunds(db, { includeDisabled: true }).length, 2); + + const got = getTrackedFund(db, second.id); + assert.equal(got!.enabled, false); +}); + +test('registry: enable/disable toggle and delete', () => { + const db = freshDb(); + const id = seedFund(db); + assert.equal(setFundEnabled(db, id, false), true); + assert.equal(listTrackedFunds(db).length, 0); + assert.equal(deleteTrackedFund(db, id), true); + assert.equal(getTrackedFund(db, id), null); +}); + +// ─── Live Book / Mirror Target Rule (sub-task 2) ───────────────────────── +test('liveBook: most recent record per (fund, symbol) by as_of, any source', () => { + const db = freshDb(); + const fund = seedFund(db); + + // Capture dated Feb (beats the Dec 31 13F). + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 750000, as_of: '2026-02-10', source: 'capture' }); + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', value_usd: 30000000, as_of: '2025-12-31', source: '13f' }); + // A later 13F (Mar 31) dated after the Feb capture should beat it. + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 900000, value_usd: 27000000, as_of: '2026-03-31', source: '13f' }); + + const book = liveBook(db, fund); + assert.equal(book.length, 1); + assert.equal(book[0].symbol, 'IREN'); + assert.equal(book[0].shares, 900000); // Mar 31 13F is newest + assert.equal(book[0].source, '13f'); + + // A fresh capture after that wins again. + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 950000, cost_basis: 22.4, as_of: '2026-05-01', source: 'capture' }); + const book2 = liveBook(db, fund); + assert.equal(book2[0].source, 'capture'); + assert.equal(book2[0].shares, 950000); +}); + +test('live: multiple symbols independent recency', () => { + const db = freshDb(); + const fund = seedFund(db); + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 100, as_of: '2026-01-01', source: 'capture' }); + insertPositionRecord(db, { fund_id: fund, symbol: 'CIFR', shares: 200, as_of: '2026-01-02', source: 'capture' }); + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 300, as_of: '2026-01-03', source: 'claim' }); + const book = liveBook(db, fund); + assert.equal(book.length, 2); + const bySym = Object.fromEntries(book.map((p) => [p.symbol, p])); + assert.equal(bySym.IREN.shares, 300); + assert.equal(bySym.CIFR.shares, 200); +}); + +test('records: append-only timeline is never overwritten', () => { + const db = freshDb(); + const fund = seedFund(db); + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 100, as_of: '2026-01-01', source: 'capture' }); + insertPositionRecord(db, { fund_id: fund, symbol: 'IREN', shares: 150, as_of: '2026-01-02', source: 'capture' }); + const all = listRecords(db, fund); + assert.equal(all.length, 2); // history preserved +}); \ No newline at end of file diff --git a/app/server/src/db/__tests__/portfolioRepository.test.ts b/app/server/src/db/__tests__/portfolioRepository.test.ts index eda3635..fd4d816 100644 --- a/app/server/src/db/__tests__/portfolioRepository.test.ts +++ b/app/server/src/db/__tests__/portfolioRepository.test.ts @@ -267,6 +267,35 @@ test('removeHolding returns false when no matching holding exists', () => { db.close(); }); +test('addHolding reopens a soft-closed holding with the new lot (not VWAP against closed qty)', () => { + const db = freshDb(); + + addHolding(db, 'user_1', 'NVDA', 10, 50); + removeHolding(db, 'user_1', 'NVDA'); + assert.equal(listHoldings(db, 'user_1').length, 0); + + // Re-add after close — UI "Record holding" / Portfolio "+ Add Holding" path. + const created = addHolding(db, 'user_1', 'NVDA', 7, 110); + + assert.equal(created, true); + const holdings = listHoldings(db, 'user_1'); + assert.equal(holdings.length, 1); + assert.equal(holdings[0].symbol, 'NVDA'); + assert.equal(holdings[0].shares, 7); + assert.equal(holdings[0].avg_cost, 110); + + // Still a single row (unique owner+symbol), now open again. + const allRows = db.prepare( + 'SELECT * FROM portfolio_holdings WHERE owner_id = ? AND symbol = ?', + ).all('user_1', 'NVDA') as Array<{ status: string; qty: number; avg_cost: number }>; + assert.equal(allRows.length, 1); + assert.equal(allRows[0].status, 'open'); + assert.equal(allRows[0].qty, 7); + assert.equal(allRows[0].avg_cost, 110); + + db.close(); +}); + // --------------------------------------------------------------------------- // Tests — listHoldings // --------------------------------------------------------------------------- diff --git a/app/server/src/db/__tests__/watchlistRepository.test.ts b/app/server/src/db/__tests__/watchlistRepository.test.ts index 7ba80bc..02ce988 100644 --- a/app/server/src/db/__tests__/watchlistRepository.test.ts +++ b/app/server/src/db/__tests__/watchlistRepository.test.ts @@ -5,7 +5,16 @@ import { readFileSync } from 'node:fs'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { addSymbol, removeSymbol, listSymbols } from '../watchlistRepository.ts'; +import { + addSymbol, + removeSymbol, + listSymbols, + listWatchlists, + listSymbolsByWatchlist, + materializePortfolioWatchlist, + materializeClassificationWatchlists, +} from '../watchlistRepository.ts'; +import { addHolding, removeHolding } from '../portfolioRepository.ts'; // --------------------------------------------------------------------------- // Test helpers @@ -238,3 +247,67 @@ test('listSymbols handles multiple watchlists (default + named)', () => { db.close(); }); + +// --------------------------------------------------------------------------- +// Tests — system Portfolio watchlist +// --------------------------------------------------------------------------- + +test('materializePortfolioWatchlist creates empty Portfolio list with no holdings', () => { + const db = freshDb(); + + materializePortfolioWatchlist(db, 'user_1'); + + const lists = listWatchlists(db, 'user_1'); + const portfolio = lists.find((w) => w.kind === 'portfolio'); + assert.ok(portfolio); + assert.equal(portfolio!.name, 'Portfolio'); + assert.equal(portfolio!.symbol_count, 0); + + db.close(); +}); + +test('materializePortfolioWatchlist mirrors open holdings symbols', () => { + const db = freshDb(); + + addHolding(db, 'user_1', 'NVDA', 10, 50); + addHolding(db, 'user_1', 'AAPL', 5, 100); + materializePortfolioWatchlist(db, 'user_1'); + + const entries = listSymbolsByWatchlist(db, 'user_1', 'Portfolio'); + const symbols = entries.map((e) => e.symbol).sort(); + assert.deepEqual(symbols, ['AAPL', 'NVDA']); + + removeHolding(db, 'user_1', 'NVDA'); + materializePortfolioWatchlist(db, 'user_1'); + const afterClose = listSymbolsByWatchlist(db, 'user_1', 'Portfolio').map((e) => e.symbol); + assert.deepEqual(afterClose, ['AAPL']); + + db.close(); +}); + +test('listWatchlists always materializes Portfolio system list', () => { + const db = freshDb(); + + addHolding(db, 'user_1', 'TSLA', 2, 200); + const lists = listWatchlists(db, 'user_1'); + const portfolio = lists.find((w) => w.kind === 'portfolio' && w.name === 'Portfolio'); + assert.ok(portfolio); + assert.equal(portfolio!.symbol_count, 1); + + db.close(); +}); + +test('classification materialize does not wipe Portfolio system list', () => { + const db = freshDb(); + + addHolding(db, 'user_1', 'MSFT', 1, 300); + materializePortfolioWatchlist(db, 'user_1'); + // No user-list symbols → classification path prunes sector/theme lists only. + materializeClassificationWatchlists(db, 'user_1'); + + const portfolio = listWatchlists(db, 'user_1').find((w) => w.kind === 'portfolio'); + assert.ok(portfolio); + assert.equal(portfolio!.symbol_count, 1); + + db.close(); +}); diff --git a/app/server/src/db/alertSubscriptionRepository.ts b/app/server/src/db/alertSubscriptionRepository.ts index 2b41092..5435766 100644 --- a/app/server/src/db/alertSubscriptionRepository.ts +++ b/app/server/src/db/alertSubscriptionRepository.ts @@ -20,6 +20,32 @@ export interface CreateSubscriptionInput { params?: string; } +/** Catalog of alert types shown in the per-type toggle list. */ +export const ALERT_TYPE_CATALOG: Array<{ type: string; label: string; description: string }> = [ + { type: 'informed_buy', label: 'Insider Buy', description: 'Insider Form 4 purchases outside 10b5-1 plans.' }, + { type: 'informed_sell', label: 'Insider Sell', description: 'Insider Form 4 sales outside 10b5-1 plans.' }, + { type: 'new_13da', label: 'New 13D/G Filing', description: 'New institutional 13D/13G positions and 13F-HR filings.' }, + { type: 'rotation_incipient', label: 'Rotation Signal', description: 'Incipient sector rotation detected in the rotation engine.' }, + { type: 'regime_shift', label: 'Regime Shift', description: 'Market regime transition affecting portfolio risk.' }, + { type: 'conviction_unlock', label: 'Conviction Unlock', description: 'A new conviction tier unlocks from trading history.' }, + { type: 'thesis_broken', label: 'Thesis Broken', description: 'Thesis invalidation criteria met.' }, + { type: 'thesis_weakening', label: 'Thesis Weakening', description: 'Signals suggest a thesis may be weakening.' }, + { type: 'cluster_breach', label: 'Cluster Breach', description: 'Cluster exposure exceeds the recommended cap.' }, + { type: 'drawdown_halt', label: 'Drawdown Halt', description: 'Drawdown tolerance breached; circuit breaker paused entries.' }, + { type: 'asymmetry_warning', label: 'Asymmetry Warning', description: 'Portfolio reward-to-risk below threshold.' }, + { type: 'fund_capture', label: 'Fund Capture', description: 'A tracked fund posted a position update.' }, + { type: 'fund_13f', label: 'Tracked Fund 13F', description: 'A tracked fund filed a new 13F.' }, + { type: 'mirror_diff', label: 'Mirror Diff', description: 'The mirror target book changed materially.' }, + { type: 'vix_level', label: 'VIX Level', description: 'VIX moves into a new historical volatility band.' }, +]; + +export interface AlertTypeToggle { + type: string; + label: string; + description: string; + enabled: boolean; +} + function stmts(db: DatabaseSync) { return { insert: db.prepare( @@ -171,3 +197,103 @@ export function getEffectiveAlertSubscription( if (globalRow) return mapRow(globalRow); return null; } + +/** + * Seed a global default subscription (symbol NULL, watchlist NULL) for every + * catalog alert type for admin users (is_admin=1). Idempotent: only creates a + * default when the user has zero enabled subscriptions of that type, so a toggled + * preference (any ticker/list/global row) is never overwritten. Runs on every + * producer tick (cheap: no-op once seeded). + */ +export function seedAdminDefaultAlertSubscriptions(db: DatabaseSync): number { + let inserted = 0; + try { + const admins = db.prepare( + `SELECT id FROM users + WHERE is_admin = 1 + AND id NOT IN ('anonymous','system') + AND email NOT LIKE 'anonymous@%'`, + ).all() as Array<{ id: string }>; + if (admins.length === 0) return 0; + + const insert = db.prepare( + `INSERT OR IGNORE INTO alerts (id, owner_id, watchlist_id, symbol, alert_type, enabled, params, created_at) + VALUES (?, ?, NULL, NULL, ?, 1, '{}', ?)`, + ); + const hasEnabled = db.prepare( + `SELECT COUNT(*) AS n FROM alerts WHERE owner_id = ? AND alert_type = ? AND enabled = 1`, + ); + const now = new Date().toISOString(); + + for (const admin of admins) { + for (const { type } of ALERT_TYPE_CATALOG) { + const { n } = hasEnabled.get(admin.id, type) as { n: number }; + if (n > 0) continue; + const id = `auto-global-${admin.id}-${type}`; + const r = insert.run(id, admin.id, type, now); + if (Number(r.changes) > 0) inserted += 1; + } + } + } catch { + /* schema may lack tables in tests */ + } + return inserted; +} + +/** + * Per-type enable state for the current user. A type is "on" if any enabled + * subscription of that type exists (ticker-level, list-level, or global). + */ +export function listAlertTypeToggles(db: DatabaseSync, userId: string): AlertTypeToggle[] { + let rows: Array<{ alert_type: string; max_enabled: number }> = []; + try { + rows = db.prepare( + `SELECT alert_type, MAX(enabled) AS max_enabled + FROM alerts WHERE owner_id = ? + GROUP BY alert_type`, + ).all(userId) as Array<{ alert_type: string; max_enabled: number }>; + } catch { + /* schema may lack alerts table in tests */ + } + const enabledByType = new Map(); + for (const r of rows) enabledByType.set(r.alert_type, r.max_enabled === 1); + return ALERT_TYPE_CATALOG.map(({ type, label, description }) => ({ + type, + label, + description, + enabled: enabledByType.get(type) ?? false, + })); +} + +/** + * Toggle a whole alert type for a user. Flipping ON creates a global default + * subscription (symbol NULL, watchlist NULL) unless one exists; flipping OFF + * disables every subscription the user has of that type (ticker + list + + * global) so the toggle is authoritative. Never deletes rows. + */ +export function toggleAlertType( + db: DatabaseSync, + userId: string, + alertType: string, + enabled: boolean, +): AlertTypeToggle[] { + const catalog = ALERT_TYPE_CATALOG.find((c) => c.type === alertType); + if (!catalog) throw new Error(`Unknown alert type: ${alertType}`); + + if (enabled) { + const existing = db.prepare( + `SELECT id FROM alerts WHERE owner_id = ? AND alert_type = ? AND symbol IS NULL AND watchlist_id IS NULL LIMIT 1`, + ).get(userId, alertType) as { id: string } | undefined; + if (existing) { + db.prepare('UPDATE alerts SET enabled = 1 WHERE id = ?').run(existing.id); + } else { + createAlertSubscription(db, userId, { alertType }); + } + } else { + db.prepare( + 'UPDATE alerts SET enabled = 0 WHERE owner_id = ? AND alert_type = ?', + ).run(userId, alertType); + } + + return listAlertTypeToggles(db, userId); +} diff --git a/app/server/src/db/client.ts b/app/server/src/db/client.ts index 133381a..d5ff6a2 100644 --- a/app/server/src/db/client.ts +++ b/app/server/src/db/client.ts @@ -67,9 +67,24 @@ function runMigrations(db: DatabaseSync): void { `ALTER TABLE rotation_state ADD COLUMN signal_since TEXT`, `ALTER TABLE users ADD COLUMN modules TEXT NOT NULL DEFAULT '["research","settings"]'`, `UPDATE users SET modules='["research","execution","analytics","settings"]' WHERE modules='["research","settings"]'`, + // Workspace profile (adaptive management density) + `ALTER TABLE users ADD COLUMN experience_stage TEXT`, + `ALTER TABLE users ADD COLUMN goal TEXT`, + `ALTER TABLE users ADD COLUMN horizon TEXT`, + `ALTER TABLE users ADD COLUMN density TEXT NOT NULL DEFAULT 'focused'`, + `ALTER TABLE users ADD COLUMN jargon_comfort TEXT NOT NULL DEFAULT 'plain'`, + // Backfill density / experience_stage from legacy complexity (idempotent) + `UPDATE users SET density='standard', experience_stage=COALESCE(experience_stage, 'some_stocks') WHERE complexity='intermediate' AND (experience_stage IS NULL OR density='focused')`, + `UPDATE users SET density='full', experience_stage=COALESCE(experience_stage, 'active_self_directed') WHERE complexity='advanced' AND (experience_stage IS NULL OR density IN ('focused','standard'))`, + `UPDATE users SET experience_stage=COALESCE(experience_stage, 'never_invested') WHERE complexity='beginner' AND experience_stage IS NULL`, `ALTER TABLE watchlists ADD COLUMN kind TEXT NOT NULL DEFAULT 'user'`, `ALTER TABLE watchlists ADD COLUMN class_key TEXT`, `ALTER TABLE watchlists ADD COLUMN class_label TEXT`, + // System-pinned demand symbols (rotation universe / benchmarks) survive unsubscribe + `ALTER TABLE symbol_demand ADD COLUMN system_pin INTEGER NOT NULL DEFAULT 0`, + // Extended-hours quote fields (pre/post last vs RTH close) + `ALTER TABLE quotes ADD COLUMN session TEXT`, + `ALTER TABLE quotes ADD COLUMN regular_price REAL`, `DROP INDEX IF EXISTS idx_watchlists_owner_name`, `CREATE UNIQUE INDEX IF NOT EXISTS idx_watchlists_owner_kind_name ON watchlists(owner_id, kind, name)`, `CREATE TABLE IF NOT EXISTS rotation_state ( @@ -114,10 +129,89 @@ function runMigrations(db: DatabaseSync): void { acquired_at TEXT NOT NULL, note TEXT )`, + `CREATE INDEX IF NOT EXISTS idx_option_legs_owner_status + ON portfolio_option_legs(owner_id, status)`, `CREATE INDEX IF NOT EXISTS idx_option_legs_owner_status ON portfolio_option_legs(owner_id, status)`, // 13F put/call indicator for options positions. `ALTER TABLE institution_filings ADD COLUMN put_call TEXT`, + // Mirror Portfolio (M21) — tracked funds + position records. + `CREATE TABLE IF NOT EXISTS tracked_funds ( + id TEXT PRIMARY KEY, + ci_key TEXT NOT NULL, + fund_name TEXT NOT NULL, + manager_name TEXT NOT NULL, + x_handle TEXT, + paywall_status TEXT NOT NULL DEFAULT 'unknown', + enabled INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL, + updated_at TEXT + )`, + `CREATE UNIQUE INDEX IF NOT EXISTS uq_tracked_funds_cik ON tracked_funds(ci_key)`, + `CREATE UNIQUE INDEX IF NOT EXISTS uq_tracked_funds_name ON tracked_funds(fund_name)`, + `CREATE TABLE IF NOT EXISTS fund_position_records ( + id TEXT PRIMARY KEY, + fund_id TEXT NOT NULL REFERENCES tracked_funds(id) ON DELETE CASCADE, + symbol TEXT NOT NULL, + shares REAL, + value_usd REAL, + cost_basis REAL, + as_of TEXT NOT NULL, + source TEXT NOT NULL, + evidence_url TEXT, + created_at TEXT NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS idx_fund_records_fund_asof ON fund_position_records(fund_id, symbol, as_of DESC)`, + // Symbol Search Index (M22) — issuer CIK on symbols + lookup index. + `ALTER TABLE symbols ADD COLUMN cik TEXT`, + `CREATE INDEX IF NOT EXISTS idx_symbols_cik ON symbols(cik)`, + // Dealer Flow velocity snapshots (PR2) + `CREATE TABLE IF NOT EXISTS dealer_map_snapshots ( + symbol TEXT NOT NULL, + as_of_date TEXT NOT NULL, + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (symbol, as_of_date) + )`, + `CREATE INDEX IF NOT EXISTS idx_dealer_map_snapshots_symbol ON dealer_map_snapshots(symbol, as_of_date DESC)`, + `CREATE TABLE IF NOT EXISTS user_llm_endpoints ( + user_id TEXT PRIMARY KEY, + base_url TEXT NOT NULL, + api_key_enc TEXT, + model TEXT NOT NULL DEFAULT 'gpt-4o-mini', + key_hint TEXT, + updated_at TEXT NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS idx_alert_events_type_time ON alert_events(type, created_at DESC)`, + `CREATE INDEX IF NOT EXISTS idx_alerts_owner_type ON alerts(owner_id, alert_type)`, + `CREATE TABLE IF NOT EXISTS producer_run_log ( + id TEXT PRIMARY KEY, + producer TEXT NOT NULL, + frequency TEXT NOT NULL, + events_fired INTEGER NOT NULL DEFAULT 0, + inspected INTEGER NOT NULL DEFAULT 0, + duration_ms INTEGER NOT NULL DEFAULT 0, + error TEXT, + run_at TEXT NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS idx_producer_run_log_producer ON producer_run_log(producer, run_at DESC)`, + `CREATE INDEX IF NOT EXISTS idx_adapter_queue_status ON adapter_queue(status, last_attempt)`, + `CREATE INDEX IF NOT EXISTS idx_queue_errors_key_ts ON queue_errors(queue_key, ts DESC)`, + `CREATE TABLE IF NOT EXISTS notification_outbox ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + type TEXT NOT NULL, + severity TEXT NOT NULL, + title TEXT NOT NULL, + description TEXT NOT NULL, + symbol TEXT, + created_at TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', + attempt INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + sent_at TEXT + )`, + `CREATE INDEX IF NOT EXISTS idx_notification_outbox_pending ON notification_outbox(status, created_at)`, ]; for (const sql of migrations) { try { db.exec(sql); } catch { /* column already exists */ } @@ -233,7 +327,9 @@ function runMigrations(db: DatabaseSync): void { `); } catch { /* ignore */ } - // 10. FINRA config table for admin-configurable download URL. + // 10. FINRA config + tables + schedule update (daily + bi-monthly). + try { db.exec(`UPDATE queue_schedules SET interval_ms=86400000 WHERE source_kind='finra-bulk'`); } catch { /* table may not exist yet */ } + try { db.exec(`INSERT OR IGNORE INTO queue_schedules (source_kind, interval_ms, last_enqueued, next_enqueue) VALUES ('finra-si', 1209600000, NULL, ?)`); } catch { /* ignore */ } try { db.exec(` CREATE TABLE IF NOT EXISTS finra_config ( @@ -243,6 +339,242 @@ function runMigrations(db: DatabaseSync): void { ) `); } catch { /* ignore */ } + try { + db.exec(` + CREATE TABLE IF NOT EXISTS finra_short_interest_biweekly ( + symbol TEXT NOT NULL, settlement_date TEXT NOT NULL, issue_name TEXT, + exchange_code TEXT, market_class TEXT, current_short_position REAL, + previous_short_position REAL, avg_daily_volume REAL, days_to_cover REAL, + change_percent REAL, change_previous REAL, revision_flag TEXT, + source_file TEXT, ingested_at TEXT NOT NULL, + PRIMARY KEY (symbol, settlement_date) + ) + `); + } catch { /* ignore */ } + try { db.exec(`CREATE INDEX IF NOT EXISTS idx_finra_si_symbol ON finra_short_interest_biweekly(symbol)`); } catch { /* ignore */ } + try { db.exec(`CREATE INDEX IF NOT EXISTS idx_finra_si_settlement ON finra_short_interest_biweekly(settlement_date)`); } catch { /* ignore */ } + + // 11. Strategy lab + execution redesign tables and columns. + try { db.exec(`ALTER TABLE strategies ADD COLUMN preset_id TEXT`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE strategies ADD COLUMN scope TEXT NOT NULL DEFAULT 'single'`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE strategies ADD COLUMN components TEXT NOT NULL DEFAULT '[]'`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE strategy_presets ADD COLUMN example_tickers TEXT`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE strategies ADD COLUMN exit_profile TEXT`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE strategies ADD COLUMN risk_policy TEXT`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE trades ADD COLUMN realized_pnl REAL`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE trades ADD COLUMN allocation_plan_id TEXT`); } catch { /* ignore */ } + try { db.exec(`ALTER TABLE trades ADD COLUMN allocation_role TEXT`); } catch { /* ignore */ } + try { db.exec(`CREATE TABLE IF NOT EXISTS strategy_presets (id TEXT PRIMARY KEY, name TEXT NOT NULL, category TEXT NOT NULL, complexity TEXT NOT NULL, scope TEXT NOT NULL DEFAULT 'single', description TEXT NOT NULL, entry_rules TEXT NOT NULL DEFAULT '[]', exit_profile TEXT NOT NULL DEFAULT '[]', risk_policy TEXT NOT NULL DEFAULT '[]', instrument_type TEXT NOT NULL DEFAULT 'stock', portfolio_template TEXT, conviction_tier_default TEXT, requires_options_unlock INTEGER NOT NULL DEFAULT 0, forkable INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL)`); } catch { /* ignore */ } + try { db.exec(`CREATE TABLE IF NOT EXISTS allocation_plans (id TEXT PRIMARY KEY, owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, strategy_id TEXT NOT NULL REFERENCES strategies(id) ON DELETE CASCADE, name TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'building', target_alloc TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL)`); } catch { /* ignore */ } + try { db.exec(`CREATE TABLE IF NOT EXISTS dividend_fundamentals (symbol TEXT PRIMARY KEY, dividend_yield REAL, payout_ratio REAL, dividend_rate REAL, ex_dividend_date TEXT, trailing_eps REAL, forward_pe REAL, growth_streak_years INTEGER, observed_at TEXT NOT NULL)`); } catch { /* ignore */ } + try { db.exec(`CREATE INDEX IF NOT EXISTS idx_strategies_preset ON strategies(preset_id)`); } catch { /* ignore */ } + try { db.exec(`CREATE INDEX IF NOT EXISTS idx_alloc_owner ON allocation_plans(owner_id)`); } catch { /* ignore */ } + + // Seed strategy presets (idempotent). + const now = new Date().toISOString(); + const seedPresets = [ + { + id: 'dca_accumulation', name: 'DCA / Accumulation', category: 'growth', complexity: 'beginner', scope: 'single', + description: 'Build a position steadily over time by investing fixed amounts at regular intervals, regardless of price. Removes the stress of timing the market.', + entry_rules: JSON.stringify([{ conditions: ['Time-based: scheduled buy day', 'Cash available >= planned amount'] }]), + exit_profile: JSON.stringify([ + { action: 'Take profit', reason: 'Price reaches target return (e.g. +20%)', threshold: 0.2 }, + { action: 'Stop loss', reason: 'Price drops below cost basis by max drawdown tolerance', threshold: -0.15 }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.15, stopLossPct: 0.15, takeProfitPct: 0.2, maxDrawdownPct: 0.2 }), + instrument_type: 'stock', portfolio_template: null, conviction_tier_default: 'B', requires_options_unlock: 0, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'VOO', name: 'S&P 500 ETF', role: 'Core holding', allocationPct: 60, reason: 'Broad market low-cost ETF' }, + { symbol: 'IVV', name: 'S&P 500 ETF (iShares)', role: 'Core holding', allocationPct: 40, reason: 'Alternative broad market ETF' }, + ]), + }, + { + id: 'dividend_growth', name: 'Dividend Growth', category: 'income', complexity: 'beginner', scope: 'single', + description: 'Own companies that consistently pay and grow their dividends. Provides income while you hold and benefits from compounding over time.', + entry_rules: JSON.stringify([{ conditions: ['Dividend yield >= 2%', 'Payout ratio <= 60%', 'Dividend growth streak >= 5 years', 'Forward P/E <= 25'] }]), + exit_profile: JSON.stringify([ + { action: 'Sell', reason: 'Dividend cut or suspension', threshold: null }, + { action: 'Sell', reason: 'Payout ratio exceeds 80%', threshold: 0.8 }, + { action: 'Take profit', reason: 'Position grows beyond portfolio allocation target', threshold: null }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.1, stopLossPct: 0.2, maxDrawdownPct: 0.25 }), + instrument_type: 'stock', portfolio_template: null, conviction_tier_default: 'B', requires_options_unlock: 0, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'SCHD', name: 'Schwab US Dividend Equity ETF', role: 'Dividend growth', allocationPct: 40, reason: 'Strong dividend growth history' }, + { symbol: 'JNJ', name: 'Johnson & Johnson', role: 'Dividend growth', allocationPct: 20, reason: '60+ years of dividend growth' }, + { symbol: 'KO', name: 'Coca-Cola', role: 'Dividend growth', allocationPct: 20, reason: '60+ years of dividend growth' }, + { symbol: 'PG', name: 'Procter & Gamble', role: 'Dividend growth', allocationPct: 20, reason: '130+ years of dividends' }, + ]), + }, + { + id: 'core_satellite', name: 'Core + Satellite', category: 'portfolio', complexity: 'intermediate', scope: 'portfolio', + description: 'Put the majority in a broad market core (e.g. VOO, VTI) for stable growth, then add smaller satellite positions around your best ideas.', + entry_rules: JSON.stringify([{ conditions: ['Core: broad market ETF (VOO, VTI, IVV, etc.)', 'Satellites: individual stocks or sector ETFs', 'Each satellite <= target allocation %'] }]), + exit_profile: JSON.stringify([ + { action: 'Rebalance', reason: 'Satellite drifts beyond allocation band', threshold: 0.05 }, + { action: 'Sell satellite', reason: 'Thesis invalidated or position no longer conviction', threshold: null }, + { action: 'Harvest', reason: 'Satellite hits profit target', threshold: null }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.05, stopLossPct: 0.2 }), + instrument_type: 'stock', portfolio_template: JSON.stringify([ + { role: 'Core (broad market)', target: 70, min: 60, max: 80 }, + { role: 'Satellite - Growth', target: 15, min: 5, max: 25 }, + { role: 'Satellite - Value/Income', target: 15, min: 5, max: 25 }, + ]), conviction_tier_default: 'B', requires_options_unlock: 0, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'VOO', name: 'S&P 500 ETF', role: 'Core', allocationPct: 50, reason: 'Broad market core holding' }, + { symbol: 'VXUS', name: 'Total International Stock ETF', role: 'Core', allocationPct: 20, reason: 'International diversification' }, + { symbol: 'QQQ', name: 'Nasdaq 100 ETF', role: 'Satellite - Growth', allocationPct: 15, reason: 'Tech-heavy growth exposure' }, + { symbol: 'SCHD', name: 'Schwab US Dividend Equity ETF', role: 'Satellite - Income', allocationPct: 15, reason: 'Dividend growth complement' }, + ]), + }, + { + id: 'barbell_strategy', name: 'Bar-bell', category: 'portfolio', complexity: 'intermediate', scope: 'portfolio', + description: 'Keep most of your capital in very safe assets (broad market ETFs, bonds) and put a small portion in high-growth bets. Limits downside while preserving upside.', + entry_rules: JSON.stringify([{ conditions: ['Safe portion: broad market ETF or bond ETF', 'Growth portion: high-conviction growth stocks or sector bets', 'Growth allocation <= max risk budget'] }]), + exit_profile: JSON.stringify([ + { action: 'Rebalance', reason: 'Growth portion exceeds allocation band due to outsized gains', threshold: 0.05 }, + { action: 'Cut growth position', reason: 'Growth thesis breaks or stop loss triggered', threshold: null }, + { action: 'Trim', reason: 'Growth position exceeds max single-stock allocation', threshold: null }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.05, stopLossPct: 0.2, maxDrawdownPct: 0.15 }), + instrument_type: 'stock', portfolio_template: JSON.stringify([ + { role: 'Safe (broad market / bonds)', target: 80, min: 70, max: 90 }, + { role: 'Growth bets', target: 20, min: 10, max: 30 }, + ]), conviction_tier_default: 'B', requires_options_unlock: 0, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'VOO', name: 'S&P 500 ETF', role: 'Safe core', allocationPct: 50, reason: 'Broad market core' }, + { symbol: 'BND', name: 'Total Bond Market ETF', role: 'Safe bonds', allocationPct: 30, reason: 'Bond diversification' }, + { symbol: 'NVDA', name: 'NVIDIA Corporation', role: 'Growth bet', allocationPct: 10, reason: 'High-growth tech leader' }, + { symbol: 'AMZN', name: 'Amazon.com Inc', role: 'Growth bet', allocationPct: 10, reason: 'High-growth consumer/cloud leader' }, + ]), + }, + { + id: 'covered_call_wheel', name: 'Covered Call Income', category: 'income', complexity: 'advanced', scope: 'single', + description: 'Generate regular income from stocks you plan to hold by selling call options against them. Requires understanding of options basics.', + entry_rules: JSON.stringify([{ conditions: ['Own at least 100 shares of underlying', 'Stock is one you want to hold long-term', 'Option liquidity: bid-ask spread < $0.20', 'IV percentile < 70 (avoid selling into low vol)'] }]), + exit_profile: JSON.stringify([ + { action: 'Roll', reason: 'Stock approaches strike price before expiration', threshold: null }, + { action: 'Let shares get called away', reason: 'Stock closes above strike at expiration', threshold: null }, + { action: 'Close early', reason: 'Captured most of the premium, time value decayed', threshold: null }, + { action: 'Sell put', reason: 'After shares called away, sell cash-secured put to re-enter', threshold: null }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.1, stopLossPct: 0.2, maxDrawdownPct: 0.3 }), + instrument_type: 'option', portfolio_template: null, conviction_tier_default: 'C', requires_options_unlock: 1, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'AAPL', name: 'Apple Inc.', role: 'Covered call', allocationPct: 30, reason: 'High liquidity, strong options market' }, + { symbol: 'MSFT', name: 'Microsoft Corporation', role: 'Covered call', allocationPct: 30, reason: 'High liquidity, stable price' }, + { symbol: 'NVDA', name: 'NVIDIA Corporation', role: 'Covered call', allocationPct: 20, reason: 'Active options chain' }, + { symbol: 'AMZN', name: 'Amazon.com Inc', role: 'Covered call', allocationPct: 20, reason: 'High liquidity options market' }, + ]), + }, + { + id: 'concentrated_value', name: 'Concentrated Value', category: 'growth', complexity: 'advanced', scope: 'portfolio', + description: 'Own a small number of high-conviction positions you understand deeply. Buy at a significant discount to intrinsic value, hold through volatility, and let your best ideas drive returns. Based on the Alpine Fox Capital playbook: board-level research, contrarian entry, permanent capital mindset.', + entry_rules: JSON.stringify([{ conditions: ['Deep research completed — understand the business at board level', 'Price at significant discount to estimated intrinsic value (margin of safety)', 'Contrarian — buying when out of favor or misunderstood', 'Management quality and capital allocation verified', 'Thesis is multi-year, not a trade'] }]), + exit_profile: JSON.stringify([ + { action: 'Sell', reason: 'Thesis invalidated — core assumption broken', threshold: null }, + { action: 'Sell', reason: 'Full valuation reached with no further upside', threshold: null }, + { action: 'Hold', reason: '50%+ drawdown is expected; re-check thesis, do not panic-sell', threshold: null }, + ]), + risk_policy: JSON.stringify({ maxPositionPct: 0.40, stopLossPct: null, maxDrawdownPct: 0.50 }), + instrument_type: 'stock', portfolio_template: JSON.stringify([ + { role: 'Core thesis positions', target: 80, min: 60, max: 95 }, + { role: 'Satellite / ballast', target: 20, min: 5, max: 40 }, + ]), conviction_tier_default: 'A', requires_options_unlock: 0, forkable: 1, + example_tickers: JSON.stringify([ + { symbol: 'CIFR', name: 'Cipher Mining', role: 'Core thesis', allocationPct: 25, reason: 'Bitcoin miner pivoting to AI/HPC infrastructure — largest position' }, + { symbol: 'IREN', name: 'IREN Limited', role: 'Core thesis', allocationPct: 20, reason: 'Bitcoin miner to AI cloud, board-level insight, held since $2' }, + { symbol: 'BKKT', name: 'Bakkt Holdings', role: 'Core thesis', allocationPct: 20, reason: 'Newest high-conviction build — 865K shares added in Q2 2026' }, + { symbol: 'IBIT', name: 'iShares Bitcoin Trust', role: 'Core thesis', allocationPct: 15, reason: 'Bitcoin ETF exposure, shares + calls for capital efficiency' }, + { symbol: 'OPEN', name: 'Opendoor Technologies', role: 'Core thesis', allocationPct: 10, reason: 'Contrarian bet — massive QoQ add (+6,581%)' }, + { symbol: 'NVO', name: 'Novo Nordisk', role: 'Satellite / ballast', allocationPct: 10, reason: 'Consumer staple ballast, consistent grower' }, + ]), + }, + ]; + for (const p of seedPresets) { + try { + db.prepare(`INSERT OR IGNORE INTO strategy_presets (id, name, category, complexity, scope, description, entry_rules, exit_profile, risk_policy, instrument_type, portfolio_template, conviction_tier_default, requires_options_unlock, forkable, created_at, example_tickers) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`) + .run(p.id, p.name, p.category, p.complexity, p.scope, p.description, p.entry_rules, p.exit_profile, p.risk_policy, p.instrument_type, p.portfolio_template, p.conviction_tier_default, p.requires_options_unlock, p.forkable, now, p.example_tickers); + } catch { /* row may already exist */ } + // Backfill example_tickers for rows that already existed before this column was added. + try { + db.prepare(`UPDATE strategy_presets SET example_tickers=? WHERE id=? AND example_tickers IS NULL`) + .run(p.example_tickers, p.id); + } catch { /* ignore */ } + } + + // Dealer Flow Study Desk (educational practice setups + auto-grade). + try { + db.exec(` + CREATE TABLE IF NOT EXISTS dealer_study_setups ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + symbol TEXT NOT NULL, + hypothesis_id TEXT NOT NULL, + title TEXT NOT NULL, + bias TEXT NOT NULL, + map_reading TEXT NOT NULL, + study_question TEXT NOT NULL, + entry REAL NOT NULL, + invalidation REAL NOT NULL, + target REAL NOT NULL, + horizon_days INTEGER NOT NULL, + confidence REAL, + map_snapshot TEXT NOT NULL DEFAULT '{}', + status TEXT NOT NULL DEFAULT 'open', + grade TEXT, + score REAL, + grade_summary TEXT, + grade_path TEXT, + graded_at TEXT, + logged_at TEXT NOT NULL, + disclaimer TEXT NOT NULL + ) + `); + db.exec(`CREATE INDEX IF NOT EXISTS idx_dealer_study_owner ON dealer_study_setups(owner_id, logged_at DESC)`); + db.exec(`CREATE INDEX IF NOT EXISTS idx_dealer_study_symbol ON dealer_study_setups(owner_id, symbol)`); + } catch { /* ignore */ } + + // Mentor call ledger (local path-match grades for harvested sources). + try { + db.exec(` + CREATE TABLE IF NOT EXISTS mentor_sources ( + handle TEXT PRIMARY KEY, + enabled INTEGER NOT NULL DEFAULT 1, + notes TEXT, + updated_at TEXT NOT NULL + ) + `); + db.exec(` + CREATE TABLE IF NOT EXISTS mentor_calls ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + handle TEXT NOT NULL, + post_id TEXT NOT NULL, + posted_at TEXT, + symbol TEXT, + bias TEXT NOT NULL, + claim_type TEXT NOT NULL, + entry REAL NOT NULL, + invalidation REAL NOT NULL, + target REAL NOT NULL, + horizon_days INTEGER NOT NULL, + excerpt TEXT, + extract_confidence REAL, + status TEXT NOT NULL DEFAULT 'draft', + grade TEXT, + score REAL, + grade_summary TEXT, + grade_path TEXT, + graded_at TEXT, + logged_at TEXT NOT NULL, + UNIQUE(owner_id, handle, post_id) + ) + `); + db.exec(`CREATE INDEX IF NOT EXISTS idx_mentor_calls_owner ON mentor_calls(owner_id, logged_at DESC)`); + db.exec(`CREATE INDEX IF NOT EXISTS idx_mentor_calls_handle ON mentor_calls(owner_id, handle)`); + } catch { /* ignore */ } } let _db: DatabaseSync | null = null; @@ -257,10 +589,37 @@ export function db(): DatabaseSync { _db.prepare("INSERT OR IGNORE INTO users (id, email, pw_hash, created_at) VALUES (?, ?, ?, ?)") .run('anonymous', 'anonymous@investor-flow.local', '', new Date().toISOString()); } catch { /* table may not exist yet or already exists */ } + seedTrackedFunds(_db); } return _db; } +/** Seed the operator-curated tracked-fund registry (v1: Alpine Fox Capital LLC). */ +function seedTrackedFunds(db: DatabaseSync): void { + const alpine: (string | number | null)[] = [ + 'alpine-fox-capital', + '0002096493', // SEC CIK (filer key), Alpine Fox Capital LLC + 'Alpine Fox Capital LLC', + 'Mike Alfred', + 'mikealfred', // manager X handle (no @) + 'paywalled', // verified: subscribe-only capture posts return tombstones + 1, + new Date().toISOString(), + null, + ]; + try { + db.prepare(`INSERT OR IGNORE INTO tracked_funds + (id, ci_key, fund_name, manager_name, x_handle, paywall_status, enabled, created_at, updated_at) + VALUES (?,?,?,?,?,?,?,?,?)`).run(...alpine); + } catch { /* row may already exist */ } + // Keep the hero row (fund) in sync with changes we may want to ship. + try { + db.prepare(`UPDATE tracked_funds SET + x_handle='mikealfred', manager_name='Mike Alfred', paywall_status='paywalled', enabled=1 + WHERE id='alpine-fox-capital'`).run(); + } catch { /* ignore */ } +} + /** CLI entry: `node src/db/client.ts` initializes the on-disk database. */ const isMain = process.argv[1] ? resolve(process.argv[1]) === fileURLToPath(import.meta.url) : false; if (isMain) { diff --git a/app/server/src/db/fundRepository.ts b/app/server/src/db/fundRepository.ts new file mode 100644 index 0000000..f75c595 --- /dev/null +++ b/app/server/src/db/fundRepository.ts @@ -0,0 +1,271 @@ +// Investor Flow — Mirror Portfolio (M21) repository. +// +// Owns the operator-curated tracked-fund registry (`tracked_funds`) and the +// append-only position-record timeline (`fund_position_records`). The Live Book +// (Mirror Target Rule) = most recent record per (fund, symbol) by `as_of`, ANY +// source. Sources compete on recency, never authority — 13F (quarter-end as-of) +// and captures/claims (post-date) are just records in one timeline. + +import type { DatabaseSync } from 'node:sqlite'; +import { randomUUID } from 'node:crypto'; + +export interface TrackedFund { + id: string; + ci_key: string; + fund_name: string; + manager_name: string; + x_handle: string | null; + paywall_status: 'unknown' | 'open' | 'paywalled'; + enabled: boolean; + created_at: string; + updated_at: string | null; +} + +export type PositionSource = '13f' | 'capture' | 'claim'; + +export interface FundPositionRecord { + id: string; + fund_id: string; + symbol: string; + shares: number | null; + value_usd: number | null; + cost_basis: number | null; + as_of: string; + source: PositionSource; + evidence_url: string | null; + notes: string | null; + created_at: string; +} + +/** A position's current state in the fund's Live Book (one row per symbol). */ +export interface LiveBookPosition { + symbol: string; + shares: number | null; + value_usd: number | null; + cost_basis: number | null; + as_of: string; + source: PositionSource; + evidence_url: string | null; + notes: string | null; + post_text: string | null; +} + +// ─── Tracked fund registry ─────────────────────────────────────────────────── + +function rowToFund(row: Record): TrackedFund { + return { + id: row.id, + ci_key: row.ci_key, + fund_name: row.fund_name, + manager_name: row.manager_name, + x_handle: row.x_handle ?? null, + paywall_status: row.paywall_status ?? 'unknown', + enabled: Boolean(row.enabled), + created_at: row.created_at, + updated_at: row.updated_at ?? null, + }; +} + +/** List all tracked funds, enabled first then by name. */ +export function listTrackedFunds(db: DatabaseSync, opts: { includeDisabled?: boolean } = {}): TrackedFund[] { + const where = opts.includeDisabled ? '' : 'WHERE enabled = 1'; + const rows = db.prepare( + `SELECT * FROM tracked_funds ${where} ORDER BY enabled DESC, fund_name ASC`, + ).all() as Array>; + return rows.map(rowToFund); +} + +export function getTrackedFund(db: DatabaseSync, id: string): TrackedFund | null { + const row = db.prepare('SELECT * FROM tracked_funds WHERE id = ?').get(id) as Record | undefined; + return row ? rowToFund(row) : null; +} + +export interface NewTrackedFund { + ci_key: string; + fund_name: string; + manager_name: string; + x_handle?: string | null; + paywall_status?: 'unknown' | 'open' | 'paywalled'; + enabled?: boolean; +} + +/** Create or update a tracked fund (upsert by ci_key; idempotent re-seed). */ +export function upsertTrackedFund(db: DatabaseSync, input: NewTrackedFund): TrackedFund { + const now = new Date().toISOString(); + const existing = db.prepare('SELECT id FROM tracked_funds WHERE ci_key = ?').get(input.ci_key) as + | { id: string } + | undefined; + const id = existing?.id ?? randomUUID(); + const enabled = input.enabled ?? true; + const paywall = input.paywall_status ?? 'unknown'; + if (existing) { + db.prepare( + `UPDATE tracked_funds SET fund_name=?, manager_name=?, x_handle=?, paywall_status=?, + enabled=?, updated_at=? WHERE id=?`, + ).run(input.fund_name, input.manager_name, input.x_handle ?? null, paywall, enabled ? 1 : 0, now, id); + } else { + db.prepare( + `INSERT INTO tracked_funds (id, ci_key, fund_name, manager_name, x_handle, paywall_status, enabled, created_at) + VALUES (?,?,?,?,?,?,?,?)`, + ).run(id, input.ci_key, input.fund_name, input.manager_name, input.x_handle ?? null, paywall, enabled ? 1 : 0, now); + } + const fund = getTrackedFund(db, id); + if (!fund) throw new Error('fund insert failed'); + return fund; +} + +export function setFundEnabled(db: DatabaseSync, id: string, enabled: boolean): boolean { + const r = db.prepare('UPDATE tracked_funds SET enabled=?, updated_at=? WHERE id=?') + .run(enabled ? 1 : 0, new Date().toISOString(), id); + return r.changes > 0; +} + +export function deleteTrackedFund(db: DatabaseSync, id: string): boolean { + const r = db.prepare('DELETE FROM tracked_funds WHERE id = ?').run(id); + return r.changes > 0; +} + +// ─── Position records / Live Book ───────────────────────────────────────────── + +function rowToRecord(row: Record): FundPositionRecord { + return { + id: row.id, + fund_id: row.fund_id, + symbol: row.symbol, + shares: row.shares ?? null, + value_usd: row.value_usd ?? null, + cost_basis: row.cost_basis ?? null, + as_of: row.as_of, + source: row.source, + evidence_url: row.evidence_url ?? null, + notes: row.notes ?? null, + created_at: row.created_at, + }; +} + +export interface NewPositionRecord { + fund_id: string; + symbol: string; + shares?: number | null; + value_usd?: number | null; + cost_basis?: number | null; + as_of: string; + source: PositionSource; + evidence_url?: string | null; + id?: string; // optional stable id (e.g. derived from 13F accession + symbol) for idempotency +} + +/** Insert one record into the append-only timeline. */ +export function insertPositionRecord(db: DatabaseSync, input: NewPositionRecord): FundPositionRecord { + const id = input.id ?? randomUUID(); + const created_at = new Date().toISOString(); + db.prepare( + `INSERT INTO fund_position_records (id, fund_id, symbol, shares, value_usd, cost_basis, as_of, source, evidence_url, created_at) + VALUES (?,?,?,?,?,?,?,?,?,?)`, + ).run( + id, input.fund_id, input.symbol, + input.shares ?? null, input.value_usd ?? null, input.cost_basis ?? null, + input.as_of, input.source, input.evidence_url ?? null, created_at, + ); + const row = db.prepare('SELECT * FROM fund_position_records WHERE id = ?').get(id) as Record; + return rowToRecord(row); +} + +/** Full append-only timeline for a fund (oldest → newest). */ +export function listRecords(db: DatabaseSync, fundId: string): FundPositionRecord[] { + const rows = db.prepare( + `SELECT * FROM fund_position_records WHERE fund_id = ? ORDER BY as_of ASC, created_at ASC`, + ).all(fundId) as Array>; + return rows.map(rowToRecord); +} + +/** + * The Live Book — most recent record per (fund, symbol) by `as_of`, any source. + * The Mirror Target Rule: captures and 13F compete on recency, not authority. + */ +export function liveBook(db: DatabaseSync, fundId: string): LiveBookPosition[] { + const rows = db.prepare( + `SELECT fpr.*, xcp.body_text AS post_text, xcp.posted_at AS post_date + FROM fund_position_records fpr + LEFT JOIN x_cookie_posts xcp ON ( + fpr.source = 'capture' + AND xcp.post_id = SUBSTR(fpr.evidence_url, INSTR(fpr.evidence_url, '/status/') + 8) + ) + JOIN ( + SELECT symbol, MAX(as_of) AS max_as_of + FROM fund_position_records + WHERE fund_id = ? + GROUP BY symbol + ) latest ON latest.symbol = fpr.symbol AND latest.max_as_of = fpr.as_of + WHERE fpr.fund_id = ? + ORDER BY fpr.as_of DESC, fpr.symbol ASC`, + ).all(fundId, fundId) as Array>; + return rows.map((r) => ({ + symbol: r.symbol, + shares: r.shares ?? null, + value_usd: r.value_usd ?? null, + cost_basis: r.cost_basis ?? null, + as_of: r.as_of, + source: r.source as PositionSource, + evidence_url: r.evidence_url ?? null, + notes: r.notes ?? null, + post_text: r.post_text ?? null, + })); +} + +// ─── 13F sync (per-fund-13f-ingest, sub-task 3) ────────────────────────────── + +/** + * Populate the fund's Live-Book records from its 13F rows in `institution_filings`. + * Idempotent per (fund, symbol, accession): maps a 13F-HR quarter to a `13f` record + * with as_of = reported quarter-end (the calendar quarter-end date). + */ +export function sync13FIntoRecords(db: DatabaseSync, fundId: string): number { + const fund = db.prepare('SELECT ci_key FROM tracked_funds WHERE id = ?').get(fundId) as + | { ci_key: string } + | undefined; + if (!fund) return 0; + const rows = db.prepare( + `SELECT symbol, shares, value_usd, reported_quarter, accession + FROM institution_filings + WHERE filer_cik = ? AND form = '13F-HR' AND shares IS NOT NULL`, + ).all(fund.ci_key) as Array>; + + let inserted = 0; + let updated = 0; + for (const r of rows) { + const asOf = quarterEnd(r.reported_quarter as string); + const evidence = r.accession ? `sec|${r.accession}` : null; + if (evidence) { + // Converge: refresh the record when the 13F source row changed. + const upd = db.prepare( + `UPDATE fund_position_records SET shares=?, value_usd=?, as_of=? WHERE fund_id=? AND symbol=? AND source='13f' AND evidence_url=?`, + ).run(r.shares, r.value_usd, asOf, fundId, r.symbol, evidence); + if (upd.changes > 0) { + updated++; + continue; + } + } else { + const existing = db.prepare( + `SELECT id FROM fund_position_records WHERE fund_id=? AND symbol=? AND source='13f' AND evidence_url IS NULL AND as_of=?`, + ).get(fundId, r.symbol, asOf) as { id?: string } | undefined; + if (existing) continue; + } + insertPositionRecord(db, { + fund_id: fundId, symbol: r.symbol, shares: r.shares, value_usd: r.value_usd, + as_of: asOf, source: '13f', evidence_url: evidence, + }); + inserted++; + } + return inserted; +} + +/** Convert a `YYYY-Qn` quarter string to its calendar quarter-end date. */ +export function quarterEnd(quarter: string): string { + const m = /^(\d{4})-Q([1-4])$/.exec(quarter || ''); + if (!m) return (quarter || '').slice(0, 10); + const year = Number(m[1]); + const q = Number(m[2]); + const dd = { 1: '03-31', 2: '06-30', 3: '09-30', 4: '12-31' }[q]; + return `${year}-${dd}`; +} \ No newline at end of file diff --git a/app/server/src/db/portfolioRepository.ts b/app/server/src/db/portfolioRepository.ts index 1803e03..32fcac0 100644 --- a/app/server/src/db/portfolioRepository.ts +++ b/app/server/src/db/portfolioRepository.ts @@ -89,6 +89,16 @@ function stmts(db: DatabaseSync) { deleteByOwnerAndSymbol: db.prepare( `DELETE FROM portfolio_holdings WHERE owner_id = ? AND symbol = ?`, ), + + /** Reopen a soft-closed holding with a fresh lot (replace qty/avg_cost; do not VWAP closed history). */ + reopenHolding: db.prepare( + `UPDATE portfolio_holdings + SET qty = ?, + avg_cost = ?, + status = 'open', + acquired_at = ? + WHERE owner_id = ? AND symbol = ?`, + ), }; } @@ -97,15 +107,20 @@ function stmts(db: DatabaseSync) { // --------------------------------------------------------------------------- /** - * Add or accumulate a holding for the user. If a holding for (owner_id, symbol) - * already exists, the new shares are blended into the existing position — avg_cost - * is recalculated as a volume-weighted average. Idempotent: re-adding the same - * (userId, symbol) with identical values is a no-op on the row. + * Add or accumulate a holding for the user. + * + * - No existing row: insert open holding. + * - Existing open row: blend new shares via VWAP (volume-weighted avg cost). + * - Existing closed row: reopen with the new lot (replace qty/avg_cost; do not + * VWAP against soft-closed history). Soft-delete leaves a unique (owner, symbol) + * row, so re-add must flip status back to open or the position never reappears + * in listHoldings. * * Per ADR-0007, "holding" is neutral descriptive language — the system never * generates directional or trade-verb text. * - * @returns true if a new row was inserted, false if the existing row was updated. + * @returns true if a new open holding was created or reopened, false if an open + * holding was accumulated into. */ export function addHolding( db: DatabaseSync, @@ -123,6 +138,7 @@ export function addHolding( const s = stmts(db); const upper = symbol.toUpperCase(); + const now = new Date().toISOString(); // Check if a holding already exists for this user + symbol. const existing = s.selectByOwnerAndSymbol.all(userId, upper) as unknown as PortfolioRow[]; @@ -130,14 +146,22 @@ export function addHolding( if (existing.length === 0) { // New holding — insert with a generated id and current timestamp. const id = generateId(); - const now = new Date().toISOString(); s.insertHolding.run(id, userId, upper, shares, avgCost, now); return true; } - // Existing holding — route through insertHolding so ON CONFLICT does VWAP accumulation. + const row = existing[0]; + + // Soft-closed rows still occupy the unique (owner_id, symbol) slot. Re-adding must + // reopen them with the new lot. Accumulating via ON CONFLICT left status='closed', + // so the API returned success while listHoldings (status='open' only) stayed empty. + if (row.status === 'closed') { + s.reopenHolding.run(shares, avgCost, now, userId, upper); + return true; + } + + // Existing open holding — route through insertHolding so ON CONFLICT does VWAP accumulation. const id = generateId(); - const now = new Date().toISOString(); s.insertHolding.run(id, userId, upper, shares, avgCost, now); // We already knew the row existed (existing.length > 0), so this is an accumulation. diff --git a/app/server/src/db/schema.sql b/app/server/src/db/schema.sql index 2f30627..a207c92 100644 --- a/app/server/src/db/schema.sql +++ b/app/server/src/db/schema.sql @@ -16,10 +16,16 @@ CREATE TABLE IF NOT EXISTS users ( is_admin INTEGER NOT NULL DEFAULT 0, -- 1 = operator (admin CLI/route access, Slice 25); never user-editable oauth_subject TEXT, -- nullable; social OAuth link (slice 2) oauth_provider TEXT, -- 'github' | 'google' | NULL - complexity TEXT NOT NULL DEFAULT 'beginner', -- beginner|intermediate|advanced + complexity TEXT NOT NULL DEFAULT 'beginner', -- beginner|intermediate|advanced (engine map) risk_tolerance TEXT NOT NULL DEFAULT 'moderate', drawdown_tolerance REAL, -- onboarding max-drawdown % (beginner -20, etc.) convexity_posture TEXT NOT NULL DEFAULT 'off', -- off|covered_income|cash_secured_entry|insurance_sleeve|leaps_conviction + -- Workspace profile (interview): management UI density + context, not a course enrollment + experience_stage TEXT, -- never_invested|some_stocks|active_self_directed + goal TEXT, -- grow|income|protect + horizon TEXT, -- short|medium|long + density TEXT NOT NULL DEFAULT 'focused', -- focused|standard|full + jargon_comfort TEXT NOT NULL DEFAULT 'plain', -- plain|mixed|technical backup_codes_hashed TEXT, -- JSON array of scrypt-hashed backup codes (slice 2) status TEXT NOT NULL DEFAULT 'active', -- active|pending_approval|rejected (user provisioning) modules TEXT NOT NULL DEFAULT '["research","settings"]', -- JSON array of allowed module keys @@ -50,6 +56,7 @@ CREATE TABLE IF NOT EXISTS symbols ( exchange TEXT, ticker_kind TEXT NOT NULL, -- equity|crypto|etf|index (gates module applicability) peers TEXT, -- JSON array + cik TEXT, -- issuer CIK (the company the ticker represents), SEC company_tickers.json updated_at TEXT ); @@ -64,14 +71,16 @@ CREATE TABLE IF NOT EXISTS price_candles ( ); CREATE TABLE IF NOT EXISTS quotes ( - symbol TEXT PRIMARY KEY, - price REAL, - bid REAL, - ask REAL, - change REAL, - change_pct REAL, - iv REAL, -- implied vol when available - observed_at TEXT NOT NULL + symbol TEXT PRIMARY KEY, + price REAL, + bid REAL, + ask REAL, + change REAL, + change_pct REAL, + iv REAL, -- implied vol when available + session TEXT, -- regular | pre | post (which print `price` is) + regular_price REAL, -- RTH last when price is extended + observed_at TEXT NOT NULL ); CREATE TABLE IF NOT EXISTS kv_cache ( @@ -92,6 +101,26 @@ CREATE TABLE IF NOT EXISTS options_chains ( PRIMARY KEY (symbol, expiry, strike, type, ts) ); +-- Daily dealer heat snapshots for velocity (GEX/VEX map day-over-day). +CREATE TABLE IF NOT EXISTS dealer_map_snapshots ( + symbol TEXT NOT NULL, + as_of_date TEXT NOT NULL, -- YYYY-MM-DD + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (symbol, as_of_date) +); +CREATE INDEX IF NOT EXISTS idx_dealer_map_snapshots_symbol ON dealer_map_snapshots(symbol, as_of_date DESC); + +-- Per-user OpenAI-compatible LLM endpoint (base_url + encrypted key + model). +CREATE TABLE IF NOT EXISTS user_llm_endpoints ( + user_id TEXT PRIMARY KEY, + base_url TEXT NOT NULL, + api_key_enc TEXT, + model TEXT NOT NULL DEFAULT 'gpt-4o-mini', + key_hint TEXT, + updated_at TEXT NOT NULL +); + CREATE TABLE IF NOT EXISTS filings ( accession TEXT PRIMARY KEY, -- SEC accession number form TEXT NOT NULL, -- 10-K|10-Q|8-K|13F-HR|13D|13G|4|... @@ -224,6 +253,11 @@ CREATE TABLE IF NOT EXISTS adapter_queue ( scheduled_for TEXT -- ISO timestamp; job won't run before this ); +-- Drain loop scans pending/backoff every 2s; this keeps the scan on an index +-- instead of a full-table filter over 1k+ rows. +CREATE INDEX IF NOT EXISTS idx_adapter_queue_status + ON adapter_queue(status, last_attempt); + CREATE TABLE IF NOT EXISTS queue_errors ( id INTEGER PRIMARY KEY AUTOINCREMENT, queue_key TEXT NOT NULL, @@ -233,6 +267,10 @@ CREATE TABLE IF NOT EXISTS queue_errors ( ts TEXT NOT NULL ); +-- queue_errors retention: prune per-key (keep newest N) + global age purge. +CREATE INDEX IF NOT EXISTS idx_queue_errors_key_ts + ON queue_errors(queue_key, ts DESC); + CREATE TABLE IF NOT EXISTS queue_schedules ( source_kind TEXT PRIMARY KEY, interval_ms INTEGER NOT NULL, @@ -250,7 +288,8 @@ CREATE TABLE IF NOT EXISTS symbol_demand ( refcount INTEGER NOT NULL DEFAULT 0, ticker_kind TEXT NOT NULL, in_demand INTEGER NOT NULL DEFAULT 1, -- gate flag for the adapter queue - last_refreshed_at TEXT + last_refreshed_at TEXT, + system_pin INTEGER NOT NULL DEFAULT 0 -- 1 = rotation/benchmark; survives unsubscribe ); -- ===== Tier C — Per-user (ownerId NOT NULL) ===== @@ -263,7 +302,8 @@ CREATE TABLE IF NOT EXISTS watchlists ( sort_order INTEGER NOT NULL DEFAULT 0, kind TEXT NOT NULL DEFAULT 'user', -- 'user' | 'sector' | 'thematic' | 'style' | 'region' class_key TEXT, -- ETF symbol for derived lists (XLK, SMH...); NULL for user - class_label TEXT -- human label synced with market outlook; NULL for user + class_label TEXT, -- human label synced with market outlook; NULL for user + UNIQUE(owner_id, kind, name) ); CREATE TABLE IF NOT EXISTS portfolio_holdings ( @@ -317,18 +357,10 @@ CREATE TABLE IF NOT EXISTS trades ( stop_loss REAL, targets TEXT, -- JSON [{price, reason}] justifications TEXT, -- JSON {entryWhy, stopWhy, targetWhy} - checklist_completed INTEGER NOT NULL DEFAULT 0 -); - -CREATE TABLE IF NOT EXISTS strategies ( - id TEXT PRIMARY KEY, - owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, - name TEXT NOT NULL, - regime_gate TEXT NOT NULL, -- JSON: gate config - setup TEXT NOT NULL, -- JSON: entry/exit checklist w/ params - risk_policy TEXT NOT NULL, -- JSON: sizing rules - created_at TEXT NOT NULL, - is_public INTEGER NOT NULL DEFAULT 0 -- shareable read-only (later phase) + checklist_completed INTEGER NOT NULL DEFAULT 0, + realized_pnl REAL, -- net PnL at close, feeds conviction unlock + allocation_plan_id TEXT, -- for portfolio-scope strategies + allocation_role TEXT -- safe | convex | core | satellite | null ); CREATE TABLE IF NOT EXISTS screener_filters ( @@ -571,6 +603,10 @@ CREATE TABLE IF NOT EXISTS alert_events ( CREATE INDEX IF NOT EXISTS idx_alert_events_user ON alert_events(user_id, created_at DESC); CREATE INDEX IF NOT EXISTS idx_alert_events_dedup ON alert_events(dedup_key); CREATE INDEX IF NOT EXISTS idx_alert_events_throttle ON alert_events(throttle_key); +CREATE INDEX IF NOT EXISTS idx_alert_events_type_time ON alert_events(type, created_at DESC); + +-- Per-type subscription lookup (toggle list + producer fan-out). +CREATE INDEX IF NOT EXISTS idx_alerts_owner_type ON alerts(owner_id, alert_type); -- ===== Alert comparison state (post-fetch change detection) ===== CREATE TABLE IF NOT EXISTS alert_comparison_state ( @@ -581,6 +617,38 @@ CREATE TABLE IF NOT EXISTS alert_comparison_state ( PRIMARY KEY (symbol, alert_type) ); +-- ===== Producer run log (alert observability) ===== +CREATE TABLE IF NOT EXISTS producer_run_log ( + id TEXT PRIMARY KEY, + producer TEXT NOT NULL, + frequency TEXT NOT NULL, -- realtime|per-fetch|batched + events_fired INTEGER NOT NULL DEFAULT 0, + inspected INTEGER NOT NULL DEFAULT 0, + duration_ms INTEGER NOT NULL DEFAULT 0, + error TEXT, + run_at TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_producer_run_log_producer + ON producer_run_log(producer, run_at DESC); + +-- ===== Email notification outbox (SMTP decoupled from alert loop) ===== +CREATE TABLE IF NOT EXISTS notification_outbox ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL, + type TEXT NOT NULL, + severity TEXT NOT NULL, + title TEXT NOT NULL, + description TEXT NOT NULL, + symbol TEXT, + created_at TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', -- pending|sent|failed + attempt INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + sent_at TEXT +); +CREATE INDEX IF NOT EXISTS idx_notification_outbox_pending + ON notification_outbox(status, created_at); + -- ===== SMTP config for email alerts ===== CREATE TABLE IF NOT EXISTS smtp_config ( id TEXT PRIMARY KEY DEFAULT 'singleton', @@ -595,16 +663,61 @@ CREATE TABLE IF NOT EXISTS smtp_config ( updated_at TEXT NOT NULL ); --- ===== Slice 12 — Strategy Lab + Backtest ===== +-- ===== Slice 12 — Strategy Lab + Backtest + Execution Redesign ===== CREATE TABLE IF NOT EXISTS strategies ( id TEXT PRIMARY KEY, owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, name TEXT NOT NULL, - components TEXT NOT NULL DEFAULT '[]', -- JSON array of StrategyComponent + preset_id TEXT, -- traces origin if forked from a preset + scope TEXT NOT NULL DEFAULT 'single', -- single | portfolio + components TEXT NOT NULL DEFAULT '[]', -- JSON StrategyComponent + exit_profile TEXT, -- JSON exit rules (customized from preset) + risk_policy TEXT, -- JSON sizing rules unlocked INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL ); +CREATE TABLE IF NOT EXISTS strategy_presets ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + category TEXT NOT NULL, -- income | growth | hedging | speculation | portfolio + complexity TEXT NOT NULL, -- beginner | intermediate | advanced + scope TEXT NOT NULL DEFAULT 'single', + description TEXT NOT NULL, + entry_rules TEXT NOT NULL DEFAULT '[]', + exit_profile TEXT NOT NULL DEFAULT '[]', + risk_policy TEXT NOT NULL DEFAULT '[]', + instrument_type TEXT NOT NULL DEFAULT 'stock', -- stock | option | both + portfolio_template TEXT, -- JSON role-slot allocation (portfolio scope) + conviction_tier_default TEXT, + requires_options_unlock INTEGER NOT NULL DEFAULT 0, + forkable INTEGER NOT NULL DEFAULT 1, + example_tickers TEXT, -- JSON array of {symbol, name, role, allocationPct, reason} + created_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS allocation_plans ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + strategy_id TEXT NOT NULL REFERENCES strategies(id) ON DELETE CASCADE, + name TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'building', -- building | active | rebalancing | closed + target_alloc TEXT NOT NULL DEFAULT '{}', -- JSON role->pct + created_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS dividend_fundamentals ( + symbol TEXT PRIMARY KEY, + dividend_yield REAL, + payout_ratio REAL, + dividend_rate REAL, + ex_dividend_date TEXT, + trailing_eps REAL, + forward_pe REAL, + growth_streak_years INTEGER, + observed_at TEXT NOT NULL +); + CREATE TABLE IF NOT EXISTS backtest_results ( id TEXT PRIMARY KEY, strategy_id TEXT NOT NULL REFERENCES strategies(id) ON DELETE CASCADE, @@ -619,6 +732,71 @@ CREATE TABLE IF NOT EXISTS backtest_results ( CREATE INDEX IF NOT EXISTS idx_strategies_owner ON strategies(owner_id); CREATE INDEX IF NOT EXISTS idx_backtest_strategy ON backtest_results(strategy_id); +-- ===== Dealer Flow Study Desk (educational practice setups + auto-grade) ===== +CREATE TABLE IF NOT EXISTS dealer_study_setups ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + symbol TEXT NOT NULL, + hypothesis_id TEXT NOT NULL, + title TEXT NOT NULL, + bias TEXT NOT NULL, + map_reading TEXT NOT NULL, + study_question TEXT NOT NULL, + entry REAL NOT NULL, + invalidation REAL NOT NULL, + target REAL NOT NULL, + horizon_days INTEGER NOT NULL, + confidence REAL, + map_snapshot TEXT NOT NULL DEFAULT '{}', + status TEXT NOT NULL DEFAULT 'open', + grade TEXT, + score REAL, + grade_summary TEXT, + grade_path TEXT, + graded_at TEXT, + logged_at TEXT NOT NULL, + disclaimer TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_dealer_study_owner ON dealer_study_setups(owner_id, logged_at DESC); +CREATE INDEX IF NOT EXISTS idx_dealer_study_symbol ON dealer_study_setups(owner_id, symbol); + +-- Mentor call ledger (local; grade path-match like Study Desk) +CREATE TABLE IF NOT EXISTS mentor_sources ( + handle TEXT PRIMARY KEY, + enabled INTEGER NOT NULL DEFAULT 1, + notes TEXT, + updated_at TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS mentor_calls ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + handle TEXT NOT NULL, + post_id TEXT NOT NULL, + posted_at TEXT, + symbol TEXT, + bias TEXT NOT NULL, + claim_type TEXT NOT NULL, + entry REAL NOT NULL, + invalidation REAL NOT NULL, + target REAL NOT NULL, + horizon_days INTEGER NOT NULL, + excerpt TEXT, + extract_confidence REAL, + status TEXT NOT NULL DEFAULT 'draft', + grade TEXT, + score REAL, + grade_summary TEXT, + grade_path TEXT, + graded_at TEXT, + logged_at TEXT NOT NULL, + UNIQUE(owner_id, handle, post_id) +); + +CREATE INDEX IF NOT EXISTS idx_mentor_calls_owner ON mentor_calls(owner_id, logged_at DESC); +CREATE INDEX IF NOT EXISTS idx_mentor_calls_handle ON mentor_calls(owner_id, handle); + -- ===== Slice 13 — Saved Filter Sets ===== CREATE TABLE IF NOT EXISTS saved_filters ( id TEXT PRIMARY KEY, @@ -677,12 +855,27 @@ CREATE TABLE IF NOT EXISTS rotation_state ( checked_at TEXT ); --- ===== Slice 22 — FINRA Short Interest (bulk ingest, twice-monthly) ===== +-- User-added custom ETF symbols for the rotation table (kind='custom') +CREATE TABLE IF NOT EXISTS rotation_custom_symbols ( + id TEXT PRIMARY KEY, + owner_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + symbol TEXT NOT NULL, + name TEXT, + grp TEXT NOT NULL DEFAULT 'Custom', + created_at TEXT NOT NULL +); +CREATE UNIQUE INDEX IF NOT EXISTS uq_rotation_custom_owner_symbol + ON rotation_custom_symbols(owner_id, symbol); + +-- ===== Slice 22 — FINRA Short Interest (two datasets) ===== +-- finra_short_interest: DAILY short-sale VOLUME (flow, CNMSshvol) +-- finra_short_interest_biweekly: BI-MONTHLY outstanding short INTEREST (stock, shrt) +-- FINRA publishes both separately. Volume ≠ interest (FINRA warns against conflating). CREATE TABLE IF NOT EXISTS finra_short_interest ( symbol TEXT NOT NULL, - settlement_date TEXT NOT NULL, -- YYYY-MM-DD of the short interest report - exchange TEXT, -- NASDAQ | NYSE | NYSE_AMERICAN | NYSE_ARCA | ALL - short_volume REAL, -- shares sold short + settlement_date TEXT NOT NULL, -- YYYY-MM-DD of the daily volume report + exchange TEXT, -- B,Q,N,ALL... + short_volume REAL, -- shares sold short that day short_exempt REAL, -- short exempt volume total_volume REAL, -- short + exempt avg_daily_vol REAL, -- average daily volume (for days-to-cover) @@ -692,8 +885,28 @@ CREATE TABLE IF NOT EXISTS finra_short_interest ( PRIMARY KEY (symbol, settlement_date, exchange) ); +CREATE TABLE IF NOT EXISTS finra_short_interest_biweekly ( + symbol TEXT NOT NULL, + settlement_date TEXT NOT NULL, -- YYYY-MM-DD of bi-monthly report + issue_name TEXT, + exchange_code TEXT, -- R, etc. mapped from issuerServicesGroupExchangeCode + market_class TEXT, -- NNM|NYSE|ARCA|OTC|... + current_short_position REAL, -- outstanding short shares + previous_short_position REAL, + avg_daily_volume REAL, + days_to_cover REAL, + change_percent REAL, + change_previous REAL, + revision_flag TEXT, + source_file TEXT, + ingested_at TEXT NOT NULL, + PRIMARY KEY (symbol, settlement_date) +); + CREATE INDEX IF NOT EXISTS idx_finra_symbol ON finra_short_interest(symbol); CREATE INDEX IF NOT EXISTS idx_finra_settlement ON finra_short_interest(settlement_date); +CREATE INDEX IF NOT EXISTS idx_finra_si_symbol ON finra_short_interest_biweekly(symbol); +CREATE INDEX IF NOT EXISTS idx_finra_si_settlement ON finra_short_interest_biweekly(settlement_date); -- FINRA download configuration (admin-configurable URL). CREATE TABLE IF NOT EXISTS finra_config ( @@ -701,3 +914,51 @@ CREATE TABLE IF NOT EXISTS finra_config ( base_url TEXT NOT NULL DEFAULT 'https://cdn.finra.org/equity/regsho/daily', updated_at TEXT NOT NULL ); + +-- ─── Mirror Portfolio (M21) ──────────────────────────────────────────────── +-- Operator-curated list of tracked funds. Fund (CIK) and manager (X handle) are +-- SEPARATE entities; one fund → one manager for v1. Fund data is shared, public +-- (ADR-0004) — no owner_id on these rows. +CREATE TABLE IF NOT EXISTS tracked_funds ( + id TEXT PRIMARY KEY, + ci_key TEXT NOT NULL, -- SEC CIK, zero-padded key form (e.g. '0002096493') + fund_name TEXT NOT NULL, + manager_name TEXT NOT NULL, + x_handle TEXT, -- manager X handle WITHOUT '@' + paywall_status TEXT NOT NULL DEFAULT 'unknown', -- unknown | open | paywalled + enabled INTEGER NOT NULL DEFAULT 1, -- 1 = eligible for ingestion + mirror + created_at TEXT NOT NULL, + updated_at TEXT +); +CREATE UNIQUE INDEX IF NOT EXISTS uq_tracked_funds_cik ON tracked_funds(ci_key); +CREATE UNIQUE INDEX IF NOT EXISTS uq_tracked_funds_name ON tracked_funds(fund_name); + +-- Append-only timeline of position records for a tracked fund. Sources compete on +-- recency (Mirror Target Rule), never authority. Live Book = most recent record +-- per (fund, symbol) by as_of, any source. +CREATE TABLE IF NOT EXISTS fund_position_records ( + id TEXT PRIMARY KEY, + fund_id TEXT NOT NULL REFERENCES tracked_funds(id) ON DELETE CASCADE, + symbol TEXT NOT NULL, + shares REAL, -- total shares (capture) OR delta (claim) + value_usd REAL, + cost_basis REAL, -- avg cost per share (capture moat) + as_of TEXT NOT NULL, -- date the record is "as of" (post date for X, quarter-end for 13F) + source TEXT NOT NULL, -- 13f | capture | claim + evidence_url TEXT, -- tweet URL for capture/claim + notes TEXT, -- e.g. 'book_reset' for position timeline markers + created_at TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_fund_records_fund_asof ON fund_position_records(fund_id, symbol, as_of DESC); + +-- Time-varying shares outstanding / public float per stock (sourced from Yahoo Finance). +-- Separate table so it doesn't pollute the static symbols metadata and supports history. +CREATE TABLE IF NOT EXISTS stock_float ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + symbol TEXT NOT NULL, + shares_outstanding REAL, -- total shares outstanding + float_shares REAL, -- publicly tradable float + as_of TEXT NOT NULL DEFAULT (datetime('now')), -- when this snapshot was taken + UNIQUE(symbol, as_of) +); +CREATE INDEX IF NOT EXISTS idx_stock_float_symbol ON stock_float(symbol); diff --git a/app/server/src/db/watchlistRepository.ts b/app/server/src/db/watchlistRepository.ts index f32d6df..220418d 100644 --- a/app/server/src/db/watchlistRepository.ts +++ b/app/server/src/db/watchlistRepository.ts @@ -257,6 +257,7 @@ export function listSymbols( export function listWatchlists(db: DatabaseSync, userId: string): WatchlistMeta[] { materializeClassificationWatchlists(db, userId); + materializePortfolioWatchlist(db, userId); const s = stmts(db); const rows = s.selectAllByOwner.all(userId) as unknown as WatchlistRow[]; return rows.map((row) => { @@ -406,6 +407,44 @@ function generateId(): string { return randomUUID(); } +/** + * Materialize the system "Portfolio" watchlist from open portfolio_holdings. + * Read-only in the UI; membership is driven by the position book, not manual edits. + * Always present (may be empty) so the list is discoverable. + */ +export function materializePortfolioWatchlist(db: DatabaseSync, userId: string): void { + const s = stmts(db); + const openRows = db.prepare( + `SELECT symbol FROM portfolio_holdings WHERE owner_id = ? AND status = 'open' ORDER BY acquired_at DESC`, + ).all(userId) as Array<{ symbol: string }>; + + const symbols = openRows + .map((r) => (r.symbol ?? '').toUpperCase()) + .filter((sym) => sym.length > 0); + // Dedupe while preserving order (first occurrence wins). + const seen = new Set(); + const unique = symbols.filter((sym) => { + if (seen.has(sym)) return false; + seen.add(sym); + return true; + }); + + const now = new Date().toISOString(); + const id = generateId(); + // sort_order -1 keeps Portfolio above sector/theme lists within the system group. + s.upsert.run( + id, + userId, + 'Portfolio', + JSON.stringify(unique), + now, + -1, + 'portfolio', + 'PORTFOLIO', + 'Portfolio holdings', + ); +} + /** Materialize system watchlists (sector, thematic, style, region) from user's symbol metadata. Idempotent; auto-prunes empty lists. */ export function materializeClassificationWatchlists(db: DatabaseSync, userId: string): void { const s = stmts(db); @@ -420,8 +459,10 @@ export function materializeClassificationWatchlists(db: DatabaseSync, userId: st } } if (symbolSet.size === 0) { - // Prune all system lists for this user. - db.prepare(`DELETE FROM watchlists WHERE owner_id = ? AND kind != 'user'`).run(userId); + // Prune classification lists only — never wipe the system Portfolio list. + db.prepare( + `DELETE FROM watchlists WHERE owner_id = ? AND kind IN ('sector', 'thematic', 'style', 'region')`, + ).run(userId); return; } diff --git a/app/server/src/derisking/DeriskingEngine.ts b/app/server/src/derisking/DeriskingEngine.ts index a167453..510651b 100644 --- a/app/server/src/derisking/DeriskingEngine.ts +++ b/app/server/src/derisking/DeriskingEngine.ts @@ -13,14 +13,15 @@ export type DeriskingStrategyId = | 'thesis_based_partial' | 'option_protected_collar' | 'regime_cut' - | 'correlation_driven'; + | 'correlation_driven' + | 'dividend_at_risk'; export interface DeriskingStrategy { id: DeriskingStrategyId; name: string; description: string; alfredFraming: string; - requiresOptionsSleeve: boolean; + requiresOptionsSleeve?: boolean; // true if this strategy requires options unlock } export const DERISKING_STRATEGIES: DeriskingStrategy[] = [ @@ -29,21 +30,18 @@ export const DERISKING_STRATEGIES: DeriskingStrategy[] = [ name: 'Scale Out at Targets', description: 'Reduce partial position size as price reaches predefined profit targets. This adds flexibility — some position remains if the thesis continues to play out.', alfredFraming: 'Winners have flexibility. Scaling out at targets locks in some gains while leaving room for further upside.', - requiresOptionsSleeve: false, }, { id: 'stop_trail_ema21_50', name: 'Stop-Trail EMA 21/50', description: 'Trail a stop above the 21-day or 50-day EMA. This adapts the stop to volatility, allowing the position to breathe while protecting gains.', alfredFraming: 'Trailing stops adapt to changing conditions. A static stop ignores new information.', - requiresOptionsSleeve: false, }, { id: 'thesis_based_partial', name: 'Thesis-Based Partial Reduction', description: 'If the thesis is weakening but not fully invalidated, consider reducing position size rather than fully exiting. This acknowledges uncertainty without abandoning the thesis entirely.', alfredFraming: 'When the thesis weakens but is not broken, flexibility means adjusting size — not all or nothing.', - requiresOptionsSleeve: false, }, { id: 'option_protected_collar', @@ -57,14 +55,18 @@ export const DERISKING_STRATEGIES: DeriskingStrategy[] = [ name: 'Regime-Cut Reduction', description: 'When the market regime shifts to trending-down, consider reducing exposure across the portfolio. This is a macro-level derisking response to changed conditions.', alfredFraming: 'When the tide turns, reduce exposure. Fighting a regime shift is a common way winners become losers.', - requiresOptionsSleeve: false, }, { id: 'correlation_driven', name: 'Correlation-Driven Reduction', description: 'When multiple positions become highly correlated, consider reducing the most correlated ones. Concentrated correlation creates hidden portfolio risk.', alfredFraming: 'When everything moves together, you do not have diversification — you have one bet wearing many masks.', - requiresOptionsSleeve: false, + }, + { + id: 'dividend_at_risk', + name: 'Dividend at Risk', + description: 'When a dividend-paying stock shows signs of a potential cut — high payout ratio, declining yield, or skipped ex-date — consider reducing exposure before the cut is announced.', + alfredFraming: 'Dividend cuts usually arrive after the warning signs are visible. The market often prices a cut before it is announced.', }, ]; @@ -80,8 +82,10 @@ export interface DeriskingInput { thesisStatus?: 'intact' | 'weakening' | 'broken'; currentRegime?: 'trending-up' | 'trending-down' | 'range-bound'; portfolioCorrelation?: number; // 0-1, how correlated this position is with others - optionsUnlockState?: number; // 0-4, from ConvexityGate profitTargets?: number[]; // price levels + dividendYield?: number; + payoutRatio?: number; + optionsUnlockState?: number; // 0 = locked, 3+ = unlocked for option strategies } export interface DeriskingSuggestion { @@ -152,13 +156,14 @@ function evaluateStrategy( } case 'option_protected_collar': { - const hasOptionsSleeve = (input.optionsUnlockState ?? 0) >= 3; + // Gated by options unlock state (0 = locked, 3+ = unlocked) + const hasOptionsUnlock = (input.optionsUnlockState ?? 0) >= 3; return { strategy: strat, - applicable: hasOptionsSleeve && inProfit, - reason: hasOptionsSleeve - ? 'Options insurance sleeve is unlocked. A collar may define downside risk while partially financing protection.' - : 'Options insurance sleeve is not unlocked. Collar strategy requires the insurance sleeve tier.', + applicable: inProfit && hasOptionsUnlock, + reason: !hasOptionsUnlock + ? 'Option collar requires options unlock level 3 or higher.' + : 'A collar may define downside risk while partially financing protection.', details: strat.alfredFraming, }; } @@ -186,5 +191,21 @@ function evaluateStrategy( details: strat.alfredFraming, }; } + + case 'dividend_at_risk': { + const dy = input.dividendYield; + const pr = input.payoutRatio; + const atRisk = (dy !== undefined && dy < 0.005) || (pr !== undefined && pr > 0.95); + return { + strategy: strat, + applicable: atRisk, + reason: atRisk + ? (dy !== undefined && dy < 0.005 + ? `Dividend yield is very low (${(dy * 100).toFixed(2)}%). This may indicate a recent cut or unsustainable payout.` + : `Payout ratio is very high (${((pr ?? 0) * 100).toFixed(0)}%). Earnings may not cover the dividend.`) + : 'Dividend fundamentals appear healthy.', + details: strat.alfredFraming, + }; + } } } diff --git a/app/server/src/index.ts b/app/server/src/index.ts index d584221..0c579bd 100644 --- a/app/server/src/index.ts +++ b/app/server/src/index.ts @@ -5,24 +5,39 @@ import { fetchRequestHandler } from '@trpc/server/adapters/fetch'; import { db } from './db/client.ts'; import { createCacheRepository, type SourceKind } from './cache/CacheRepository.ts'; import { YFinanceAdapter } from './adapters/YFinanceAdapter.ts'; +import { OptionsAdapter } from './adapters/OptionsAdapter.ts'; import { NasdaqAdapter } from './adapters/NasdaqAdapter.ts'; import { FinraBulkAdapter } from './adapters/FinraBulkAdapter.ts'; +import { FinraShortInterestAdapter } from './adapters/FinraShortInterestAdapter.ts'; import { SecFetchAdapter } from './adapters/SecFetchAdapter.ts'; +import { SecCompanyTickersAdapter } from './adapters/SecCompanyTickersAdapter.ts'; import { SecLintAdapter } from './adapters/SecLintAdapter.ts'; import { XCookieAdapter } from './adapters/XCookieAdapter.ts'; +import { FredAdapterImpl } from './macro/FredAdapter.ts'; import type { SourceFetch } from './adapters/SourceAdapter.ts'; +import { composeYFinanceWithOptions } from './options/OptionsChainRouter.ts'; import cryptoMod from './lib/crypto.ts'; import { AdapterQueue } from './queue/AdapterQueue.ts'; +import { seedCuratedCusips } from './services/cusipRegistry.ts'; +import { seedAdminDefaultAlertSubscriptions } from './db/alertSubscriptionRepository.ts'; import { makeCreateContext } from './trpc/context.ts'; import { appRouter } from './trpc/router.ts'; +import type { Alert } from './alerts/AlertEngine.ts'; const PORT = Number(process.env.PORT ?? 3001); const database = db(); +const yfinanceAdapter = composeYFinanceWithOptions( + new YFinanceAdapter({ db: database }) as unknown as SourceFetch, + new OptionsAdapter() as unknown as SourceFetch, +); const adapters = new Map([ - ['yfinance' as const, new YFinanceAdapter() as unknown as SourceFetch], + ['yfinance' as const, yfinanceAdapter], ['nasdaq' as const, new NasdaqAdapter() as unknown as SourceFetch], ['finra-bulk' as const, new FinraBulkAdapter(database) as unknown as SourceFetch], + ['finra-si' as const, new FinraShortInterestAdapter(database) as unknown as SourceFetch], ['sec-fetch' as const, new SecFetchAdapter(database) as unknown as SourceFetch], + ['sec-sc-fetch' as const, new SecFetchAdapter(database) as unknown as SourceFetch], + ['sec-tickers' as const, new SecCompanyTickersAdapter(database) as unknown as SourceFetch], ['sec-lint-holders' as const, new SecLintAdapter(() => database, 'sec-lint-holders') as unknown as SourceFetch], ['sec-lint-insiders' as const, new SecLintAdapter(() => database, 'sec-lint-insiders') as unknown as SourceFetch], ]); @@ -45,13 +60,59 @@ let xAdapter: XCookieAdapter | null = null; console.log('[investor-flow] X adapter registered (credentials configured)'); })(); +// Register FredAdapter when a FRED API key is configured (encrypted in x_credentials). +// FRED series are warmed by the queue schedule (fred_macro tier), never on the +// market.condition request path. +(function initFredAdapter() { + const row = database.prepare('SELECT fred_api_key_enc FROM x_credentials WHERE id=?').get('singleton') as { fred_api_key_enc?: string | null } | undefined; + if (!row?.fred_api_key_enc) return; + let apiKey: string; + try { apiKey = cryptoMod.decrypt(row.fred_api_key_enc); } catch { return; } + if (!apiKey) return; + adapters.set('fred' as const, new FredAdapterImpl(apiKey) as unknown as SourceFetch); + console.log('[investor-flow] FRED adapter registered (series warm-up via queue schedule)'); +})(); + const queue = new AdapterQueue({ db: database, adapters }); const cache = createCacheRepository({ db: database, scheduler: queue }); queue.cache = cache; // break the cache<->scheduler cycle -// Seed default schedules (noop if already seeded) +// Seed / migrate tiered schedules (safe every boot) queue.seedDefaultSchedules(); +// Offline CUSIP registry → kv_cache so institutional alerts never depend solely on EFTS. +try { + const seeded = seedCuratedCusips(database); + if (seeded > 0) console.log(`[investor-flow] seeded ${seeded} curated sec:cusip cache entries`); +} catch (e) { + console.error('[investor-flow] cusip seed failed', e); +} + +// Admin default alert subscriptions: every catalog alert type ON for is_admin=1 +// (idempotent; respects any toggled preference). Seeded now, not just on the +// 10-min batch producer tick, so operators receive alerts immediately. +try { + const seeded = seedAdminDefaultAlertSubscriptions(database); + if (seeded > 0) console.log(`[investor-flow] seeded ${seeded} admin default alert subscription(s)`); +} catch (e) { + console.error('[investor-flow] admin alert seed failed', e); +} + +// Demand hygiene: junk test symbols + inflated refcounts from page-view subscribe spam +try { + const hygiene = queue.cleanupDemandHygiene(); + if (hygiene.removedJunk || hygiene.cappedRefcounts) { + console.log(`[investor-flow] demand hygiene: removedJunk=${hygiene.removedJunk} cappedRefcounts=${hygiene.cappedRefcounts}`); + } +} catch (e) { + console.error('[investor-flow] demand hygiene failed', e); +} + +// Pin rotation universe + SPY + VIX (no refcount inflation) +queue.pinSystemUniverse().then(() => { + console.log('[investor-flow] system pins (rotation universe) ready'); +}).catch((e) => console.error('[investor-flow] pinSystemUniverse failed', e)); + // Startup recovery: any job left 'in_flight' was interrupted by a restart/crash. // Reset to 'pending' so the drain loop reprocesses it. const recovered = database.prepare("UPDATE adapter_queue SET status='pending', error=NULL, retry_count=0 WHERE status='in_flight'").run(); @@ -70,28 +131,49 @@ const SCHEDULE_MS = 30_000; const scheduleTimer = setInterval(() => { queue.enqueueDueSchedules().catch((e) => console.error('[schedule error]', e)); }, SCHEDULE_MS); scheduleTimer.unref(); -// Per-fetch alert check: run alongside the schedule cycle for critical alert types. +// Realtime + per-fetch alert check: run alongside the schedule cycle for +// low-latency alert types (e.g. VIX band crosses). Enqueues to the email +// outbox only; SMTP is drained on its own timer so it never blocks this loop. const perFetchAlertTimer = setInterval(async () => { try { - const alerts = await runProducers(database, 'per-fetch'); + const alerts: Alert[] = [ + ...(await runProducers(database, 'realtime')), + ...(await runProducers(database, 'per-fetch')), + ]; if (alerts.length > 0) { - console.log(`[alert] ${alerts.length} per-fetch alert(s) created`); - const { sendAlertEmail } = await import('./services/emailAlertService.ts'); - alerts.forEach((a) => sendAlertEmail(database, a).catch(() => {})); + console.log(`[alert] ${alerts.length} realtime alert(s) created`); + const { enqueueAlertEmail } = await import('./services/emailAlertService.ts'); + alerts.forEach((a) => enqueueAlertEmail(database, a)); } } catch (e) { - console.error('[alert] per-fetch check failed:', e); + console.error('[alert] realtime check failed:', e); } }, SCHEDULE_MS); perFetchAlertTimer.unref(); // Register alert producers. -import { registerProducer, runProducers } from './alerts/producers/index.ts'; +import { registerProducer, runProducers, registerMirrorProducers } from './alerts/producers/index.ts'; import { informedBuyProducer, informedSellProducer } from './alerts/producers/insiderProducer.ts'; -import { new13daProducer } from './alerts/producers/new13daProducer.ts'; +import { new13daProducer, new13fFilingProducer } from './alerts/producers/new13daProducer.ts'; +import { vixLevelProducer } from './alerts/producers/vixLevelProducer.ts'; +import { rotationIncipientProducer, regimeShiftProducer } from './alerts/producers/rotationProducer.ts'; +import { convictionUnlockProducer } from './alerts/producers/unlockProducer.ts'; +import { thesisBrokenProducer, thesisWeakeningProducer } from './alerts/producers/thesisProducer.ts'; +import { clusterBreachProducer, drawdownHaltProducer, asymmetryWarningProducer } from './alerts/producers/portfolioRiskProducer.ts'; registerProducer(informedBuyProducer); registerProducer(informedSellProducer); registerProducer(new13daProducer); +registerProducer(new13fFilingProducer); +registerProducer(vixLevelProducer); +registerProducer(rotationIncipientProducer); +registerProducer(regimeShiftProducer); +registerProducer(convictionUnlockProducer); +registerProducer(thesisBrokenProducer); +registerProducer(thesisWeakeningProducer); +registerProducer(clusterBreachProducer); +registerProducer(drawdownHaltProducer); +registerProducer(asymmetryWarningProducer); +registerMirrorProducers(); // fund_capture / fund_13f / mirror_diff (lazy, offline-safe) // Batched alert check: run every 10 minutes for non-critical producers. const ALERT_BATCH_MS = 10 * 60 * 1000; @@ -99,17 +181,45 @@ const alertBatchTimer = setInterval(async () => { try { const alerts = await runProducers(database, 'batched'); if (alerts.length > 0) console.log(`[alert] ${alerts.length} batch alert(s) created`); - // Send email for each new alert. - const { sendAlertEmail } = await import('./services/emailAlertService.ts'); - for (const alert of alerts) { - sendAlertEmail(database, alert).catch((e) => console.error('[alert:email] send error:', e)); - } + // Enqueue email for each new alert (SMTP drained separately). + const { enqueueAlertEmail } = await import('./services/emailAlertService.ts'); + for (const alert of alerts) enqueueAlertEmail(database, alert); } catch (e) { console.error('[alert] batch check failed:', e); } }, ALERT_BATCH_MS); alertBatchTimer.unref(); +// Email outbox drain: decouples SMTP latency/errors from the alert loops. +const OUTBOX_DRAIN_MS = 60_000; +const outboxDrainTimer = setInterval(async () => { + try { + const { drainEmailOutbox } = await import('./services/emailAlertService.ts'); + const sent = await drainEmailOutbox(database); + if (sent > 0) console.log(`[alert:email] drained ${sent} outbox email(s)`); + } catch (e) { + console.error('[alert:email] outbox drain failed:', e); + } +}, OUTBOX_DRAIN_MS); +outboxDrainTimer.unref(); + +// Daily queue housekeeping: purge stale 'done' rows and age out queue_errors. +// Both tables previously grew unbounded (live DB had 819k error rows). +const DAILY_HOUSEKEEP_MS = 24 * 60 * 60 * 1000; +const housekeepTimer = setInterval(() => { + try { + const cleared = queue.clearDone(); + const pruned = queue.pruneQueueErrors(); + if (cleared > 0 || pruned > 0) { + console.log(`[queue] housekeeping: cleared ${cleared} done jobs, pruned ${pruned} old queue_errors`); + } + } catch (e) { + console.error('[queue] daily housekeeping failed:', e); + } +}, DAILY_HOUSEKEEP_MS); +housekeepTimer.unref(); + + function readBody(req: IncomingMessage): Promise { return new Promise((resolve, reject) => { let data = ''; diff --git a/app/server/src/llm/__tests__/openaiCompatible.test.ts b/app/server/src/llm/__tests__/openaiCompatible.test.ts new file mode 100644 index 0000000..67e9511 --- /dev/null +++ b/app/server/src/llm/__tests__/openaiCompatible.test.ts @@ -0,0 +1,72 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { extractContent, resolveChatCompletionsUrl } from '../openaiCompatible.ts'; + +test('resolveChatCompletionsUrl: bare host', () => { + assert.equal( + resolveChatCompletionsUrl('https://api.openai.com'), + 'https://api.openai.com/v1/chat/completions', + ); +}); + +test('resolveChatCompletionsUrl: with /v1', () => { + assert.equal( + resolveChatCompletionsUrl('http://127.0.0.1:11434/v1'), + 'http://127.0.0.1:11434/v1/chat/completions', + ); +}); + +test('resolveChatCompletionsUrl: full path unchanged', () => { + assert.equal( + resolveChatCompletionsUrl('https://example.com/v1/chat/completions'), + 'https://example.com/v1/chat/completions', + ); +}); + +test('resolveChatCompletionsUrl: adds https if missing', () => { + assert.equal( + resolveChatCompletionsUrl('api.openai.com'), + 'https://api.openai.com/v1/chat/completions', + ); +}); + +test('resolveChatCompletionsUrl: local IP with /v1', () => { + assert.equal( + resolveChatCompletionsUrl('http://10.37.0.165:30081/v1'), + 'http://10.37.0.165:30081/v1/chat/completions', + ); +}); + +test('extractContent: standard OpenAI shape', () => { + const r = extractContent({ + choices: [{ message: { role: 'assistant', content: 'hello' } }], + }); + assert.equal(r.content, 'hello'); +}); + +test('extractContent: empty content + reasoning + finish length', () => { + const r = extractContent({ + choices: [{ + finish_reason: 'length', + message: { + role: 'assistant', + content: '', + reasoning_content: 'thinking…', + }, + }], + }); + assert.equal(r.content, null); + assert.match(r.reason ?? '', /token budget|reasoning/i); +}); + +test('extractContent: multimodal content array', () => { + const r = extractContent({ + choices: [{ + message: { + role: 'assistant', + content: [{ type: 'text', text: 'part1 ' }, { type: 'text', text: 'part2' }], + }, + }], + }); + assert.equal(r.content, 'part1 part2'); +}); diff --git a/app/server/src/llm/openaiCompatible.ts b/app/server/src/llm/openaiCompatible.ts new file mode 100644 index 0000000..8292892 --- /dev/null +++ b/app/server/src/llm/openaiCompatible.ts @@ -0,0 +1,246 @@ +// Thin OpenAI-compatible chat client (ADR-0008 shape). +// POST {baseUrl}/v1/chat/completions — works with OpenAI, Ornith, Ollama, Azure-compatible hosts. +// +// Reasoning models (e.g. Ornith / llama.cpp) often fill `message.reasoning_content` +// first and only then `message.content`. With a tiny max_tokens budget, content stays +// empty and finish_reason is "length". We use larger default budgets and parse both fields. + +export interface ChatMessage { + role: 'system' | 'user' | 'assistant'; + content: string; +} + +export interface ChatCompletionsInput { + baseUrl: string; + apiKey?: string | null; + model: string; + messages: ChatMessage[]; + temperature?: number; + maxTokens?: number; + timeoutMs?: number; +} + +export interface ChatCompletionsResult { + content: string; + model: string; + raw?: unknown; +} + +/** Normalize user-entered base URL to a chat/completions endpoint. */ +export function resolveChatCompletionsUrl(baseUrl: string): string { + let u = baseUrl.trim().replace(/\/+$/, ''); + if (!u) throw new Error('Base URL is required'); + if (!/^https?:\/\//i.test(u)) { + u = `https://${u}`; + } + if (u.endsWith('/chat/completions')) return u; + if (u.endsWith('/v1')) return `${u}/chat/completions`; + if (u.includes('/v1/')) return u.endsWith('/') ? `${u}chat/completions` : `${u}/chat/completions`; + return `${u}/v1/chat/completions`; +} + +/** + * Extract assistant text from OpenAI-compatible and reasoning-model responses. + * Exported for unit tests. + */ +export function extractContent(json: unknown): { content: string | null; reason: string | null } { + if (!json || typeof json !== 'object') { + return { content: null, reason: 'response is not an object' }; + } + const root = json as { + choices?: Array<{ + finish_reason?: string; + text?: string; + message?: { + content?: unknown; + reasoning_content?: unknown; + reasoning?: unknown; + }; + delta?: { content?: unknown }; + }>; + // some servers + content?: unknown; + output_text?: unknown; + }; + + const choices = root.choices; + if (!Array.isArray(choices) || choices.length === 0) { + // Non-choice shapes + if (typeof root.content === 'string' && root.content.trim()) { + return { content: root.content, reason: null }; + } + if (typeof root.output_text === 'string' && root.output_text.trim()) { + return { content: root.output_text, reason: null }; + } + return { content: null, reason: 'no choices in response' }; + } + + const choice = choices[0]; + const msg = choice.message; + + // 1) Standard message.content (string) + if (typeof msg?.content === 'string' && msg.content.trim()) { + return { content: msg.content, reason: null }; + } + + // 2) Content as array of parts (OpenAI multimodal / some gateways) + if (Array.isArray(msg?.content)) { + const joined = (msg!.content as Array<{ text?: string; type?: string; content?: string }>) + .map((p) => { + if (typeof p === 'string') return p; + if (typeof p?.text === 'string') return p.text; + if (typeof p?.content === 'string') return p.content; + return ''; + }) + .join('') + .trim(); + if (joined) return { content: joined, reason: null }; + } + + // 3) Legacy / completions-style choice.text + if (typeof choice.text === 'string' && choice.text.trim()) { + return { content: choice.text, reason: null }; + } + + // 4) Streaming leftover shape + if (typeof choice.delta?.content === 'string' && choice.delta.content.trim()) { + return { content: choice.delta.content, reason: null }; + } + + // 5) Reasoning models: content empty, reasoning_content filled (often finish_reason=length) + const reasoning = + (typeof msg?.reasoning_content === 'string' && msg.reasoning_content) || + (typeof msg?.reasoning === 'string' && msg.reasoning) || + ''; + if (reasoning.trim()) { + const finish = choice.finish_reason ?? ''; + if (finish === 'length') { + return { + content: null, + reason: + 'model spent the token budget on reasoning and returned empty content ' + + '(raise max tokens; reasoning models like Ornith need headroom after thinking)', + }; + } + // Some hosts only return reasoning; use a trimmed tail as last-resort text + // only when finish is stop/end and content truly empty — better than hard fail for Test. + if (finish === 'stop' || finish === 'end_turn' || finish === '') { + // Prefer not to treat pure reasoning as final answer for product explains; + // still return null with a clear reason so callers can raise max_tokens. + return { + content: null, + reason: + 'response has reasoning_content but empty message.content ' + + '(increase max_tokens so the model can emit the final answer)', + }; + } + } + + if (typeof msg?.content === 'string' && msg.content === '') { + return { + content: null, + reason: + choice.finish_reason === 'length' + ? 'empty content with finish_reason=length (token budget too low for this model)' + : 'message.content is empty', + }; + } + + return { content: null, reason: 'unrecognized message shape' }; +} + +export async function chatCompletions(input: ChatCompletionsInput): Promise { + const url = resolveChatCompletionsUrl(input.baseUrl); + // Reasoning models need larger budgets; 900 was often only enough for thinking. + const maxTokens = input.maxTokens ?? 2048; + const timeoutMs = input.timeoutMs ?? 120_000; + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + + const headers: Record = { + 'content-type': 'application/json', + }; + // Never send Authorization: Bearer none/null placeholders (breaks some local hosts). + const key = input.apiKey?.trim(); + if (key && !/^(none|null|undefined|n\/a|-)$/i.test(key)) { + headers.authorization = `Bearer ${key}`; + } + + try { + const res = await fetch(url, { + method: 'POST', + headers, + body: JSON.stringify({ + model: input.model, + messages: input.messages, + temperature: input.temperature ?? 0.3, + max_tokens: maxTokens, + }), + signal: controller.signal, + }); + + const text = await res.text(); + if (!res.ok) { + const snippet = text.slice(0, 240).replace(/\s+/g, ' '); + throw new Error(`LLM HTTP ${res.status}: ${snippet || res.statusText}`); + } + + let json: unknown; + try { + json = JSON.parse(text); + } catch { + throw new Error('LLM returned non-JSON body'); + } + + const { content, reason } = extractContent(json); + if (!content) { + throw new Error( + reason + ? `LLM response missing message content: ${reason}` + : 'LLM response missing message content', + ); + } + const model = + typeof (json as { model?: string }).model === 'string' + ? (json as { model: string }).model + : input.model; + + return { content, model, raw: json }; + } catch (e) { + if (e instanceof Error && e.name === 'AbortError') { + throw new Error(`LLM request timed out after ${timeoutMs}ms`); + } + throw e; + } finally { + clearTimeout(timer); + } +} + +/** Tiny probe for Settings "Test" — short completion with reasoning headroom. */ +export async function testEndpoint(opts: { + baseUrl: string; + apiKey?: string | null; + model: string; +}): Promise<{ ok: true; model: string; sample: string } | { ok: false; error: string }> { + try { + const r = await chatCompletions({ + baseUrl: opts.baseUrl, + apiKey: opts.apiKey, + model: opts.model, + messages: [ + { + role: 'system', + content: 'You are a connectivity probe. Reply with exactly: ok', + }, + { role: 'user', content: 'ping' }, + ], + // Ornith spends ~150–250 tokens reasoning before "ok"; 8 was always empty. + maxTokens: 512, + temperature: 0, + timeoutMs: 90_000, + }); + return { ok: true, model: r.model, sample: r.content.slice(0, 80) }; + } catch (e) { + return { ok: false, error: e instanceof Error ? e.message : String(e) }; + } +} diff --git a/app/server/src/llm/userLlmEndpoint.ts b/app/server/src/llm/userLlmEndpoint.ts new file mode 100644 index 0000000..0569563 --- /dev/null +++ b/app/server/src/llm/userLlmEndpoint.ts @@ -0,0 +1,132 @@ +// Per-user OpenAI-compatible endpoint storage (encrypted API key). + +import type { DatabaseSync } from 'node:sqlite'; +import cryptoMod from '../lib/crypto.ts'; + +export interface UserLlmEndpointPublic { + configured: boolean; + baseUrl: string | null; + model: string | null; + keyHint: string | null; + updatedAt: string | null; +} + +export interface UserLlmEndpointSecret { + baseUrl: string; + apiKey: string | null; + model: string; +} + +export function ensureUserLlmTable(db: DatabaseSync): void { + db.exec(` + CREATE TABLE IF NOT EXISTS user_llm_endpoints ( + user_id TEXT PRIMARY KEY, + base_url TEXT NOT NULL, + api_key_enc TEXT, + model TEXT NOT NULL DEFAULT 'gpt-4o-mini', + key_hint TEXT, + updated_at TEXT NOT NULL + ); + `); +} + +export function getUserLlmStatus(db: DatabaseSync, userId: string): UserLlmEndpointPublic { + ensureUserLlmTable(db); + const row = db.prepare( + 'SELECT base_url, model, key_hint, updated_at FROM user_llm_endpoints WHERE user_id=?', + ).get(userId) as + | { base_url: string; model: string; key_hint: string | null; updated_at: string } + | undefined; + if (!row) { + return { configured: false, baseUrl: null, model: null, keyHint: null, updatedAt: null }; + } + return { + configured: true, + baseUrl: row.base_url, + model: row.model, + keyHint: row.key_hint, + updatedAt: row.updated_at, + }; +} + +/** Treat blank / placeholder values as "no API key" (local Ornith often needs none). */ +export function normalizeApiKey(raw: string | null | undefined): string | null { + if (raw == null) return null; + const t = raw.trim(); + if (!t) return null; + if (/^(none|null|undefined|n\/a|-)$/i.test(t)) return null; + return t; +} + +export function loadUserLlmSecret(db: DatabaseSync, userId: string): UserLlmEndpointSecret | null { + ensureUserLlmTable(db); + const row = db.prepare( + 'SELECT base_url, api_key_enc, model FROM user_llm_endpoints WHERE user_id=?', + ).get(userId) as + | { base_url: string; api_key_enc: string | null; model: string } + | undefined; + if (!row) return null; + let apiKey: string | null = null; + if (row.api_key_enc) { + try { + apiKey = normalizeApiKey(cryptoMod.decrypt(row.api_key_enc)); + } catch { + apiKey = null; + } + } + return { baseUrl: row.base_url, apiKey, model: row.model }; +} + +export function upsertUserLlmEndpoint( + db: DatabaseSync, + userId: string, + input: { baseUrl: string; apiKey?: string | null; model?: string; keepExistingKey?: boolean }, +): UserLlmEndpointPublic { + ensureUserLlmTable(db); + const baseUrl = input.baseUrl.trim(); + if (!baseUrl) throw new Error('base_url is required'); + const model = (input.model ?? 'gpt-4o-mini').trim() || 'gpt-4o-mini'; + const now = new Date().toISOString(); + + const existing = db.prepare( + 'SELECT api_key_enc, key_hint FROM user_llm_endpoints WHERE user_id=?', + ).get(userId) as { api_key_enc: string | null; key_hint: string | null } | undefined; + + let apiKeyEnc: string | null = existing?.api_key_enc ?? null; + let keyHint: string | null = existing?.key_hint ?? null; + + // apiKey provided: + // - real key → encrypt + // - blank / "none" / "null" → clear stored key (local hosts like Ornith) + // - omitted + keepExistingKey → leave prior key + if (input.apiKey !== undefined && input.apiKey !== null) { + const normalizedKey = normalizeApiKey(input.apiKey); + if (normalizedKey) { + apiKeyEnc = cryptoMod(normalizedKey); + keyHint = normalizedKey.length <= 4 ? '****' : normalizedKey.slice(-4); + } else { + // Empty or placeholder string - clear unless keepExistingKey and field was omitted + // (here apiKey was explicitly sent, so clear). + apiKeyEnc = null; + keyHint = null; + } + } + + db.prepare( + `INSERT INTO user_llm_endpoints (user_id, base_url, api_key_enc, model, key_hint, updated_at) + VALUES (?,?,?,?,?,?) + ON CONFLICT(user_id) DO UPDATE SET + base_url=excluded.base_url, + api_key_enc=excluded.api_key_enc, + model=excluded.model, + key_hint=excluded.key_hint, + updated_at=excluded.updated_at`, + ).run(userId, baseUrl, apiKeyEnc, model, keyHint, now); + + return getUserLlmStatus(db, userId); +} + +export function clearUserLlmEndpoint(db: DatabaseSync, userId: string): void { + ensureUserLlmTable(db); + db.prepare('DELETE FROM user_llm_endpoints WHERE user_id=?').run(userId); +} diff --git a/app/server/src/macro/FredAdapter.ts b/app/server/src/macro/FredAdapter.ts index afccb5a..925091a 100644 --- a/app/server/src/macro/FredAdapter.ts +++ b/app/server/src/macro/FredAdapter.ts @@ -4,6 +4,8 @@ // Cache with weekly staleness (macro data is slow-moving). import type { CacheEntry } from '../cache/CacheRepository.ts'; +import type { FetchResult } from '../adapters/SourceAdapter.ts'; +import { vendorFetch } from '../services/vendorGate.ts'; export interface FredSeries { seriesId: string; @@ -14,7 +16,7 @@ export interface FredSeries { export interface FredAdapter { sourceKind: 'fred'; - fetchOne(key: string): Promise>; + fetchOne(key: string): Promise; series(seriesId: string): Promise>; } @@ -30,12 +32,17 @@ export class FredAdapterImpl implements FredAdapter { this.apiKey = apiKey ?? process.env.FRED_API_KEY ?? ''; } - async fetchOne(key: string): Promise> { + async fetchOne(key: string): Promise { // key format: "fred:series:{seriesId}" const parts = key.split(':'); if (parts.length >= 3 && parts[0] === 'fred' && parts[1] === 'series') { const seriesId = parts[2]; - return this.series(seriesId); + const entry = await this.series(seriesId); + return { + value: entry.value as FredSeries, + ttlClass: 'fred_macro', + provenance: entry.provenance ?? { fetchedAt: new Date().toISOString(), sourceKind: 'fred' }, + }; } throw new Error(`Unknown FRED cache key: ${key}`); } @@ -46,7 +53,10 @@ export class FredAdapterImpl implements FredAdapter { } const url = `${FRED_BASE}/series/observations?series_id=${seriesId}&api_key=${this.apiKey}&file_type=json&limit=100&sort_order=desc`; - const resp = await fetch(url); + const resp = await vendorFetch('fred', url, { + accept: 'application/json', + hostAllowlist: /api\.stlouisfed\.org/i, + }); if (!resp.ok) { let detail = ''; try { @@ -93,4 +103,5 @@ export const FRED_SERIES = { FED_FUNDS: 'FEDFUNDS', HEAVY_TRUCK_SALES: 'HTRUCKSSAAR', MANUFACTURING_ACTIVITY: 'IPMAN', + CONSUMER_SENTIMENT: 'UMCSENT', } as const; diff --git a/app/server/src/macro/MacroRegime.ts b/app/server/src/macro/MacroRegime.ts index 5b7ce75..93d5620 100644 --- a/app/server/src/macro/MacroRegime.ts +++ b/app/server/src/macro/MacroRegime.ts @@ -185,7 +185,7 @@ export function generateMacroCommentary( ? shortTermWithEvents(upcomingEvents.join(', ')) : SHORT_TERM_NO_EVENTS, longTerm: LONG_TERM_MAP[classification.regime], - disclaimer: 'This commentary is for educational purposes only and is not investment advice. ' + disclaimer: 'Educational observation only. ' + 'Past performance does not guarantee future results. Always do your own research ' + 'before making investment decisions.', }; diff --git a/app/server/src/mirror/__tests__/captureParser.test.ts b/app/server/src/mirror/__tests__/captureParser.test.ts new file mode 100644 index 0000000..2838c09 --- /dev/null +++ b/app/server/src/mirror/__tests__/captureParser.test.ts @@ -0,0 +1,133 @@ +// Capture pipeline classification/parser tests (sub-task 4). +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { classifyPost, extractCaptures } from '../captureParser.ts'; + +test('capture: total-position post with numbers', () => { + const r = classifyPost('IREN position: 750,000 shares, avg cost $22.40, market value $28.3M.'); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'IREN'); + assert.equal(r.shares, 750000); + assert.ok(Math.abs(r.cost_basis! - 22.4) < 0.01); + assert.ok(Math.abs(r.value_usd! - 28_300_000) < 1000); +}); + +test('capture: totals without explicit marker still parse via position/holdings', () => { + const r = classifyPost('$ASST current position: 3,150,345 shares, cost basis $0.74.'); + assert.equal(r.class, 'capture'); + assert.equal(r.shares, 3150345); + assert.ok(Math.abs(r.cost_basis! - 0.74) < 0.001); +}); + +test('claim: "added n shares at x average price"', () => { + const r = classifyPost('Added 865,000 shares of BKKT at $3.10 average price today.'); + assert.equal(r.class, 'claim'); + assert.equal(r.symbol, 'BKKT'); + assert.equal(r.shares, 865000); + assert.ok(Math.abs(r.cost_basis! - 3.1) < 0.001); +}); + +test('claim: bought / sold with quantity', () => { + const bought = classifyPost('Bought 10,000 shares of IREN at $22.'); + assert.equal(bought.class, 'claim'); + assert.equal(bought.shares, 10000); + const sold = classifyPost('Sold 2,000 shares of CIFR today.'); + assert.equal(sold.class, 'claim'); +}); + +test('commentary: no numbers → never a record', () => { + const r = classifyPost('Huge week for the portfolio. Love the setup here.'); + assert.equal(r.class, 'none'); +}); + +test('commentary: numbers but a pundit take, not a position', () => { + const r = classifyPost('Bitcoin at $100k is a great risk/reward in my view.'); + assert.equal(r.class, 'none'); +}); +// ─── Real Mike Alfred capture posts (subscriber-only, fetched via user-tweets) ─── +// These shapes come straight from @mikealfred's timeline; regression-locked so the +// capture pipeline never writes garbage from the live feed. + +test('real: "Took OPEN over 5.7M shares now. Brought average down to $4.42"', () => { + const r = classifyPost('**Real time position update\n\nTook OPEN over 5.7M shares now. Brought average down to $4.42. Still focused on longer term execution but don\'t want to miss the chance to keep adding below cost basis in the short term.'); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'OPEN'); + assert.equal(r.shares, 5_700_000); + assert.ok(Math.abs(r.cost_basis! - 4.42) < 0.001); +}); + +test('real: "Holding 3.09M shares at a $1.25 average" (SLNH)', () => { + const r = classifyPost('Real time SLNH position update\n\nHolding 3.09M shares at a $1.25 average. Fantastic dinner with the team tonight.'); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'SLNH'); + assert.equal(r.shares, 3_090_000); + assert.ok(Math.abs(r.cost_basis! - 1.25) < 0.001); +}); + +test('real: "Real time position update on ANY — holding 200,000 shares at a $1.75 average"', () => { + const r = classifyPost('Real time position update on ANY\n\nI am holding 200,000 shares at a $1.75 average. All shares are long.'); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'ANY'); + assert.equal(r.shares, 200000); + assert.ok(Math.abs(r.cost_basis! - 1.75) < 0.001); +}); + +test('real: trade narrative with two quantities is refused, not misrecorded', () => { + // "reduced from 200,000 to 10,000" — old size must not become the record. + const r = classifyPost('I had previously reduced the position from 200,000 shares to 10,000 shares over the last month in advance of earnings... Today I have added some shares back just above $46.'); + assert.equal(r.class, 'none'); +}); + +test('real: "Real time position update" header alone (no numbers) is not a record', () => { + const r = classifyPost('**Real time position update\n\nWhat a day. More below for subscribers.'); + assert.equal(r.class, 'none'); +}); + +test('real: BKKT board seat post — not prose word IS', () => { + const r = classifyPost( + '@MAGASIKH My primary exposure to crypto brokerage / stablecoins is via BKKT where I also serve on the board.\n\nMy publicly disclosed position is 905,000 shares and I was the 4th largest institutional shareholder as of the last reporting period.', + ); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'BKKT'); + assert.equal(r.shares, 905000); + assert.equal(r.instrument ?? 'equity', 'equity'); +}); + +test('real: WULF Sept calls — not prose word IN; marked as call', () => { + const r = classifyPost( + '**Real time position initiation\n\nI am buying a very small initial position in $20 Sept WULF calls. I think WULF may be the most undervalued name in my coverage universe and since I currently have no spot position, I wanted to have some exposure.', + ); + assert.equal(r.class, 'capture'); + assert.equal(r.symbol, 'WULF'); + assert.equal(r.instrument, 'call'); + assert.equal(r.shares, undefined); +}); + +test('prose: "position is N shares" without a real ticker is not a capture', () => { + const r = classifyPost('My publicly disclosed position is 905,000 shares as of last quarter.'); + // No extractable ticker — refuse rather than invent IS/IN/AT. + assert.equal(r.symbol, undefined); +}); + +test('prose: slang "bought this BS" is not a ticker capture', () => { + const r = classifyPost('@sunny051488 Agree. I never bought this BS maxi ideology. Frauds.'); + assert.equal(r.class, 'none'); +}); + +test('real: multi-name position updates — OPEN total + SLNH fill (Mike 2026-08-07)', () => { + const text = `A few quick position updates. + +Just filled a 100,000 share order in SLNH at $1.12. Have another for 100,000 shares at $1.11. Market makers moved the price up a couple ticks. + +OPEN now over 6.45M shares. Have taken it up on the weakness. RKT/Z also weak. Sector wide. I'm buying.`; + // extractCaptures is the multi-name path (classifyPost is single-symbol only). + const multi = extractCaptures(text); + const open = multi.find((c) => c.symbol === 'OPEN'); + assert.ok(open, 'OPEN total must extract'); + assert.equal(open!.class, 'capture'); + assert.equal(open!.shares, 6_450_000); + const slnh = multi.find((c) => c.symbol === 'SLNH'); + assert.ok(slnh, 'SLNH fill should extract as claim'); + assert.equal(slnh!.class, 'claim'); + assert.equal(slnh!.shares, 100_000); +}); diff --git a/app/server/src/mirror/__tests__/mirrorEngine.test.ts b/app/server/src/mirror/__tests__/mirrorEngine.test.ts new file mode 100644 index 0000000..20168d4 --- /dev/null +++ b/app/server/src/mirror/__tests__/mirrorEngine.test.ts @@ -0,0 +1,96 @@ +// Mirror Portfolio (M21) — mirror engine tests (sub-task 5). +// Covers: target math (weight × base), floor behavior, diff vs holdings, +// tick rounding, cost-basis lens, locked sleeve, and ADR-0010 mechanical strings. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { mirrorBook } from '../mirrorEngine.ts'; + +function book(symbol: string, shares: number | null, value?: number | null, cost?: number | null) { + return { symbol, shares, value_usd: value ?? null, cost_basis: cost ?? null }; +} + +test('target = weight × base; fund AUM is irrelevant', () => { + const book = [ + { symbol: 'IREN', shares: 750, value_usd: 30000, cost_basis: 20 }, + { symbol: 'CIFR', shares: 250, value_usd: 5000, cost_basis: 6 }, + ]; + // total book value = 750*40 + 250*20 = 30000+5000 + const prices = { IREN: 40, CIFR: 20 }; + const r = mirrorBook(book, [], prices, { base: 10000 }); + assert.equal(r.rows.length, 2); + const iren = r.rows.find((x) => x.symbol === 'IREN')!; + // weight = 30000/35000 ≈ 0.8571 → target = 8571 + assert.ok(Math.abs(iren.target_value - 8571.42) < 1); + // target qty = target value / price + assert.ok(112 <= iren.target_qty! && iren.target_qty! <= 220); + assert.equal(iren.fund_avg_cost, 20); // cost lens present +}); + +test('floor behavior: below floor → excluded tail; above floor → on book', () => { + const book = [ + { symbol: 'BIG', shares: 1, value_usd: 9000, cost_basis: null }, // 90% + { symbol: 'SMALL', shares: 1, value_usd: 1000, cost_basis: null }, // 10% + ]; + const prices = { BIG: 9000, SMALL: 1000 }; + const r = mirrorBook(book, [], prices, { base: 10000, minWeightPct: 3 }); + // floor = max(500, 3% * 10000 = 300) = 500. SMALL target 1000 → kept. + assert.equal(r.rows.some((x) => x.symbol === 'SMALL'), true); + assert.equal(r.excluded_tail, null); + + const r2 = mirrorBook(book, [], prices, { base: 20000, minWeightPct: 0.5, minPositionUsd: 500 }); + // SMALL target = 0.1 * 20000 = 2000 > 500, kept again. Use a 1%-style lower base. + const r3 = mirrorBook(book, [], prices, { base: 3000, minPositionUsd: 500 }); + // SMALL target = 0.1*3000 = 300 < 500 → excluded + assert.equal(r3.rows.some((x) => x.symbol === 'SMALL'), false); + assert.equal(r3.excluded_tail!.count, 1); + assert.ok(Math.abs(r3.excluded_tail!.value_usd - 300) < 0.01); +}); + +test('delta vs holdings: sell trim when you hold more than target', () => { + const book = [{ symbol: 'IREN', shares: 100, value_usd: 5000, cost_basis: 40 }]; + const prices = { IREN: 50 }; + const r = mirrorBook(book, [{ symbol: 'IREN', qty: 120, avg_cost: 60 }], prices, { base: 10000, minPositionUsd: 1 }); + // weight = 1, target = 10000, targetQty = 200. actual 120 → delta +80 (buy). + const pos = r.rows[0]; + assert.equal(pos.delta_qty, 80); + assert.ok(pos.delta_value > 0); + + const r2 = mirrorBook(book, [{ symbol: 'IREN', qty: 300, avg_cost: 60 }], prices, { base: 10000, minPositionUsd: 1 }); + assert.equal(r2.rows[0].delta_qty, -100); // trim +}); + +test('rounding: whole-share tick by default, fractional with tick 0.01', () => { + const book = [{ symbol: 'IREN', shares: 1000, value_usd: 41000, cost_basis: 41 }]; + const r = mirrorBook(book, [], { IREN: 41.7 }, { base: 20000 }); + // target value = 20000, qty = 479.616 → round to 480 whole shares + assert.equal(r.rows[0].target_qty, 480); + const r2 = mirrorBook(book, [], { IREN: 41.7 }, { base: 20000, tickSize: 0.01 }); + assert.ok(Math.abs(r2.rows[0].target_qty! - 479.62) < 0.011); +}); + +test('ADR-0010: messages are mechanical (never advice-shaped)', () => { + const book = [{ symbol: 'IREN', shares: 100, value_usd: 5000, cost_basis: 40 }]; + const prices = { IREN: 50 }; + const r = mirrorBook(book, [], prices, { base: 20000, minPositionUsd: 1 }); + const msg = r.rows[0].message; + // must not contain imperative "you should buy/sell" + assert.ok(!/\byou should\b/i.test(msg)); + assert.ok(!/^\s*(buy|sell)\s/i.test(msg)); + assert.match(msg, /delta/i); + assert.match(msg, /shares/); + // Excluded line is also mechanical. + const rSmall = mirrorBook( + [{ symbol: 'BIG', shares: 1, value_usd: 9900, cost_basis: null }, { symbol: 'TINY', shares: 1, value_usd: 100, cost_basis: null }], + [], { BIG: 9900, TINY: 100 }, { base: 20000, minPositionUsd: 1000 }); + // TINY target = 0.01 * 20000 = 200 < 1000 floor → excluded. + assert.equal(rSmall.excluded_tail!.count, 1); + assert.match(rSmall.excluded[0].message, /floor/i); +}); + +test('locked: base pinned (fixed-size sleeve) vs default rebaseline flag', () => { + const book = [{ symbol: 'IREN', shares: 100, value_usd: 5000, cost_basis: 40 }]; + const prices = { IREN: 50 }; + const r = mirrorBook(book, [], prices, { base: 10000, locked: true }); + assert.equal(r.locked, true); + assert.equal(r.base, 10000); +}); \ No newline at end of file diff --git a/app/server/src/mirror/__tests__/primaryRuleLint.test.ts b/app/server/src/mirror/__tests__/primaryRuleLint.test.ts new file mode 100644 index 0000000..a8b5394 --- /dev/null +++ b/app/server/src/mirror/__tests__/primaryRuleLint.test.ts @@ -0,0 +1,64 @@ +// M21 Primary-Rule lint (sub-task 9) — ADR-0010 enforcement. +// +// Mirrors the project's Primary-Rule lint: every M21 user-facing string source +// is STATICALLY scanned and FAILS on any sentence shaped like "you should buy/sell +// X because did". The Mirror module is a CALCULATOR, never advice. +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { readFileSync } from 'node:fs'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { mirrorBook } from '../mirrorEngine.ts'; +import { classifyPost } from '../captureParser.ts'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const ROOT = resolve(__dirname, '..', '..'); // app/server/src + +const TARGETS = [ + join(ROOT, 'mirror', 'mirrorEngine.ts'), + join(ROOT, 'mirror', 'captureParser.ts'), + join(ROOT, 'alerts', 'producers', 'mirrorAlertProducers.ts'), + join(ROOT, 'alerts', 'AlertEngine.ts'), + // M22 Symbol Search Index — user-facing copy in the autocomplete / holders strip. + join(ROOT, 'trpc', 'router.ts'), +]; + +// Advice-shaped patterns forbidden in any user-facing string. +const FORBIDDEN = [ + /you should (buy|sell|add|trim|reduce|exit|close)/i, + /i (would|recommend) (buy|sell|add|trim)/i, + /\brecommend(ed|ation)?( to)? (buy|sell)\b/i, + /\b(buy|sell|add|trim) .+\b(because|since|as) \b(fund|he|the manager)\b/i, + /^\s*(buy|sell)\s+\$?[A-Z]/m, +]; + +test('statics: no advice-shaped string in M21 templates (ADR-0010)', () => { + for (const file of TARGETS) { + const src = readFileSync(file, 'utf8'); + for (const pat of FORBIDDEN) { + const m = pat.exec(src); + assert.ok(!m, `ADR-0010 violation in ${file} → /${m ? m[0] : ''}/`); + } + } +}); + +test('engine output is arithmetic, never "you should X because the fund did"', () => { + const book = [ + { symbol: 'IREN', shares: 100, value_usd: 5000, cost_basis: 40 }, + { symbol: 'CIFR', shares: 25, value_usd: 500, cost_basis: 12 }, + ]; + const r = mirrorBook(book, [{ symbol: 'IREN', qty: 120, avg_cost: 60 }], { IREN: 50, CIFR: 20 }, { base: 20000 }); + for (const row of r.rows) { + assert.match(row.message, /delta/i); // arithmetic framing + assert.match(row.message, /\d/); // contains a number + assert.ok(!/you should/i.test(row.message)); + assert.ok(!/because .*(did|bought|sold|added|trimmed)/i.test(row.message)); + assert.ok(!/recommend|advisory/i.test(row.message)); + } +}); + +test('commentary/advice posts never become records (capture rail)', () => { + assert.equal(classifyPost('This hidden gem is a huge opportunity. We are positioned size in BTC.').class, 'none'); + assert.equal(classifyPost('Strong buy here — the setup is compelling.').class, 'none'); +}); \ No newline at end of file diff --git a/app/server/src/mirror/captureParser.ts b/app/server/src/mirror/captureParser.ts new file mode 100644 index 0000000..167937b --- /dev/null +++ b/app/server/src/mirror/captureParser.ts @@ -0,0 +1,338 @@ +// Investor Flow — Mirror Portfolio (M21) position-capture parser/classifier. +// +// Classifies a fund manager's X post into position-capture | trade-claim | commentary +// and, for capture/claim, extracts {symbol, shares, value_usd, cost_basis, date}. +// +// Captures are self-reported *total current position* state (tot a screenshot); +// claims are dated deltas ("added N shares at $X avg"). Commentary is never a record. +// v1 uses a deterministic, greppable rule set (no LLM round-trip so it runs offline +// and is trivially testable); an LLM gateway can be swapped in for ambiguous posts +// later (ADR-0006 local-only). + +export type PostClass = 'capture' | 'claim' | 'none'; + +export type InstrumentKind = 'equity' | 'call' | 'put'; + +export interface ParsedCapture { + class: PostClass; + symbol?: string; + shares?: number; // total shares (capture) OR delta shares (claim) + value_usd?: number; // market value when stated + cost_basis?: number; // avg cost per share when stated + date?: string; // ISO date if a date was stated, else inferred from the post + instrument?: InstrumentKind; // equity (default) | call | put +} + +// Words that look like tickers but are prose / common finance jargon. +// Never emit these as a symbol. Includes short prepositions that the old +// "position " pattern used to steal (IS, IN, AT, ON, TO, ...). +const STOPWORDS = new Set([ + 'REAL', 'TIME', 'THIS', 'THAT', 'THESE', 'THOSE', 'THE', 'MY', 'OUR', 'YOUR', + 'ALL', 'STILL', 'TOTAL', 'CURRENT', 'HOLDING', 'HOLDINGS', 'POSITION', 'POSITIONS', + 'AVERAGE', 'MARKET', 'VALUE', 'BOOK', 'UPDATE', 'DAY', 'WEEK', 'MONTH', 'YEAR', + 'NEW', 'OLD', 'JUST', 'NOW', 'AND', 'OR', 'BUT', 'FOR', 'FROM', 'WITH', 'VIA', + 'INTO', 'OVER', 'UNDER', 'ABOUT', 'AFTER', 'BEFORE', 'BETWEEN', 'DURING', + 'WITHOUT', 'WITHIN', 'AMONG', 'ACROSS', 'BEHIND', 'BEYOND', 'HAVE', 'HAS', + 'HAD', 'WILL', 'WOULD', 'COULD', 'SHOULD', 'MIGHT', 'MUST', 'SHALL', 'CAN', + 'NOT', 'THAN', 'THEN', 'WHEN', 'WHERE', 'WHAT', 'WHO', 'WHY', 'HOW', 'ARE', + 'WAS', 'WERE', 'BEEN', 'BEING', 'AM', 'BE', 'DO', 'DID', 'DOES', 'DONE', + // Short prepositions / pronouns — classic false positives from "position is/in" + 'IS', 'IN', 'AT', 'ON', 'TO', 'OF', 'AS', 'BY', 'IF', 'IT', 'NO', 'SO', 'UP', + 'WE', 'ME', 'HE', 'SHE', 'THEY', 'THEM', 'HIS', 'HER', 'ITS', 'AN', 'A', + // Finance / corporate jargon often uppercased in posts + 'CEO', 'CFO', 'COO', 'CTO', 'IPO', 'ETF', 'ETFS', 'SEC', 'FDA', 'USD', 'ATH', + 'ATL', 'EOD', 'EOW', 'IMO', 'TBH', 'FYI', 'LLC', 'INC', 'LTD', 'PLC', 'AI', + 'API', 'GDP', 'CPI', 'PPI', 'FOMC', 'YOY', 'QOQ', 'EPS', 'PE', 'NAV', 'AUM', + 'SEPT', 'SEP', 'OCT', 'NOV', 'DEC', 'JAN', 'FEB', 'MAR', 'APR', 'JUN', 'JUL', + 'AUG', 'MON', 'TUE', 'WED', 'THU', 'FRI', 'SAT', 'SUN', + 'CALL', 'CALLS', 'PUT', 'PUTS', 'SHARE', 'SHARES', 'SPOT', 'LONG', 'SHORT', + 'SMALL', 'LARGE', 'INITIAL', 'PRIMARY', 'PUBLIC', 'BOARD', 'NAME', + // Slang / interjections often uppercased in posts (not tickers in this context) + 'BS', 'OK', 'LOL', 'OMG', 'WTF', 'IMO', 'NGL', 'FR', 'IDK', 'SMH', 'FOMO', + 'HODL', 'HFSP', 'GM', 'GN', 'WAGMI', 'NGMI', 'IYKYK', 'GOAT', 'RIP', +]); + +function isTickerCandidate(raw: string | undefined | null): raw is string { + if (!raw) return false; + const s = raw.toUpperCase(); + if (s.length < 1 || s.length > 5) return false; + if (STOPWORDS.has(s)) return false; + // Must be letters only (no digits — those are strikes/sizes). + if (!/^[A-Z]+$/.test(s)) return false; + return true; +} + +function toNum(raw: string): number | undefined { + const cleaned = raw.replace(/[,$]/g, '').replace(/\.$/, ''); // drop separators + trailing period + const n = Number(cleaned); + return Number.isFinite(n) ? n : undefined; +} + +function findQty(text: string): number | undefined { + // "750,000 shares", "5.7M shares", "3.09M shares", "N shares" + const m = text.match(/([\d.,]+)\s*([MBKmbk]?)\s*(?:shares|share)\b/i); + if (!m) return undefined; + const base = toNum(m[1]); + if (base === undefined) return undefined; + const suffix = (m[2] || '').toUpperCase(); + if (suffix === 'B') return base * 1_000_000_000; + if (suffix === 'M') return base * 1_000_000; + if (suffix === 'K') return base * 1_000; + return base; +} + +/** Distinct share-quantity mentions. >1 means a trade narrative ("from X to Y"), not a clean capture. */ +function shareMentionCount(text: string): number { + return (text.match(/([\d.,]+\s*[MBK]?\s*shares)/gi) ?? []).length; +} + +function findValue(text: string): number | undefined { + // "value $30M" / "$5.7M value" / "market value ~$28M" / "$1.28B" + const m = text.match(/(?:value|worth|market value|mv)[^\d$]{0,12}\$?\s*([\d.,]+)\s*([MBKmbk])?/i) + || text.match(/\$\s*([\d.,]+)\s*([MBKmbk])\b.*?(?:value|worth|market)/i); + if (!m) return undefined; + const base = toNum(m[1]); + if (base === undefined) return undefined; + const suffix = (m[2] || '').toUpperCase(); + if (suffix === 'B') return base * 1_000_000_000; + if (suffix === 'M') return base * 1_000_000; + if (suffix === 'K') return base * 1_000; + return base; +} + +function findCostBasis(text: string): number | undefined { + // "at $22.40 avg" / "avg cost $22.40" / "cost basis $22.40" / "@ $7" / + // "average down to $4.42" / "at a $1.25 average" + const m = text.match(/(?:at|avg(?: cost)?|average|cost basis|basis|bought at)[^$\d]{0,16}\$\s*([\d.,]+)/i) + || text.match(/at\s+(?:a\s+)?\$\s*([\d.,]+)\s*(?:average|avg)\b/i); + return m ? toNum(m[1]) : undefined; +} + +function findInstrument(text: string): InstrumentKind { + // "WULF calls", "$20 Sept WULF calls", "buying calls on WULF" + if (/\b[A-Z]{1,5}\s+calls?\b/.test(text) || /\bcalls?\s+(?:on|in)\s+/i.test(text) || /\bcall options?\b/i.test(text)) { + return 'call'; + } + if (/\b[A-Z]{1,5}\s+puts?\b/.test(text) || /\bputs?\s+(?:on|in)\s+/i.test(text) || /\bput options?\b/i.test(text)) { + return 'put'; + } + return 'equity'; +} + +function findSymbol(text: string): string | undefined { + // 1. Cashtag — highest confidence ($OPEN, $wulf). + const cashtag = text.match(/\$([A-Za-z]{1,5})(?=[^A-Za-z0-9]|$)/); + if (cashtag && isTickerCandidate(cashtag[1])) return cashtag[1].toUpperCase(); + + // 2. Options underlyings: "WULF calls", "Sept WULF calls", "IREN puts". + const optUnder = text.match(/\b([A-Z]{1,5})\s+(?:calls?|puts?)\b/); + if (optUnder && isTickerCandidate(optUnder[1])) return optUnder[1]; + + // 3. "position update on ANY" / "update on ABC" — strong, specific. + const on = text.match(/\b(?:update|position|book)\s+on\s+([A-Z]{1,5})\b/); + if (on && isTickerCandidate(on[1])) return on[1]; + + // 4. "via BKKT" / "in BKKT" when the ticker is already uppercase in source. + const via = text.match(/\b(?:via|in|into|on)\s+([A-Z]{2,5})\b/); + if (via && isTickerCandidate(via[1])) return via[1]; + + // 5. "shares of BKKT at $3". + const of = text.match(/(?:shares of|of)\s+([A-Z]{1,5})\b\s*(?:at\s*\$|\$|today|average|\d)/); + if (of && isTickerCandidate(of[1])) return of[1]; + + // 6. "Took OPEN over 5.7M shares now" / "Holding 3.09M shares". + // Capture group must be uppercase in source (no /i on the ticker itself). + const took = text.match( + /\b(?:took|taken|added to|increased|holding|hold|own|bought(?: into)?)\s+([A-Z]{1,5})\b\s*(?:over|about|~|more than)?\s*[\d.,]+\s*[MBK]?\s*shares/i, + ); + if (took && isTickerCandidate(took[1])) return took[1].toUpperCase(); + + // 7. "positions AEVA" / "position INTC" — ONLY when the next token is + // already uppercase. "position is" / "position in" must NEVER match. + const posWord = text.match(/\bpositions?\s+([A-Z]{2,5})\b/); + if (posWord && isTickerCandidate(posWord[1])) return posWord[1]; + + // 8. "SLNH position update" / "ANY position" — ticker immediately before + // position language. + const pos = text.match(/\b([A-Z]{1,5})\b[^\w]{0,3}(?:position|holdings|book)\b/); + if (pos && isTickerCandidate(pos[1])) return pos[1]; + + // No bare-word last resort. Grabbing a lone uppercase token from the post + // ("BS", "AI", random ALLCAPS) caused false live-book rows. Every accepted + // symbol must sit in a structured finance pattern above. + return undefined; +} + +const CLAIM_MARKERS = [ + /added\s+[\d.,]+\s*shares/i, + /\bbought\b|\bpurchased\b|\btrimmed?\b|\bsold\s+[\d.,]+\s*shares/i, + /augmented\s+[\d.,]+\s*shares/i, + /added\b.*\bat\s*\$|bought\b.*\bat\s*\$|aliases:.*\$/i, +]; + +const CAPTURE_MARKERS = [ + /\bposition\b/i, + /\btotal\b/i, + /\bcurrent\b.*\bholdings\b/i, + /\bstill\b.*\bshares\b/i, + /\bwe hold\b/i, + /\bholdings\b/i, + /market value/i, + /\bstill holding\b/i, + /\bsold some\b/i, + /\btrim(?:med)?\b.*\brest\b/i, + /\bcleaning the book\b/i, + /\bwhat stayed\b/i, +]; + +function qtyFromParts(num: string, suffix: string | undefined): number | undefined { + const base = toNum(num); + if (base === undefined) return undefined; + const s = (suffix || '').toUpperCase(); + if (s === 'B') return base * 1_000_000_000; + if (s === 'M') return base * 1_000_000; + if (s === 'K') return base * 1_000; + return base; +} + +/** + * Multi-name book updates (Mike Alfred style): one post, several symbols. + * "OPEN now over 6.45M shares" + "filled a 100,000 share order in SLNH at $1.12". + * + * Totals ("TICKER now over N shares") → capture. + * Order fills / "added N shares of X" → claim (ingest still skips claims in v1). + */ +export function extractCaptures(text: string): ParsedCapture[] { + const t = text || ''; + const out: ParsedCapture[] = []; + const seenSym = new Set(); + + const push = (p: ParsedCapture) => { + if (!p.symbol || seenSym.has(p.symbol)) return; + // Prefer capture over claim if both appear for same ticker. + seenSym.add(p.symbol); + out.push(p); + }; + + // Totals: "OPEN now over 6.45M shares" / "SLNH now at 3.09M shares" + for (const m of t.matchAll( + /\b([A-Z]{1,5})\s+now\s+(?:over|at|above|about|under|~|more than)?\s*([\d.,]+)\s*([MBKmbk])?\s*shares\b/g, + )) { + if (!isTickerCandidate(m[1])) continue; + const shares = qtyFromParts(m[2], m[3]); + if (shares === undefined) continue; + push({ + class: 'capture', + symbol: m[1].toUpperCase(), + shares, + instrument: 'equity', + }); + } + + // Totals: "Took OPEN over 5.7M shares" / "Holding 3.09M shares" near ticker already handled by classifyPost + for (const m of t.matchAll( + /\b(?:took|taken|holding|hold|own)\s+([A-Z]{1,5})\s+(?:over|at|about|~|more than)?\s*([\d.,]+)\s*([MBKmbk])?\s*shares\b/gi, + )) { + const sym = m[1].toUpperCase(); + if (!isTickerCandidate(sym)) continue; + const shares = qtyFromParts(m[2], m[3]); + if (shares === undefined) continue; + push({ class: 'capture', symbol: sym, shares, instrument: 'equity' }); + } + + // Order fills / clips (claims, not book totals): "filled a 100,000 share order in SLNH at $1.12" + for (const m of t.matchAll( + /\b(?:filled|fill(?:ing)?|bought|purchased)\s+(?:a\s+)?([\d.,]+)\s*([MBKmbk])?\s*share\s+orders?\s+(?:in|of|for)\s+([A-Z]{1,5})\b(?:[^$\n]{0,40}?\$\s*([\d.,]+))?/gi, + )) { + const sym = m[3].toUpperCase(); + if (!isTickerCandidate(sym)) continue; + if (seenSym.has(sym)) continue; // already have a total for this name + const shares = qtyFromParts(m[1], m[2]); + const cost = m[4] ? toNum(m[4]) : undefined; + push({ + class: 'claim', + symbol: sym, + shares, + cost_basis: cost, + instrument: 'equity', + }); + } + + if (out.length > 0) { + // Attach cost / value only when the post is a single capture (no sibling + // claims). Multi-name posts often have "$1.12" on a fill that must not + // stick to another ticker's total. + if (out.length === 1 && out[0].class === 'capture') { + const cost = findCostBasis(t); + const value = findValue(t); + if (cost !== undefined && out[0].cost_basis === undefined) out[0].cost_basis = cost; + if (value !== undefined && out[0].value_usd === undefined) out[0].value_usd = value; + } + return out; + } + + // Single-capture path + const one = classifyPost(t); + return one.class === 'none' ? [] : [one]; +} + +/** Classify a post and extract structured numbers. Deterministic. */ +export function classifyPost(text: string): ParsedCapture { + const t = text || ''; + const qty = findQty(t); + const cost = findCostBasis(t); + const value = findValue(t); + const symbol = findSymbol(t); + const instrument = findInstrument(t); + // Options are not share counts — never treat option posts as equity share qty. + const shares = instrument === 'equity' ? qty : undefined; + const hasAccountNumbers = (instrument === 'equity' && qty !== undefined) + || cost !== undefined + || value !== undefined; + + // Trade narratives with multiple quantity mentions (e.g. "reduced from + // 200,000 to 10,000 shares") are NOT clean position snapshots — unless the + // multi-name extractor already pulled per-ticker totals (see extractCaptures). + const multiQty = shareMentionCount(t) > 1; + const fromToNarrative = /\bfrom\b[\s\S]{0,40}\bto\b/i.test(t) || /\breduced\b[\s\S]{0,40}\bto\b/i.test(t); + if (multiQty && (fromToNarrative || !/\b[A-Z]{2,5}\s+now\s+/m.test(t))) { + // Allow multi-qty when post has "TICKER now … shares" style multi-book updates; + // those are handled by extractCaptures. Here we only hard-refuse single-name + // trade stories. + if (fromToNarrative || !/\bposition updates?\b/i.test(t)) { + return { class: 'none' }; + } + } + + const base = { symbol, shares, cost_basis: cost, value_usd: value, instrument }; + + const claimHit = CLAIM_MARKERS.some((re) => re.test(t)); + if (claimHit && (qty !== undefined || cost !== undefined)) { + // Claims stay equity-oriented for v1 (delta folding is future work). + return { class: 'claim', ...base, shares: qty }; + } + + const captureHit = CAPTURE_MARKERS.some((re) => re.test(t)); + if (captureHit && hasAccountNumbers && symbol) { + return { class: 'capture', ...base }; + } + // Capture marker + numbers but no symbol → refuse (don't invent IS/IN). + if (captureHit && hasAccountNumbers && !symbol) { + return { class: 'none' }; + } + + // Conversational captures / option initiations: "buying ... WULF calls", + // "bought $X", "initiating position". + const strongAction = /\b(?:bought|added|adding|buying|buy|initiat(?:ing|ed)|new (?:position|entry)|we (?:buy|adding))\b/i.test(t); + if (strongAction && symbol) { + return { class: 'capture', ...base }; + } + + // Position list patterns: "what stayed: $X $Y $Z" / "kept: $X $Y" + const listPattern = /\b(?:what stayed|kept|still (?:holding|long)|remain(?:ing)?)\s*:/i.test(t); + if (listPattern && symbol) { + return { class: 'capture', ...base }; + } + + return { class: 'none' }; +} \ No newline at end of file diff --git a/app/server/src/mirror/fund13fFetcher.ts b/app/server/src/mirror/fund13fFetcher.ts new file mode 100644 index 0000000..254a8bf --- /dev/null +++ b/app/server/src/mirror/fund13fFetcher.ts @@ -0,0 +1,148 @@ +// Investor Flow — M21 per-fund 13F ingestion (sub-task 3, real implementation). +// +// Walks a tracked fund's FULL 13F history via the existing EDGAR machinery +// (filings_index + form13f_holdings), maps CUSIP → symbol (curated + name +// fallback against the symbols table), upserts into `institution_filings` +// (idempotent on uq_inst_filings: filer_cik, symbol, reported_quarter, form), +// then refreshes the fund's Live-Book records via sync13FIntoRecords. +// +// ADR-0009 rate-limit-first: the EDGAR adapter already owns caching/ETags and +// the queue path; this is called on-demand from the admin "Sync 13F" button. + +import type { DatabaseSync } from 'node:sqlite'; +import { EdgarAdapter } from '../adapters/EdgarAdapter.ts'; +import { + curatedSymbolForCusip, + writeCachedCusip, +} from '../services/cusipRegistry.ts'; + +function normalize(s: string): string { + return s.toUpperCase().replace(/[^A-Z0-9]+/g, ' '); +} + +/** + * Resolve a 13F holding's CUSIP/issuer to a ticker symbol. Uses the curated + * CUSIP map first, then fuzzy name match against the symbols table. + * Side effect: seeds sec:cusip cache when mapping succeeds (feeds alert pipeline). + */ +export function cusipToSymbol(db: DatabaseSync, cusip: string, issuerName: string): string | null { + const override = curatedSymbolForCusip(cusip); + if (override) { + writeCachedCusip(db, override, cusip); + return override; + } + const name = normalize(issuerName); + if (!name) return null; + const rows = db.prepare('SELECT symbol, name FROM symbols WHERE name IS NOT NULL').all() as + Array<{ symbol: string; name: string }>; + for (const r of rows) { + if (normalize(r.name) === name) { + writeCachedCusip(db, r.symbol, cusip); + return r.symbol; + } + } + const token = name.split(' ').filter((t) => t.length > 2); + for (const r of rows) { + const rn = normalize(r.name); + const hit = token.filter((t) => rn.includes(t)).length >= 2; + if (hit) { + writeCachedCusip(db, r.symbol, cusip); + return r.symbol; + } + } + return null; +} + +function quarterOf(dateIso: string): string { + const m = /^(\d{4})-(\d{2})/.exec(dateIso || ''); + if (!m) return dateIso; + const year = m[1]; + const month = Number(m[2]); + const q = month <= 3 ? 1 : month <= 6 ? 2 : month <= 9 ? 3 : 4; + return `${year}-Q${q}`; +} + +export interface Fund13FFetchResult { + fetched_filings: number; + holdings_seen: number; + symbols_upserted: number; + records_inserted: number; + unresolved: Array<{ cusip: string; issuerName: string }>; +} + +/** + * Fetch the fund's full 13F history into institution_filings and refresh its + * Live-Book records. Idempotent; safe to re-run (SEC cadence). + */ +export async function fetchFund13F(db: DatabaseSync, fundId: string): Promise { + const fund = db.prepare('SELECT id, ci_key FROM tracked_funds WHERE id = ? AND enabled = 1').get(fundId) as + | { id: string; ci_key: string } + | undefined; + if (!fund) throw new Error('Tracked fund not found or disabled.'); + + const edgar = new EdgarAdapter(); + const indexRes = await edgar.filings_index(fund.ci_key, { formTypes: ['13F-HR'] }); + const filings = (indexRes.value ?? []) as Array>; + + const unresolved: Fund13FFetchResult['unresolved'] = []; + let holdingsSeen = 0; + let upserted = 0; + const now = new Date().toISOString(); + const upsert = db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at, accession, put_call) + VALUES (?, ?, ?, '13F-HR', ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(filer_cik, symbol, reported_quarter, form) DO UPDATE SET + shares = excluded.shares, + value_usd = excluded.value_usd, + accession = excluded.accession, + put_call = excluded.put_call, + filed_at = excluded.filed_at, + fetched_at = excluded.fetched_at`, + ); + + for (const filing of filings) { + const accession = String(filing.accessionNumber ?? filing.accession ?? filing.adsh ?? ''); + const reportDate = String(filing.reportDate ?? filing.periodEnding ?? filing.period_of_report ?? ''); + const fileDate = String(filing.fileDate ?? filing.filingDate ?? now); + const quarter = quarterOf(reportDate); + if (!accession || !quarter) continue; + + const holdRes = await edgar.form13f_holdings(fund.ci_key, accession); + const holdings = (holdRes.value?.holdings ?? []) as Array<{ + cusip: string; issuerName: string; value: number; sshPrnamt: number; putCall: string; + }>; + holdingsSeen += holdings.length; + + // Aggregate per symbol (13F lists shares + separate put/call rows for the + // same name; uq_inst_filings allows one row per (cik, symbol, quarter, form)). + const bySymbol = new Map(); + for (const h of holdings) { + const symbol = cusipToSymbol(db, h.cusip, h.issuerName); + if (!symbol) { + unresolved.push({ cusip: h.cusip, issuerName: h.issuerName }); + continue; + } + const agg = bySymbol.get(symbol) ?? { shares: 0, value: 0, putCall: '' }; + agg.shares += h.sshPrnamt ?? 0; + agg.value += h.value ?? 0; + if (h.putCall) agg.putCall = h.putCall; + bySymbol.set(symbol, agg); + } + + for (const [symbol, agg] of bySymbol) { + if (agg.shares <= 0 && agg.value <= 0) continue; + upsert.run( + fund.ci_key, null, symbol, agg.shares, agg.value, quarter, fileDate, now, accession, + agg.putCall || null, + ); + upserted++; + } + } + + // Refresh Live-Book records from the (now complete) 13F rows. + const { sync13FIntoRecords } = await import('../db/fundRepository.ts'); + const recordsInserted = sync13FIntoRecords(db, fundId); + + return { fetched_filings: filings.length, holdings_seen: holdingsSeen, symbols_upserted: upserted, records_inserted: recordsInserted, unresolved }; +} \ No newline at end of file diff --git a/app/server/src/mirror/mirrorEngine.ts b/app/server/src/mirror/mirrorEngine.ts new file mode 100644 index 0000000..cb038d3 --- /dev/null +++ b/app/server/src/mirror/mirrorEngine.ts @@ -0,0 +1,191 @@ +// Investor Flow — Mirror Portfolio (M21) engine. +// +// PURE deep module. No I/O. Everything is a deterministic function of its inputs. +// Mirrors a fund's disclosed book onto the user's own base capital: target value +// per position = fund weight × user base. Base = user-entered seed by default and +// re-baselines to actual equity; the `locked` option pins a fixed-size sleeve. +// +// ADR-0010: the ONLY strings this module can produce are MECHANICAL (arithmetic) +// statements — "to match your mirror target, the delta is N shares (~$X)". It never +// forms a recommending imperative ("act on this because the fund did"). The +// ADR-0010 boundary is also enforced by the M21 Primary-Rule lint (sub-task 9). + +export interface BookPosition { + symbol: string; + shares: number | null; + value_usd: number | null; // quarter-end market value (13F) or capture market value + cost_basis: number | null; // avg cost per share (capture) — the construction lens +} + +export interface Holding { + symbol: string; + qty: number; + avg_cost: number; +} + +export interface MirrorConfig { + /** Capital base in USD. */ + base: number; + /** Absolute floor under a kept position. Default 500. */ + minPositionUsd?: number; + /** Relative floor (% of base) under a kept position. Default 0.5. */ + minWeightPct?: number; + /** Share rounding tick. Default 1 (whole shares). */ + tickSize?: number; + /** True = base is a fixed sleeve; false = re-baselines to live equity. */ + locked?: boolean; +} + +export interface PriceMap { + [symbol: string]: number | undefined; +} + +export interface MirrorRow { + symbol: string; + weight: number; // fund weight in the book (fraction 0..1), by market value + fund_avg_cost: number | null; + target_value: number; + target_qty: number | null; // tick-rounded + actual_qty: number; + actual_value: number; + delta_qty: number; // target − actual (tick-rounded); positive = increase, negative = trim + delta_value: number; // signed USD delta + message: string; // mechanical, ADR-0010 + excluded: boolean; +} + +export interface MirrorResult { + base: number; + locked: boolean; + tick_size: number; + min_position_usd: number; + min_weight_pct: number; + rows: MirrorRow[]; + excluded: MirrorRow[]; + excluded_tail: { count: number; value_usd: number } | null; + placed_value: number; + book_value: number; +} + +const DEFAULT_MIN_USD = 500; +const DEFAULT_MIN_WEIGHT_PCT = 0.5; +const DEFAULT_TICK = 1; + +function roundTick(value: number, tick: number): number { + if (!(tick > 0)) return value; + return Math.round(value / tick) * tick; +} + +function fmtUsd(value: number): string { + return value.toLocaleString('en-US', { + minimumFractionDigits: 2, + maximumFractionDigits: 2, + }); +} + +/** Mechanical arithmetic string (ADR-0010). No advice shape. */ +function buildMessage(symbol: string, deltaQty: number, deltaValue: number): string { + const qty = Math.abs(Math.round(deltaQty * 100) / 100); + const verb = deltaQty >= 0 ? 'increase' : 'trim'; + return `To match your mirror target for ${symbol}, the delta is ${verb} of ${qty} shares (about $${fmtUsd(Math.abs(deltaValue))}).`; +} + +/** + * Compute the mirror diff between a fund's book and the user's holdings. + * + * @param book fund Live Book rows (post-Mirror-Target-Rule). + * @param actual the user's open equity holdings (actual side). + * @param prices current prices keyed by symbol (marks both sides). + * @param config mirror config (base + floor + rounding). + */ +export function mirrorBook( + book: BookPosition[], + actual: Holding[], + prices: PriceMap, + config: MirrorConfig, +): MirrorResult { + const base = Math.max(0, config.base); + const minUsd = Math.max(0, config.minPositionUsd ?? DEFAULT_MIN_USD); + const minWeightPct = Math.max(0, config.minWeightPct ?? DEFAULT_MIN_WEIGHT_PCT); + const tick = config.tickSize ?? DEFAULT_TICK; + const locked = Boolean(config.locked); + + const actualBy = new Map(); + for (const h of actual) actualBy.set(h.symbol.toUpperCase(), h); + + const bookValueMap = new Map(); // symbol -> fund avg cost + for (const p of book) bookValueMap.set(p.symbol.toUpperCase(), p.cost_basis ?? null); + + // Mark the fund book to current market: prefer shares × price; fall back to the + // recorded value when we lack shares or a price. + let bookValue = 0; + const val = book.map((p) => { + const sym = p.symbol.toUpperCase(); + const price = prices[sym]; + let value: number; + if (p.shares != null && typeof price === 'number' && price > 0) { + value = p.shares * price; + } else { + value = p.value_usd ?? 0; + } + bookValue += value; + return { symbol: sym, value, price: price ?? null }; + }); + + const weightTotal = bookValue > 0 ? bookValue : 1; + const floorUsd = Math.max(minUsd, (minWeightPct / 100) * base); + const rows: MirrorRow[] = []; + const excluded: MirrorRow[] = []; + + for (const v of val) { + const weight = v.value / weightTotal; + const targetValue = weight * base; + const belowFloor = targetValue < floorUsd; + const hold = actualBy.get(v.symbol); + const actualQty = hold ? hold.qty : 0; + const price = v.price; + const actualValue = typeof price === 'number' && price > 0 ? actualQty * price : 0; + + let targetQty: number | null = null; + if (price != null && price > 0) targetQty = roundTick(targetValue / price, tick); + const delta = (targetQty ?? 0) - actualQty; + const deltaVal = ((targetQty ?? 0) - actualQty) * (price ?? 0); + + const row: MirrorRow = { + symbol: v.symbol, + weight, + fund_avg_cost: bookValueMap.get(v.symbol) ?? null, + target_value: targetValue, + target_qty: targetQty, + actual_qty: actualQty, + actual_value: actualValue, + delta_qty: roundTick(delta, tick), + delta_value: deltaVal, + message: belowFloor + ? `Excluded (below minimum): ${v.symbol} falls under the $${fmtUsd(floorUsd)} floor.` + : buildMessage(v.symbol, delta, deltaVal), + excluded: belowFloor, + }; + (belowFloor ? excluded : rows).push(row); + } + + const order = (a: MirrorRow, b: MirrorRow) => b.weight - a.weight || a.symbol.localeCompare(b.symbol); + rows.sort(order); + excluded.sort(order); + + const excludedValue = excluded.reduce((s, r) => s + r.target_value, 0); + const placedValue = rows.reduce((s, r) => s + r.target_value, 0); + + return { + base, + locked, + tick_size: tick, + min_position_usd: minUsd, + min_weight_pct: minWeightPct, + rows, + excluded, + excluded_tail: excluded.length > 0 ? { count: excluded.length, value_usd: excludedValue } : null, + placed_value: placedValue, + book_value: bookValue, + }; +} \ No newline at end of file diff --git a/app/server/src/onboarding/__tests__/starter.test.ts b/app/server/src/onboarding/__tests__/starter.test.ts index faa9189..f71c182 100644 --- a/app/server/src/onboarding/__tests__/starter.test.ts +++ b/app/server/src/onboarding/__tests__/starter.test.ts @@ -3,108 +3,135 @@ import * as assert from "node:assert/strict"; import { STARTER_WATCHLIST, + STARTER_PACK_FOCUSED, + STARTER_PACK_STANDARD, ONBOARDING_DISCLAIMER, defaultDrawdownTolerancePct, defaultRiskTolerance, + starterPackFor, + resolveWorkspaceDefaults, + complexityFromExperience, + defaultDensity, type StarterSymbol, } from "../starter.ts"; +const FORBIDDEN = [ + /buy\b/i, + /sell\b/i, + /you should\b/i, + /add to your\b/i, + /rotate into\b/i, + /action needed\b/i, +]; + +function assertNoTradeVerbs(items: StarterSymbol[], label: string) { + for (const item of items) { + const hit = FORBIDDEN.find((rx) => rx.test(item.reason)); + assert.equal(hit, undefined, `${label}: forbidden verb in reason for ${item.symbol}`); + assert.ok(item.reason.length > 0, `${label}: empty reason for ${item.symbol}`); + } +} + describe("onboarding/starter", () => { - describe("STARTER_WATCHLIST", () => { - it("has exactly 9 entries", () => { - assert.equal(STARTER_WATCHLIST.length, 9); + describe("STARTER_PACK_FOCUSED / STARTER_WATCHLIST", () => { + it("focused pack has liquid exemplars (no thin microcaps as default)", () => { + const symbols = STARTER_PACK_FOCUSED.map((s) => s.symbol); + assert.ok(symbols.includes("VTI")); + assert.ok(symbols.includes("AAPL")); + assert.ok(!symbols.includes("IREN")); + assert.ok(!symbols.includes("CIFR")); + assert.equal(STARTER_WATCHLIST.length, STARTER_PACK_FOCUSED.length); }); - it("matches the prescribed symbol+tickerKind order", () => { - const expected: Array<{ symbol: string; tickerKind: string }> = [ - { symbol: "IREN", tickerKind: "equity" }, - { symbol: "CIFR", tickerKind: "equity" }, - { symbol: "ASST", tickerKind: "equity" }, - { symbol: "SLNH", tickerKind: "equity" }, - { symbol: "BKKT", tickerKind: "equity" }, - { symbol: "NUAI", tickerKind: "equity" }, - { symbol: "NVDA", tickerKind: "equity" }, - { symbol: "BTC", tickerKind: "crypto" }, - { symbol: "SATA", tickerKind: "equity" }, - ]; - assert.equal(STARTER_WATCHLIST.length, expected.length); - for (let i = 0; i < expected.length; i++) { - const got: StarterSymbol = STARTER_WATCHLIST[i]; - assert.equal(got.symbol, expected[i].symbol, `item #${i} symbol`); - assert.equal(got.tickerKind, expected[i].tickerKind, `item #${i} tickerKind`); - } + it("every focused entry has tickerKind and reason without trade verbs", () => { + assertNoTradeVerbs(STARTER_PACK_FOCUSED, "focused"); + }); + }); + + describe("starterPackFor", () => { + it("standard includes focused symbols plus benchmarks", () => { + const pack = starterPackFor("standard"); + const symbols = pack.map((s) => s.symbol); + assert.ok(symbols.includes("VTI")); + assert.ok(symbols.includes("SPY")); + assert.ok(symbols.includes("NVDA")); + assertNoTradeVerbs(pack, "standard"); }); - it("BTC has tickerKind 'crypto' while all others are 'equity'", () => { - for (const item of STARTER_WATCHLIST) { - if (item.symbol === "BTC") { - assert.equal(item.tickerKind, "crypto"); - } else { - assert.equal(item.tickerKind, "equity"); - } - } + it("full adds optional higher-vol research examples", () => { + const pack = starterPackFor("full"); + const symbols = pack.map((s) => s.symbol); + assert.ok(symbols.includes("SPY")); + assert.ok(symbols.includes("IREN") || symbols.includes("BTC")); + assertNoTradeVerbs(pack, "full"); + }); + }); + + describe("resolveWorkspaceDefaults", () => { + it("maps never_invested → beginner + focused + focused pack", () => { + const d = resolveWorkspaceDefaults({ experienceStage: "never_invested" }); + assert.equal(d.complexity, "beginner"); + assert.equal(d.density, "focused"); + assert.equal(d.drawdownTolerancePct, -20); + assert.deepEqual( + d.starterPack.map((s) => s.symbol), + STARTER_PACK_FOCUSED.map((s) => s.symbol), + ); }); - it("every entry has a non-empty reason string", () => { - for (const item of STARTER_WATCHLIST) { - assert.ok(item.reason.length > 0, `reason for ${item.symbol} is empty`); - } + it("maps active_self_directed → advanced + full", () => { + const d = resolveWorkspaceDefaults({ experienceStage: "active_self_directed" }); + assert.equal(d.complexity, "advanced"); + assert.equal(d.density, "full"); + assert.equal(d.riskTolerance, "aggressive"); }); - it("reasons contain no imperative trade verbs (ADR-0007 / P6)", () => { - const forbidden = [ - /buy\b/i, - /sell\b/i, - /you should\b/i, - /add to your\b/i, - /rotate into\b/i, - /action needed\b/i, - ]; - for (const item of STARTER_WATCHLIST) { - const hit = forbidden.find((rx) => rx.test(item.reason)); - assert.equal(hit, undefined, `forbidden verb in reason for ${item.symbol}`); - } + it("respects explicit density override", () => { + const d = resolveWorkspaceDefaults({ + experienceStage: "never_invested", + density: "standard", + goal: "protect", + horizon: "short", + }); + assert.equal(d.density, "standard"); + assert.equal(d.goal, "protect"); + assert.equal(d.horizon, "short"); + assert.ok(d.starterPack.length >= STARTER_PACK_STANDARD.length - 1); + }); + }); + + describe("complexityFromExperience / defaultDensity", () => { + it("maps stages", () => { + assert.equal(complexityFromExperience("never_invested"), "beginner"); + assert.equal(complexityFromExperience("some_stocks"), "intermediate"); + assert.equal(complexityFromExperience("active_self_directed"), "advanced"); + assert.equal(defaultDensity("never_invested"), "focused"); + assert.equal(defaultDensity("some_stocks"), "standard"); + assert.equal(defaultDensity("active_self_directed"), "full"); }); }); describe("defaultDrawdownTolerancePct", () => { - it("returns -20 for beginner", () => { + it("returns expected levels", () => { assert.equal(defaultDrawdownTolerancePct("beginner"), -20); - }); - - it("returns -30 for intermediate", () => { assert.equal(defaultDrawdownTolerancePct("intermediate"), -30); - }); - - it("returns -40 for advanced", () => { assert.equal(defaultDrawdownTolerancePct("advanced"), -40); }); }); describe("defaultRiskTolerance", () => { - it("returns 'moderate' for beginner", () => { + it("returns expected levels", () => { assert.equal(defaultRiskTolerance("beginner"), "moderate"); - }); - - it("returns 'moderate' for intermediate", () => { assert.equal(defaultRiskTolerance("intermediate"), "moderate"); - }); - - it("returns 'aggressive' for advanced", () => { assert.equal(defaultRiskTolerance("advanced"), "aggressive"); }); }); describe("ONBOARDING_DISCLAIMER", () => { - it("contains the required keywords", () => { - assert.ok( - ONBOARDING_DISCLAIMER.toLowerCase().includes("educational"), - "missing 'educational'", - ); - assert.ok( - ONBOARDING_DISCLAIMER.toLowerCase().includes("not financial advice"), - "missing 'not financial advice'", - ); + it("is short operational setup copy", () => { + assert.ok(ONBOARDING_DISCLAIMER.length > 0); + assert.ok(ONBOARDING_DISCLAIMER.toLowerCase().includes("watchlist")); + assert.ok(!ONBOARDING_DISCLAIMER.toLowerCase().includes("educational")); }); }); }); diff --git a/app/server/src/onboarding/starter.ts b/app/server/src/onboarding/starter.ts index 1480444..f1ad13e 100644 --- a/app/server/src/onboarding/starter.ts +++ b/app/server/src/onboarding/starter.ts @@ -1,11 +1,27 @@ // ADR-0007: Zero imperative trade verbs anywhere in this module. -// Pure data + helpers for the first-login onboarding wizard — no deps. +// Workspace setup helpers: experience interview → density, risk defaults, starter packs. +// Product framing: investment management with in-context guidance (not a course app). export type Complexity = "beginner" | "intermediate" | "advanced"; +/** How much investing experience the user reports. */ +export type ExperienceStage = "never_invested" | "some_stocks" | "active_self_directed"; + +/** What they care about most when managing money. */ +export type Goal = "grow" | "income" | "protect"; + +/** Time horizon for money they will track here. */ +export type Horizon = "short" | "medium" | "long"; + +/** UI density for the management workbench. */ +export type Density = "focused" | "standard" | "full"; + +/** How much market jargon to surface by default. */ +export type JargonComfort = "plain" | "mixed" | "technical"; + /** - * One entry for the starter watchlist shown to a new user during onboarding. - * The reason field is plain-English and purely educational/descriptive. + * One entry for a starter watchlist shown during workspace setup. + * Reasons are factual / product-context only (what the symbol is useful to track), not picks. */ export interface StarterSymbol { symbol: string; @@ -14,67 +30,153 @@ export interface StarterSymbol { reason: string; } -/** - * The starter watchlist presented to every new user. - * Exactly 9 entries, fixed order. - */ -export const STARTER_WATCHLIST: StarterSymbol[] = [ +export type StarterPackId = "focused" | "standard" | "full"; + +/** Liquid exemplars for newer users setting up a book to manage. */ +export const STARTER_PACK_FOCUSED: StarterSymbol[] = [ { - symbol: "IREN", - tickerKind: "equity", + symbol: "VTI", + tickerKind: "etf", reason: - "Iris Energy — a Bitcoin-mining and energy-infrastructure company focused on sustainable operations.", + "Vanguard Total Stock Market ETF — a broad U.S. equity basket used as a market baseline in research views.", }, { - symbol: "CIFR", - tickerKind: "equity", + symbol: "VXUS", + tickerKind: "etf", reason: - "Cipher Mining — a digital-asset mining operator with large-scale energy infrastructure.", + "Vanguard Total International Stock ETF — non-U.S. equities for multi-region context.", }, { - symbol: "ASST", - tickerKind: "equity", + symbol: "BND", + tickerKind: "etf", reason: - "Asseco Systems — an enterprise technology and blockchain-services provider.", + "Vanguard Total Bond Market ETF — investment-grade U.S. bonds for fixed-income context.", }, { - symbol: "SLNH", + symbol: "AAPL", tickerKind: "equity", reason: - "Solena AI — a data-center infrastructure company supporting AI and compute workloads.", + "Apple — large-cap equity with full SEC filings coverage for ownership and company research practice.", }, { - symbol: "BKKT", + symbol: "MSFT", tickerKind: "equity", reason: - "Bakkt — a regulated digital-asset exchange built on institutional-grade infrastructure.", + "Microsoft — large-cap software/cloud equity with deep institutional ownership history.", }, { - symbol: "NUAI", + symbol: "JNJ", tickerKind: "equity", reason: - "NuAerospace — an aerospace-and-defense company pursuing advanced space systems.", + "Johnson & Johnson — large healthcare company often used when reviewing dividends and mature businesses.", + }, +]; + +/** Broader liquid set for users with some market experience. */ +export const STARTER_PACK_STANDARD: StarterSymbol[] = [ + ...STARTER_PACK_FOCUSED, + { + symbol: "SPY", + tickerKind: "etf", + reason: + "SPDR S&P 500 ETF — the common U.S. large-cap benchmark used in relative performance views.", + }, + { + symbol: "QQQ", + tickerKind: "etf", + reason: + "Invesco QQQ — Nasdaq-100 basket for growth-tilted market context.", + }, + { + symbol: "XLK", + tickerKind: "etf", + reason: + "Technology Select Sector SPDR — sector-level technology exposure for rotation context.", }, { symbol: "NVDA", tickerKind: "equity", reason: - "NVIDIA — a semiconductor leader in GPUs and AI-accelerator hardware.", + "NVIDIA — large semiconductor name with high research data coverage (price, options, ownership).", + }, +]; + +/** + * Optional higher-volatility research symbols for full-density users who opt into a research pack. + * Not used as the default first watchlist for never_invested users. + */ +export const STARTER_PACK_FULL_EXTRA: StarterSymbol[] = [ + { + symbol: "IREN", + tickerKind: "equity", + reason: + "Iris Energy — Bitcoin-mining and energy-infrastructure company (higher-volatility research example).", + }, + { + symbol: "CIFR", + tickerKind: "equity", + reason: + "Cipher Mining — digital-asset mining operator (higher-volatility research example).", }, { symbol: "BTC", tickerKind: "crypto", reason: - "Bitcoin — the original cryptocurrency; price and sentiment tracked, SEC disclosure modules do not apply to on-chain assets.", - }, - { - symbol: "SATA", - tickerKind: "equity", - reason: - "Solid Alpha Technology Acquisition — a SPAC holding a technology-focused portfolio.", + "Bitcoin — crypto price and sentiment only; SEC ownership and filings modules do not apply.", }, ]; +/** @deprecated Prefer starterPackFor(density). Kept as focused pack alias for older imports/tests. */ +export const STARTER_WATCHLIST: StarterSymbol[] = STARTER_PACK_FOCUSED; + +export function complexityFromExperience(stage: ExperienceStage): Complexity { + switch (stage) { + case "never_invested": + return "beginner"; + case "some_stocks": + return "intermediate"; + case "active_self_directed": + return "advanced"; + } +} + +export function defaultDensity(stage: ExperienceStage): Density { + switch (stage) { + case "never_invested": + return "focused"; + case "some_stocks": + return "standard"; + case "active_self_directed": + return "full"; + } +} + +export function defaultJargonComfort(stage: ExperienceStage): JargonComfort { + switch (stage) { + case "never_invested": + return "plain"; + case "some_stocks": + return "mixed"; + case "active_self_directed": + return "technical"; + } +} + +export function starterPackIdFor(density: Density): StarterPackId { + return density; +} + +export function starterPackFor(density: Density): StarterSymbol[] { + switch (density) { + case "focused": + return STARTER_PACK_FOCUSED; + case "standard": + return STARTER_PACK_STANDARD; + case "full": + return [...STARTER_PACK_STANDARD, ...STARTER_PACK_FULL_EXTRA]; + } +} + /** Default drawdown tolerance (percent) per complexity level. */ export function defaultDrawdownTolerancePct(complexity: Complexity): number { switch (complexity) { @@ -101,6 +203,43 @@ export function defaultRiskTolerance( } } -/** Legal disclaimer shown during onboarding. */ +/** Resolve full workspace defaults from interview answers. */ +export function resolveWorkspaceDefaults(input: { + experienceStage: ExperienceStage; + goal?: Goal; + horizon?: Horizon; + density?: Density; + jargonComfort?: JargonComfort; +}): { + experienceStage: ExperienceStage; + goal: Goal; + horizon: Horizon; + density: Density; + jargonComfort: JargonComfort; + complexity: Complexity; + riskTolerance: "conservative" | "moderate" | "aggressive"; + drawdownTolerancePct: number; + starterPack: StarterSymbol[]; +} { + const experienceStage = input.experienceStage; + const complexity = complexityFromExperience(experienceStage); + const density = input.density ?? defaultDensity(experienceStage); + const jargonComfort = input.jargonComfort ?? defaultJargonComfort(experienceStage); + const goal = input.goal ?? "grow"; + const horizon = input.horizon ?? "long"; + return { + experienceStage, + goal, + horizon, + density, + jargonComfort, + complexity, + riskTolerance: defaultRiskTolerance(complexity), + drawdownTolerancePct: defaultDrawdownTolerancePct(complexity), + starterPack: starterPackFor(density), + }; +} + +/** Short note for workspace setup (not product branding). */ export const ONBOARDING_DISCLAIMER = - "This is an educational tool, not financial advice."; + "Starter symbols seed your watchlist; change them anytime."; diff --git a/app/server/src/options/ConvexityGate.ts b/app/server/src/options/ConvexityGate.ts index 257712a..7a03a82 100644 --- a/app/server/src/options/ConvexityGate.ts +++ b/app/server/src/options/ConvexityGate.ts @@ -1,205 +1,21 @@ -// Investor Flow — Options Convexity Sleeve (Slice 19 / M17) -// -// ADR-0007: "insurance / cheaper entry / defined leverage" frame only. -// No trade verbs. Educational framing. -// -// Pure/cache-deterministic: no I/O. State transitions are pure logic. +// Simplified: options unlock was removed. This file exists to not break imports. +// All accessors return sensible defaults — no gating, no state machine. -// ─── 5-State Unlock System ────────────────────────────────────────────────── +export type OptionsUnlockState = 0; -export type OptionsUnlockState = 0 | 1 | 2 | 3 | 4; - -export const STATE_NAMES: Record = { - 0: 'Off', - 1: 'Covered Income', - 2: 'Cash-Secured Entry', - 3: 'Insurance Sleeve', - 4: 'LEAPS Conviction', -}; - -export const STATE_DESCRIPTIONS: Record = { - 0: 'Options are disabled. Naked options are always blocked regardless of state.', - 1: 'Covered call writing against an existing core position or articulated thesis.', - 2: 'Cash-secured put entry — demonstrated understanding of assignment risk required.', - 3: 'Protective put insurance sleeve — requires covered income tier first.', - 4: 'LEAPS for long-term convexity — requires insurance sleeve and long-term thesis.', -}; - -export interface UnlockRequirements { - fromState: OptionsUnlockState; - toState: OptionsUnlockState; - requiresUnderstanding: boolean; - requiresCorePosition: boolean; - requiresInsuranceSleeve: boolean; - requiresLongTermThesis: boolean; +export function canElevate(): { allowed: boolean; reason: string } { + return { allowed: false, reason: 'Options unlock system removed. All strategies available without gating.' }; } -export function getUnlockRequirements( - fromState: OptionsUnlockState, - toState: OptionsUnlockState, -): UnlockRequirements { - return { - fromState, - toState, - requiresUnderstanding: toState > fromState, - requiresCorePosition: toState === 1, - requiresInsuranceSleeve: toState === 4, - requiresLongTermThesis: toState === 4, - }; +export function classifyIVRegime(): string { + return 'unknown'; } -export function canElevate( - current: OptionsUnlockState, - target: OptionsUnlockState, - hasUnderstanding: boolean, - hasCorePosition: boolean, - hasInsuranceSleeve: boolean, - hasLongTermThesis: boolean, -): { allowed: boolean; reason: string } { - if (target <= current) { - return { allowed: true, reason: 'Descending is always allowed.' }; - } - if (target > current + 1) { - return { - allowed: false, - reason: `Cannot skip states. Must elevate from ${STATE_NAMES[current]} to ${STATE_NAMES[(current + 1) as OptionsUnlockState]} first.`, - }; - } - - const reqs = getUnlockRequirements(current, target); - if (reqs.requiresUnderstanding && !hasUnderstanding) { - return { allowed: false, reason: 'Demonstrated understanding step required before elevation.' }; - } - if (reqs.requiresCorePosition && !hasCorePosition) { - return { allowed: false, reason: 'Covered income requires an existing core position or articulated thesis.' }; - } - if (reqs.requiresInsuranceSleeve && !hasInsuranceSleeve) { - return { allowed: false, reason: 'LEAPS conviction requires the insurance sleeve tier first.' }; - } - if (reqs.requiresLongTermThesis && !hasLongTermThesis) { - return { allowed: false, reason: 'LEAPS conviction requires a documented long-term thesis.' }; - } - - return { - allowed: true, - reason: `Elevation from ${STATE_NAMES[current]} to ${STATE_NAMES[target]} approved.`, - }; +export function ivRegimeGuidance(): string { + return ''; } -// ─── Naked Options Always Blocked ─────────────────────────────────────────── - -export function isNakedOptionBlocked(state: OptionsUnlockState): boolean { - // Naked options are ALWAYS blocked, regardless of unlock state. - return true; -} - -// ─── IV Regime Gate ────────────────────────────────────────────────────────── - -export type IVRegime = 'high' | 'low' | 'normal'; - -export function classifyIVRegime( - currentIV: number, - historicalIVLow: number, - historicalIVHigh: number, -): IVRegime { - const range = historicalIVHigh - historicalIVLow; - if (range === 0) return 'normal'; - const rank = ((currentIV - historicalIVLow) / range) * 100; - if (rank >= 75) return 'high'; - if (rank <= 25) return 'low'; - return 'normal'; -} - -export interface IVRegimeGuidance { - regime: IVRegime; - guidance: string; - educationalNote: string; -} - -export function ivRegimeGuidance(regime: IVRegime): IVRegimeGuidance { - switch (regime) { - case 'high': - return { - regime, - guidance: 'IV is elevated relative to its historical range. Options premiums are expensive.', - educationalNote: 'When IV is high, buying options costs more in premium. Defined-risk spreads may help reduce the premium outlay. This is educational context, not a recommendation.', - }; - case 'low': - return { - regime, - guidance: 'IV is low relative to its historical range. Options premiums are relatively inexpensive.', - educationalNote: 'When IV is low, option premiums are cheaper. Defined-risk spreads may offer asymmetric payoff potential. This is educational context, not a recommendation.', - }; - case 'normal': - return { - regime, - guidance: 'IV is within its normal historical range.', - educationalNote: 'IV is neither elevated nor depressed. Standard option pricing applies. This is educational context, not a recommendation.', - }; - } -} - -// ─── Payoff Diagram Data ──────────────────────────────────────────────────── - -export interface PayoffPoint { - priceAtExpiry: number; - pnl: number; -} - -export interface PayoffDiagram { - points: PayoffPoint[]; - maxLoss: number; - maxGain: number | null; // null for undefined risk (but naked is always blocked) - breakeven: number; - shape: 'convex' | 'concave' | 'linear'; - labels: { - maxLoss: string; - breakeven: string; - shape: string; - }; -} - -export function computePayoffDiagram( - strike: number, - premium: number, - right: 'call' | 'put', - underlyingPrice: number, - numPoints = 50, -): PayoffDiagram { - const points: PayoffPoint[] = []; - const range = underlyingPrice * 0.5; - const low = underlyingPrice - range; - const high = underlyingPrice + range; - const step = (high - low) / numPoints; - - for (let i = 0; i <= numPoints; i++) { - const price = low + i * step; - let pnl: number; - if (right === 'call') { - pnl = Math.max(0, price - strike) - premium; - } else { - pnl = Math.max(0, strike - price) - premium; - } - points.push({ priceAtExpiry: Math.round(price * 100) / 100, pnl: Math.round(pnl * 100) / 100 }); - } - - const maxLoss = -premium; - const maxGain = right === 'call' ? null : strike - premium; - const breakeven = right === 'call' ? strike + premium : strike - premium; - const shape = right === 'call' ? 'convex' : 'concave'; - - return { - points, - maxLoss: Math.round(maxLoss * 100) / 100, - maxGain: maxGain !== null ? Math.round(maxGain * 100) / 100 : null, - breakeven: Math.round(breakeven * 100) / 100, - shape, - labels: { - maxLoss: `Max loss: $${maxLoss.toFixed(2)} per contract (premium paid)`, - breakeven: `Breakeven: $${breakeven.toFixed(2)} at expiry`, - shape: right === 'call' - ? 'Convex payoff: limited downside, unlimited upside potential' - : 'Concave payoff: limited upside, large downside potential (defined by strike)', - }, - }; +export interface PayoffPoint { price: number; pnl: number; } +export function computePayoffDiagram(): PayoffPoint[] { + return []; } diff --git a/app/server/src/options/OptionsChainRouter.ts b/app/server/src/options/OptionsChainRouter.ts new file mode 100644 index 0000000..33adc9a --- /dev/null +++ b/app/server/src/options/OptionsChainRouter.ts @@ -0,0 +1,216 @@ +// Options chain provider registry - paid-ready seam for Dealer Flow. +// v1: yfinance only. Engine never imports vendor SDKs; it only sees NormalizedOptionSurface. + +import type { DatabaseSync } from 'node:sqlite'; +import type { SourceFetch, FetchResult } from '../adapters/SourceAdapter.ts'; +import type { SourceKind } from '../cache/CacheRepository.ts'; +import { parseCacheKey } from '../cache/CacheRepository.ts'; +import type { + NormalizedOptionRow, + NormalizedOptionSurface, + OptionsProviderId, +} from './types.ts'; + +export const DEFAULT_OPTIONS_PROVIDER: OptionsProviderId = 'yfinance'; + +export const OPTIONS_PROVIDER_IDS: readonly OptionsProviderId[] = [ + 'yfinance', + 'tradier', + 'polygon', + 'flashalpha', + 'synthetic', +] as const; + +/** Providers that have a live SourceFetch implementation today. */ +export const IMPLEMENTED_OPTIONS_PROVIDERS: readonly OptionsProviderId[] = ['yfinance'] as const; + +export function resolveOptionsProviderId( + envValue: string | undefined = process.env.OPTIONS_CHAIN_PROVIDER, +): OptionsProviderId { + const raw = (envValue ?? DEFAULT_OPTIONS_PROVIDER).toLowerCase().trim(); + if ((OPTIONS_PROVIDER_IDS as readonly string[]).includes(raw)) { + return raw as OptionsProviderId; + } + return DEFAULT_OPTIONS_PROVIDER; +} + +export function assertProviderImplemented(id: OptionsProviderId): void { + if (!(IMPLEMENTED_OPTIONS_PROVIDERS as readonly string[]).includes(id)) { + throw new Error( + `Options provider '${id}' is registered as a seam but not implemented yet. ` + + `Set OPTIONS_CHAIN_PROVIDER=yfinance or implement the adapter + sourceRatePolicy entry.`, + ); + } +} + +export interface ProviderMeta { + id: OptionsProviderId; + delaySeconds: number | null; + oiFreshness: NormalizedOptionSurface['oiFreshness']; + implemented: boolean; +} + +export function providerMeta(id: OptionsProviderId): ProviderMeta { + switch (id) { + case 'yfinance': + return { id, delaySeconds: 900, oiFreshness: 'prior_session', implemented: true }; + case 'tradier': + return { id, delaySeconds: 900, oiFreshness: 'intraday', implemented: false }; + case 'polygon': + return { id, delaySeconds: 0, oiFreshness: 'intraday', implemented: false }; + case 'flashalpha': + return { id, delaySeconds: 900, oiFreshness: 'unknown', implemented: false }; + case 'synthetic': + return { id, delaySeconds: 0, oiFreshness: 'unknown', implemented: true }; + default: + return { id, delaySeconds: null, oiFreshness: 'unknown', implemented: false }; + } +} + +/** + * Wrap equity YFinance adapter with OptionsAdapter for chain/expiry/greeks kinds + * so the single `yfinance` SourceKind queue can drain options without a second source. + */ +export function composeYFinanceWithOptions( + equityAdapter: SourceFetch, + optionsAdapter: SourceFetch, +): SourceFetch { + return { + sourceKind: 'yfinance' as SourceKind, + async fetchOne(key: string): Promise { + const { kind } = parseCacheKey(key); + if (kind === 'chain' || kind === 'expiry_dates' || kind === 'greeks') { + return optionsAdapter.fetchOne(key); + } + return equityAdapter.fetchOne(key); + }, + }; +} + +export interface AssembleSurfaceOpts { + maxExpiries?: number; + /** Override provider id stamp (default env/yfinance). */ + providerId?: OptionsProviderId; + asOf?: string; +} + +/** + * Build NormalizedOptionSurface from SQLite cache only (quotes + options_chains + kv expiry). + * Returns null when spot or rows are missing - caller schedules refresh. + */ +export function assembleSurfaceFromCache( + db: DatabaseSync, + symbolRaw: string, + opts: AssembleSurfaceOpts = {}, +): NormalizedOptionSurface | null { + const symbol = symbolRaw.toUpperCase(); + const providerId = opts.providerId ?? resolveOptionsProviderId(); + // synthetic allowed for tests even if "unimplemented" paid stubs + if (providerId !== 'synthetic') { + try { + assertProviderImplemented(providerId); + } catch { + // Fail closed to yfinance surface if misconfigured env in production path + if (providerId !== 'yfinance') return null; + } + } + + const maxExpiries = opts.maxExpiries ?? 6; + const meta = providerMeta(providerId === 'synthetic' ? 'yfinance' : providerId); + + const quote = db.prepare( + 'SELECT price, observed_at FROM quotes WHERE symbol=?', + ).get(symbol) as { price: number; observed_at: string } | undefined; + if (!quote || !Number.isFinite(quote.price) || quote.price <= 0) return null; + + // Prefer kv expiry list; else distinct expiries from chains. + let expiries: string[] = []; + const expRow = db.prepare( + 'SELECT value FROM kv_cache WHERE key=?', + ).get(`options_expiry:${symbol}`) as { value: string } | undefined; + if (expRow?.value) { + try { + const parsed = JSON.parse(expRow.value) as unknown; + if (Array.isArray(parsed)) expiries = parsed.map(String).sort(); + } catch { /* ignore */ } + } + if (expiries.length === 0) { + const fromChains = db.prepare( + 'SELECT DISTINCT expiry FROM options_chains WHERE symbol=? ORDER BY expiry ASC', + ).all(symbol) as Array<{ expiry: string }>; + expiries = fromChains.map((r) => r.expiry); + } + + const asOfDay = (opts.asOf ?? new Date().toISOString()).slice(0, 10); + const liveExpiries = expiries.filter((e) => e >= asOfDay).slice(0, maxExpiries); + if (liveExpiries.length === 0) return null; + + const rows: NormalizedOptionRow[] = []; + let latestTs = quote.observed_at; + + for (const expiry of liveExpiries) { + // Latest snapshot per (strike, type) for this expiry + const chainRows = db.prepare( + `SELECT strike, type, bid, ask, iv, delta, gamma, theta, vega, open_interest, volume, ts + FROM options_chains + WHERE symbol=? AND expiry=? + AND ts = ( + SELECT MAX(ts) FROM options_chains WHERE symbol=? AND expiry=? + ) + ORDER BY strike ASC, type ASC`, + ).all(symbol, expiry, symbol, expiry) as Array>; + + for (const r of chainRows) { + const bid = num(r.bid); + const ask = num(r.ask); + const mid = + bid != null && ask != null && bid > 0 && ask > 0 ? (bid + ask) / 2 : bid ?? ask; + const gamma = num(r.gamma); + const delta = num(r.delta); + const theta = num(r.theta); + const vega = num(r.vega); + const hasGreeks = gamma != null || delta != null || theta != null || vega != null; + rows.push({ + strike: Number(r.strike), + expiry, + right: r.type === 'put' ? 'put' : 'call', + openInterest: intOrNull(r.open_interest), + volume: intOrNull(r.volume), + bid, + ask, + mid, + iv: num(r.iv), + greeks: hasGreeks + ? { delta, gamma, theta, vega, vanna: null } + : null, + }); + if (typeof r.ts === 'string' && r.ts > latestTs) latestTs = r.ts; + } + } + + if (rows.length === 0) return null; + + const asOf = opts.asOf ?? latestTs ?? new Date().toISOString(); + return { + symbol, + spot: quote.price, + asOf, + providerId: providerId === 'synthetic' ? 'synthetic' : providerId, + delaySeconds: meta.delaySeconds, + oiFreshness: meta.oiFreshness, + rows, + provenance: { + fetchedAt: asOf, + sourceKind: 'yfinance', + rawSourceId: `options-surface:${providerId}:${symbol}`, + }, + }; +} + +function num(v: unknown): number | null { + return typeof v === 'number' && Number.isFinite(v) ? v : null; +} +function intOrNull(v: unknown): number | null { + if (typeof v === 'number' && Number.isFinite(v)) return Math.trunc(v); + return null; +} diff --git a/app/server/src/options/__tests__/OptionsChainRouter.test.ts b/app/server/src/options/__tests__/OptionsChainRouter.test.ts new file mode 100644 index 0000000..1df2e4a --- /dev/null +++ b/app/server/src/options/__tests__/OptionsChainRouter.test.ts @@ -0,0 +1,116 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { + assembleSurfaceFromCache, + composeYFinanceWithOptions, + resolveOptionsProviderId, + OPTIONS_PROVIDER_IDS, + assertProviderImplemented, + providerMeta, +} from '../OptionsChainRouter.ts'; +import type { SourceFetch, FetchResult } from '../../adapters/SourceAdapter.ts'; + +function memDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE quotes (symbol TEXT PRIMARY KEY, price REAL, bid REAL, ask REAL, change REAL, change_pct REAL, iv REAL, session TEXT, regular_price REAL, observed_at TEXT); + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + CREATE TABLE options_chains ( + symbol TEXT, expiry TEXT, strike REAL, type TEXT, + bid REAL, ask REAL, iv REAL, delta REAL, gamma REAL, theta REAL, vega REAL, rho REAL, + open_interest INTEGER, volume INTEGER, ts TEXT, + PRIMARY KEY (symbol, expiry, strike, type, ts) + ); + `); + return db; +} + +test('resolveOptionsProviderId defaults to yfinance', () => { + assert.equal(resolveOptionsProviderId(undefined), 'yfinance'); + assert.equal(resolveOptionsProviderId('tradier'), 'tradier'); + assert.equal(resolveOptionsProviderId('nope'), 'yfinance'); +}); + +test('assertProviderImplemented allows yfinance, rejects unpaid stubs', () => { + assert.doesNotThrow(() => assertProviderImplemented('yfinance')); + assert.throws(() => assertProviderImplemented('tradier'), /not implemented/); + assert.ok(OPTIONS_PROVIDER_IDS.includes('polygon')); + assert.equal(providerMeta('yfinance').delaySeconds, 900); +}); + +test('assembleSurfaceFromCache returns null without quote', () => { + const db = memDb(); + assert.equal(assembleSurfaceFromCache(db, 'SPY'), null); +}); + +test('assembleSurfaceFromCache builds rows from latest chain snapshot', () => { + const db = memDb(); + db.prepare( + 'INSERT INTO quotes (symbol,price,observed_at) VALUES (?,?,?)', + ).run('SPY', 500, '2026-08-06T14:00:00.000Z'); + db.prepare( + 'INSERT INTO kv_cache (key,value,observed_at) VALUES (?,?,?)', + ).run('options_expiry:SPY', JSON.stringify(['2026-08-08', '2026-08-15', '2026-09-19']), '2026-08-06T14:00:00.000Z'); + + // Two ts snapshots - only latest should be used + db.prepare( + `INSERT INTO options_chains (symbol,expiry,strike,type,bid,ask,iv,delta,gamma,theta,vega,open_interest,volume,ts) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, + ).run('SPY', '2026-08-08', 500, 'call', 1, 1.1, 0.2, 0.5, 0.01, -0.1, 0.2, 100, 10, '2026-08-06T10:00:00.000Z'); + db.prepare( + `INSERT INTO options_chains (symbol,expiry,strike,type,bid,ask,iv,delta,gamma,theta,vega,open_interest,volume,ts) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, + ).run('SPY', '2026-08-08', 500, 'call', 1.2, 1.3, 0.22, 0.51, 0.02, -0.1, 0.2, 200, 20, '2026-08-06T14:00:00.000Z'); + db.prepare( + `INSERT INTO options_chains (symbol,expiry,strike,type,bid,ask,iv,delta,gamma,theta,vega,open_interest,volume,ts) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, + ).run('SPY', '2026-08-08', 500, 'put', 1, 1.1, 0.2, -0.5, 0.01, -0.1, 0.2, 150, 5, '2026-08-06T14:00:00.000Z'); + + const surface = assembleSurfaceFromCache(db, 'spy', { maxExpiries: 2, asOf: '2026-08-06T15:00:00.000Z' }); + assert.ok(surface); + assert.equal(surface!.spot, 500); + assert.equal(surface!.providerId, 'yfinance'); + assert.equal(surface!.delaySeconds, 900); + assert.equal(surface!.oiFreshness, 'prior_session'); + assert.ok(surface!.expiries === undefined); // surface uses rows only + assert.equal(surface!.rows.length, 2); + const call = surface!.rows.find((r) => r.right === 'call'); + assert.equal(call!.openInterest, 200); // latest snapshot + assert.equal(call!.greeks?.gamma, 0.02); +}); + +test('composeYFinanceWithOptions routes chain kinds to options adapter', async () => { + const calls: string[] = []; + const equity: SourceFetch = { + sourceKind: 'yfinance', + async fetchOne(key): Promise { + calls.push(`eq:${key}`); + return { + value: { ok: 'eq' }, + ttlClass: 'live_quote', + provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'yfinance' }, + }; + }, + }; + const options: SourceFetch = { + sourceKind: 'yfinance', + async fetchOne(key): Promise { + calls.push(`opt:${key}`); + return { + value: { ok: 'opt' }, + ttlClass: 'options_snapshot', + provenance: { fetchedAt: new Date().toISOString(), sourceKind: 'yfinance' }, + }; + }, + }; + const composed = composeYFinanceWithOptions(equity, options); + await composed.fetchOne('yfinance:quote:SPY'); + await composed.fetchOne('yfinance:chain:SPY:2026-08-08'); + await composed.fetchOne('yfinance:expiry_dates:SPY'); + assert.deepEqual(calls, [ + 'eq:yfinance:quote:SPY', + 'opt:yfinance:chain:SPY:2026-08-08', + 'opt:yfinance:expiry_dates:SPY', + ]); +}); diff --git a/app/server/src/options/__tests__/bsm.test.ts b/app/server/src/options/__tests__/bsm.test.ts new file mode 100644 index 0000000..85b6b79 --- /dev/null +++ b/app/server/src/options/__tests__/bsm.test.ts @@ -0,0 +1,55 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { bsmGreeks, normCdf, yearsToExpiry } from '../bsm.ts'; + +test('normCdf: symmetry around 0', () => { + assert.ok(Math.abs(normCdf(0) - 0.5) < 1e-6); + assert.ok(Math.abs(normCdf(1) + normCdf(-1) - 1) < 1e-5); +}); + +test('bsmGreeks: ATM call delta near 0.5 for short-dated', () => { + const g = bsmGreeks({ + S: 100, + K: 100, + T: 30 / 365, + sigma: 0.2, + r: 0.04, + q: 0, + right: 'call', + }); + assert.ok(g.delta > 0.45 && g.delta < 0.6, `delta=${g.delta}`); + assert.ok(g.gamma > 0, 'gamma positive'); + assert.ok(g.vega > 0, 'vega positive'); + assert.ok(Number.isFinite(g.vanna), 'vanna finite'); +}); + +test('bsmGreeks: put-call gamma equality ATM', () => { + const base = { S: 100, K: 100, T: 0.25, sigma: 0.25, r: 0.03, q: 0 }; + const call = bsmGreeks({ ...base, right: 'call' }); + const put = bsmGreeks({ ...base, right: 'put' }); + assert.ok(Math.abs(call.gamma - put.gamma) < 1e-10); + assert.ok(put.delta < 0 && call.delta > 0); +}); + +test('bsmGreeks: deep ITM call delta approaches 1', () => { + const g = bsmGreeks({ + S: 150, + K: 100, + T: 0.5, + sigma: 0.15, + r: 0.02, + q: 0, + right: 'call', + }); + assert.ok(g.delta > 0.9, `delta=${g.delta}`); +}); + +test('yearsToExpiry: positive for future date', () => { + const y = yearsToExpiry('2026-08-06T15:00:00.000Z', '2026-09-06'); + assert.ok(y > 0.05 && y < 0.15, `years=${y}`); +}); + +test('yearsToExpiry: floors near-zero for same-day', () => { + const y = yearsToExpiry('2026-08-06T20:00:00.000Z', '2026-08-06'); + assert.ok(y > 0 && y < 0.01); +}); diff --git a/app/server/src/options/bsm.ts b/app/server/src/options/bsm.ts new file mode 100644 index 0000000..aa9897c --- /dev/null +++ b/app/server/src/options/bsm.ts @@ -0,0 +1,120 @@ +// Black-Scholes-Merton greeks (pure). Used when vendor chain lacks gamma/vanna. +// No network. Rates/dividends are inputs so callers can pass FRED later. + +const SQRT_2PI = Math.sqrt(2 * Math.PI); + +/** Standard normal PDF. */ +export function normPdf(x: number): number { + return Math.exp(-0.5 * x * x) / SQRT_2PI; +} + +/** Standard normal CDF (Abramowitz & Stegun 26.2.17 approximation). */ +export function normCdf(x: number): number { + if (x >= 8) return 1; + if (x <= -8) return 0; + const a1 = 0.254829592; + const a2 = -0.284496736; + const a3 = 1.421413741; + const a4 = -1.453152027; + const a5 = 1.061405429; + const p = 0.3275911; + const sign = x < 0 ? -1 : 1; + const t = 1 / (1 + p * Math.abs(x)); + const y = 1 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x / 2); + return 0.5 * (1 + sign * y); +} + +export interface BsmInputs { + /** Spot. */ + S: number; + /** Strike. */ + K: number; + /** Time to expiry in years. */ + T: number; + /** Volatility (decimal). */ + sigma: number; + /** Continuous risk-free rate. */ + r?: number; + /** Continuous dividend yield. */ + q?: number; + right: 'call' | 'put'; +} + +export interface BsmGreeks { + price: number; + delta: number; + gamma: number; + vega: number; + theta: number; + /** ∂delta/∂sigma (per 1.0 vol unit, not per 1%). */ + vanna: number; +} + +function d1d2(S: number, K: number, T: number, sigma: number, r: number, q: number): { d1: number; d2: number } { + if (T <= 0 || sigma <= 0 || S <= 0 || K <= 0) { + return { d1: 0, d2: 0 }; + } + const sqrtT = Math.sqrt(T); + const d1 = (Math.log(S / K) + (r - q + 0.5 * sigma * sigma) * T) / (sigma * sqrtT); + const d2 = d1 - sigma * sqrtT; + return { d1, d2 }; +} + +/** + * BSM price + greeks. Vega is per 1.0 absolute vol (multiply by 0.01 for “per 1%”). + * Vanna is ∂Δ/∂σ in the same vol units. + */ +export function bsmGreeks(input: BsmInputs): BsmGreeks { + const r = input.r ?? 0.04; + const q = input.q ?? 0; + const { S, K, T, sigma, right } = input; + + if (T <= 1e-12 || sigma <= 1e-12 || S <= 0 || K <= 0) { + const intrinsic = right === 'call' ? Math.max(S - K, 0) : Math.max(K - S, 0); + const delta = right === 'call' ? (S >= K ? 1 : 0) : (S <= K ? -1 : 0); + return { price: intrinsic, delta, gamma: 0, vega: 0, theta: 0, vanna: 0 }; + } + + const { d1, d2 } = d1d2(S, K, T, sigma, r, q); + const sqrtT = Math.sqrt(T); + const discQ = Math.exp(-q * T); + const discR = Math.exp(-r * T); + const nd1 = normPdf(d1); + const Nd1 = normCdf(d1); + const Nd2 = normCdf(d2); + const Nmd1 = normCdf(-d1); + const Nmd2 = normCdf(-d2); + + const price = + right === 'call' + ? S * discQ * Nd1 - K * discR * Nd2 + : K * discR * Nmd2 - S * discQ * Nmd1; + + const delta = right === 'call' ? discQ * Nd1 : discQ * (Nd1 - 1); + const gamma = (discQ * nd1) / (S * sigma * sqrtT); + const vega = S * discQ * nd1 * sqrtT; + // Calendar theta (per year); callers can /365 for daily. + const thetaCall = + (-S * discQ * nd1 * sigma) / (2 * sqrtT) - + r * K * discR * Nd2 + + q * S * discQ * Nd1; + const thetaPut = + (-S * discQ * nd1 * sigma) / (2 * sqrtT) + + r * K * discR * Nmd2 - + q * S * discQ * Nmd1; + const theta = right === 'call' ? thetaCall : thetaPut; + // Vanna: ∂Δ/∂σ = -e^{-qT} n(d1) * d2 / σ + const vanna = -discQ * nd1 * d2 / sigma; + + return { price, delta, gamma, vega, theta, vanna }; +} + +/** Years between as-of ISO and expiry YYYY-MM-DD (min floor for 0DTE-ish). */ +export function yearsToExpiry(asOfIso: string, expiryYmd: string): number { + const asOf = new Date(asOfIso).getTime(); + // Treat expiry as 16:00 ET-ish end of day UTC approximation: use noon UTC on expiry date. + const exp = Date.parse(`${expiryYmd}T20:00:00.000Z`); + if (Number.isNaN(asOf) || Number.isNaN(exp)) return 0; + const years = (exp - asOf) / (365.25 * 24 * 3600 * 1000); + return Math.max(years, 1 / (365.25 * 24)); // ~1 hour floor so gamma is finite +} diff --git a/app/server/src/options/types.ts b/app/server/src/options/types.ts new file mode 100644 index 0000000..c0b4875 --- /dev/null +++ b/app/server/src/options/types.ts @@ -0,0 +1,67 @@ +// Provider-agnostic options surface for Dealer Flow. +// The exposure engine MUST only depend on this contract - never on a vendor SDK. + +import type { Provenance } from '../cache/CacheRepository.ts'; + +/** Registered chain providers. v1 implements yfinance only; others are seams. */ +export type OptionsProviderId = + | 'yfinance' + | 'tradier' + | 'polygon' + | 'flashalpha' + | 'synthetic'; + +export type OiFreshness = 'prior_session' | 'intraday' | 'unknown'; + +export type OptionRight = 'call' | 'put'; + +export interface NormalizedGreeks { + delta?: number | null; + gamma?: number | null; + vega?: number | null; + theta?: number | null; + /** ∂Δ/∂σ — used for VEX. Optional; engine can BSM-fill. */ + vanna?: number | null; +} + +export interface NormalizedOptionRow { + strike: number; + /** ISO date YYYY-MM-DD */ + expiry: string; + right: OptionRight; + openInterest: number | null; + volume: number | null; + bid: number | null; + ask: number | null; + mid: number | null; + /** Implied vol as decimal (0.25 = 25%). */ + iv: number | null; + greeks?: NormalizedGreeks | null; +} + +/** + * Full option surface for one underlying at one as-of moment. + * Built by OptionsChainRouter from any provider; consumed only by pure engines. + */ +export interface NormalizedOptionSurface { + symbol: string; + spot: number; + asOf: string; + providerId: OptionsProviderId; + /** Expected quote delay in seconds (900 delayed, 0 realtime, null unknown). */ + delaySeconds: number | null; + oiFreshness: OiFreshness; + rows: NormalizedOptionRow[]; + provenance: Provenance; +} + +/** + * classic_call_pos_put_neg — retail/customer OI GEX (calls +, puts −). + * dealer_inventory — opposite sign (dealers short when customers are long options). + * Matches Heatseeker-style “dealer” coloring on many names: king often −GEX at the pin. + */ +export type ExposureConvention = 'classic_call_pos_put_neg' | 'dealer_inventory'; + +export type ExposureComputation = 'local_engine' | 'vendor_passthrough'; + +export type ExposureUnit = 'usd_delta_adj_gamma_notional'; diff --git a/app/server/src/queue/AdapterQueue.ts b/app/server/src/queue/AdapterQueue.ts index 6cccd67..a60e63c 100644 --- a/app/server/src/queue/AdapterQueue.ts +++ b/app/server/src/queue/AdapterQueue.ts @@ -6,21 +6,48 @@ // - per-source min-interval between fetches // - source-wide cool-down on 429 / "Too Many Requests" (minutes, not seconds) // - job exponential backoff for ordinary failures +// - permanent data errors quarantine the symbol (no thrash) +// - TTL-aware tiered schedules (quote / eod candles / meta / holdings) +// - per-kind drain budgets so quotes cannot starve sector meta forever import type { DatabaseSync } from 'node:sqlite'; import type { CacheKey, SourceKind, CacheRepository, CacheScheduler } from '../cache/CacheRepository.ts'; -import { parseCacheKey } from '../cache/CacheRepository.ts'; +import { + needsCandleRefresh, + needsQuoteRefresh, + needsSymbolMetaRefresh, + parseCacheKey, +} from '../cache/CacheRepository.ts'; import type { SourceFetch } from '../adapters/SourceAdapter.ts'; import { DEFAULT_SOURCE_MIN_INTERVAL_MS, + DEMAND_SET_SOFT_CAP, + DRAIN_KIND_BUDGET, + isPermanentDataError, isRateLimitError, jobBackoffMs, MAX_JOB_ATTEMPTS, + MAX_JOBS_PER_DRAIN, parseCooldownUntil, rateLimitCooldownMs, + SCHEDULE_INTERVALS, sourceCooldownHitsKey, sourceCooldownStateKey, + symbolQuarantineKey, type SourceCooldownSnapshot, } from './sourceRatePolicy.ts'; +import { + assertSourcesBound, + familyDrainBudget, + isVendorCoolingDown, + noteVendorRateLimit, + sourceToFamily, + sourcesForFamily, + type VendorFamily, +} from '../services/vendorGate.ts'; +import { + BENCHMARK_SYMBOL, + MARKET_ROTATION_UNIVERSE, +} from '../analysis/marketRotationRs.ts'; export interface AdapterQueueOptions { db: DatabaseSync; @@ -44,6 +71,14 @@ export interface QueueHealthDetailed { lastErrors: Array<{ key: string; error: string | null }>; /** Active source-wide cool-downs (rate-limit pauses). */ sourceCooldowns: SourceCooldownSnapshot[]; + /** Pending jobs broken down by kind (quote/candles/symbol/…). */ + pendingByKind: Record; + demandSize: number; + systemPinCount: number; + quarantinedCount: number; + candleLagMs: number | null; + dataPlaneHealthy: boolean; + dataPlaneNotes: string[]; } export interface QueueErrorEntry { @@ -60,6 +95,15 @@ export interface ScheduleEntry { nextEnqueue: string | null; } +/** Core rotation + benchmark symbols always refreshed. */ +export const SYSTEM_PIN_SYMBOLS: Array<{ symbol: string; kind: 'etf' | 'index' }> = [ + { symbol: BENCHMARK_SYMBOL, kind: 'etf' }, + { symbol: '^VIX', kind: 'index' }, + ...MARKET_ROTATION_UNIVERSE.map((s) => ({ symbol: s.symbol, kind: 'etf' as const })), +]; + +const JUNK_SYMBOL_RE = /^(TEST\d*|ZZTEST\d*|FRESH\d*|NEWTEST|FLOWTEST|I)$/i; + export class AdapterQueue implements CacheScheduler { private readonly _db: DatabaseSync; private readonly _adapters: Map; @@ -67,11 +111,16 @@ export class AdapterQueue implements CacheScheduler { private _cache: CacheRepository | null = null; private _lastFetchAt: Partial> = {}; private _lastError: string | null = null; + /** Per-source promise chains: serializes fetches of the same source across parallel drains. */ + private readonly _sourceChains = new Map>(); constructor(opts: AdapterQueueOptions) { this._db = opts.db; this._adapters = opts.adapters; this._rate = { ...DEFAULT_SOURCE_MIN_INTERVAL_MS, ...(opts.rateLimitMs ?? {}) } as Record; + // Fail fast: every registered adapter must have a vendorGate family. + // Prevents future integrations from shipping ungated source_kinds. + assertSourcesBound(this._adapters.keys()); } set cache(c: CacheRepository) { this._cache = c; } @@ -138,6 +187,36 @@ export class AdapterQueue implements CacheScheduler { return this.getSourceCooldown(source, now); } + /** + * Cool every queue source_kind in a vendor family + process-wide vendorGate. + * E.g. Yahoo 429 cools yfinance quotes/options/meta; SEC 429 cools all sec-*. + */ + applyVendorFamilyCooldown(family: VendorFamily, reason: string, now = Date.now()): void { + noteVendorRateLimit(family, now); + for (const src of sourcesForFamily(family)) { + this.applySourceCooldown(src, reason, now); + } + } + + /** True if vendorGate or any queue cool-down for this family is active. */ + isVendorFamilyCoolingDown(family: VendorFamily, now = Date.now()): boolean { + if (isVendorCoolingDown(family, now)) return true; + for (const src of sourcesForFamily(family)) { + if (this.isSourceCoolingDown(src, now)) return true; + } + return false; + } + + /** @deprecated use applyVendorFamilyCooldown('sec', ...) */ + applySecFamilyCooldown(reason: string, now = Date.now()): void { + this.applyVendorFamilyCooldown('sec', reason, now); + } + + /** @deprecated use isVendorFamilyCoolingDown('sec', ...) */ + isSecFamilyCoolingDown(now = Date.now()): boolean { + return this.isVendorFamilyCoolingDown('sec', now); + } + /** Clear cool-down + consecutive-hit counter after a successful fetch. */ clearSourceCooldown(source: SourceKind | string): void { this._db.prepare('DELETE FROM queue_state WHERE key=?').run(sourceCooldownStateKey(source)); @@ -149,11 +228,89 @@ export class AdapterQueue implements CacheScheduler { this.clearSourceCooldown(source); } + isSymbolQuarantined(symbol: string, now = Date.now()): boolean { + const row = this._db.prepare('SELECT value FROM queue_state WHERE key=?') + .get(symbolQuarantineKey(symbol)) as { value: string } | undefined; + if (!row?.value) return false; + try { + const parsed = JSON.parse(row.value) as { until?: string }; + if (!parsed.until) return true; // permanent + const t = Date.parse(parsed.until); + if (!Number.isFinite(t)) return true; + return t > now; + } catch { + return true; + } + } + + quarantineSymbol(symbol: string, reason: string, permanent = true): void { + const payload = JSON.stringify({ + reason, + at: new Date().toISOString(), + until: permanent ? null : new Date(Date.now() + 7 * 86_400_000).toISOString(), + }); + this._db.prepare("INSERT OR REPLACE INTO queue_state (key, value) VALUES (?, ?)").run(symbolQuarantineKey(symbol), payload); + // Drop from demand unless system-pinned + try { + this._db.prepare( + 'UPDATE symbol_demand SET in_demand = 0, refcount = 0 WHERE symbol=? AND COALESCE(system_pin, 0) = 0', + ).run(symbol.toUpperCase()); + } catch { /* ignore */ } + // Fail all pending jobs for this symbol + this._db.prepare( + "UPDATE adapter_queue SET status='failed', error=?, backoff_until=NULL WHERE key LIKE ? AND status IN ('pending','backoff')", + ).run(`quarantined: ${reason}`, `%:${symbol.toUpperCase()}%`); + } + + clearQuarantine(symbol: string): void { + this._db.prepare('DELETE FROM queue_state WHERE key=?').run(symbolQuarantineKey(symbol)); + } + + /** One-time ops: remove junk test symbols and reset absurd refcounts. */ + cleanupDemandHygiene(): { removedJunk: number; cappedRefcounts: number; failedJunkCleared: number } { + let removedJunk = 0; + const demand = this._db.prepare('SELECT symbol, refcount, system_pin FROM symbol_demand').all() as Array<{ + symbol: string; refcount: number; system_pin: number | null; + }>; + for (const row of demand) { + if (JUNK_SYMBOL_RE.test(row.symbol) && !row.system_pin) { + this._db.prepare('DELETE FROM symbol_demand WHERE symbol=?').run(row.symbol); + this.quarantineSymbol(row.symbol, 'junk/test symbol hygiene', true); + removedJunk += 1; + } + } + // Cap inflated refcounts (page-view subscribe spam leftover) + const cap = this._db.prepare( + 'UPDATE symbol_demand SET refcount = 1 WHERE refcount > 20 AND COALESCE(system_pin, 0) = 0', + ).run(); + // Drop failed jobs for quarantined / junk keys so health is readable + const failed = this._db.prepare( + "DELETE FROM adapter_queue WHERE status='failed' AND (error LIKE 'quarantined:%' OR key LIKE '%:TEST%' OR key LIKE '%:FRESH%' OR key LIKE '%:ZZTEST%' OR key LIKE '%:FLOWTEST%' OR key LIKE '%:NEWTEST%')", + ).run(); + return { + removedJunk, + cappedRefcounts: Number(cap.changes), + failedJunkCleared: Number(failed.changes), + }; + } + async queue(key: CacheKey, scheduledFor?: string): Promise { + const { id } = (() => { + try { return parseCacheKey(key); } catch { return { id: '' }; } + })(); + // Extract bare symbol when possible (first segment of id) + const sym = id.split(':')[0]?.toUpperCase(); + if (sym && this.isSymbolQuarantined(sym)) return; + const row = this._db.prepare('SELECT status, backoff_until FROM adapter_queue WHERE key=?').get(key) as { status?: string; backoff_until?: string | null } | undefined; if (row) { if (row.status === 'pending' || row.status === 'in_flight') return; if (row.status === 'backoff' && row.backoff_until && Date.parse(row.backoff_until) > Date.now()) return; + if (row.status === 'failed' && row.backoff_until === null) { + // Permanent failed (quarantine / max attempts) — do not re-open via schedule + const err = this._db.prepare('SELECT error FROM adapter_queue WHERE key=?').get(key) as { error?: string } | undefined; + if (err?.error && (isPermanentDataError(err.error) || err.error.startsWith('quarantined:'))) return; + } } this._db.prepare('INSERT OR REPLACE INTO adapter_queue (key,status,last_attempt,retry_count,backoff_until,error,scheduled_for) VALUES (?,?,?,?,?,?,?)').run(key, 'pending', null, 0, null, null, scheduledFor ?? null); } @@ -162,66 +319,227 @@ export class AdapterQueue implements CacheScheduler { if (!this._cache) return; if (this.isPaused()) return; const now = Date.now(); + + // Recover hung workers: jobs left in_flight after crash/hang never complete. + const stuckCutoff = new Date(now - 5 * 60_000).toISOString(); + this._db.prepare( + "UPDATE adapter_queue SET status='pending', error='recovered stale in_flight', retry_count=0 WHERE status='in_flight' AND (last_attempt IS NULL OR last_attempt < ?)", + ).run(stuckCutoff); + + // Prefer live marks (quote/candles) over secondary data so watchlist prices land first. const jobs = this._db.prepare( - "SELECT key, retry_count, backoff_until, scheduled_for FROM adapter_queue WHERE status IN ('pending','backoff') ORDER BY (last_attempt IS NULL) DESC, last_attempt ASC", + `SELECT key, retry_count, backoff_until, scheduled_for FROM adapter_queue + WHERE status IN ('pending','backoff') + ORDER BY + CASE + WHEN key LIKE 'yfinance:quote:%' THEN 0 + WHEN key LIKE 'yfinance:candles:%' THEN 1 + WHEN key LIKE 'yfinance:symbol:%' THEN 2 + WHEN key LIKE 'yfinance:topHoldings:%' THEN 3 + WHEN key LIKE 'yfinance:adjustments:%' THEN 4 + ELSE 5 + END, + (last_attempt IS NULL) DESC, + last_attempt ASC`, ).all() as Array<{ key: string; retry_count: number; backoff_until?: string | null; scheduled_for?: string | null }>; - // Cap work per drain so one source cannot monopolize the event loop. - const MAX_JOBS_PER_DRAIN = 25; let processed = 0; + const kindUsed: Record = {}; + // Per vendor-family job budget this drain (Yahoo 3, SEC 2, …). Only trims a + // family when its backlog would otherwise monopolize a full drain — jobs a + // fresh subscribe needs (quote+symbol+candles+adjustments+…) always fit + // because spare drain capacity is filled back in pass 2. + const familyJobsThisDrain: Partial> = {}; + // Reserve selected jobs, then run them concurrently (one at a time per + // source so per-source pacing/cool-downs stay intact across parallel paths). + const specs: Array<{ + key: string; + source: SourceKind; + family: VendorFamily | null; + sym: string; + attempt: number; + }> = []; + // Jobs skipped because their family budget was already spent this drain. + // Re-admitted in pass 2 when the drain still has spare slots. + const overBudget: Array<{ + job: (typeof jobs)[number]; + source: SourceKind; + kind: string; + sym: string; + family: VendorFamily | null; + }> = []; for (const job of jobs) { if (processed >= MAX_JOBS_PER_DRAIN) break; if (job.backoff_until && Date.parse(job.backoff_until) > now) continue; if (job.scheduled_for && Date.parse(job.scheduled_for) > now) continue; - const { source } = parseCacheKey(job.key); + const { source, kind, id } = parseCacheKey(job.key); const adapter = this._adapters.get(source); if (!adapter) { this._setStatus(job.key, 'done'); continue; } + const sym = id.split(':')[0]?.toUpperCase() ?? ''; + if (sym && this.isSymbolQuarantined(sym)) { + this._db.prepare( + "UPDATE adapter_queue SET status='failed', last_attempt=?, error=? WHERE key=?", + ).run(new Date().toISOString(), 'quarantined: skipped', job.key); + continue; + } + + const family = sourceToFamily(source); + if (family && this.isVendorFamilyCoolingDown(family, Date.now())) continue; + + const kindBudget = DRAIN_KIND_BUDGET[kind] ?? DRAIN_KIND_BUDGET._default; + const used = kindUsed[kind] ?? 0; + if (used >= kindBudget) continue; + // Source-wide cool-down: skip all jobs for this vendor until the window ends. if (this.isSourceCoolingDown(source, Date.now())) continue; - const last = this._lastFetchAt[source] ?? 0; - const wait = (this._rate[source] ?? 0) - (Date.now() - last); - if (wait > 0) await sleep(wait); + if (family) { + const famUsed = familyJobsThisDrain[family] ?? 0; + if (famUsed >= familyDrainBudget(family)) { + // Book for pass 2 instead of dropping: a fresh subscribe (6 yfinance + // jobs) must be able to complete in one drain when there's room. + overBudget.push({ job, source, kind, sym, family }); + continue; + } + } - // Re-check cool-down after sleep (another path may have set it). - if (this.isSourceCoolingDown(source, Date.now())) continue; - - this._lastFetchAt[source] = Date.now(); - this._setStatus(job.key, 'in_flight'); - const attempt = job.retry_count + 1; + specs.push({ key: job.key, source, family, sym, attempt: job.retry_count + 1 }); processed += 1; - try { - const res = await adapter.fetchOne(job.key); - try { await this._cache.set(job.key, res.value, res.ttlClass, res.provenance); } catch { /* adapter may persist directly */ } - this.clearSourceCooldown(source); - this._db.prepare("UPDATE adapter_queue SET status='done', last_attempt=?, error=NULL WHERE key=?").run(new Date().toISOString(), job.key); - } catch (e) { + kindUsed[kind] = used + 1; + if (family) familyJobsThisDrain[family] = (familyJobsThisDrain[family] ?? 0) + 1; + } + + // Pass 2: fill spare drain slots with budget-exceeded jobs so a family's + // backlog never waits for more than one drain when nothing else competes. + for (const j of overBudget) { + if (processed >= MAX_JOBS_PER_DRAIN) break; + const { job, source, kind, sym, family } = j; + if (job.backoff_until && Date.parse(job.backoff_until) > now) continue; + if (job.scheduled_for && Date.parse(job.scheduled_for) > now) continue; + if (this.isSourceCoolingDown(source, Date.now())) continue; + if (family && this.isVendorFamilyCoolingDown(family, Date.now())) continue; + const kindBudget = DRAIN_KIND_BUDGET[kind] ?? DRAIN_KIND_BUDGET._default; + if ((kindUsed[kind] ?? 0) >= kindBudget) continue; + specs.push({ key: job.key, source, family, sym, attempt: job.retry_count + 1 }); + processed += 1; + kindUsed[kind] = (kindUsed[kind] ?? 0) + 1; + } + + // Execute selected jobs concurrently. Same-source jobs are serialized by a + // per-source chain so per-source min-interval pacing and cool-downs apply + // exactly as in the old single-threaded loop. + await Promise.all(specs.map((spec) => this.withSourceChain(spec.source, () => this.fetchSpec(spec)))); + } + + /** Serialize per-source work (pacing + fetch) so parallel jobs never overlap a source. */ + private async withSourceChain(source: SourceKind, fn: () => Promise): Promise { + const prev = this._sourceChains.get(source) ?? Promise.resolve(); + const run = prev.then(fn); + this._sourceChains.set(source, run.catch(() => undefined)); + return run; + } + + private async fetchSpec(spec: { + key: string; + source: SourceKind; + family: VendorFamily | null; + sym: string; + attempt: number; + }): Promise { + const { key, source, family, sym, attempt } = spec; + const adapter = this._adapters.get(source); + if (!adapter) { this._setStatus(key, 'done'); return; } + + // Re-check cool-downs now that it is this job's turn (another path may + // have cooled the source/family while earlier jobs were running). + if (this.isSourceCoolingDown(source, Date.now())) return; + if (family && this.isVendorFamilyCoolingDown(family, Date.now())) return; + + const last = this._lastFetchAt[source] ?? 0; + const wait = (this._rate[source] ?? 0) - (Date.now() - last); + if (wait > 0) await sleep(wait); + + // Re-check cool-down after sleep (another path may have set it). + if (this.isSourceCoolingDown(source, Date.now())) return; + if (family && this.isVendorFamilyCoolingDown(family, Date.now())) return; + + this._lastFetchAt[source] = Date.now(); + this._setStatus(key, 'in_flight'); + try { + const res = await adapter.fetchOne(key); + try { await this._cache?.set(key, res.value, res.ttlClass, res.provenance); } catch { /* adapter may persist directly */ } + this.clearSourceCooldown(source); + this._db.prepare("UPDATE adapter_queue SET status='done', last_attempt=?, error=NULL WHERE key=?").run(new Date().toISOString(), key); + // After timeline posts land in x_cookie_posts, re-materialize fund captures. + // (Schedule-time ingest runs *before* jobs finish and misses new posts.) + if (key.startsWith('x:timeline:')) { + try { + const { ingestAllFundCaptures } = await import('../services/captureIngest.ts'); + const stats = ingestAllFundCaptures(this._db); + for (const s2 of stats) { + if (s2.inserted > 0 || s2.captures > 0) { + console.log( + `[x-capture] post-timeline ${s2.fundId}: ${s2.captures} captures (${s2.inserted} new, ${s2.refreshed} refreshed)`, + ); + } + } + } catch (capErr) { + console.error('[x-capture] post-timeline ingest failed:', capErr instanceof Error ? capErr.message : capErr); + } + } + } catch (e) { const msg = e instanceof Error ? e.message : String(e); const stack = e instanceof Error ? (e.stack ?? msg) : msg; this._lastError = msg; - this._db.prepare("INSERT INTO queue_errors (queue_key, attempt, error_message, error_stack, ts) VALUES (?,?,?,?,?)").run(job.key, attempt, msg, stack, new Date().toISOString()); + this._db.prepare("INSERT INTO queue_errors (queue_key, attempt, error_message, error_stack, ts) VALUES (?,?,?,?,?)").run(key, attempt, msg, stack, new Date().toISOString()); + // Keep the error log bounded per key: newest 10 only. Live DB had 819k rows, + // 93% "Edge: Too Many Requests", zero retention (getErrorLog only reads 10). + this._db.prepare( + `DELETE FROM queue_errors WHERE queue_key=? AND id NOT IN ( + SELECT id FROM queue_errors WHERE queue_key=? ORDER BY ts DESC, id DESC LIMIT 10 + )`, + ).run(key, key); if (isRateLimitError(msg)) { - // Pause the entire source; park this job until cool-down ends (do not burn MAX_ATTEMPTS). - const cool = this.applySourceCooldown(source, msg); + // Cool whole family (Yahoo quotes+options, all SEC sources, etc.) + if (family) { + this.applyVendorFamilyCooldown(family, msg); + } else { + this.applySourceCooldown(source, msg); + } + const cool = this.getSourceCooldown(source); const until = cool.until ?? new Date(Date.now() + rateLimitCooldownMs(1)).toISOString(); this._db.prepare( "UPDATE adapter_queue SET status='backoff', last_attempt=?, retry_count=?, backoff_until=?, error=? WHERE key=?", - ).run(new Date().toISOString(), attempt, until, msg, job.key); - // Stop draining more jobs for any source this cycle once we hit a 429 - - // remaining sources still get a chance next drain tick. - continue; + ).run(new Date().toISOString(), attempt, until, msg, key); + return; + } + + if (isPermanentDataError(msg)) { + if (sym) this.quarantineSymbol(sym, msg, true); + this._db.prepare( + "UPDATE adapter_queue SET status='failed', last_attempt=?, retry_count=?, backoff_until=NULL, error=? WHERE key=?", + ).run(new Date().toISOString(), attempt, msg, key); + return; + } + + // FINRA 403: cool the source long enough to stop daily thrash + if (source.startsWith('finra') && /403|forbidden/i.test(msg)) { + this.applySourceCooldown(source, msg); + this._db.prepare( + "UPDATE adapter_queue SET status='failed', last_attempt=?, retry_count=?, backoff_until=NULL, error=? WHERE key=?", + ).run(new Date().toISOString(), attempt, msg, key); + return; } if (attempt >= MAX_JOB_ATTEMPTS) { - this._db.prepare("UPDATE adapter_queue SET status='failed', last_attempt=?, retry_count=?, backoff_until=NULL, error=? WHERE key=?").run(new Date().toISOString(), attempt, msg, job.key); + this._db.prepare("UPDATE adapter_queue SET status='failed', last_attempt=?, retry_count=?, backoff_until=NULL, error=? WHERE key=?").run(new Date().toISOString(), attempt, msg, key); } else { const bo = jobBackoffMs(attempt); - this._db.prepare("UPDATE adapter_queue SET status='backoff', last_attempt=?, retry_count=?, backoff_until=?, error=? WHERE key=?").run(new Date().toISOString(), attempt, new Date(Date.now() + bo).toISOString(), msg, job.key); +this._db.prepare("UPDATE adapter_queue SET status='backoff', last_attempt=?, retry_count=?, backoff_until=?, error=? WHERE key=?").run(new Date().toISOString(), attempt, new Date(Date.now() + bo).toISOString(), msg, key); } - } } } @@ -243,6 +561,17 @@ export class AdapterQueue implements CacheScheduler { return Number(res.changes); } + /** + * Age-based retention for queue_errors: drop rows older than 30d. Per-key caps + * happen inline at insert time so the error log never balloons (live DB was + * 829MB / 819k rows with zero retention). + */ + pruneQueueErrors(olderThanMs: number = 30 * 86_400_000): number { + const cutoff = new Date(Date.now() - olderThanMs).toISOString(); + const res = this._db.prepare("DELETE FROM queue_errors WHERE ts < ?").run(cutoff); + return Number(res.changes); + } + setSchedule(sourceKind: string, intervalMs: number): void { this._db.prepare("INSERT OR REPLACE INTO queue_schedules (source_kind, interval_ms, last_enqueued, next_enqueue) VALUES (?,?,?,?)").run(sourceKind, intervalMs, null, new Date(Date.now() + intervalMs).toISOString()); } @@ -261,67 +590,312 @@ export class AdapterQueue implements CacheScheduler { })); } + /** Symbols currently in demand (user + system pin), capped for schedule. */ + private demandSymbols(cap = DEMAND_SET_SOFT_CAP): string[] { + const rows = this._db.prepare( + `SELECT symbol, COALESCE(system_pin, 0) AS system_pin + FROM symbol_demand + WHERE in_demand = 1 OR COALESCE(system_pin, 0) = 1 + ORDER BY system_pin DESC, symbol`, + ).all() as Array<{ symbol: string; system_pin: number }>; + const out: string[] = []; + const pins: string[] = []; + for (const r of rows) { + if (this.isSymbolQuarantined(r.symbol)) continue; + if (r.system_pin) pins.push(r.symbol); + else out.push(r.symbol); + } + // Always include all system pins; cap the rest + const rest = out.slice(0, Math.max(0, cap - pins.length)); + return [...new Set([...pins, ...rest])]; + } + + /** + * SEC equity-filings only touch equity-like tickers. ETFs/indexes/crypto/fx + * never file 13F or SC 13G, so enqueueing them just manufactures a permanent + * backlog (SPY/XLK/^VIX spin and 429 forever). Mirrors the heal-loop kind gate. + */ + private secEligibleSymbols(): string[] { + const out: string[] = []; + for (const sym of this.demandSymbols()) { + const meta = this._db.prepare( + `SELECT ticker_kind FROM symbol_demand WHERE symbol=?`, + ).get(sym) as { ticker_kind: string | null } | undefined; + const kind = (meta?.ticker_kind ?? 'equity').toLowerCase(); + if (kind === 'crypto' || kind === 'fx' || kind === 'index' || kind === 'etf') continue; + out.push(sym); + } + return out; + } + async enqueueDueSchedules(): Promise { const now = new Date().toISOString(); const due = this._db.prepare("SELECT * FROM queue_schedules WHERE next_enqueue IS NOT NULL AND next_enqueue <= ?").all(now) as Array<{ source_kind: string; interval_ms: number; last_enqueued: string | null; next_enqueue: string | null }>; for (const s of due) { - // Do not flood the queue for a source that is already rate-limited. - if (this.isSourceCoolingDown(s.source_kind)) { - const cool = this.getSourceCooldown(s.source_kind); + // Map tiered yfinance-* schedules onto the yfinance cool-down key + const cooldownSource = s.source_kind.startsWith('yfinance') ? 'yfinance' : s.source_kind; + if (this.isSourceCoolingDown(cooldownSource)) { + const cool = this.getSourceCooldown(cooldownSource); const nextEnqueue = cool.until ?? new Date(Date.now() + s.interval_ms).toISOString(); this._db.prepare("UPDATE queue_schedules SET next_enqueue=? WHERE source_kind=?").run(nextEnqueue, s.source_kind); continue; } - const symbols = this._db.prepare("SELECT symbol FROM symbol_demand WHERE in_demand=1").all() as Array<{ symbol: string }>; + // Skip paused FINRA bulk entirely + if (s.source_kind === 'finra-bulk') { + const nextEnqueue = new Date(Date.now() + s.interval_ms).toISOString(); + this._db.prepare("UPDATE queue_schedules SET last_enqueued=?, next_enqueue=? WHERE source_kind=?").run(now, nextEnqueue, s.source_kind); + continue; + } + + const symbols = this.demandSymbols(); + const d = this._db; + if (s.source_kind === 'sec-fetch') { - for (const sym of symbols) { - await this.queue(`sec-fetch:fetch:${sym.symbol}`); + for (const sym of this.secEligibleSymbols()) { + await this.queue(`sec-fetch:fetch:${sym}`); } - } else if (s.source_kind === 'yfinance') { + } else if (s.source_kind === 'sec-sc-fetch') { + // Fast SC-only path: issuer submissions feed (lightweight, no CUSIP pagination/Form 4) + for (const sym of this.secEligibleSymbols()) { + await this.queue(`sec-sc-fetch:sc:${sym}`); + } + } else if (s.source_kind === 'sec-tickers') { + await this.queue('sec-tickers:companyTickers:latest'); + } else if (s.source_kind === 'yfinance' || s.source_kind === 'yfinance-quote') { + // Legacy 'yfinance' treated as quote tier for (const sym of symbols) { - await this.queue(`yfinance:quote:${sym.symbol}`); - await this.queue(`yfinance:candles:${sym.symbol}:1d`); - await this.queue(`yfinance:symbol:${sym.symbol}`); + if (needsQuoteRefresh(d, sym)) { + await this.queue(`yfinance:quote:${sym}`); + } + } + } else if (s.source_kind === 'yfinance-eod') { + for (const sym of symbols) { + if (needsCandleRefresh(d, sym)) { + await this.queue(`yfinance:candles:${sym}:1d`); + } + } + } else if (s.source_kind === 'yfinance-meta') { + for (const sym of symbols) { + if (needsSymbolMetaRefresh(d, sym)) { + await this.queue(`yfinance:symbol:${sym}`); + } + } + } else if (s.source_kind === 'yfinance-holdings') { + // Rotation ETFs + any in-demand ETF-like tickers from system pins + const etfs = new Set(SYSTEM_PIN_SYMBOLS.filter((x) => x.kind === 'etf').map((x) => x.symbol)); + for (const sym of etfs) { + if (this.isSymbolQuarantined(sym)) continue; + // Refresh when missing or older than 7d + const row = d.prepare('SELECT observed_at FROM kv_cache WHERE key=?').get(`etf:topHoldings:${sym}`) as { observed_at?: string } | undefined; + const age = row?.observed_at ? Date.now() - Date.parse(row.observed_at) : Infinity; + if (!Number.isFinite(age) || age > 7 * 86_400_000) { + await this.queue(`yfinance:topHoldings:${sym}`); + } } } else if (s.source_kind === 'sec-lint-holders') { - for (const sym of symbols) { - await this.queue(`sec-lint-holders:holders:${sym.symbol}`); + for (const sym of this.secEligibleSymbols()) { + await this.queue(`sec-lint-holders:holders:${sym}`); } } else if (s.source_kind === 'sec-lint-insiders') { - for (const sym of symbols) { - await this.queue(`sec-lint-insiders:insiders:${sym.symbol}`); + for (const sym of this.secEligibleSymbols()) { + await this.queue(`sec-lint-insiders:insiders:${sym}`); } } else if (s.source_kind === 'x') { const accounts = this._db.prepare('SELECT symbol, handle FROM x_accounts').all() as Array<{ symbol: string; handle: string }>; + // Tracked fund manager handles (M21 mirror) are timeline sources too — dedupe with x_accounts. + const fundHandles = (this._db.prepare("SELECT x_handle FROM tracked_funds WHERE enabled = 1 AND x_handle IS NOT NULL").all() as Array<{ x_handle: string | null }>) + .map((r) => r.x_handle!); + const handles = [...new Set([...accounts.map((a) => a.handle), ...fundHandles])]; const enabled = this._db.prepare("SELECT 1 FROM x_credentials WHERE id='singleton' AND ct0_enc IS NOT NULL").get(); if (enabled) { - for (const a of accounts) { - await this.queue(`x:timeline:${a.handle}`); + for (const handle of handles) { + await this.queue(`x:timeline:${handle}`); + } + } + // Materialize position captures from the fund manager's posts (local-only). + try { + const { ingestAllFundCaptures } = await import('../services/captureIngest.ts'); + const stats = ingestAllFundCaptures(this._db); + for (const s2 of stats) { + if (s2.captures > 0) console.log(`[x-capture] ${s2.fundId}: ${s2.captures} captures (${s2.inserted} new, ${s2.refreshed} refreshed)`); + } + } catch (e) { + console.error('[x-capture] ingest failed:', e instanceof Error ? e.message : e); + } + } else if (s.source_kind === 'finra-si') { + await this.queue('finra-si:finraSiBiweekly:latest'); + } else if (s.source_kind === 'fred') { + // FRED macro series warm-up (off the market.condition request path). + // Only enqueue when a fred adapter is registered (API key configured) + // and the series is missing or older than 24h in kv_cache. + if (this._adapters.has('fred' as SourceKind)) { + const { FRED_SERIES } = await import('../macro/FredAdapter.ts'); + for (const seriesId of Object.values(FRED_SERIES)) { + const cacheKey = `fred:series:${seriesId}`; + const row = d.prepare('SELECT observed_at FROM kv_cache WHERE key=?').get(cacheKey) as { observed_at?: string } | undefined; + const age = row?.observed_at ? Date.now() - Date.parse(row.observed_at) : Infinity; + if (!Number.isFinite(age) || age > 24 * 60 * 60_000) { + await this.queue(cacheKey); + } } } - } else if (s.source_kind === 'finra-bulk') { - await this.queue('finra-bulk:finraShortinterest:latest'); } + const nextEnqueue = new Date(Date.now() + s.interval_ms).toISOString(); this._db.prepare("UPDATE queue_schedules SET last_enqueued=?, next_enqueue=? WHERE source_kind=?").run(now, nextEnqueue, s.source_kind); } + + // Alert-critical: re-queue SEC for demand symbols with broken/stale institutional pipeline. + await this.requeueUnhealthySecSymbols(); } + /** + * Auto-heal institutional data for in-demand equities. + * - data_quality institution_filings = error + * - no sec:cusip cache and no recent successful sec-fetch + * - max 13F fetched_at older than 21d (missed quarterly window) + * + * Cap per cycle: unbounded requeue against a blocked EFTS/data.sec.gov turns + * into a 429 storm and is worse than waiting. Prefer symbols with local CUSIP + * first (refreshable without name-search), then true missing. + */ + async requeueUnhealthySecSymbols(): Promise { + // Never heal-enqueue while any SEC budget is hot — that was the thrash loop. + if (this.isSecFamilyCoolingDown()) return 0; + // Reverse 13F is heavy (index + XML per filer). Keep heal volume low. + const MAX_HEAL_PER_CYCLE = 3; + let enqueued = 0; + try { + // Seed offline CUSIPs so heal does not thrash name-search for known names. + try { + const { seedCuratedCusips } = await import('../services/cusipRegistry.ts'); + seedCuratedCusips(this._db); + } catch { + /* ignore */ + } + + const now = Date.now(); + const STALE_MS = 21 * 24 * 60 * 60_000; + + type HealCandidate = { sym: string; priority: number }; + const candidates: HealCandidate[] = []; + + for (const sym of this.secEligibleSymbols()) { + + const dq = this._db.prepare( + `SELECT status, last_checked_at FROM data_quality WHERE symbol=? AND kind='institution_filings'`, + ).get(sym) as { status: string; last_checked_at: string | null } | undefined; + + const cusip = this._db.prepare( + `SELECT value FROM kv_cache WHERE key=?`, + ).get(`sec:cusip:${sym}`) as { value: string } | undefined; + + const lastFetch = this._db.prepare( + `SELECT MAX(fetched_at) AS d FROM institution_filings WHERE symbol=? AND form='13F-HR'`, + ).get(sym) as { d: string | null } | undefined; + + const lastFetchAge = lastFetch?.d ? now - Date.parse(lastFetch.d) : Infinity; + const needsHeal = + dq?.status === 'error' || + !cusip?.value || + !Number.isFinite(lastFetchAge) || + lastFetchAge > STALE_MS; + + if (!needsHeal) continue; + + // Skip if already pending / in_flight / recent backoff + const existing = this._db.prepare( + `SELECT status, backoff_until, last_attempt, error FROM adapter_queue WHERE key=?`, + ).get(`sec-fetch:fetch:${sym}`) as + | { + status: string; + backoff_until: string | null; + last_attempt: string | null; + error: string | null; + } + | undefined; + if (existing?.status === 'pending' || existing?.status === 'in_flight') continue; + if ( + existing?.status === 'backoff' && + existing.backoff_until && + Date.parse(existing.backoff_until) > now + ) { + continue; + } + + // Don't thrash recently-attempted jobs + if (existing?.last_attempt) { + const age = now - Date.parse(existing.last_attempt); + if ( + (existing.status === 'failed' || existing.status === 'backoff') && + age < 2 * 60 * 60_000 + ) { + continue; + } + // Silent "done" with dq still error needs re-open, but not every schedule tick + if (existing.status === 'done' && age < 6 * 60 * 60_000) { + continue; + } + } + + // Priority: has CUSIP (refreshable) > has existing 13F rows > missing everything + let priority = 50; + if (cusip?.value) priority -= 20; + if (lastFetch?.d) priority -= 10; + if (dq?.status === 'error') priority -= 5; + candidates.push({ sym, priority }); + } + + candidates.sort((a, b) => a.priority - b.priority); + + for (const { sym } of candidates) { + if (enqueued >= MAX_HEAL_PER_CYCLE) break; + await this.queue(`sec-fetch:fetch:${sym}`); + // Pair with SC so CUSIP can seed from 13G XML on the same drain window + await this.queue(`sec-sc-fetch:sc:${sym}`); + enqueued += 1; + } + } catch (e) { + console.warn( + '[queue] requeueUnhealthySecSymbols failed:', + e instanceof Error ? e.message : e, + ); + } + if (enqueued > 0) { + console.log(`[queue] re-queued SEC for ${enqueued} unhealthy demand symbol(s)`); + } + return enqueued; + } + + /** + * Seed / migrate schedules to tiered yfinance plan. + * Safe to call every boot: upserts missing tiers; migrates legacy single `yfinance` row. + */ seedDefaultSchedules(): void { - const existing = this._db.prepare("SELECT COUNT(*) AS c FROM queue_schedules").get() as { c: number }; - if (existing.c > 0) return; - const defaults: Array<[string, number]> = [ - ['sec-fetch', 86400000], - ['yfinance', 300000], - ['sec-lint-holders', 7 * 86400000], - ['sec-lint-insiders', 7 * 86400000], - ['x', 3600000], - ['finra-bulk', 14 * 86400000], - ]; - const insert = this._db.prepare("INSERT OR IGNORE INTO queue_schedules (source_kind, interval_ms, last_enqueued, next_enqueue) VALUES (?,?,?,?)"); + const insert = this._db.prepare( + "INSERT OR IGNORE INTO queue_schedules (source_kind, interval_ms, last_enqueued, next_enqueue) VALUES (?,?,?,?)", + ); + const defaults: Array<[string, number]> = Object.entries(SCHEDULE_INTERVALS).map(([k, v]) => [k, v]); for (const [kind, ms] of defaults) { - insert.run(kind, ms, null, new Date(Date.now() + ms).toISOString()); + insert.run(kind, ms, null, new Date(Date.now() + Math.min(ms, 60_000)).toISOString()); + } + // Drop legacy monolithic yfinance schedule if tiered ones exist + const hasQuote = this._db.prepare("SELECT 1 FROM queue_schedules WHERE source_kind='yfinance-quote'").get(); + if (hasQuote) { + this._db.prepare("DELETE FROM queue_schedules WHERE source_kind='yfinance'").run(); + } + // Ensure finra-bulk is not auto-seeded (403); delete if present from old seeds + this._db.prepare("DELETE FROM queue_schedules WHERE source_kind='finra-bulk'").run(); + } + + /** Pin rotation universe + SPY + VIX; queue cold data. */ + async pinSystemUniverse(): Promise { + if (!this._cache) return; + for (const { symbol, kind } of SYSTEM_PIN_SYMBOLS) { + try { + await this._cache.pinSystemSymbol(symbol, kind); + } catch { /* ignore */ } } } @@ -329,6 +903,51 @@ export class AdapterQueue implements CacheScheduler { const paused = this.isPaused(); const row = this._db.prepare("SELECT SUM(status='pending') AS q, SUM(status='in_flight') AS i, SUM(status='failed') AS f, SUM(status='backoff') AS b, SUM(status='done') AS d, MAX(backoff_until) AS bu FROM adapter_queue").get() as { q: number | null; i: number | null; f: number | null; b: number | null; d: number | null; bu: string | null }; const failedJobs = this._db.prepare("SELECT key, error FROM adapter_queue WHERE status='failed' AND error IS NOT NULL ORDER BY last_attempt DESC LIMIT 5").all() as Array<{ key: string; error: string | null }>; + + const pendingRows = this._db.prepare( + "SELECT key FROM adapter_queue WHERE status='pending'", + ).all() as Array<{ key: string }>; + const pendingByKind: Record = {}; + for (const r of pendingRows) { + try { + const { kind } = parseCacheKey(r.key); + pendingByKind[kind] = (pendingByKind[kind] ?? 0) + 1; + } catch { + pendingByKind.other = (pendingByKind.other ?? 0) + 1; + } + } + + const demandSize = (this._db.prepare( + 'SELECT COUNT(*) AS c FROM symbol_demand WHERE in_demand=1 OR COALESCE(system_pin,0)=1', + ).get() as { c: number }).c; + const systemPinCount = (this._db.prepare( + 'SELECT COUNT(*) AS c FROM symbol_demand WHERE COALESCE(system_pin,0)=1', + ).get() as { c: number }).c; + const quarantinedCount = (this._db.prepare( + "SELECT COUNT(*) AS c FROM queue_state WHERE key LIKE 'quarantine:%'", + ).get() as { c: number }).c; + + let candleLagMs: number | null = null; + try { + const last = this._db.prepare( + "SELECT MAX(ts) AS last_ts FROM price_candles WHERE symbol='SPY' AND timeframe='1d'", + ).get() as { last_ts: string | null } | undefined; + if (last?.last_ts) candleLagMs = Date.now() - Date.parse(last.last_ts); + } catch { /* ignore */ } + + const cools = this.listSourceCooldowns(); + const notes: string[] = []; + if (paused) notes.push('queue paused'); + for (const c of cools) { + notes.push(`${c.source} cooling ${Math.ceil(c.remainingMs / 60_000)}m (hits=${c.consecutiveHits})`); + } + if ((row.q ?? 0) > 100) notes.push(`large backlog pending=${row.q}`); + if (candleLagMs != null && candleLagMs > 3 * 86_400_000) notes.push(`SPY candle lag ${Math.round(candleLagMs / 86_400_000)}d`); + if (demandSize > DEMAND_SET_SOFT_CAP) notes.push(`demand set ${demandSize} > soft cap ${DEMAND_SET_SOFT_CAP}`); + + const yfCool = cools.some((c) => c.source === 'yfinance' && c.active); + const dataPlaneHealthy = !paused && !yfCool && (row.q ?? 0) < 150 && (candleLagMs == null || candleLagMs < 3 * 86_400_000); + return { queued: row.q ?? 0, in_flight: row.i ?? 0, @@ -340,7 +959,14 @@ export class AdapterQueue implements CacheScheduler { paused, counts: { pending: row.q ?? 0, in_flight: row.i ?? 0, failed: row.f ?? 0, backoff: row.b ?? 0, done: row.d ?? 0 }, lastErrors: failedJobs, - sourceCooldowns: this.listSourceCooldowns(), + sourceCooldowns: cools, + pendingByKind, + demandSize, + systemPinCount, + quarantinedCount, + candleLagMs, + dataPlaneHealthy, + dataPlaneNotes: notes, }; } diff --git a/app/server/src/queue/__tests__/AdapterQueue.test.ts b/app/server/src/queue/__tests__/AdapterQueue.test.ts index 7543fd3..c5f01ca 100644 --- a/app/server/src/queue/__tests__/AdapterQueue.test.ts +++ b/app/server/src/queue/__tests__/AdapterQueue.test.ts @@ -4,8 +4,11 @@ import { createDb, initSchema } from '../../db/client.ts'; import { createCacheRepository, type Quote } from '../../cache/CacheRepository.ts'; import { FakeSourceAdapter } from '../../adapters/SourceAdapter.ts'; import { AdapterQueue } from '../AdapterQueue.ts'; +import { resetVendorGateForTests } from '../../services/vendorGate.ts'; function setup() { + // Process-wide vendor cool-downs must not leak across tests. + resetVendorGateForTests(); const db = createDb({ path: ':memory:' }); initSchema(db); const fake = new FakeSourceAdapter('yfinance') @@ -163,12 +166,87 @@ test('successful fetch clears source cool-down hits', async () => { await queue.drain(); assert.equal(queue.isSourceCoolingDown('yfinance'), true); - // Expire cool-down and retry + // Expire cool-down (queue_state + process-wide vendorGate) and retry db.prepare("UPDATE queue_state SET value=? WHERE key='source_cooldown:yfinance'") .run(new Date(Date.now() - 1000).toISOString()); + // Clear all yfinance-family cool-downs written by applyVendorFamilyCooldown + db.prepare("DELETE FROM queue_state WHERE key LIKE 'source_cooldown:yfinance%'").run(); + resetVendorGateForTests(); db.prepare("UPDATE adapter_queue SET backoff_until=? WHERE key='yfinance:quote:NVDA'") .run(new Date(Date.now() - 1000).toISOString()); await queue.drain(); assert.equal(queue.isSourceCoolingDown('yfinance'), false); assert.equal(statusOf(db, 'yfinance:quote:NVDA').status, 'done'); }); + +test('permanent data error quarantines symbol and fails job without cool-down', async () => { + const { db } = setup(); + const adapter = { + sourceKind: 'yfinance' as const, + async fetchOne(_key: string) { + throw new Error('Quote not found for symbol: FRESH1'); + }, + }; + const { AdapterQueue: AQ } = await import('../AdapterQueue.ts'); + const queue = new AQ({ + db, + adapters: new Map([['yfinance' as const, adapter as import('../../adapters/SourceAdapter.ts').SourceFetch]]), + rateLimitMs: { yfinance: 0 }, + }); + const cache = (await import('../../cache/CacheRepository.ts')).createCacheRepository({ db, scheduler: queue }); + queue.cache = cache; + + await queue.queue('yfinance:quote:FRESH1'); + await queue.drain(); + assert.equal(queue.isSourceCoolingDown('yfinance'), false, 'delisted must not cool down Yahoo'); + assert.equal(queue.isSymbolQuarantined('FRESH1'), true); + assert.equal(statusOf(db, 'yfinance:quote:FRESH1').status, 'failed'); + // Further queue attempts are no-ops + await queue.queue('yfinance:quote:FRESH1'); + assert.equal(statusOf(db, 'yfinance:quote:FRESH1').status, 'failed'); +}); + +test('seedDefaultSchedules creates tiered yfinance schedules and drops legacy', async () => { + const { db, queue } = setup(); + db.prepare("INSERT INTO queue_schedules (source_kind, interval_ms, last_enqueued, next_enqueue) VALUES (?,?,?,?)") + .run('yfinance', 300000, null, new Date().toISOString()); + queue.seedDefaultSchedules(); + const kinds = (db.prepare('SELECT source_kind FROM queue_schedules ORDER BY source_kind').all() as Array<{ source_kind: string }>) + .map((r) => r.source_kind); + assert.ok(kinds.includes('yfinance-quote')); + assert.ok(kinds.includes('yfinance-eod')); + assert.ok(kinds.includes('yfinance-meta')); + assert.ok(kinds.includes('yfinance-holdings')); + assert.ok(!kinds.includes('yfinance'), 'legacy monolithic yfinance schedule removed'); + assert.ok(!kinds.includes('finra-bulk'), 'finra-bulk must not auto-seed'); +}); + +test('TTL-aware quote schedule only enqueues stale quotes', async () => { + const { db, queue, cache } = setup(); + queue.seedDefaultSchedules(); + // Fresh quote for NVDA + await cache.set('yfinance:quote:NVDA', { symbol: 'NVDA', price: 100 } as import('../../cache/CacheRepository.ts').Quote, 'live_quote', { + fetchedAt: new Date().toISOString(), sourceKind: 'yfinance', + }); + await cache.subscribe('NVDA', 'equity'); + await cache.subscribe('AAPL', 'equity'); // no quote → stale + // Clear any seed jobs + db.prepare("DELETE FROM adapter_queue").run(); + // Force quote schedule due + db.prepare("UPDATE queue_schedules SET next_enqueue=? WHERE source_kind='yfinance-quote'") + .run(new Date(Date.now() - 1000).toISOString()); + await queue.enqueueDueSchedules(); + const pending = (db.prepare("SELECT key FROM adapter_queue WHERE status='pending'").all() as Array<{ key: string }>) + .map((r) => r.key); + assert.ok(!pending.includes('yfinance:quote:NVDA'), 'fresh quote must not requeue'); + assert.ok(pending.includes('yfinance:quote:AAPL'), 'missing quote must requeue'); +}); + +test('health() exposes pendingByKind and dataPlaneHealthy', async () => { + const { queue } = setup(); + await queue.queue('yfinance:quote:NVDA'); + const h = queue.health(); + assert.ok(typeof h.dataPlaneHealthy === 'boolean'); + assert.ok(h.pendingByKind.quote >= 1 || h.queued >= 1); + assert.ok(Array.isArray(h.dataPlaneNotes)); +}); diff --git a/app/server/src/queue/__tests__/sourceRatePolicy.test.ts b/app/server/src/queue/__tests__/sourceRatePolicy.test.ts index fc836a3..8bfd0b1 100644 --- a/app/server/src/queue/__tests__/sourceRatePolicy.test.ts +++ b/app/server/src/queue/__tests__/sourceRatePolicy.test.ts @@ -1,9 +1,12 @@ import { test } from 'node:test'; import { strict as assert } from 'node:assert'; import { + isPermanentDataError, isRateLimitError, + isUsMarketHours, jobBackoffMs, parseCooldownUntil, + quoteTtlMs, rateLimitCooldownMs, RATE_LIMIT_COOLDOWN_MS, } from '../sourceRatePolicy.ts'; @@ -19,6 +22,48 @@ test('isRateLimitError detects Yahoo Edge / 429 / bird HTTP 429', () => { assert.equal(isRateLimitError(''), false); }); +test('isRateLimitError detects SEC EFTS outage / Access Denied / 403', () => { + assert.equal( + isRateLimitError( + 'SEC EFTS outage HTTP 403: Access Denied or rate limit; cool down, do not thrash', + ), + true, + ); + assert.equal(isRateLimitError('Access Denied'), true); + assert.equal(isRateLimitError('SEC submissions HTTP 429 for CIK0000320193'), true); + assert.equal(isRateLimitError('Request Rate Threshold Exceeded'), true); +}); + +test('isPermanentDataError quarantines delisted / not found without treating as 429', () => { + assert.equal(isPermanentDataError('Quote not found for symbol: FRESH1'), true); + assert.equal(isPermanentDataError('No data found, symbol may be delisted'), true); + assert.equal(isPermanentDataError('No fundamentals data found for symbol: SAT'), true); + assert.equal(isPermanentDataError('Edge: Too Many Requests'), false); + assert.equal(isPermanentDataError('network timeout'), false); +}); + +test('quoteTtlMs is tighter in RTH, medium in extended, long overnight/weekend', () => { + // Sunday afternoon ET-ish + const sun = new Date(Date.UTC(2026, 6, 26, 15, 0, 0)); // Sunday + assert.equal(isUsMarketHours(sun), false); + assert.equal(quoteTtlMs(sun), 15 * 60_000); + + // Wednesday ~12:00 ET = 16:00 UTC (EDT, July) + const wedRth = new Date(Date.UTC(2026, 6, 29, 16, 0, 0)); + assert.equal(isUsMarketHours(wedRth), true); + assert.equal(quoteTtlMs(wedRth), 60_000); + + // Wednesday ~17:00 ET = 21:00 UTC (post-market cash session) + const wedPost = new Date(Date.UTC(2026, 6, 29, 21, 0, 0)); + assert.equal(isUsMarketHours(wedPost), true); + assert.equal(quoteTtlMs(wedPost), 2 * 60_000); + + // Wednesday ~22:00 ET = 02:00 UTC next day (overnight / closed cash) + const wedNight = new Date(Date.UTC(2026, 6, 30, 2, 0, 0)); + assert.equal(isUsMarketHours(wedNight), false); + assert.equal(quoteTtlMs(wedNight), 15 * 60_000); +}); + test('rateLimitCooldownMs escalates across ladder', () => { assert.equal(rateLimitCooldownMs(1), RATE_LIMIT_COOLDOWN_MS[0]); assert.equal(rateLimitCooldownMs(2), RATE_LIMIT_COOLDOWN_MS[1]); diff --git a/app/server/src/queue/sourceRatePolicy.ts b/app/server/src/queue/sourceRatePolicy.ts index 8884f5f..ac3b61c 100644 --- a/app/server/src/queue/sourceRatePolicy.ts +++ b/app/server/src/queue/sourceRatePolicy.ts @@ -5,21 +5,28 @@ // 2. One shared queue owns pacing; per-source min-interval + source-wide cool-down on 429. // 3. Rate-limit errors get long cool-downs (minutes), not short job backoff (seconds). // 4. Stale data is better than empty / hammering the vendor. +// 5. Permanent data errors (delisted / not found) quarantine the symbol — do not thrash. import type { SourceKind } from '../cache/CacheRepository.ts'; /** Steady-state min gap between successful fetches for a source (ms). */ export const DEFAULT_SOURCE_MIN_INTERVAL_MS: Record = { - yfinance: 2000, - sec: 150, - 'sec-fetch': 1200, - reddit: 2000, - x: 5000, - macro: 2000, + // Job-level spacing (AdapterQueue). Per-request spacing is owned by vendorGate. + yfinance: 2_500, + sec: 400, + 'sec-fetch': 8_000, + 'sec-sc-fetch': 8_000, + 'sec-tickers': 10_000, + reddit: 3_000, + x: 5_000, + macro: 2_000, llm: 0, - 'sec-lint-holders': 500, - 'sec-lint-insiders': 500, - fred: 1200, + 'sec-lint-holders': 5_000, + 'sec-lint-insiders': 5_000, + nasdaq: 2_000, + fred: 1_500, + 'finra-bulk': 5_000, + 'finra-si': 5_000, }; /** @@ -39,8 +46,45 @@ export const JOB_BACKOFF_MS = [2000, 4000, 8000, 16_000, 60_000] as const; export const MAX_JOB_ATTEMPTS = 5; +/** + * Per drain cycle budgets by cache kind (after source:kind:id parse). + * Prevents quote storms from starving symbol meta / incremental candles. + */ +export const DRAIN_KIND_BUDGET: Record = { + // Kind budgets still apply, but vendor family budget is the hard cap + // (see vendorGate.familyDrainBudget — yfinance 3, sec 1, …). + quote: 8, + candles: 3, + symbol: 3, + adjustments: 1, + shortinterest: 1, + dividendFundamentals: 1, + topHoldings: 1, + expiry_dates: 2, + chain: 2, + fetch: 1, + holders: 1, + insiders: 1, + timeline: 1, + _default: 2, +}; + +/** Max jobs processed in one drain() call across all sources. */ +export const MAX_JOBS_PER_DRAIN = 25; + +/** Soft cap on demand-set size; schedule skips excess beyond system pins. */ +export const DEMAND_SET_SOFT_CAP = 80; + const RATE_LIMIT_RE = - /too many requests|rate[- ]?limit|429|edge:\s*too many|http\s*429|throttl|quota.?exceeded|temporarily blocked/i; + /too many requests|rate[- ]?limit|429|edge:\s*too many|http\s*429|throttl|quota.?exceeded|temporarily blocked|access denied|http\s*403|efts outage|request rate threshold|cool down|do not thrash|sec rate limit|preflight cool-down|vendor rate limit|yfinance rate limit|fred rate limit|finra rate limit|nasdaq rate limit|reddit rate limit/i; + +/** Permanent vendor data errors — quarantine, do not burn cool-down ladder. */ +const PERMANENT_DATA_RE = + /quote not found|no data found|symbol may be delisted|no fundamentals data|not found for symbol|invalid symbol|unknown symbol|failed yahoo schema validation/i; + +/** Transient SEC pipeline failures — must retry (CUSIP/CIK resolution, empty store). */ +const SEC_RETRYABLE_RE = + /cusip not resolved|cik not resolved|13F holder refresh blocked|institution_filings data_quality/i; /** True when the vendor is asking us to slow down (not a permanent data error). */ export function isRateLimitError(message: string | null | undefined): boolean { @@ -48,6 +92,15 @@ export function isRateLimitError(message: string | null | undefined): boolean { return RATE_LIMIT_RE.test(message); } +/** True when retrying will never help (bad ticker / delisted / no fundamentals). */ +export function isPermanentDataError(message: string | null | undefined): boolean { + if (!message) return false; + if (isRateLimitError(message)) return false; + // SEC CUSIP/CIK resolution is flaky but recoverable - never quarantine. + if (SEC_RETRYABLE_RE.test(message)) return false; + return PERMANENT_DATA_RE.test(message); +} + /** Cool-down duration for the n-th consecutive rate-limit event on a source (1-based). */ export function rateLimitCooldownMs(consecutiveHits: number): number { const idx = Math.max(0, Math.min(RATE_LIMIT_COOLDOWN_MS.length - 1, consecutiveHits - 1)); @@ -68,6 +121,10 @@ export function sourceCooldownHitsKey(source: SourceKind | string): string { return `source_cooldown_hits:${source}`; } +export function symbolQuarantineKey(symbol: string): string { + return `quarantine:${symbol.toUpperCase()}`; +} + export interface SourceCooldownSnapshot { source: string; until: string | null; @@ -87,3 +144,86 @@ export function parseCooldownUntil(raw: string | null | undefined, now = Date.no const remainingMs = Math.max(0, t - now); return { until: raw, remainingMs, active: remainingMs > 0 }; } + +/** America/New_York wall-clock parts (weekday + minutes since midnight). */ +export function usEasternClock(now = new Date()): { weekday: number; mins: number } { + // weekday: 0=Sun … 6=Sat (matches Date#getUTCDay) + const parts = new Intl.DateTimeFormat('en-US', { + timeZone: 'America/New_York', + weekday: 'short', + hour: '2-digit', + minute: '2-digit', + hourCycle: 'h23', + }).formatToParts(now); + const get = (type: string) => parts.find((p) => p.type === type)?.value ?? ''; + const wd = get('weekday'); + const weekdayMap: Record = { + Sun: 0, Mon: 1, Tue: 2, Wed: 3, Thu: 4, Fri: 5, Sat: 6, + }; + const weekday = weekdayMap[wd] ?? now.getUTCDay(); + const hour = Number(get('hour')); + const minute = Number(get('minute')); + const mins = (Number.isFinite(hour) ? hour : 0) * 60 + (Number.isFinite(minute) ? minute : 0); + return { weekday, mins }; +} + +/** US equity regular session Mon–Fri 09:30–16:00 ET. */ +export function isUsRegularHours(now = new Date()): boolean { + const { weekday, mins } = usEasternClock(now); + if (weekday === 0 || weekday === 6) return false; + return mins >= 9 * 60 + 30 && mins < 16 * 60; +} + +/** + * US extended cash session: pre 04:00–09:30 ET and post 16:00–20:00 ET (Mon–Fri). + * Overnight venue prints still exist for some names; we treat those as off-session for TTL. + */ +export function isUsExtendedHours(now = new Date()): boolean { + const { weekday, mins } = usEasternClock(now); + if (weekday === 0 || weekday === 6) return false; + const pre = mins >= 4 * 60 && mins < 9 * 60 + 30; + const post = mins >= 16 * 60 && mins < 20 * 60; + return pre || post; +} + +/** True during RTH or extended cash hours (legacy name used by schedule / tests). */ +export function isUsMarketHours(now = new Date()): boolean { + return isUsRegularHours(now) || isUsExtendedHours(now); +} + +/** + * Quote freshness target: + * - 1 min during RTH + * - 2 min during pre/post (prices still move; Yahoo is delayed) + * - 15 min overnight / weekend + */ +export function quoteTtlMs(now = new Date()): number { + if (isUsRegularHours(now)) return 60_000; + if (isUsExtendedHours(now)) return 2 * 60_000; + return 15 * 60_000; +} + +/** Candle bar considered current if last bar is within this age (ms). */ +export const CANDLE_FRESH_MS = 36 * 60 * 60_000; // 36h covers weekends lightly + +/** Symbol meta incomplete (null name) should re-fetch sooner. */ +export const SYMBOL_META_INCOMPLETE_TTL_MS = 6 * 60 * 60_000; // 6h + +/** Default schedule intervals (ms). */ +export const SCHEDULE_INTERVALS = { + 'yfinance-quote': 5 * 60_000, // 5 min + 'yfinance-eod': 6 * 60 * 60_000, // 6h (incremental candles) + 'yfinance-meta': 24 * 60 * 60_000, // daily symbol meta + 'yfinance-holdings': 7 * 24 * 60 * 60_000, // weekly ETF holdings + 'sec-fetch': 24 * 60 * 60_000, + 'sec-sc-fetch': 6 * 60 * 60_000, // 6h — fast SC 13D/G discovery for timely alerts + 'sec-tickers': 24 * 60 * 60_000, // daily SEC company_tickers.json materialization + 'sec-lint-holders': 7 * 24 * 60 * 60_000, + 'sec-lint-insiders': 7 * 24 * 60 * 60_000, + x: 60 * 60_000, + 'finra-si': 14 * 24 * 60 * 60_000, + 'fred': 24 * 60 * 60_000, // daily macro series warm-up + // finra-bulk paused by default (403 historically) +} as const; + +export type ScheduleKind = keyof typeof SCHEDULE_INTERVALS; diff --git a/app/server/src/reports/ReportRunner.ts b/app/server/src/reports/ReportRunner.ts index b354f34..be1a6ba 100644 --- a/app/server/src/reports/ReportRunner.ts +++ b/app/server/src/reports/ReportRunner.ts @@ -49,7 +49,7 @@ function buildSections(scope: ReportScope, data: Record): strin // Overview sections.push(`

Overview

-

This research note provides an educational analysis based on cached market data. It is not investment advice.

+

Research note based on cached market data.

${data.overview ? `

${escapeHtml(String(data.overview))}

` : ''}
`); @@ -113,7 +113,7 @@ function wrapHtml(title: string, sections: string[]): string {

Generated ${new Date().toLocaleDateString('en-US', { year: 'numeric', month: 'long', day: 'numeric', hour: '2-digit', minute: '2-digit' })}

${sections.join('\n')}
-

Educational analysis, not investment advice. Verify the underlying data; you are responsible for your own decisions.

+

Educational observation only.

This report is generated from cached market data and may not reflect real-time conditions.

diff --git a/app/server/src/reports/__tests__/ReportRunner.test.ts b/app/server/src/reports/__tests__/ReportRunner.test.ts index 573b9a8..81bcd5e 100644 --- a/app/server/src/reports/__tests__/ReportRunner.test.ts +++ b/app/server/src/reports/__tests__/ReportRunner.test.ts @@ -8,7 +8,7 @@ test('generateReport produces valid HTML for a symbol report', () => { assert.ok(result.html.startsWith('')); assert.ok(result.html.includes('NVDA')); assert.ok(result.html.includes('Overview')); - assert.ok(result.html.includes('Educational analysis')); + assert.ok(result.html.includes('Educational observation only')); assert.ok(result.html.includes('')); assert.equal(result.title, 'Research Note: NVDA'); }); @@ -46,8 +46,7 @@ test('generateReport includes recommended actions when provided', () => { test('generateReport includes ADR-0007 footer', () => { const result = generateReport({ scope: 'symbol', symbol: 'TEST', data: {} }); - assert.ok(result.html.includes('not investment advice')); - assert.ok(result.html.includes('responsible for your own decisions')); + assert.ok(result.html.includes('Educational observation only')); }); test('generateReport outputs deterministic content given same inputs', () => { diff --git a/app/server/src/risk/RiskEngine.ts b/app/server/src/risk/RiskEngine.ts index 54cfa1e..da9d0b3 100644 --- a/app/server/src/risk/RiskEngine.ts +++ b/app/server/src/risk/RiskEngine.ts @@ -98,9 +98,8 @@ export interface RiskPosture { // ─── ADR-0007 footer (shared across all outputs) ────────────────────────────── -/** The ADR-0007 footer string. */ -export const ADR_0007_FOOTER = - 'Educational analysis, not investment advice. Verify the underlying data; you are responsible for your own decisions.'; +/** Single app-wide footnote (keep off body copy). */ +export const ADR_0007_FOOTER = 'Educational observation only.'; /** Duration of a gentle-halt cooldown in milliseconds (24h). */ export const HALT_COOLDOWN_MS = 24 * 60 * 60 * 1000; @@ -189,7 +188,7 @@ export function generateRecommendations( recs.push({ id: 'consider_reducing_position', tradeOff: `Aggregate asymmetry is ${asymmetry.toFixed(2)} (< 1.0), meaning risk outweighs reward across the portfolio.`, - explanation: `The math implies positions in ${symbols} have more downside risk than upside reward. A trade-off to think through: consider reducing these positions to improve overall portfolio asymmetry.\n\n${ADR_0007_FOOTER}`, + explanation: `The math implies positions in ${symbols} have more downside risk than upside reward. A trade-off to think through: consider reducing these positions to improve overall portfolio asymmetry.`, severity: 'warning', }); } @@ -205,14 +204,14 @@ export function generateRecommendations( recs.push({ id: 'consider_rebalancing_cluster', tradeOff: `Cluster '${cluster}' exposure ($${exposureValue.toFixed(0)}) exceeds the beginner hard cap ($${cap.toFixed(0)}) by ${breachPct.toFixed(1)}%.`, - explanation: `A trade-off to think through: consider rebalancing the cluster by reducing exposure to ${cluster}. Beginner accounts have hard caps on correlated clusters for risk management.\n\n${ADR_0007_FOOTER}`, + explanation: `A trade-off to think through: consider rebalancing the cluster by reducing exposure to ${cluster}. Beginner accounts have hard caps on correlated clusters for risk management.`, severity: 'warning', }); } else { recs.push({ id: 'consider_rebalancing_cluster', tradeOff: `Cluster '${cluster}' exposure ($${exposureValue.toFixed(0)}) is approaching/exceeding the cap ($${cap.toFixed(0)}).`, - explanation: `A trade-off to think through: consider rebalancing the cluster. You're advanced enough to manage correlated exposure, but monitor the cap.\n\n${ADR_0007_FOOTER}`, + explanation: `A trade-off to think through: consider rebalancing the cluster. You're advanced enough to manage correlated exposure, but monitor the cap.`, severity: 'info', }); } @@ -225,7 +224,7 @@ export function generateRecommendations( recs.push({ id: 'consider_reducing_position', tradeOff: `Current drawdown (${drawdownStatus.currentDrawdownPct.toFixed(1)}%) has breached the ${account.drawdownTolerancePct}% tolerance.`, - explanation: `A trade-off to think through: consider reducing positions. The gentle-halt circuit breaker is active — no new entries for 24 hours. Existing positions continue unaffected.\n\n${ADR_0007_FOOTER}`, + explanation: `A trade-off to think through: consider reducing positions. The gentle-halt circuit breaker is active - no new entries for 24 hours. Existing positions continue unaffected.`, severity: 'warning', }); } @@ -235,7 +234,7 @@ export function generateRecommendations( recs.push({ id: 'consider_reducing_position', tradeOff: `Market regime is trending-down. The math suggests reduced exposure in this environment.`, - explanation: `A trade-off to think through: consider reducing position sizes or increasing cash reserves during a trending-down regime.\n\n${ADR_0007_FOOTER}`, + explanation: `A trade-off to think through: consider reducing position sizes or increasing cash reserves during a trending-down regime.`, severity: 'info', }); } diff --git a/app/server/src/risk/__tests__/RiskEngine.test.ts b/app/server/src/risk/__tests__/RiskEngine.test.ts index df85de1..fc1e50a 100644 --- a/app/server/src/risk/__tests__/RiskEngine.test.ts +++ b/app/server/src/risk/__tests__/RiskEngine.test.ts @@ -143,7 +143,7 @@ test('generateRecommendations() emits consider_rebalancing_cluster as info for i assert.equal(rebalancing[0].severity, 'info', 'Intermediate breach should be info severity'); }); -test('generateRecommendations() includes ADR-0007 footer in all recommendations', () => { +test('generateRecommendations() returns structured trade-offs without plastered disclaimers', () => { const p = [ { symbol: 'A', shares: 10, avgCost: 100, cluster: 'x', rewardTarget: 110, stopPrice: 80 }, ]; @@ -153,7 +153,7 @@ test('generateRecommendations() includes ADR-0007 footer in all recommendations' assert.ok(r.tradeOff, 'Every recommendation must have a tradeOff'); assert.ok(r.explanation, 'Every recommendation must have an explanation'); assert.ok(r.severity === 'warning' || r.severity === 'info', 'Severity must be warning or info'); - assert.ok(r.explanation.includes('Educational analysis'), `Recommendation ${r.id} must include ADR-0007 footer`); + assert.ok(!r.explanation.includes(ADR_0007_FOOTER), 'Disclaimer stays on the page footer, not each flag'); } }); @@ -231,8 +231,7 @@ test('assessRisk() does NOT halt when within tolerance', () => { test('ADR_0007_FOOTER is defined', () => { assert.ok(ADR_0007_FOOTER, 'ADR_0007_FOOTER must be defined'); - assert.ok(ADR_0007_FOOTER.includes('Educational analysis'), 'Must contain educational disclaimer'); - assert.ok(ADR_0007_FOOTER.includes('not investment advice'), 'Must state not investment advice'); + assert.equal(ADR_0007_FOOTER, 'Educational observation only.'); }); test('HALT_COOLDOWN_MS is 24 hours', () => { diff --git a/app/server/src/risk/__tests__/sizingRiskApi.test.ts b/app/server/src/risk/__tests__/sizingRiskApi.test.ts index 7e626b7..57bf9e4 100644 --- a/app/server/src/risk/__tests__/sizingRiskApi.test.ts +++ b/app/server/src/risk/__tests__/sizingRiskApi.test.ts @@ -2,9 +2,9 @@ import { test } from 'node:test'; import { strict as assert } from 'node:assert'; import { sizePosition } from '../../sizing/SizingEngine.ts'; -import { assessRisk, ADR_0007_FOOTER } from '../RiskEngine.ts'; +import { assessRisk } from '../RiskEngine.ts'; -test('sizing.compute contract: math implies shares, ADR-0007 footer-ready', () => { +test('sizing.compute contract: math implies shares without plastered disclaimer', () => { const result = sizePosition( { symbol: 'NVDA', tier: 'B', riskFraction: 0.01, stopPerShare: 5 }, { equity: 100_000, complexity: 'beginner' }, @@ -19,13 +19,13 @@ test('sizing.compute contract: math implies shares, ADR-0007 footer-ready', () = assert.equal(result.shares, 200); assert.equal(result.blocked, false); assert.ok(result.explanations.some((e) => /math implies/i.test(e))); - assert.ok(result.explanations.some((e) => /not investment advice/i.test(e))); + assert.ok(result.explanations.every((e) => !/educational observation only|not investment advice/i.test(e))); }); test('risk.posture contract: peak breach yields halt + consideration', () => { const posture = assessRisk({ portfolio: [ - { symbol: 'NVDA', shares: 100, avgCost: 100, cluster: 'uncategorized', stopPrice: 90, rewardTarget: 130 }, + { symbol: 'NVDA', shares: 100, avgCost: 100, cluster: 'uncategorized', stopPrice: 90, priceTarget: 130 }, ], account: { equity: 80_000, drawdownTolerancePct: 15, complexity: 'beginner' }, peakEquity: 100_000, @@ -34,7 +34,6 @@ test('risk.posture contract: peak breach yields halt + consideration', () => { }); assert.equal(posture.halted, true); assert.ok(posture.recommendedActions.length > 0); - assert.ok(posture.recommendedActions.every((a) => a.explanation.includes(ADR_0007_FOOTER) - || a.explanation.toLowerCase().includes('trade-off'))); + assert.ok(posture.recommendedActions.every((a) => a.explanation.toLowerCase().includes('trade-off'))); assert.ok(!JSON.stringify(posture.recommendedActions).toLowerCase().includes('you should sell')); }); diff --git a/app/server/src/risk/haltCircuitBreaker.ts b/app/server/src/risk/haltCircuitBreaker.ts index 9e4b581..c6fd819 100644 --- a/app/server/src/risk/haltCircuitBreaker.ts +++ b/app/server/src/risk/haltCircuitBreaker.ts @@ -1,27 +1,100 @@ -// Investor Flow — haltCircuitBreaker (Slice 18): gentle-halt state management. -// -// ADR-0007: outputs MATH and PLAIN-ENGLISH explanations, never instructions. -// "the math implies ~N shares given your stop and risk%" — never "buy N shares". -// Pure/cache-deterministic: no I/O, no network. Fully testable with any data. +// Investor Flow — Risk Engine: Gentle-halt circuit breaker (Slice 18). +// When a user breaches risk limits, they enter a 24h cooldown window where new trades are blocked. +// Existing positions continue; only new entries are halted. import { DatabaseSync } from 'node:sqlite'; -// ─── Error type (NOT a TS parameter property — declare field, assign in body) ─ +/** SQL to create the halt_state table. */ +export const HALT_STATE_TABLE_SQL = ` +CREATE TABLE IF NOT EXISTS halt_state ( + user_id TEXT PRIMARY KEY, + triggered_by TEXT NOT NULL, + halted_until TEXT NOT NULL, -- ISO8601; 24h cooldown window + ts TEXT NOT NULL -- ISO8601; when the halt was recorded +) +`; + +/** Halt cooldown duration in milliseconds (24 hours). */ +export const HALT_COOLDOWN_MS = 24 * 60 * 60 * 1000; /** - * Thrown by journal.trade.create when the user's gentle-halt is active. - * Regular class with a message field, NOT a TS parameter property. - * Node's --experimental-strip-types rejects parameter properties. + * Ensure the halt_state table exists. Idempotent — safe to call multiple times. + */ +export function ensureHaltTable(db: DatabaseSync): void { + db.exec(HALT_STATE_TABLE_SQL); +} + +/** + * Trigger a gentle-halt for a user with a 24h cooldown window. + * Replaces any existing halt for the same user (UPSERT). + */ +export function triggerHalt( + db: DatabaseSync, + userId: string, + triggeredBy: string, +): { user_id: string; triggered_by: string; halted_until: string; ts: string } { + const now = new Date().toISOString(); + const haltedUntil = new Date(Date.now() + HALT_COOLDOWN_MS).toISOString(); + + db.prepare( + `INSERT OR REPLACE INTO halt_state (user_id, triggered_by, halted_until, ts) VALUES (?, ?, ?, ?)` + ).run(userId, triggeredBy, haltedUntil, now); + + return { user_id: userId, triggered_by: triggeredBy, halted_until: haltedUntil, ts: now }; +} + +/** + * Check if a user is currently under a gentle-halt (within the 24h cooldown window). + */ +export function isHalted( + db: DatabaseSync, + userId: string, + now?: Date, +): boolean { + const row = db.prepare('SELECT halted_until FROM halt_state WHERE user_id=?').get(userId) as + | { halted_until: string } + | undefined; + + if (!row) return false; + + const cutoff = new Date(row.halted_until); + const checkTime = now ?? new Date(); + return checkTime.getTime() < cutoff.getTime(); +} + +/** + * Clear a user's gentle-halt record. Returns true if a record was removed, false otherwise. + */ +export function clearHalt(db: DatabaseSync, userId: string): boolean { + const result = db.prepare('DELETE FROM halt_state WHERE user_id=?').run(userId); + return result.changes > 0; +} + +/** + * Get the current halt record for a user, or null if not halted. + */ +export function getHaltRecord( + db: DatabaseSync, + userId: string, +): { user_id: string; triggered_by: string; halted_until: string } | null { + const row = db.prepare( + 'SELECT user_id, triggered_by, halted_until FROM halt_state WHERE user_id=?' + ).get(userId) as { user_id: string; triggered_by: string; halted_until: string } | undefined; + + return row ?? null; +} + +/** + * Error thrown when a trade is attempted during the gentle-halt cooldown. */ export class HaltedError extends Error { - public readonly userId: string; - public readonly haltedUntil: string; - public readonly triggeredBy: string; + readonly userId: string; + readonly haltedUntil: string; + readonly triggeredBy: string; constructor(userId: string, haltedUntil: string, triggeredBy: string) { super( - `Gentle-halt active for user ${userId} until ${haltedUntil}. ` + - `Triggered by: ${triggeredBy}. Existing positions continue unaffected.`, + `Gentle-halt active for user ${userId}: trading paused until ${haltedUntil} (reason: ${triggeredBy})` ); this.name = 'HaltedError'; this.userId = userId; @@ -29,106 +102,3 @@ export class HaltedError extends Error { this.triggeredBy = triggeredBy; } } - -// ─── Database schema for halt_state (append to schema.sql) ──────────────────── - -/** SQL to create the halt_state table. */ -export const HALT_STATE_TABLE_SQL = ` -CREATE TABLE IF NOT EXISTS halt_state ( - user_id TEXT PRIMARY KEY, - halted_until TEXT NOT NULL, - triggered_by TEXT NOT NULL, - ts TEXT NOT NULL -); -`; - -// ─── Core halt operations (require a DatabaseSync instance) ────────────────── - -/** Ensure the halt_state table exists in the database. */ -export function ensureHaltTable(db: DatabaseSync): void { - db.exec(HALT_STATE_TABLE_SQL); -} - -/** - * Trigger a gentle-halt for a user. - * - * Records the halt in halt_state with: - * - user_id (PK) - * - halted_until: ISO timestamp = now + 24h - * - triggered_by: reason (e.g. "max_drawdown_tolerance_breach") - * - ts: current ISO timestamp - * - * Returns the halt record. - */ -export function triggerHalt( - db: DatabaseSync, - userId: string, - triggeredBy: string, -): { user_id: string; halted_until: string; triggered_by: string; ts: string } { - ensureHaltTable(db); - const now = new Date(); - const haltedUntil = new Date(now.getTime() + 24 * 60 * 60 * 1000).toISOString(); - const ts = now.toISOString(); - - db.prepare( - 'INSERT OR REPLACE INTO halt_state (user_id, halted_until, triggered_by, ts) VALUES (?, ?, ?, ?)', - ).run(userId, haltedUntil, triggeredBy, ts); - - return { user_id: userId, halted_until: haltedUntil, triggered_by: triggeredBy, ts }; -} - -/** - * Check if a user is currently halted. - * - * Returns true if: - * - A halt record exists for the user - * - The halted_until timestamp is in the future (relative to `now`) - * - * If `now` is not provided, uses the current time. - */ -export function isHalted( - db: DatabaseSync, - userId: string, - now?: Date, -): boolean { - ensureHaltTable(db); - const checkTime = now ?? new Date(); - const row = db.prepare( - 'SELECT halted_until FROM halt_state WHERE user_id = ?', - ).get(userId) as { halted_until: string } | undefined; - - if (!row) return false; - - const haltedUntil = new Date(row.halted_until); - return checkTime.getTime() < haltedUntil.getTime(); -} - -/** - * Clear a user's halt state (remove the record). - * - * Returns true if a record was removed, false otherwise. - */ -export function clearHalt(db: DatabaseSync, userId: string): boolean { - ensureHaltTable(db); - const result = db.prepare( - 'DELETE FROM halt_state WHERE user_id = ?', - ).run(userId); - return result.changes > 0; -} - -/** - * Get the halt record for a user (if any). - * - * Returns null if no halt is active. - */ -export function getHaltRecord( - db: DatabaseSync, - userId: string, -): { user_id: string; halted_until: string; triggered_by: string; ts: string } | null { - ensureHaltTable(db); - const row = db.prepare( - 'SELECT user_id, halted_until, triggered_by, ts FROM halt_state WHERE user_id = ?', - ).get(userId) as { user_id: string; halted_until: string; triggered_by: string; ts: string } | undefined; - - return row ?? null; -} diff --git a/app/server/src/risk/optionRiskContribution.ts b/app/server/src/risk/optionRiskContribution.ts index b6caf7d..766df51 100644 --- a/app/server/src/risk/optionRiskContribution.ts +++ b/app/server/src/risk/optionRiskContribution.ts @@ -3,7 +3,7 @@ // ADR-0007: considerations only, never trade instructions. import type { PortfolioOptionLeg } from '../db/portfolioOptionRepository.ts'; -import { ADR_0007_FOOTER, type Recommendation } from './RiskEngine.ts'; +import { type Recommendation } from './RiskEngine.ts'; export interface EquityHoldingLite { symbol: string; @@ -166,7 +166,7 @@ function buildOptionRecommendations(input: { id: 'consider_reducing_position', tradeOff: `${input.uncoveredShortCallCount} short call leg(s) lack matching long stock at contract size (shares needed = contracts × multiplier).`, explanation: - `Short calls without covering stock have an undefined-risk shape in the teaching model. A trade-off to think through: whether the position is intentionally structured elsewhere, or whether risk is open-ended relative to a covered call.\n\n${ADR_0007_FOOTER}`, + `Short calls without covering stock have an undefined-risk shape in this model. Check whether stock cover lives in another account, or whether short-call risk is open-ended relative to a covered call.`, severity: 'warning', }); } @@ -177,7 +177,7 @@ function buildOptionRecommendations(input: { id: 'consider_reducing_position', tradeOff: `Long option premium at risk is ~${pct}% of stated equity ($${input.premiumAtRiskUsd.toFixed(0)}).`, explanation: - `Debit option premium is capital that can go to zero if contracts expire worthless. A trade-off to think through: whether that premium share of equity matches your stated risk process.\n\n${ADR_0007_FOOTER}`, + `Debit option premium is capital that can go to zero if contracts expire worthless. Compare that premium share of equity to your stated risk process.`, severity: 'info', }); } @@ -188,7 +188,7 @@ function buildOptionRecommendations(input: { id: 'consider_rebalancing_cluster', tradeOff: `Cash reserved for cash-secured puts is ~${pct}% of stated equity ($${input.cashReservedUsd.toFixed(0)}).`, explanation: - `CSP collateral is capital committed to potential assignment at the strike. A trade-off to think through: concentration of reserved cash versus other uses of that capital.\n\n${ADR_0007_FOOTER}`, + `CSP collateral is capital committed to potential assignment at the strike. Compare reserved cash concentration to other uses of that capital.`, severity: 'info', }); } diff --git a/app/server/src/screener/SectorCrosslink.ts b/app/server/src/screener/SectorCrosslink.ts index b54c751..9c86b26 100644 --- a/app/server/src/screener/SectorCrosslink.ts +++ b/app/server/src/screener/SectorCrosslink.ts @@ -60,8 +60,7 @@ export function crossLinkSector( matchedSymbols, rotationSignal: rotationInto, sectorConfirmationReason, - educationalDisclaimer: - 'This cross-link shows symbols matching both sector rotation and strategy conditions. It is for discovery and learning, not a ready-made action list. Always verify against your own thesis and risk tolerance.', + educationalDisclaimer: 'Educational observation only.', }; } diff --git a/app/server/src/services/FinraIngestService.ts b/app/server/src/services/FinraIngestService.ts index 637f2d6..d07466a 100644 --- a/app/server/src/services/FinraIngestService.ts +++ b/app/server/src/services/FinraIngestService.ts @@ -90,9 +90,11 @@ export async function downloadAndIngestFinra( const ingestedAt = new Date().toISOString(); console.log(`[finra] downloading ${url}`); - const resp = await fetch(url, { + const { vendorFetch } = await import('./vendorGate.ts'); + const resp = await vendorFetch('finra', url, { headers: { 'User-Agent': 'InvestorFlow/1.0 (research) node' }, signal: AbortSignal.timeout(30_000), + hostAllowlist: /finra\.org|cdn\.finra\.org|files\.finra\.org/i, }); if (!resp.ok) throw new Error(`FINRA download failed: ${resp.status} ${resp.statusText}`); const body = await resp.text(); @@ -153,4 +155,159 @@ export function latestFinraSettlementDate(db: DatabaseSync): string | null { 'SELECT settlement_date FROM finra_short_interest ORDER BY settlement_date DESC LIMIT 1' ).get() as { settlement_date: string } | undefined; return r?.settlement_date ?? null; +} + +/** Backfill FINRA files for the last N calendar days. Skips 404s (weekends/holidays) with 500ms rate-limit. */ +export async function backfillFinra( + db: DatabaseSync, + days: number = 30, + baseUrl?: string, +): Promise<{ filesAttempted: number; filesStored: number; firstDate: string | null; lastDate: string | null; errors: string[] }> { + const errors: string[] = []; + let filesAttempted = 0; + let filesStored = 0; + let firstDate: string | null = null; + let lastDate: string | null = null; + + const existing = new Set( + (db.prepare('SELECT DISTINCT settlement_date FROM finra_short_interest').all() as Array<{ settlement_date: string }>) + .map((r) => r.settlement_date), + ); + + for (let i = days; i >= 0; i--) { + const d = new Date(Date.now() - i * 86400000); + const dateStr = d.toISOString().slice(0, 10); + if (existing.has(dateStr)) continue; + + try { + await downloadAndIngestFinra(db, dateStr, baseUrl); + filesStored += 1; + if (firstDate === null) firstDate = dateStr; + lastDate = dateStr; + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (msg.includes('404') || msg.includes('40')) { + // Weekend/holiday — expected + } else { + errors.push(`${dateStr}: ${msg}`); + } + } + filesAttempted += 1; + await new Promise((r) => setTimeout(r, 500)); + } + return { filesAttempted, filesStored, firstDate, lastDate, errors }; +} + +// ============================================================================ +// FINRA bi-monthly short INTEREST (outstanding positions) — shrt{DATE}.csv +// Separate dataset from daily volume; comparable to NASDAQ/Yahoo. +// ============================================================================ + +const FINRA_SI_BASE = 'https://cdn.finra.org/equity/otcmarket/biweekly'; +function finraSiFilename(settlementDate: string): string { + return `shrt${settlementDate.replace(/-/g, '')}.csv`; +} + +function parseFinraSiFile(body: string, settlementDate: string, ingestedAt: string, sourceFile: string): Array<{ + symbol: string; settlementDate: string; issueName: string | null; exchangeCode: string | null; + marketClass: string | null; currentShortPosition: number; previousShortPosition: number | null; + avgDailyVolume: number | null; daysToCover: number | null; changePercent: number | null; + changePrevious: number | null; revisionFlag: string | null; +}> { + const lines = body.split(/\r?\n/); + type SiRow = { + symbol: string; settlementDate: string; issueName: string | null; exchangeCode: string | null; + marketClass: string | null; currentShortPosition: number; previousShortPosition: number | null; + avgDailyVolume: number | null; daysToCover: number | null; changePercent: number | null; + changePrevious: number | null; revisionFlag: string | null; + }; + const rows: SiRow[] = []; + let headerFound = false; + for (const raw of lines) { + const line = raw.trim(); + if (!line || line.startsWith('#')) continue; + if (line.includes('accountingYearMonthNumber|symbolCode')) { headerFound = true; continue; } + if (!headerFound) continue; + const c = line.split('|').map((x) => x.trim()); + if (c.length < 6 || !c[1]) continue; + const symbol = c[1].replace(/\/.*$/, '')?.toUpperCase(); + if (!symbol) continue; + const cur = parseFloat((c[5] ?? '0').replace(/,/g, '')); + if (Number.isNaN(cur)) continue; + rows.push({ + symbol, settlementDate, + issueName: c[2] || null, exchangeCode: c[3] || null, marketClass: c[4] || null, + currentShortPosition: cur, + previousShortPosition: c[6] ? parseFloat(c[6].replace(/,/g, '')) || null : null, + avgDailyVolume: c[8] ? parseFloat(c[8].replace(/,/g, '')) || null : null, + daysToCover: c[9] ? parseFloat(c[9].replace(/,/g, '')) || null : null, + changePercent: c[11] ? parseFloat(c[11].replace(/,/g, '')) || null : null, + changePrevious: c[12] ? parseFloat(c[12].replace(/,/g, '')) || null : null, + revisionFlag: c[10] || null, + }); + } + return rows; +} + +export async function downloadAndIngestFinraSi( + db: DatabaseSync, + settlementDate: string, +): Promise<{ symbolsStored: number; sourceFile: string }> { + const filename = finraSiFilename(settlementDate); + const url = `${FINRA_SI_BASE}/${filename}`; + const ingestedAt = new Date().toISOString(); + console.log(`[finra-si] downloading ${url}`); + const { vendorFetch } = await import('./vendorGate.ts'); + const resp = await vendorFetch('finra', url, { + headers: { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36' }, + signal: AbortSignal.timeout(30_000), + hostAllowlist: /finra\.org|cdn\.finra\.org|files\.finra\.org/i, + }); + if (!resp.ok) throw new Error(`FINRA SI download failed: ${resp.status}`); + const body = await resp.text(); + if (!body.trim()) throw new Error('FINRA SI file empty'); + const rows = parseFinraSiFile(body, settlementDate, ingestedAt, filename); + if (!rows.length) throw new Error('No FINRA SI rows parsed'); + db.exec('BEGIN'); + let inserted = 0; + for (const r of rows) { + try { + db.exec(`INSERT OR REPLACE INTO finra_short_interest_biweekly (symbol,settlement_date,issue_name,exchange_code,market_class,current_short_position,previous_short_position,avg_daily_volume,days_to_cover,change_percent,change_previous,revision_flag,source_file,ingested_at) VALUES ('${r.symbol.replace(/'/g,"''")}','${r.settlementDate}','${(r.issueName??'').replace(/'/g,"''")}','${(r.exchangeCode??'').replace(/'/g,"''")}','${(r.marketClass??'').replace(/'/g,"''")}',${r.currentShortPosition},${r.previousShortPosition??'NULL'},${r.avgDailyVolume??'NULL'},${r.daysToCover??'NULL'},${r.changePercent??'NULL'},${r.changePrevious??'NULL'},${r.revisionFlag??'NULL'},'${filename.replace(/'/g,"''")}','${ingestedAt}')`); + inserted++; + } catch (re) { console.log(`[finra-si] skip row ${r.symbol} ${(re as Error).message}`); } + } + db.exec('COMMIT'); + console.log(`[finra-si] ingested ${inserted}/${rows.length} rows from ${filename}`); + return { symbolsStored: inserted, sourceFile: filename }; +} + +export function latestFinraSiSettlementDate(db: DatabaseSync): string | null { + const r = db.prepare('SELECT settlement_date FROM finra_short_interest_biweekly ORDER BY settlement_date DESC LIMIT 1').get() as { settlement_date: string } | undefined; + return r?.settlement_date ?? null; +} + +/** Backfill FINRA bi-monthly files: probe date windows around 15th and month-end, skip 403s. */ +export async function backfillFinraSi(db: DatabaseSync, months: number = 24): Promise<{ filesStored: number; firstDate: string | null; lastDate: string | null; errors: string[] }> { + const errors: string[] = []; + let filesStored = 0; + let firstDate: string | null = null; + let lastDate: string | null = null; + const existing = new Set((db.prepare('SELECT DISTINCT settlement_date FROM finra_short_interest_biweekly').all() as Array<{ settlement_date: string }>).map(r => r.settlement_date)); + const now = new Date(); + const candidates: string[] = []; + for (let m = 0; m < months; m++) { + const d = new Date(now.getFullYear(), now.getMonth() - m, 1); + for (let day of [13,14,15,16,17]) candidates.push(new Date(d.getFullYear(), d.getMonth(), day).toISOString().slice(0,10)); + const lastDay = new Date(d.getFullYear(), d.getMonth()+1, 0).getDate(); + for (let day of [28,29,30,31]) if (day <= lastDay) candidates.push(new Date(d.getFullYear(), d.getMonth(), day).toISOString().slice(0,10)); + for (let day of [1,2,3]) candidates.push(new Date(d.getFullYear(), d.getMonth()+1, day).toISOString().slice(0,10)); + } + const valid = [...new Set(candidates)].filter((x): x is string => !!x).sort().reverse(); + for (const dateStr of valid) { + if (existing.has(dateStr)) continue; + try { await downloadAndIngestFinraSi(db, dateStr); filesStored++; if (!firstDate) firstDate = dateStr; lastDate = dateStr; } + catch (e) { const m = e instanceof Error ? e.message : String(e); if (!m.includes('403')) errors.push(`${dateStr}: ${m}`); } + await new Promise(r => setTimeout(r, 500)); + } + return { filesStored, firstDate, lastDate, errors }; } \ No newline at end of file diff --git a/app/server/src/services/__tests__/captureIngest.test.ts b/app/server/src/services/__tests__/captureIngest.test.ts new file mode 100644 index 0000000..41f73e3 --- /dev/null +++ b/app/server/src/services/__tests__/captureIngest.test.ts @@ -0,0 +1,127 @@ +// Capture materializer tests — X posts → fund_position_records (source='capture'). +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { DatabaseSync } from 'node:sqlite'; +import { ingestFundCaptures, ingestAllFundCaptures, listTrackedFundHandles } from '../captureIngest.ts'; + +function makeDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE tracked_funds ( + id TEXT PRIMARY KEY, ci_key TEXT NOT NULL, fund_name TEXT NOT NULL, + manager_name TEXT NOT NULL, x_handle TEXT, paywall_status TEXT NOT NULL DEFAULT 'unknown', + enabled INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL, updated_at TEXT + ); + CREATE TABLE fund_position_records ( + id TEXT PRIMARY KEY, fund_id TEXT NOT NULL, symbol TEXT NOT NULL, + shares REAL, value_usd REAL, cost_basis REAL, as_of TEXT NOT NULL, + source TEXT NOT NULL, evidence_url TEXT, notes TEXT, created_at TEXT NOT NULL + ); + CREATE TABLE x_cookie_posts ( + post_id TEXT PRIMARY KEY, source TEXT, author_handle TEXT, cashtag TEXT, + body_text TEXT, posted_at TEXT, engagement TEXT, cached_until TEXT + ); + `); + db.prepare(`INSERT INTO tracked_funds (id, ci_key, fund_name, manager_name, x_handle, paywall_status, enabled, created_at) + VALUES ('alpine-fox-capital', '0002096493', 'Alpine Fox Capital LLC', 'Mike Alfred', 'mikealfred', 'paywalled', 1, ?)`) + .run(new Date().toISOString()); + return db; +} + +function seedPost(db: DatabaseSync, id: string, body: string, at: string): void { + db.prepare(`INSERT INTO x_cookie_posts (post_id, source, author_handle, cashtag, body_text, posted_at) + VALUES (?, 'x', 'mikealfred', NULL, ?, ?)`).run(id, body, at); +} + +test('ingestFundCaptures: materializes a total-position capture with source=capture', () => { + const db = makeDb(); + seedPost(db, 'p1', '**Real time position update\n\nTook OPEN over 5.7M shares now. Brought average down to $4.42.', '2026-07-30T19:51:12.000Z'); + + const s = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.equal(s.postsScanned, 1); + assert.equal(s.captures, 1); + assert.equal(s.inserted, 1); + + const row = db.prepare('SELECT * FROM fund_position_records').get() as any; + assert.equal(row.fund_id, 'alpine-fox-capital'); + assert.equal(row.symbol, 'OPEN'); + assert.equal(row.shares, 5_700_000); + assert.ok(Math.abs(row.cost_basis - 4.42) < 0.001); + assert.equal(row.source, 'capture'); + assert.equal(row.as_of, '2026-07-30'); + assert.equal(row.evidence_url, 'https://x.com/mikealfred/status/p1'); +}); + +test('ingestFundCaptures: idempotent — same post converges, no duplicate rows', () => { + const db = makeDb(); + seedPost(db, 'p1', '**Real time position update\n\nTook OPEN over 5.7M shares now. Brought average down to $4.42.', '2026-07-30T19:51:12.000Z'); + + ingestFundCaptures(db, 'alpine-fox-capital'); + const again = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.equal(again.inserted, 0); + assert.equal(again.refreshed, 1); + assert.equal(db.prepare('SELECT COUNT(*) FROM fund_position_records').get()!['COUNT(*)'], 1); +}); + +test('ingestFundCaptures: commentary posts and claims are never materialized', () => { + const db = makeDb(); + seedPost(db, 'c1', 'Beautiful day in the mountains. Great conversations with subscribers.', '2026-08-04T23:07:20.000Z'); + seedPost(db, 'cl1', 'Added 865,000 shares of BKKT at $3.10 average price today.', '2026-08-01T10:00:00.000Z'); + + const s = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.equal(s.skipped, 1); // commentary + assert.equal(s.claims, 1); // claim detected but skipped in v1 + assert.equal(db.prepare('SELECT COUNT(*) FROM fund_position_records').get()!['COUNT(*)'], 0); +}); + +test('ingestFundCaptures: trade narrative with two quantities is refused (no garbage)', () => { + const db = makeDb(); + seedPost(db, 'n1', 'I reduced the position from 200,000 shares to 10,000 shares over the last month. Today I added some shares back just above $46.', '2026-08-02T10:00:00.000Z'); + + const s = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.equal(s.skipped, 1); + assert.equal(db.prepare('SELECT COUNT(*) FROM fund_position_records').get()!['COUNT(*)'], 0); +}); + +test('ingestFundCaptures: disabled fund or missing handle is skipped', () => { + const db = makeDb(); + seedPost(db, 'p1', '**Real time position update\n\nTook OPEN over 5.7M shares now. Brought average down to $4.42.', '2026-07-30T19:51:12.000Z'); + db.prepare(`UPDATE tracked_funds SET enabled = 0 WHERE id = 'alpine-fox-capital'`).run(); + const s = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.equal(s.disabled, true); + assert.equal(db.prepare('SELECT COUNT(*) FROM fund_position_records').get()!['COUNT(*)'], 0); +}); + +test('ingestFundCaptures: multi-name update materializes OPEN total (SLNH claim skipped)', () => { + const db = makeDb(); + seedPost( + db, + 'p-multi', + `A few quick position updates. + +Just filled a 100,000 share order in SLNH at $1.12. Have another for 100,000 shares at $1.11. + +OPEN now over 6.45M shares. Have taken it up on the weakness.`, + '2026-08-07T15:45:54.000Z', + ); + const s = ingestFundCaptures(db, 'alpine-fox-capital'); + assert.ok(s.captures >= 1); + assert.equal(s.inserted, 1); + assert.equal(s.claims, 1); // SLNH fill recorded as claim, not book row + const open = db.prepare(`SELECT * FROM fund_position_records WHERE symbol='OPEN'`).get() as any; + assert.equal(open.shares, 6_450_000); + assert.equal(open.as_of, '2026-08-07'); + assert.equal( + db.prepare(`SELECT COUNT(*) AS n FROM fund_position_records WHERE symbol='SLNH'`).get()!['n'], + 0, + ); +}); + +test('ingestAllFundCaptures: runs across every enabled tracked fund with a handle', () => { + const db = makeDb(); + seedPost(db, 'p1', '**Real time position update\n\nTook OPEN over 5.7M shares now. Brought average down to $4.42.', '2026-07-30T19:51:12.000Z'); + const stats = ingestAllFundCaptures(db); + assert.equal(stats.length, 1); + assert.equal(stats[0].captures, 1); + assert.deepEqual(listTrackedFundHandles(db).map((f) => f.id), ['alpine-fox-capital']); +}); diff --git a/app/server/src/services/__tests__/reverse13fRefresh.test.ts b/app/server/src/services/__tests__/reverse13fRefresh.test.ts new file mode 100644 index 0000000..4da7795 --- /dev/null +++ b/app/server/src/services/__tests__/reverse13fRefresh.test.ts @@ -0,0 +1,132 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { + MAJOR_13F_FILER_CIKS, + selectMissingMajorFilers, + selectReverseFilerCiks, +} from '../reverse13fRefresh.ts'; + +function makeDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE institution_filings ( + filer_cik TEXT, filer_name TEXT, filer_sic TEXT, symbol TEXT, form TEXT, + shares REAL, value_usd REAL, reported_quarter TEXT, filed_at TEXT, + fetched_at TEXT, put_call TEXT, accession TEXT + ); + CREATE TABLE tracked_funds ( + id TEXT PRIMARY KEY, ci_key TEXT, fund_name TEXT, manager_name TEXT, + x_handle TEXT, paywall_status TEXT, enabled INTEGER, created_at TEXT, updated_at TEXT + ); + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + `); + return db; +} + +test('selectReverseFilerCiks prefers prior holders then tracked then majors', () => { + const db = makeDb(); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES (?, ?, 'IREN', '13F-HR', ?, 1, '2026-Q2', '2026-07-01', '2026-07-01')`, + ).run('0001535385', 'Artemis', 739723); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES (?, ?, 'IREN', '13F-HR', ?, 1, '2026-Q2', '2026-07-01', '2026-07-01')`, + ).run('0002012155', 'VIMA', 116245); + db.prepare( + `INSERT INTO tracked_funds + (id, ci_key, fund_name, manager_name, paywall_status, enabled, created_at) + VALUES ('alpine', '0002096493', 'Alpine Fox', 'Mike', 'open', 1, '2026-01-01')`, + ).run(); + + const filers = selectReverseFilerCiks(db, 'IREN', 10, 5); + assert.ok(filers.length >= 3); + assert.equal(filers[0].cik, '0001535385'); + assert.equal(filers[0].source, 'prior_holder'); + assert.ok(filers.some((f) => f.cik === '0002096493' && f.source === 'tracked_fund')); + assert.ok(filers.some((f) => f.source === 'major')); + // no duplicates + const ciks = filers.map((f) => f.cik); + assert.equal(ciks.length, new Set(ciks).size); +}); + +test('selectReverseFilerCiks skips non-numeric tracked fund keys', () => { + const db = makeDb(); + db.prepare( + `INSERT INTO tracked_funds + (id, ci_key, fund_name, manager_name, paywall_status, enabled, created_at) + VALUES ('mr-t', 'mr-t-invests', 'Mr T', 'T', 'open', 1, '2026-01-01')`, + ).run(); + const filers = selectReverseFilerCiks(db, 'AAPL', 5, 3); + assert.ok(filers.every((f) => /^\d{10}$/.test(f.cik))); + assert.ok(filers.length >= 1); + assert.ok(MAJOR_13F_FILER_CIKS.length >= 10); +}); + +test('MAJOR_13F includes current BlackRock Inc CIK (BLK), not old Finance CIK', () => { + const br = MAJOR_13F_FILER_CIKS.find((m) => /blackrock/i.test(m.name)); + assert.ok(br); + assert.equal(br!.cik, '0002012383'); + assert.ok(!MAJOR_13F_FILER_CIKS.some((m) => m.cik === '0001364742')); +}); + +test('selectMissingMajorFilers excludes majors already in latest quarter', () => { + const db = makeDb(); + // Seed latest quarter with BlackRock present, State Street absent + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0002012383', 'BlackRock, Inc.', 'IREN', '13F-HR', 1e7, 1, '2026-Q2', '2026-08-07', '2026-08-07')`, + ).run(); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0001535385', 'Artemis', 'IREN', '13F-HR', 1e5, 1, '2026-Q2', '2026-07-01', '2026-07-01')`, + ).run(); + + const missing = selectMissingMajorFilers(db, 'IREN'); + assert.ok(!missing.some((m) => m.cik === '0002012383'), 'BlackRock already in Q2'); + assert.ok(missing.some((m) => m.cik === '0000093751'), 'State Street still missing'); + assert.ok(missing.every((m) => m.source === 'missing_major')); +}); + +test('selectMissingMajorFilers does not thrash fresh prior-quarter majors', () => { + const db = makeDb(); + // Frontier is Q2 via a small filer; Goldman only has freshly fetched Q1 + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0001535385', 'Artemis', 'IREN', '13F-HR', 1e5, 1, '2026-Q2', '2026-07-01', '2026-08-07T12:00:00.000Z')`, + ).run(); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0000886982', 'Goldman', 'IREN', '13F-HR', 1e7, 1, '2026-Q1', '2026-05-15', '2026-08-07T18:00:00.000Z')`, + ).run(); + + const now = Date.parse('2026-08-08T00:00:00.000Z'); + const missing = selectMissingMajorFilers(db, 'IREN', { now, staleAfterMs: 10 * 24 * 3600_000 }); + assert.ok(!missing.some((m) => m.cik === '0000886982'), 'Goldman Q1 freshly fetched — wait'); + assert.ok(missing.some((m) => m.cik === '0002012383'), 'BlackRock never stored'); +}); + +test('selectMissingMajorFilers rechecks stale prior-quarter majors', () => { + const db = makeDb(); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0001535385', 'Artemis', 'IREN', '13F-HR', 1e5, 1, '2026-Q2', '2026-07-01', '2026-08-07')`, + ).run(); + db.prepare( + `INSERT INTO institution_filings + (filer_cik, filer_name, symbol, form, shares, value_usd, reported_quarter, filed_at, fetched_at) + VALUES ('0000886982', 'Goldman', 'IREN', '13F-HR', 1e7, 1, '2026-Q1', '2026-05-15', '2026-07-01T00:00:00.000Z')`, + ).run(); + + const now = Date.parse('2026-08-08T00:00:00.000Z'); + const missing = selectMissingMajorFilers(db, 'IREN', { now, staleAfterMs: 10 * 24 * 3600_000 }); + assert.ok(missing.some((m) => m.cik === '0000886982'), 'Goldman Q1 stale vs Q2 frontier'); +}); diff --git a/app/server/src/services/__tests__/secCusipCache.test.ts b/app/server/src/services/__tests__/secCusipCache.test.ts new file mode 100644 index 0000000..dda0d6d --- /dev/null +++ b/app/server/src/services/__tests__/secCusipCache.test.ts @@ -0,0 +1,66 @@ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { DatabaseSync } from 'node:sqlite'; +import { + curatedCusipForSymbol, + resolveCusipLocal, + seedCuratedCusips, +} from '../cusipRegistry.ts'; + +/** + * Unit-level tests for CUSIP cache helpers via sec fetch behavior contracts. + * Network resolve is not exercised here. + */ +test('kv_cache can store and read sec:cusip keys', () => { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + `); + const key = 'sec:cusip:IREN'; + const cusip = 'Q4982L109'; + db.prepare('INSERT OR REPLACE INTO kv_cache (key,value,observed_at) VALUES (?,?,?)').run( + key, + cusip, + new Date().toISOString(), + ); + const row = db.prepare('SELECT value FROM kv_cache WHERE key=?').get(key) as { value: string }; + assert.equal(row.value, cusip); +}); + +test('curated map resolves IREN offline', () => { + assert.equal(curatedCusipForSymbol('IREN'), 'Q4982L109'); + assert.equal(curatedCusipForSymbol('AAPL'), '037833100'); +}); + +test('resolveCusipLocal seeds cache from curated map', () => { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + `); + const c = resolveCusipLocal(db, 'IREN'); + assert.equal(c, 'Q4982L109'); + const row = db.prepare('SELECT value FROM kv_cache WHERE key=?').get('sec:cusip:IREN') as { + value: string; + }; + assert.equal(row.value, 'Q4982L109'); +}); + +test('seedCuratedCusips is idempotent', () => { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE kv_cache (key TEXT PRIMARY KEY, value TEXT NOT NULL, observed_at TEXT NOT NULL); + `); + const n1 = seedCuratedCusips(db); + const n2 = seedCuratedCusips(db); + assert.ok(n1 > 0); + assert.equal(n2, 0); +}); + +test('isPermanentDataError does not quarantine cusip failures', async () => { + const { isPermanentDataError } = await import('../../queue/sourceRatePolicy.ts'); + assert.equal( + isPermanentDataError('sec-fetch:IREN: cusip not resolved - 13F holder refresh blocked'), + false, + ); + assert.equal(isPermanentDataError('quote not found for symbol'), true); +}); diff --git a/app/server/src/services/__tests__/secHttp.test.ts b/app/server/src/services/__tests__/secHttp.test.ts new file mode 100644 index 0000000..217f197 --- /dev/null +++ b/app/server/src/services/__tests__/secHttp.test.ts @@ -0,0 +1,68 @@ +import { test, beforeEach } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + isSecHttpCoolingDown, + noteSecRateLimitHit, + resetSecHttpStateForTests, + secFetch, + SecRateLimitError, + SEC_MIN_INTERVAL_MS, +} from '../secHttp.ts'; + +beforeEach(() => { + resetSecHttpStateForTests(); +}); + +test('secFetch refuses non-SEC hosts', async () => { + await assert.rejects(() => secFetch('https://example.com/x'), /non-SEC/); +}); + +test('secFetch serializes and spaces requests', async () => { + const times: number[] = []; + const original = globalThis.fetch; + globalThis.fetch = (async () => { + times.push(Date.now()); + return new Response(JSON.stringify({ ok: true }), { status: 200 }); + }) as typeof fetch; + + try { + // Use sequential calls to verify spacing between each pair. + await secFetch('https://data.sec.gov/a'); + await secFetch('https://data.sec.gov/b'); + await secFetch('https://www.sec.gov/c'); + assert.equal(times.length, 3); + // Each sequential start must be spaced by ~SEC_MIN_INTERVAL_MS. + assert.ok(times[1] - times[0] >= SEC_MIN_INTERVAL_MS - 20, `gap1=${times[1] - times[0]}ms`); + assert.ok(times[2] - times[1] >= SEC_MIN_INTERVAL_MS - 20, `gap2=${times[2] - times[1]}ms`); + } finally { + globalThis.fetch = original; + } +}); + +test('429 sets cool-down and throws SecRateLimitError', async () => { + const original = globalThis.fetch; + globalThis.fetch = (async () => + new Response('Request Rate Threshold Exceeded', { status: 429 })) as typeof fetch; + + try { + await assert.rejects(() => secFetch('https://data.sec.gov/x', { retries: 0 }), (e: unknown) => { + assert.ok(e instanceof SecRateLimitError); + return true; + }); + assert.equal(isSecHttpCoolingDown(), true); + } finally { + globalThis.fetch = original; + } +}); + +test('noteSecRateLimitHit escalates cool-down', () => { + const a = noteSecRateLimitHit(); + const b = noteSecRateLimitHit(); + assert.ok(b >= a); + assert.equal(isSecHttpCoolingDown(), true); +}); + +test('preflight rejects while cooling', async () => { + noteSecRateLimitHit(); + await assert.rejects(() => secFetch('https://data.sec.gov/y', { retries: 0 }), /preflight cool-down|rate limit/i); +}); diff --git a/app/server/src/services/__tests__/secTickersMerge.test.ts b/app/server/src/services/__tests__/secTickersMerge.test.ts new file mode 100644 index 0000000..5d5944a --- /dev/null +++ b/app/server/src/services/__tests__/secTickersMerge.test.ts @@ -0,0 +1,107 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { DatabaseSync } from 'node:sqlite'; +import { mergeCompanyTickers } from '../secTickersIngest.ts'; + +function makeDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE symbols ( + symbol TEXT PRIMARY KEY, + name TEXT, sector TEXT, industry TEXT, exchange TEXT, + ticker_kind TEXT NOT NULL, + peers TEXT, cik TEXT, updated_at TEXT + ); + `); + return db; +} + +// A small synthetic SEC payload (shape matches company_tickers.json). +const secRows = { + '0': { cik_str: 2096493, ticker: 'IREN', title: 'IREN Limited' }, + '1': { cik_str: 320193, ticker: 'AAPL', title: 'Apple Inc.' }, + '2': { cik_str: 1652044, ticker: 'NVDA', title: 'NVIDIA CORPORATION' }, +}; + +test('mergeCompanyTickers: inserts new equity rows with zero-padded issuer CIK', () => { + const db = makeDb(); + const r = mergeCompanyTickers(db, secRows); + assert.equal(r.created, 3); + assert.equal(r.upserted, 0); + + const row = db.prepare('SELECT * FROM symbols WHERE symbol = ?').get('AAPL') as any; + assert.equal(row.cik, '0000320193'); // zero-padded to 10 + assert.equal(row.ticker_kind, 'equity'); + assert.equal(row.name, 'Apple Inc.'); + assert.equal(row.sector, null); + assert.equal(row.exchange, null); +}); + +test('mergeCompanyTickers: fills cik on existing rows, never downgrades ticker_kind', () => { + const db = makeDb(); + // Simulate an existing yfinance-hydrated ETF row WITHOUT a cik. + db.prepare( + "INSERT INTO symbols (symbol, name, sector, ticker_kind, updated_at) VALUES ('IBIT', 'iShares Bitcoin Trust', 'Crypto', 'etf', ?)", + ).run(new Date().toISOString()); + + // SEC payload includes IBIT as a listed issuer. + const r = mergeCompanyTickers(db, { + '0': { cik_str: 1990438, ticker: 'IBIT', title: 'iShares Bitcoin Trust' }, + }); + assert.equal(r.upserted, 1); + + const row = db.prepare('SELECT * FROM symbols WHERE symbol = ?').get('IBIT') as any; + assert.equal(row.cik, '0001990438'); // cik filled by SEC + assert.equal(row.ticker_kind, 'etf'); // NOT downgraded to equity + assert.equal(row.sector, 'Crypto'); // descriptive metadata untouched +}); + +test('mergeCompanyTickers: keeps richer yfinance name over SEC title on conflict', () => { + const db = makeDb(); + db.prepare( + "INSERT INTO symbols (symbol, name, ticker_kind, updated_at) VALUES ('NVDA', 'NVIDIA Corporation (Detailed)', 'equity', ?)", + ).run(new Date().toISOString()); + + const r = mergeCompanyTickers(db, { '0': { cik_str: 1652044, ticker: 'NVDA', title: 'NVIDIA CORPORATION' } }); + assert.equal(r.upserted, 1); + + const row = db.prepare('SELECT symbol, name, cik FROM symbols WHERE symbol = ?').get('NVDA') as any; + assert.equal(row.name, 'NVIDIA Corporation (Detailed)'); // existing richer name preserved + assert.equal(row.cik, '0001652044'); // cik still filled +}); + +test('mergeCompanyTickers: rows absent from SEC payload are never purged', () => { + const db = makeDb(); + db.prepare( + "INSERT INTO symbols (symbol, name, ticker_kind, updated_at) VALUES ('BTC-USD', 'Bitcoin', 'crypto', ?)", + ).run(new Date().toISOString()); + + mergeCompanyTickers(db, secRows); // BTC-USD not in payload + + const row = db.prepare('SELECT symbol FROM symbols WHERE symbol = ?').get('BTC-USD') as any; + assert.ok(row, 'crypto row must survive a SEC-materialization pass'); +}); + +test('mergeCompanyTickers: skips empty tickers and is idempotent', () => { + const db = makeDb(); + const withJunk = { ...secRows, '9': { cik_str: 1, ticker: '', title: 'No ticker' } }; + const r1 = mergeCompanyTickers(db, withJunk); + assert.equal(r1.created, 3); + assert.equal(r1.skipped, 1); + + // Second pass: no new rows, all existing rows counted as upserted. + const r2 = mergeCompanyTickers(db, secRows); + assert.equal(r2.created, 0); + assert.equal(r2.upserted, 3); +}); + +test('mergeCompanyTickers: is atomic — a failing row rolls back the batch', () => { + const db = makeDb(); + // Force a failure mid-batch: duplicate primary key cannot happen here, so instead + // verify a throwing payload leaves zero rows behind. + assert.throws(() => { + mergeCompanyTickers(db, null as unknown as Record); + }); + const count = (db.prepare('SELECT COUNT(*) AS n FROM symbols').get() as any).n; + assert.equal(count, 0); +}); diff --git a/app/server/src/services/__tests__/symbolSearch.test.ts b/app/server/src/services/__tests__/symbolSearch.test.ts new file mode 100644 index 0000000..f66a51e --- /dev/null +++ b/app/server/src/services/__tests__/symbolSearch.test.ts @@ -0,0 +1,95 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { DatabaseSync } from 'node:sqlite'; +import { mergeCompanyTickers } from '../secTickersIngest.ts'; + +function makeDb(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE symbols ( + symbol TEXT PRIMARY KEY, + name TEXT, sector TEXT, industry TEXT, exchange TEXT, + ticker_kind TEXT NOT NULL, + peers TEXT, cik TEXT, updated_at TEXT + ); + `); + return db; +} + +const secRows = { + '0': { cik_str: 1878848, ticker: 'IREN', title: 'IREN Limited' }, + '1': { cik_str: 320193, ticker: 'AAPL', title: 'Apple Inc.' }, + '2': { cik_str: 1652044, ticker: 'NVDA', title: 'NVIDIA CORPORATION' }, + '3': { cik_str: 1631761, ticker: 'YRD', title: 'Yiren Digital Ltd.' }, +}; + +// The search SQL used by the symbols.search tRPC procedure (kept in sync). +function search(db: DatabaseSync, q: string, limit = 10) { + const upper = q.trim().toUpperCase(); + const rows = db.prepare( + `SELECT symbol, name, sector, industry, exchange, ticker_kind, cik, + CASE WHEN symbol = ? THEN 0 ELSE 1 END AS rank + FROM symbols + WHERE symbol = ? OR symbol LIKE ? + ORDER BY rank ASC, LENGTH(symbol) ASC + LIMIT ?`, + ).all(upper, upper, `${upper}%`, Math.ceil(limit * 2 / 3)) as Array>; + + let results = rows.map((r) => ({ + symbol: r.symbol, name: r.name ?? null, ticker_kind: r.ticker_kind, cik: r.cik ?? null, + })); + + if (results.length < limit) { + const seen = new Set(results.map((r) => r.symbol)); + const nameRows = db.prepare( + `SELECT symbol, name, sector, industry, exchange, ticker_kind, cik + FROM symbols WHERE name IS NOT NULL AND name != '' AND name LIKE ? + ORDER BY LENGTH(name) ASC LIMIT ?`, + ).all(`%${q}%`, limit) as Array>; + const fresh = nameRows + .filter((r) => !seen.has(r.symbol as string)) + .map((r) => ({ symbol: r.symbol, name: r.name ?? null, ticker_kind: r.ticker_kind, cik: r.cik ?? null })); + results = [...results, ...fresh.slice(0, limit - results.length)]; + } + return results; +} + +test('search: exact symbol match ranks first', () => { + const db = makeDb(); + mergeCompanyTickers(db, secRows); + const r = search(db, 'IREN', 5); + assert.equal(r[0].symbol, 'IREN'); + assert.equal(r[0].cik, '0001878848'); + assert.equal(r[0].ticker_kind, 'equity'); +}); + +test('search: prefix match returns partial symbols without exact duplication', () => { + const db = makeDb(); + mergeCompanyTickers(db, secRows); + const r = search(db, 'ire', 5); + // IREN must appear exactly once. + const iren = r.filter((x) => x.symbol === 'IREN'); + assert.equal(iren.length, 1); + assert.equal(r[0].symbol, 'IREN'); // exact-prefix ranks first +}); + +test('search: name-contains fallback fills beyond symbol prefix matches', () => { + const db = makeDb(); + mergeCompanyTickers(db, secRows); + // 'yiren' matches no symbol prefix but matches Yiren Digital Ltd by name. + const r = search(db, 'yiren', 5); + assert.ok(r.some((x) => x.symbol === 'YRD')); +}); + +test('search: case-insensitive and trims', () => { + const db = makeDb(); + mergeCompanyTickers(db, secRows); + const r = search(db, ' aapl ', 5); + assert.equal(r[0]?.symbol, 'AAPL'); +}); + +test('search: empty corpus returns empty results', () => { + const db = makeDb(); + const r = search(db, 'zzz', 5); + assert.equal(r.length, 0); +}); diff --git a/app/server/src/services/__tests__/vendorGate.test.ts b/app/server/src/services/__tests__/vendorGate.test.ts new file mode 100644 index 0000000..c3f033f --- /dev/null +++ b/app/server/src/services/__tests__/vendorGate.test.ts @@ -0,0 +1,108 @@ +import { test, beforeEach } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + familyDrainBudget, + isVendorCoolingDown, + noteVendorRateLimit, + resetVendorGateForTests, + sourceToFamily, + sourcesForFamily, + vendorFetch, + withVendorGate, + VendorRateLimitError, +} from '../vendorGate.ts'; + +beforeEach(() => { + resetVendorGateForTests(); +}); + +test('sourceToFamily maps queue kinds', () => { + assert.equal(sourceToFamily('yfinance'), 'yfinance'); + assert.equal(sourceToFamily('yfinance-quote'), 'yfinance'); + assert.equal(sourceToFamily('sec-fetch'), 'sec'); + assert.equal(sourceToFamily('sec-sc-fetch'), 'sec'); + assert.equal(sourceToFamily('finra-si'), 'finra'); + assert.equal(sourceToFamily('fred'), 'fred'); + assert.equal(sourceToFamily('nasdaq'), 'nasdaq'); + assert.equal(sourceToFamily('x'), 'x'); + assert.equal(sourceToFamily('unregistered-vendor-xyz'), null); +}); + +test('sourcesForFamily includes all yahoo tiers', () => { + const yf = sourcesForFamily('yfinance'); + assert.ok(yf.includes('yfinance')); + assert.ok(yf.includes('yfinance-quote')); +}); + +test('familyDrainBudget keeps Yahoo higher than SEC', () => { + assert.ok(familyDrainBudget('yfinance') >= 2); + assert.equal(familyDrainBudget('sec'), 2); +}); + +test('withVendorGate serializes concurrent calls', async () => { + const order: number[] = []; + await Promise.all([ + withVendorGate('yfinance', async () => { + order.push(1); + await new Promise((r) => setTimeout(r, 50)); + order.push(2); + return 'a'; + }), + withVendorGate('yfinance', async () => { + order.push(3); + return 'b'; + }), + ]); + // Second call cannot interleave inside first + assert.deepEqual(order, [1, 2, 3]); +}); + +test('rate-limit throw notes cool-down', async () => { + await assert.rejects( + () => + withVendorGate('yfinance', async () => { + throw new Error('Edge: Too Many Requests'); + }), + (e: unknown) => e instanceof VendorRateLimitError, + ); + assert.equal(isVendorCoolingDown('yfinance'), true); +}); + +test('preflight blocks while cooling', async () => { + noteVendorRateLimit('fred'); + await assert.rejects( + () => withVendorGate('fred', async () => 'ok'), + /preflight cool-down|rate limit/i, + ); +}); + +test('vendorFetch refuses host outside allowlist', async () => { + await assert.rejects( + () => + vendorFetch('nasdaq', 'https://evil.example.com/x', { + hostAllowlist: /api\.nasdaq\.com/i, + }), + /not allowed/, + ); +}); + +test('vendorFetch paces allowed hosts', async () => { + const times: number[] = []; + const original = globalThis.fetch; + globalThis.fetch = (async () => { + times.push(Date.now()); + return new Response('{}', { status: 200 }); + }) as typeof fetch; + try { + await vendorFetch('nasdaq', 'https://api.nasdaq.com/a', { + hostAllowlist: /api\.nasdaq\.com/i, + }); + await vendorFetch('nasdaq', 'https://api.nasdaq.com/b', { + hostAllowlist: /api\.nasdaq\.com/i, + }); + assert.equal(times.length, 2); + assert.ok(times[1] - times[0] >= 700, `expected pacing, gap=${times[1] - times[0]}`); + } finally { + globalThis.fetch = original; + } +}); diff --git a/app/server/src/services/__tests__/vendorHttpGuard.test.ts b/app/server/src/services/__tests__/vendorHttpGuard.test.ts new file mode 100644 index 0000000..c61aa0d --- /dev/null +++ b/app/server/src/services/__tests__/vendorHttpGuard.test.ts @@ -0,0 +1,120 @@ +/** + * Structural guard: bare `fetch(` must not appear in adapter/service vendor code. + * Future integrations that skip vendorGate fail this test in CI. + * + * Allowlist: vendorGate itself, secHttp (wrapper), tests, LLM (user-configured endpoint). + */ +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { readdirSync, readFileSync, statSync } from 'node:fs'; +import { join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(fileURLToPath(new URL('../..', import.meta.url))); // app/server/src + +/** Paths relative to src/ that may call global fetch (the gate implementations). */ +const ALLOW_FETCH_PATHS = [ + 'services/vendorGate.ts', + 'services/secHttp.ts', + 'llm/openaiCompatible.ts', // user-configured LLM endpoint; gated via family 'llm' when queued +]; + +/** Directories scanned for violations. */ +const SCAN_DIRS = ['adapters', 'services', 'macro', 'mirror']; + +function walk(dir: string, out: string[] = []): string[] { + for (const name of readdirSync(dir)) { + if (name === '__tests__' || name === 'node_modules') continue; + const p = join(dir, name); + const st = statSync(p); + if (st.isDirectory()) walk(p, out); + else if (name.endsWith('.ts') && !name.endsWith('.test.ts')) out.push(p); + } + return out; +} + +const BARE_FETCH_RE = /(? { + const violations: string[] = []; + for (const sub of SCAN_DIRS) { + const dir = join(ROOT, sub); + let files: string[]; + try { + files = walk(dir); + } catch { + continue; + } + for (const file of files) { + const rel = relative(ROOT, file).replace(/\\/g, '/'); + if (ALLOW_FETCH_PATHS.includes(rel)) continue; + const text = readFileSync(file, 'utf8'); + // Strip line comments for a coarse check + const stripped = text.replace(/\/\/.*$/gm, '').replace(/\/\*[\s\S]*?\*\//g, ''); + if (BARE_FETCH_RE.test(stripped)) { + violations.push(rel); + } + BARE_FETCH_RE.lastIndex = 0; + } + } + assert.deepEqual( + violations, + [], + `Bare fetch() found (use vendorFetch / withVendorGate / secHttp):\n ${violations.join('\n ')}\n` + + `See docs/VENDOR_INTEGRATIONS.md`, + ); +}); + +test('new vendor families can be registered at runtime', async () => { + const { + registerVendorIntegration, + withVendorGate, + sourceToFamily, + resetVendorRegistryForTests, + assertSourcesBound, + } = await import('../vendorGate.ts'); + + registerVendorIntegration({ + family: 'polygon', + sourceKinds: ['polygon', 'polygon-quotes'], + policy: { minIntervalMs: 200, maxInflight: 1, drainJobBudget: 2 }, + }); + assert.equal(sourceToFamily('polygon'), 'polygon'); + assert.equal(sourceToFamily('polygon-quotes'), 'polygon'); + assertSourcesBound(['polygon', 'yfinance']); + + let ran = false; + await withVendorGate('polygon', async () => { + ran = true; + return 1; + }); + assert.equal(ran, true); + + resetVendorRegistryForTests(); + assert.equal(sourceToFamily('polygon'), null); +}); + +test('AdapterQueue rejects adapters without a vendor family', async () => { + const { createDb, initSchema } = await import('../../db/client.ts'); + const { AdapterQueue } = await import('../../queue/AdapterQueue.ts'); + const { FakeSourceAdapter } = await import('../../adapters/SourceAdapter.ts'); + const { resetVendorRegistryForTests } = await import('../vendorGate.ts'); + + resetVendorRegistryForTests(); + const db = createDb({ path: ':memory:' }); + initSchema(db); + // FakeSourceAdapter claims sourceKind yfinance which is bound — use a fake unbound kind + // by casting (simulates a future SourceKind not yet registered). + const unbound = new FakeSourceAdapter('yfinance' as never); + Object.defineProperty(unbound, 'sourceKind', { value: 'brand-new-vendor' }); + + assert.throws( + () => + new AdapterQueue({ + db, + adapters: new Map([['brand-new-vendor' as never, unbound as never]]), + }), + /no vendor family|brand-new-vendor/i, + ); + resetVendorRegistryForTests(); +}); diff --git a/app/server/src/services/analystRatingsService.ts b/app/server/src/services/analystRatingsService.ts index 4b0e30c..667f67b 100644 --- a/app/server/src/services/analystRatingsService.ts +++ b/app/server/src/services/analystRatingsService.ts @@ -57,7 +57,10 @@ async function fetchFromYahoo(symbol: string): Promise<{ ratings: AnalystRating[ let raw: Record; let timeoutId: ReturnType | null = null; try { - const yfPromise = yf.quoteSummary(symbol, { modules: ['upgradeDowngradeHistory', 'recommendationTrend'] }, { validateResult: false }); + const { withVendorGate } = await import('./vendorGate.ts'); + const yfPromise = withVendorGate('yfinance', () => + yf.quoteSummary(symbol, { modules: ['upgradeDowngradeHistory', 'recommendationTrend'] }, { validateResult: false }), + ); const timeoutPromise = new Promise((_, reject) => { timeoutId = setTimeout(() => reject(new Error('Yahoo Finance timed out')), YAHOO_TIMEOUT_MS); }); @@ -141,10 +144,24 @@ export function getAnalystRatings(db: DatabaseSync, symbol: string): { ratings: }; } -export async function fetchAndStoreAnalystRatings(db: DatabaseSync, symbol: string): Promise<{ ratings: AnalystRating[]; consensus: AnalystConsensus | null; stale: boolean } | { error: string }> { +export interface FetchAnalystOpts { + /** Called when Yahoo returns a rate-limit signal so the shared queue can cool down. */ + onRateLimit?: () => void; +} + +export async function fetchAndStoreAnalystRatings( + db: DatabaseSync, + symbol: string, + opts: FetchAnalystOpts = {}, +): Promise<{ ratings: AnalystRating[]; consensus: AnalystConsensus | null; stale: boolean } | { error: string }> { const upper = symbol.toUpperCase(); const result = await fetchFromYahoo(upper); - if ('error' in result) return { error: result.error }; + if ('error' in result) { + if (/too many requests|rate[- ]?limit|429|edge:\s*too many/i.test(result.error)) { + try { opts.onRateLimit?.(); } catch { /* ignore */ } + } + return { error: result.error }; + } const now = new Date().toISOString(); const upsert = db.prepare(` diff --git a/app/server/src/services/captureIngest.ts b/app/server/src/services/captureIngest.ts new file mode 100644 index 0000000..aef6ac0 --- /dev/null +++ b/app/server/src/services/captureIngest.ts @@ -0,0 +1,152 @@ +// Investor Flow — X position-capture materializer (M21 follow-up). +// +// Reads a tracked fund manager's posts from x_cookie_posts (already fetched by +// the `x` timeline job), classifies them with captureParser, and upserts total +// position snapshots into fund_position_records with source='capture'. +// +// Recency rule (mirrorEngine): captures and 13F compete on MAX(as_of) per symbol. +// CLAIMS are explicitly skipped in v1: a claim ("added 865,000 shares") is a +// DELTA, not a total — letting it win recency would corrupt the Live Book. +// Claim folding (delta onto latest total) is future work. +// +// Local-only: no network, no rate limiting — this is a materializer, run from +// the x schedule branch after timeline jobs are enqueued. +import type { DatabaseSync } from 'node:sqlite'; +import { extractCaptures } from '../mirror/captureParser.ts'; + +export interface CaptureIngestStats { + fundId: string; + postsScanned: number; + captures: number; // total-position posts materialized (inserted or refreshed) + inserted: number; + refreshed: number; + claims: number; // delta posts detected — recorded for observability, not materialized + skipped: number; // commentary / unparseable / missing symbol + disabled: boolean; +} + +export interface FundWithHandle { + id: string; + x_handle: string | null; + enabled: number; +} + +export function listTrackedFundHandles(db: DatabaseSync): FundWithHandle[] { + return db.prepare('SELECT id, x_handle, enabled FROM tracked_funds').all() as FundWithHandle[]; +} + +interface PostRow { + post_id: string; + body_text: string | null; + posted_at: string; +} + +/** + * Materialize captures for one fund. Idempotent: converges on + * (fund_id, symbol, source='capture', evidence_url) — a re-run with the same + * post updates the row instead of duplicating it. + */ +export function ingestFundCaptures(db: DatabaseSync, fundId: string): CaptureIngestStats { + const fund = db.prepare('SELECT id, x_handle, enabled FROM tracked_funds WHERE id = ?').get(fundId) as + | FundWithHandle + | undefined; + + const stats: CaptureIngestStats = { + fundId, + postsScanned: 0, + captures: 0, + inserted: 0, + refreshed: 0, + claims: 0, + skipped: 0, + disabled: false, + }; + if (!fund || !fund.x_handle) return stats; + if (fund.enabled !== 1) { stats.disabled = true; return stats; } + + const posts = db.prepare( + `SELECT post_id, body_text, posted_at FROM x_cookie_posts + WHERE lower(author_handle) = lower(?) + AND body_text IS NOT NULL AND body_text != '' + ORDER BY posted_at ASC`, + ).all(fund.x_handle) as PostRow[]; + stats.postsScanned = posts.length; + + const selectExisting = db.prepare( + `SELECT id FROM fund_position_records WHERE fund_id=? AND symbol=? AND source='capture' AND evidence_url=?`, + ); + const updateExisting = db.prepare( + `UPDATE fund_position_records SET shares=?, value_usd=?, cost_basis=?, as_of=?, notes=COALESCE(notes, ?) + WHERE id=?`, + ); + const insertNew = db.prepare( + `INSERT INTO fund_position_records (id, fund_id, symbol, shares, value_usd, cost_basis, as_of, source, evidence_url, notes, created_at) + VALUES (?,?,?,?,?,?,?, 'capture', ?, ?, ?)`, + ); + + for (const p of posts) { + // One post can mention several names (Mike: SLNH fill + OPEN total in one tweet). + const parsedList = extractCaptures(p.body_text ?? ''); + if (parsedList.length === 0) { + stats.skipped++; + continue; + } + + let anyMaterialized = false; + for (const parsed of parsedList) { + if (parsed.class === 'claim') { + stats.claims++; + continue; + } + if (parsed.class !== 'capture') continue; + if (!parsed.symbol) continue; + + const asOf = normalizePostedAtDate(p.posted_at); + const evidenceUrl = `https://x.com/${fund.x_handle}/status/${p.post_id}`; + // Preserve book_reset / pre_reset markers; only stamp instrument notes when empty. + const instrumentNote = parsed.instrument && parsed.instrument !== 'equity' + ? parsed.instrument + : null; + const existing = selectExisting.get(fundId, parsed.symbol, evidenceUrl) as { id?: string } | undefined; + + if (existing?.id) { + updateExisting.run( + parsed.shares ?? null, parsed.value_usd ?? null, parsed.cost_basis ?? null, asOf, + instrumentNote, existing.id, + ); + stats.refreshed++; + } else { + insertNew.run( + crypto.randomUUID(), fundId, parsed.symbol, + parsed.shares ?? null, parsed.value_usd ?? null, parsed.cost_basis ?? null, asOf, evidenceUrl, + instrumentNote, new Date().toISOString(), + ); + stats.inserted++; + } + stats.captures++; + anyMaterialized = true; + } + if (!anyMaterialized && parsedList.every((x) => x.class === 'claim')) { + // already counted as claims + } else if (!anyMaterialized) { + stats.skipped++; + } + } + + return stats; +} + +/** posted_at may be ISO or Twitter "Wed Jul 15 20:43:28 +0000 2026". */ +function normalizePostedAtDate(postedAt: string): string { + if (/^\d{4}-\d{2}-\d{2}/.test(postedAt)) return postedAt.slice(0, 10); + const t = Date.parse(postedAt); + if (Number.isFinite(t)) return new Date(t).toISOString().slice(0, 10); + return postedAt.slice(0, 10); +} + +/** Materialize captures for every enabled tracked fund with an x_handle. */ +export function ingestAllFundCaptures(db: DatabaseSync): CaptureIngestStats[] { + return listTrackedFundHandles(db) + .filter((f) => f.x_handle && f.enabled === 1) + .map((f) => ingestFundCaptures(db, f.id)); +} diff --git a/app/server/src/services/cusipRegistry.ts b/app/server/src/services/cusipRegistry.ts new file mode 100644 index 0000000..75cb3c7 --- /dev/null +++ b/app/server/src/services/cusipRegistry.ts @@ -0,0 +1,133 @@ +// Shared CUSIP ↔ symbol registry for institutional pipeline reliability. +// +// EFTS full-text (efts.sec.gov) is flaky / often 403. Alerts need CUSIP for 13F +// holder refresh and SC XML often carries issuerCusip only after a successful +// download. This module provides offline curated mappings + kv_cache helpers +// so resolve never depends solely on EFTS. + +import type { DatabaseSync } from 'node:sqlite'; + +/** Curated CUSIP → symbol (observed in 13F info tables / SC XML). */ +export const CUSIP_TO_SYMBOL: Record = { + '05759B305': 'BKKT', // Bakkt, Inc + '09175A206': 'BTM', // BitMine Immersion Technologies, Inc. + '13646K108': 'CP', // Canadian Pacific Kansas City + '17253J106': 'CIFR', // Cipher Digital Inc. + '21036P108': 'STZ', // Constellation Brands, Inc. + Q4982L109: 'IREN', // IREN Limited + '46438F101': 'IBIT', // iShares Bitcoin Trust ETF + '46438R105': 'ETHA', // iShares Ethereum Trust ETF + '670100205': 'NVO', // Novo-Nordisk A/S + '683712103': 'OPEN', // Opendoor Technologies Inc. + '713448108': 'PEP', // PepsiCo, Inc. + '75886F107': 'REGN', // Regeneron Pharmaceuticals, Inc. + '862945300': 'ASST', // Strive, Inc. + '87612E106': 'TGT', // Target Corporation + // High-demand mega-caps (standard 9-char CUSIPs) so pipeline works when EFTS is down + '037833100': 'AAPL', + '023135106': 'AMZN', + '02079K305': 'GOOGL', + '594918104': 'MSFT', + '67066G104': 'NVDA', + '88160R101': 'TSLA', + '30303M102': 'META', + '11135F101': 'AVGO', + '007903107': 'AMD', + '46090E103': 'INTC', + '166764100': 'CVX', + '30231G102': 'XOM', + '46625H100': 'JPM', + '060505104': 'BAC', + '191216100': 'KO', + '742718109': 'PG', + '931142103': 'WMT', + '084670702': 'BRK-B', + '922908769': 'VOO', + '464287200': 'IWM', + '464287655': 'IWF', + '464287499': 'IWD', + '464288281': 'EEM', + '464287465': 'EFA', + '78462F103': 'SPY', + '92189F106': 'GDX', + '922042858': 'VTI', +}; + +/** Inverted: symbol → CUSIP (first mapping wins). */ +export const SYMBOL_TO_CUSIP: Record = Object.fromEntries( + Object.entries(CUSIP_TO_SYMBOL).map(([cusip, sym]) => [sym.toUpperCase(), cusip.toUpperCase()]), +); + +export function cusipCacheKey(symbol: string): string { + return `sec:cusip:${symbol.toUpperCase()}`; +} + +export function isValidCusip(c: string | null | undefined): c is string { + return !!c && /^[0-9A-Z]{8,9}$/i.test(c); +} + +/** Offline curated CUSIP for a ticker (no network). */ +export function curatedCusipForSymbol(symbol: string): string | null { + const c = SYMBOL_TO_CUSIP[symbol.toUpperCase()]; + return isValidCusip(c) ? c.toUpperCase() : null; +} + +export function curatedSymbolForCusip(cusip: string): string | null { + const s = CUSIP_TO_SYMBOL[cusip.toUpperCase()]; + return s ? s.toUpperCase() : null; +} + +export function readCachedCusip(db: DatabaseSync | undefined, symbol: string): string | null { + if (!db) return null; + try { + const row = db.prepare('SELECT value FROM kv_cache WHERE key=?').get(cusipCacheKey(symbol)) as + | { value: string } + | undefined; + const v = row?.value?.trim(); + if (isValidCusip(v)) return v.toUpperCase(); + } catch { + /* ignore */ + } + return null; +} + +export function writeCachedCusip(db: DatabaseSync | undefined, symbol: string, cusip: string): void { + if (!db || !isValidCusip(cusip)) return; + try { + db.prepare('INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)').run( + cusipCacheKey(symbol), + cusip.toUpperCase(), + new Date().toISOString(), + ); + } catch { + /* ignore */ + } +} + +/** + * Resolve CUSIP without network: kv_cache → curated map. + * Also seeds cache when curated hits so subsequent runs are O(1). + */ +export function resolveCusipLocal(db: DatabaseSync | undefined, symbol: string): string | null { + const upper = symbol.toUpperCase(); + const cached = readCachedCusip(db, upper); + if (cached) return cached; + const curated = curatedCusipForSymbol(upper); + if (curated) { + writeCachedCusip(db, upper, curated); + return curated; + } + return null; +} + +/** Seed kv_cache for every curated symbol (idempotent). */ +export function seedCuratedCusips(db: DatabaseSync): number { + let n = 0; + for (const [sym, cusip] of Object.entries(SYMBOL_TO_CUSIP)) { + if (!readCachedCusip(db, sym)) { + writeCachedCusip(db, sym, cusip); + n += 1; + } + } + return n; +} diff --git a/app/server/src/services/emailAlertService.ts b/app/server/src/services/emailAlertService.ts index 4501b7c..afe43d7 100644 --- a/app/server/src/services/emailAlertService.ts +++ b/app/server/src/services/emailAlertService.ts @@ -129,3 +129,84 @@ export async function sendAlertEmail( return false; } } + +/** + * Write an alert into the notification outbox (pending). ID is the alert id so + * re-running a producer never duplicates an entry. Never touches SMTP. + */ +export function enqueueAlertEmail(db: DatabaseSync, alert: Alert): void { + try { + db.prepare( + `INSERT OR IGNORE INTO notification_outbox + (id, user_id, type, severity, title, description, symbol, created_at, status, attempt) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'pending', 0)`, + ).run( + alert.id, + alert.userId, + alert.type, + alert.severity, + alert.title, + alert.description, + alert.symbol ?? null, + alert.createdAt, + ); + } catch { + /* notification_outbox may not exist until migration runs */ + } +} + +/** + * Drain the outbox, sending one email per pending alert. Returns how many were + * actually sent. Failures stay pending (bounded retry) so SMTP latency never + * blocks the producer loop; this runs on its own timer. + */ +export async function drainEmailOutbox(db: DatabaseSync, limit = 25): Promise { + let sent = 0; + try { + const pending = db.prepare( + `SELECT id FROM notification_outbox + WHERE status = 'pending' AND attempt < 3 + ORDER BY created_at ASC + LIMIT ?`, + ).all(limit) as Array<{ id: string }>; + if (pending.length === 0) return 0; + + const fetchAlert = db.prepare( + `SELECT user_id, type, severity, title, description, symbol, created_at FROM notification_outbox WHERE id = ?`, + ); + const markSent = db.prepare(`UPDATE notification_outbox SET status='sent', sent_at=? WHERE id=?`); + const markFailed = db.prepare( + `UPDATE notification_outbox SET attempt = attempt + 1, last_error = ?, status = CASE WHEN attempt + 1 >= 3 THEN 'failed' ELSE 'pending' END WHERE id = ?`, + ); + + for (const { id } of pending) { + const row = fetchAlert.get(id) as + | { user_id: string; type: string; severity: string; title: string; description: string; symbol: string | null; created_at: string } + | undefined; + if (!row) { markSent.run(new Date().toISOString(), id); continue; } + const alert: Alert = { + id, + userId: row.user_id, + type: row.type as Alert['type'], + severity: row.severity as Alert['severity'], + title: row.title, + description: row.description, + symbol: row.symbol ?? undefined, + createdAt: row.created_at, + acknowledged: false, + dedupKey: '', + payload: {}, + }; + try { + const ok = await sendAlertEmail(db, alert); + if (ok) { markSent.run(new Date().toISOString(), id); sent += 1; } + else { markFailed.run('no SMTP / rate-limited', id); } + } catch (err) { + markFailed.run(err instanceof Error ? err.message : String(err), id); + } + } + } catch { + /* outbox not present yet */ + } + return sent; +} diff --git a/app/server/src/services/reverse13fRefresh.ts b/app/server/src/services/reverse13fRefresh.ts new file mode 100644 index 0000000..27694cf --- /dev/null +++ b/app/server/src/services/reverse13fRefresh.ts @@ -0,0 +1,501 @@ +// Reverse 13F holder refresh — fund-centric path that does NOT use EFTS. +// +// When efts.sec.gov is 403/blocked, symbol→holders discovery via CUSIP full-text +// search fails. This module instead: +// 1. Collects filer CIKs that already held the symbol (local history) +// 2. Adds tracked funds + a curated set of major 13F managers +// 3. Pulls each filer's latest 13F-HR via data.sec.gov (EdgarAdapter) +// 4. Extracts the target CUSIP and upserts institution_filings +// +// Completeness is lower than full EFTS pagination, but it keeps the alert +// pipeline live without thrashing a dead full-text index. +// +// Even when EFTS is up, CUSIP full-text ranking often misses mega-filers +// (e.g. BlackRock's 50k-line 13F). Call mode "missing_majors" after every +// EFTS pass so Vanguard/BlackRock/Fidelity/etc. still land. + +import type { DatabaseSync } from 'node:sqlite'; +import { EdgarAdapter, padCik } from '../adapters/EdgarAdapter.ts'; +import { writeCachedCusip } from './cusipRegistry.ts'; + +const edgar = new EdgarAdapter(); + +/** + * Large / systemically important 13F managers. + * CIKs verified against data.sec.gov submissions (active 13F-HR in 2026). + * Stale entity CIKs (e.g. old BlackRock Finance 1364742) silently miss books. + */ +export const MAJOR_13F_FILER_CIKS: ReadonlyArray<{ cik: string; name: string }> = [ + { cik: '0000102909', name: 'Vanguard Group Inc' }, + { cik: '0002012383', name: 'BlackRock, Inc.' }, // BLK; not old BlackRock Finance 1364742 + { cik: '0000093751', name: 'State Street Corp' }, + { cik: '0000315066', name: 'FMR LLC (Fidelity)' }, + { cik: '0001214717', name: 'Geode Capital Management LLC' }, + { cik: '0000019617', name: 'JPMorgan Chase & Co' }, + { cik: '0000886982', name: 'Goldman Sachs Group Inc' }, + { cik: '0000895421', name: 'Morgan Stanley' }, + { cik: '0000070858', name: 'Bank of America Corp / DE' }, + { cik: '0001423053', name: 'Citadel Advisors LLC' }, + { cik: '0001179392', name: 'Two Sigma Investments LP' }, + { cik: '0001037389', name: 'Renaissance Technologies LLC' }, + { cik: '0001009207', name: 'D. E. Shaw & Co., L.P.' }, + { cik: '0001273087', name: 'Millennium Management LLC' }, + { cik: '0001067983', name: 'Berkshire Hathaway Inc' }, + { cik: '0001422848', name: 'Capital Research Global Investors' }, + { cik: '0000080255', name: 'T. Rowe Price Associates Inc' }, + { cik: '0000914208', name: 'Invesco Ltd.' }, + { cik: '0000073124', name: 'Northern Trust Corp' }, + { cik: '0001374170', name: 'Norges Bank' }, + { cik: '0001446194', name: 'Susquehanna International Group LLP' }, + { cik: '0001595888', name: 'Jane Street Group LLC' }, + { cik: '0001610520', name: 'UBS Group AG' }, + { cik: '0000884546', name: 'Charles Schwab Investment Management Inc' }, + { cik: '0000902219', name: 'Wellington Management Group LLP' }, + { cik: '0001390777', name: 'Bank of New York Mellon Corp' }, + { cik: '0001167557', name: 'AQR Capital Management LLC' }, + { cik: '0001603466', name: 'Point72 Asset Management L.P.' }, +]; + +export interface Reverse13fResult { + filersTried: number; + filersWithMatch: number; + rowsWritten: number; + errors: string[]; + path: 'reverse_13f'; + mode?: Reverse13fMode; +} + +export type Reverse13fMode = 'balanced' | 'missing_majors' | 'majors_first'; + +function quarterFromPeriod(periodEnding: string, filedAt: string): string { + if (periodEnding && /^\d{4}-\d{2}/.test(periodEnding)) { + const month = parseInt(periodEnding.slice(5, 7), 10); + return `${periodEnding.slice(0, 4)}-Q${Math.ceil(month / 3)}`; + } + if (filedAt) { + const d = new Date(filedAt); + if (!Number.isNaN(d.getTime())) { + return `${d.getFullYear()}-Q${Math.ceil((d.getMonth() + 1) / 3)}`; + } + } + return ''; +} + +function normalizeCusip(c: string | null | undefined): string { + return (c ?? '').replace(/[^0-9A-Za-z]/g, '').toUpperCase(); +} + +export type ReverseFiler = { cik: string; name: string | null; source: string }; + +/** Build ordered unique filer CIK list for reverse refresh of `symbol`. */ +export function selectReverseFilerCiks( + db: DatabaseSync, + symbol: string, + maxPrior = 35, + maxMajors = 20, +): ReverseFiler[] { + const upper = symbol.toUpperCase(); + const out: ReverseFiler[] = []; + const seen = new Set(); + + const push = (cik: string, name: string | null, source: string) => { + const p = padCik(cik); + if (!p || p === '0000000000' || seen.has(p)) return; + // Skip non-numeric CIK placeholders (e.g. mr-t-invests) + if (!/^\d{10}$/.test(p)) return; + seen.add(p); + out.push({ cik: p, name, source }); + }; + + // 1. Prior holders of this symbol (largest recent books first) + try { + const prior = db + .prepare( + `SELECT filer_cik AS cik, filer_name AS name, MAX(shares) AS max_shares + FROM institution_filings + WHERE symbol = ? AND form = '13F-HR' AND filer_cik IS NOT NULL + GROUP BY filer_cik + ORDER BY max_shares DESC + LIMIT ?`, + ) + .all(upper, maxPrior) as Array<{ cik: string; name: string | null; max_shares: number }>; + for (const r of prior) push(r.cik, r.name, 'prior_holder'); + } catch { + /* ignore */ + } + + // 2. Tracked funds (mirror managers) + try { + const funds = db + .prepare(`SELECT ci_key, fund_name FROM tracked_funds WHERE enabled = 1`) + .all() as Array<{ ci_key: string; fund_name: string }>; + for (const f of funds) push(f.ci_key, f.fund_name, 'tracked_fund'); + } catch { + /* ignore */ + } + + // 3. Major managers (coverage for first-time / thin symbols) + let majors = 0; + for (const m of MAJOR_13F_FILER_CIKS) { + if (majors >= maxMajors) break; + const before = seen.size; + push(m.cik, m.name, 'major'); + if (seen.size > before) majors += 1; + } + + return out; +} + +/** True if a has a strictly newer calendar quarter label than b (YYYY-Qn). */ +export function quarterIsAfter(a: string | null | undefined, b: string | null | undefined): boolean { + if (!a) return false; + if (!b) return true; + const pa = /^(\d{4})-Q([1-4])$/.exec(a); + const pb = /^(\d{4})-Q([1-4])$/.exec(b); + if (!pa || !pb) return a > b; + const ya = Number(pa[1]); + const yb = Number(pb[1]); + if (ya !== yb) return ya > yb; + return Number(pa[2]) > Number(pb[2]); +} + +/** + * Majors that still need a reverse pull for this symbol. + * + * - Never stored for symbol → missing (e.g. BlackRock never in EFTS hits). + * - Present in the symbol's latest reported quarter → covered. + * - Only older quarter on file → re-check only if last fetch is older than + * ~10 days (Q2 filing season: do not re-download Goldman every job once + * Q1 is stored while peers already show Q2). + */ +export function selectMissingMajorFilers( + db: DatabaseSync, + symbol: string, + opts?: { staleAfterMs?: number; now?: number }, +): ReverseFiler[] { + const upper = symbol.toUpperCase(); + const staleAfterMs = opts?.staleAfterMs ?? 10 * 24 * 3600_000; + const now = opts?.now ?? Date.now(); + + let latestQ: string | null = null; + try { + const row = db + .prepare( + `SELECT MAX(reported_quarter) AS q FROM institution_filings + WHERE symbol = ? AND form = '13F-HR'`, + ) + .get(upper) as { q: string | null } | undefined; + latestQ = row?.q ?? null; + } catch { + latestQ = null; + } + + /** cik → { maxQ, lastFetchedMs } */ + const byCik = new Map(); + try { + const rows = db + .prepare( + `SELECT filer_cik AS cik, + MAX(reported_quarter) AS max_q, + MAX(fetched_at) AS last_fetched + FROM institution_filings + WHERE symbol = ? AND form = '13F-HR' AND filer_cik IS NOT NULL + GROUP BY filer_cik`, + ) + .all(upper) as Array<{ cik: string; max_q: string | null; last_fetched: string | null }>; + for (const r of rows) { + const p = padCik(r.cik); + if (!p) continue; + const ts = r.last_fetched ? Date.parse(r.last_fetched) : 0; + byCik.set(p, { + maxQ: r.max_q, + lastFetchedMs: Number.isFinite(ts) ? ts : 0, + }); + } + } catch { + /* ignore */ + } + + const missing: ReverseFiler[] = []; + for (const m of MAJOR_13F_FILER_CIKS) { + const p = padCik(m.cik); + if (!p) continue; + const have = byCik.get(p); + if (!have) { + missing.push({ cik: p, name: m.name, source: 'missing_major' }); + continue; + } + // Already on the frontier quarter for this symbol + if (latestQ && have.maxQ === latestQ) continue; + // Have some history; only re-poll when stale (new quarter filings may land) + if (have.lastFetchedMs > 0 && now - have.lastFetchedMs < staleAfterMs) continue; + // Never successfully timestamped — treat as missing + if (have.lastFetchedMs <= 0) { + missing.push({ cik: p, name: m.name, source: 'missing_major' }); + continue; + } + // Stale relative to frontier: re-check for a newer 13F + if (latestQ && quarterIsAfter(latestQ, have.maxQ)) { + missing.push({ cik: p, name: m.name, source: 'missing_major' }); + } + } + return missing; +} + +/** Majors first, then prior holders (used when EFTS is down cold). */ +export function selectMajorsFirstFilers( + db: DatabaseSync, + symbol: string, + maxMajors = 28, + maxPrior = 10, +): ReverseFiler[] { + const upper = symbol.toUpperCase(); + const out: ReverseFiler[] = []; + const seen = new Set(); + const push = (cik: string, name: string | null, source: string) => { + const p = padCik(cik); + if (!p || p === '0000000000' || seen.has(p) || !/^\d{10}$/.test(p)) return; + seen.add(p); + out.push({ cik: p, name, source }); + }; + + let majors = 0; + for (const m of MAJOR_13F_FILER_CIKS) { + if (majors >= maxMajors) break; + const before = seen.size; + push(m.cik, m.name, 'major'); + if (seen.size > before) majors += 1; + } + + try { + const prior = db + .prepare( + `SELECT filer_cik AS cik, filer_name AS name, MAX(shares) AS max_shares + FROM institution_filings + WHERE symbol = ? AND form = '13F-HR' AND filer_cik IS NOT NULL + GROUP BY filer_cik + ORDER BY max_shares DESC + LIMIT ?`, + ) + .all(upper, maxPrior) as Array<{ cik: string; name: string | null }>; + for (const r of prior) push(r.cik, r.name, 'prior_holder'); + } catch { + /* ignore */ + } + + return out; +} + +/** + * Refresh 13F positions for `symbol` by re-pulling known/major filers' 13F-HR + * history and matching on CUSIP. Safe under ADR-0009 (EdgarAdapter bucket). + * + * Important: 13F is a quarter-end snapshot. Pulling only the latest filing + * leaves majors with a single row and no "vs prior Q" (BlackRock IREN looked + * like a first-time report when Q1–Q4 history existed on EDGAR). + */ +/** Rank 13F filings: newest report period first; prefer non-amendments. */ +function rank13fFilings(filings: Array>): Array> { + return filings + .map((f) => { + const reportDate = String(f.reportDate ?? f.periodEnding ?? f.period_of_report ?? ''); + const fileDate = String(f.filingDate ?? f.fileDate ?? ''); + const form = String(f.form ?? '').toUpperCase(); + const isAmend = form.includes('/A') ? 0 : 1; + const reportTs = Date.parse(reportDate) || 0; + const fileTs = Date.parse(fileDate) || 0; + return { f, reportTs, fileTs, isAmend, reportDate }; + }) + .sort((a, b) => b.reportTs - a.reportTs || b.isAmend - a.isAmend || b.fileTs - a.fileTs) + .map((x) => x.f); +} + +/** One original (prefer non-/A) filing per report period, newest first, capped. */ +function pickRecent13fs( + filings: Array>, + maxQuarters: number, +): Array> { + const ranked = rank13fFilings(filings); + const out: Array> = []; + const seenPeriod = new Set(); + for (const f of ranked) { + const reportDate = String(f.reportDate ?? f.periodEnding ?? f.period_of_report ?? ''); + const periodKey = reportDate.slice(0, 10) || String(f.accessionNumber ?? f.accession ?? ''); + if (!periodKey || seenPeriod.has(periodKey)) continue; + seenPeriod.add(periodKey); + out.push(f); + if (out.length >= maxQuarters) break; + } + return out; +} + +export async function refreshHoldersViaReverse13f( + db: DatabaseSync, + symbol: string, + cusip: string, + opts?: { maxFilers?: number; mode?: Reverse13fMode; maxQuartersPerFiler?: number }, +): Promise { + const upper = symbol.toUpperCase(); + const targetCusip = normalizeCusip(cusip); + const mode: Reverse13fMode = opts?.mode ?? 'balanced'; + if (!targetCusip) { + return { + filersTried: 0, + filersWithMatch: 0, + rowsWritten: 0, + errors: ['empty cusip'], + path: 'reverse_13f', + mode, + }; + } + + writeCachedCusip(db, upper, targetCusip); + + // Keep default modest: each filer is filings_index + N × form13f XML (heavy). + // maxQuartersPerFiler > 1 is required for "vs prior Q" on majors. + const maxFilers = opts?.maxFilers ?? 15; + const maxQuartersPerFiler = Math.max(1, Math.min(opts?.maxQuartersPerFiler ?? 4, 8)); + let filers: ReverseFiler[]; + if (mode === 'missing_majors') { + filers = selectMissingMajorFilers(db, upper).slice(0, maxFilers); + } else if (mode === 'majors_first') { + filers = selectMajorsFirstFilers(db, upper).slice(0, maxFilers); + } else { + filers = selectReverseFilerCiks(db, upper).slice(0, maxFilers); + } + + const now = new Date().toISOString(); + let filersWithMatch = 0; + let rowsWritten = 0; + let tried = 0; + const errors: string[] = []; + + const insert = db.prepare(` + INSERT INTO institution_filings + (filer_cik, filer_name, filer_sic, symbol, form, shares, value_usd, reported_quarter, filed_at, accession, fetched_at, put_call) + VALUES (?, ?, NULL, ?, '13F-HR', ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(filer_cik, symbol, reported_quarter, form) DO UPDATE SET + shares = excluded.shares, + value_usd = excluded.value_usd, + filer_name = COALESCE(excluded.filer_name, institution_filings.filer_name), + accession = excluded.accession, + put_call = excluded.put_call, + filed_at = excluded.filed_at, + fetched_at = excluded.fetched_at + `); + + for (const filer of filers) { + tried += 1; + try { + const indexRes = await edgar.filings_index(filer.cik, { + formTypes: ['13F-HR', '13F-HR/A'], + }); + const filings = (indexRes.value ?? []) as Array>; + const recent = pickRecent13fs(filings, maxQuartersPerFiler); + if (!recent.length) continue; + + let matchedAnyQuarter = false; + for (const filing of recent) { + const accession = String( + filing.accessionNumber ?? filing.accession ?? filing.adsh ?? '', + ); + if (!accession) continue; + + const reportDate = String( + filing.reportDate ?? filing.periodEnding ?? filing.period_of_report ?? '', + ); + const fileDate = String(filing.filingDate ?? filing.fileDate ?? now); + const quarter = quarterFromPeriod(reportDate, fileDate); + if (!quarter) continue; + + let holdings: Array<{ + cusip: string; + issuerName?: string; + value?: number; + sshPrnamt?: number; + putCall?: string; + }>; + try { + const holdRes = await edgar.form13f_holdings(filer.cik, accession); + holdings = ((holdRes.value as { holdings?: Array> })?.holdings ?? + []) as typeof holdings; + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (/429|403|too many|rate.?limit|access denied|threshold/i.test(msg)) { + errors.push(msg); + return { + filersTried: tried, + filersWithMatch, + rowsWritten, + errors, + path: 'reverse_13f', + mode, + }; + } + if (errors.length < 8) errors.push(`${filer.cik}/${accession}: ${msg.slice(0, 100)}`); + continue; + } + + // Aggregate matches (managers often split one issuer across many lines) + let shares = 0; + let value = 0; + let putCall = ''; + let matched = false; + for (const h of holdings) { + if (normalizeCusip(h.cusip) !== targetCusip) continue; + matched = true; + shares += Number(h.sshPrnamt) || 0; + value += Number(h.value) || 0; + if (h.putCall) putCall = String(h.putCall); + } + + if (!matched) continue; + matchedAnyQuarter = true; + if (shares <= 0 && value <= 0) continue; + + const fromFiling = String(filing.companyName ?? filing.name ?? '').trim(); + const filerName = filer.name || fromFiling || null; + + insert.run( + filer.cik, + filerName, + upper, + shares, + value, + quarter, + fileDate, + accession, + now, + putCall || null, + ); + rowsWritten += 1; + } + if (matchedAnyQuarter) filersWithMatch += 1; + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + // Surface rate limits to caller so AdapterQueue can cool the source + if (/429|403|too many|rate.?limit|access denied|threshold/i.test(msg)) { + errors.push(msg); + return { + filersTried: tried, + filersWithMatch, + rowsWritten, + errors, + path: 'reverse_13f', + mode, + }; + } + // 503 on a single document is transient — skip filer, keep going + if (errors.length < 8) errors.push(`${filer.cik}: ${msg.slice(0, 120)}`); + } + } + + return { + filersTried: tried, + filersWithMatch, + rowsWritten, + errors, + path: 'reverse_13f', + mode, + }; +} diff --git a/app/server/src/services/secDataFetcher.ts b/app/server/src/services/secDataFetcher.ts index 7183ee4..7a4aa4b 100644 --- a/app/server/src/services/secDataFetcher.ts +++ b/app/server/src/services/secDataFetcher.ts @@ -6,6 +6,22 @@ import type { DatabaseSync } from 'node:sqlite'; import { EdgarAdapter, padCik } from '../adapters/EdgarAdapter.ts'; +import { + isValidCusip, + readCachedCusip, + resolveCusipLocal, + writeCachedCusip, +} from './cusipRegistry.ts'; +import { secFetch } from './secHttp.ts'; + +/** 13F CUSIPs are often 8-char (no check digit); compare leniently. */ +function cusipsEqual(a: string | null | undefined, b: string | null | undefined): boolean { + const na = (a ?? '').replace(/[^0-9A-Za-z]/g, '').toUpperCase(); + const nb = (b ?? '').replace(/[^0-9A-Za-z]/g, '').toUpperCase(); + if (!na || !nb) return false; + if (na === nb) return true; + return na.slice(0, 8) === nb.slice(0, 8); +} // --------------------------------------------------------------------------- // SC 13D / SC 13G data types @@ -29,9 +45,6 @@ const edgar = new EdgarAdapter(); // Symbol → CIK + Company Name resolution // --------------------------------------------------------------------------- -const OPERATOR_EMAIL = process.env.SEC_OPERATOR_EMAIL ?? 'research@example.com'; -const UA = `Investor Flow (${OPERATOR_EMAIL})`; - // In-memory cache for company tickers (refreshed daily) let tickersCache: Record | null = null; let tickersCacheDate: string | null = null; @@ -40,9 +53,7 @@ async function loadTickers(): Promise; @@ -51,26 +62,78 @@ async function loadTickers(): Promise { +async function resolveCik(symbol: string, db?: DatabaseSync): Promise { + const upper = symbol.toUpperCase(); + // Prefer local symbols table (seeded from company_tickers / Yahoo). + if (db) { + try { + const row = db.prepare('SELECT cik FROM symbols WHERE symbol=?').get(upper) as + | { cik: string | null } + | undefined; + if (row?.cik && String(row.cik).trim()) { + return padCik(String(row.cik)); + } + } catch { + /* ignore */ + } + } try { const tickers = await loadTickers(); for (const v of Object.values(tickers)) { - if (v.ticker === symbol.toUpperCase()) { + if (v.ticker === upper) { return String(v.cik_str).padStart(10, '0'); } } return null; - } catch { return null; } + } catch { + return null; + } } -async function resolveCompanyName(symbol: string): Promise { +async function resolveCompanyName(symbol: string, db?: DatabaseSync): Promise { + const upper = symbol.toUpperCase(); + if (db) { + try { + const row = db.prepare('SELECT name FROM symbols WHERE symbol=?').get(upper) as + | { name: string | null } + | undefined; + if (row?.name?.trim()) return row.name.trim(); + } catch { + /* ignore */ + } + } try { const tickers = await loadTickers(); for (const v of Object.values(tickers)) { - if (v.ticker === symbol.toUpperCase()) return v.title; + if (v.ticker === upper) return v.title; } return null; - } catch { return null; } + } catch { + return null; + } +} + +/** Strip legal suffixes for looser issuer matching. */ +function normalizeIssuerName(name: string): string { + return name + .toUpperCase() + .replace(/[,.\s]+/g, ' ') + .replace( + /\b(LIMITED|LTD|INC|INCORPORATED|CORP|CORPORATION|CO|PLC|SA|AG|NV|LLC|LP|HOLDINGS|GROUP)\b/g, + ' ', + ) + .replace(/\s+/g, ' ') + .trim(); +} + +/** Thrown when SEC full-text search is blocked / rate-limited (not a missing CUSIP). */ +export class SecEftsOutageError extends Error { + constructor(status: number, detail?: string) { + super( + `SEC EFTS outage HTTP ${status}${detail ? `: ${detail}` : ''} - Access Denied or rate limit; cool down, do not thrash`, + ); + this.name = 'SecEftsOutageError'; + } } // --------------------------------------------------------------------------- @@ -87,48 +150,154 @@ interface EftsFiling { /** * Resolve a stock symbol to its CUSIP by fetching a recent 13F-HR from EDGAR - * that holds the stock. Uses company name search → fetch one filing → extract CUSIP. + * that holds the stock. Caches success in kv_cache (sec:cusip:SYMBOL). + * + * Reliability: company-name EFTS alone is flaky for names like "IREN Limited". + * We try full name + core token, loose issuer match, and local cache. */ -async function resolveCusip(symbol: string): Promise { - const companyName = await resolveCompanyName(symbol); - if (!companyName) return null; - - const encoded = encodeURIComponent(`"${companyName}"`); - const url = `https://efts.sec.gov/LATEST/search-index?q=${encoded}&forms=13F-HR&_source=ciks,adsh&size=1`; - - const resp = await fetch(url, { - headers: { 'User-Agent': UA, Accept: 'application/json' }, - }); - if (!resp.ok) return null; - - const body = await resp.json() as { - hits?: { hits?: Array<{ _source: Record }> }; - }; - - const hit = body?.hits?.hits?.[0]; - if (!hit) return null; - - const ciks = hit._source.ciks as string[] | undefined; - const adsh = hit._source.adsh as string | undefined; - if (!ciks?.length || !adsh) return null; - - // Fetch the 13F-HR and extract CUSIP for our symbol +/** + * Re-download a recent SC 13G/13D for this symbol and pull issuerCusip from XML. + * Automated path: SC feed often works when 13F name search fails. + */ +async function resolveCusipFromStoredScAccessions( + db: DatabaseSync, + symbol: string, +): Promise { try { - const cik = ciks[0].padStart(10, '0'); - const holdingsResult = await edgar.form13f_holdings(cik, adsh); - const holdings = (holdingsResult.value as { holdings: Array<{ - cusip: string; - issuerName: string; - }> }).holdings; + const rows = db.prepare( + `SELECT accession FROM institution_filings + WHERE symbol=? AND form IN ('13G','13D','SC 13G','SC 13D') + AND accession IS NOT NULL AND accession != '' + ORDER BY filed_at DESC LIMIT 5`, + ).all(symbol.toUpperCase()) as Array<{ accession: string }>; - const companyNorm = companyName.toUpperCase().replace(/[,.\s]+/g, ' '); - for (const h of holdings) { - const issuerNorm = (h.issuerName ?? '').toUpperCase().replace(/[,.\s]+/g, ' '); - if (issuerNorm.includes(companyNorm) || companyNorm.includes(issuerNorm)) { - return h.cusip; + for (const r of rows) { + const accession = r.accession; + const accNoDashes = accession.replace(/-/g, ''); + const filerCik = padCik(accession.slice(0, 10)); + // Try common primary doc names + for (const doc of ['primary_doc.xml', 'xslSchedule13G-PRIMARY-DOCUMENT.xml', 'xslSchedule13D-PRIMARY-DOCUMENT.xml']) { + try { + const xmlUrl = `https://www.sec.gov/Archives/edgar/data/${filerCik}/${accNoDashes}/${doc}`; + const xmlResp = await secFetch(xmlUrl, { + accept: 'application/xml, text/xml, */*', + }); + if (!xmlResp.ok) continue; + const parsed = parseScFiling(await xmlResp.text()); + if (parsed?.cusip && isValidCusip(parsed.cusip)) { + const cusip = parsed.cusip.toUpperCase(); + writeCachedCusip(db, symbol, cusip); + return cusip; + } + } catch { + continue; + } } } - } catch { /* ignore */ } + } catch { + /* ignore */ + } + return null; +} + +async function resolveCusip(symbol: string, db?: DatabaseSync): Promise { + const upper = symbol.toUpperCase(); + + // Path 0: kv_cache + curated offline map (never depends on EFTS) + const local = resolveCusipLocal(db, upper); + if (local) return local; + + // Path A: SC 13G/13D we already stored (accessions) - reparse issuerCusip + if (db) { + const fromSc = await resolveCusipFromStoredScAccessions(db, upper); + if (fromSc) return fromSc; + } + + const companyName = await resolveCompanyName(upper, db); + if (!companyName) return null; + + const companyNorm = normalizeIssuerName(companyName); + // Tokens: full name, then first significant token (e.g. IREN from "IREN Limited") + const queries = [ + companyName, + companyNorm, + companyNorm.split(' ').find((t) => t.length >= 3) ?? companyNorm, + upper, + ].filter((q, i, a) => q && a.indexOf(q) === i); + + let sawEftsOutage: { status: number } | null = null; + + for (const q of queries) { + try { + const encoded = encodeURIComponent(`"${q}"`); + // Prefer more hits so we can find a filing that actually holds the name + const url = `https://efts.sec.gov/LATEST/search-index?q=${encoded}&forms=13F-HR&_source=ciks,adsh,display_names&size=5`; + let resp: Response; + try { + resp = await secFetch(url); + } catch (e) { + // secHttp throws SecRateLimitError on 403/429 — surface as EFTS outage + const msg = e instanceof Error ? e.message : String(e); + if (/rate limit|HTTP 403|HTTP 429|HTTP 503|access denied/i.test(msg)) { + const m = msg.match(/HTTP (\d+)/); + sawEftsOutage = { status: m ? Number(m[1]) : 403 }; + break; + } + throw e; + } + if (resp.status === 403 || resp.status === 429 || resp.status === 503) { + sawEftsOutage = { status: resp.status }; + break; // do not hammer remaining queries + } + if (!resp.ok) continue; + + const body = (await resp.json()) as { + hits?: { hits?: Array<{ _source: Record }> }; + }; + const hits = body?.hits?.hits ?? []; + + for (const hit of hits.slice(0, 5)) { + const ciks = hit._source.ciks as string[] | undefined; + const adsh = hit._source.adsh as string | undefined; + if (!ciks?.length || !adsh) continue; + + try { + const cik = ciks[0].padStart(10, '0'); + const holdingsResult = await edgar.form13f_holdings(cik, adsh); + const holdings = ( + holdingsResult.value as { + holdings: Array<{ cusip: string; issuerName: string }>; + } + ).holdings; + + for (const h of holdings) { + const issuerNorm = normalizeIssuerName(h.issuerName ?? ''); + if (!issuerNorm || !h.cusip) continue; + if ( + issuerNorm.includes(companyNorm) || + companyNorm.includes(issuerNorm) || + issuerNorm.split(' ').some((t) => t.length >= 3 && companyNorm.includes(t)) || + companyNorm.split(' ').some((t) => t.length >= 3 && issuerNorm.includes(t)) + ) { + const cusip = h.cusip.toUpperCase(); + writeCachedCusip(db, upper, cusip); + return cusip; + } + } + } catch { + continue; + } + } + } catch (e) { + if (e instanceof SecEftsOutageError) throw e; + continue; + } + } + + // EFTS blocked ≠ missing CUSIP. Surface as rate-limit so AdapterQueue cools the source. + if (sawEftsOutage) { + throw new SecEftsOutageError(sawEftsOutage.status, `resolveCusip ${upper}`); + } return null; } @@ -150,9 +319,20 @@ async function searchByCusip(cusip: string, maxResults = 2000): Promise { - const companyName = await resolveCompanyName(symbol); - if (!companyName) return 0; + const upper = symbol.toUpperCase(); + const now = new Date().toISOString(); + let storedCount = 0; - const filings = await searchByCompanyName(companyName, ['SCHEDULE 13G', 'SCHEDULE 13D']); + // Resolve issuer CIK (symbols table → company_tickers.json). + const issuerCik = await resolveCik(upper, db); + if (issuerCik) { + storedCount += await fetchAndStoreSCFilingsByIssuerCik(db, upper, issuerCik, now); + } + + // Secondary: EFTS company-name search (may catch filings the feed misses). + const companyName = await resolveCompanyName(upper, db); + if (companyName) { + storedCount += await fetchAndStoreSCFilingsByEfts(db, upper, companyName, now); + } + + return storedCount; +} + +/** + * SC 13G/13D discovery via the issuer's submissions feed. The `recent` block + * lists filings where the issuer is the subject, including SC 13G/13D filed by + * other institutions — exactly the set we want to surface as ownership events. + */ +async function fetchAndStoreSCFilingsByIssuerCik( + db: DatabaseSync, + symbol: string, + issuerCik: string, + now: string, +): Promise { + const padded = padCik(issuerCik); + const resp = await secFetch(`https://data.sec.gov/submissions/CIK${padded}.json`); + // Surface rate-limit so AdapterQueue cools the source instead of silent empty SC. + if (resp.status === 429 || resp.status === 403) { + throw new Error( + `SEC submissions HTTP ${resp.status} for CIK${padded} - rate limit / access denied`, + ); + } + if (!resp.ok) return 0; + + const body = await resp.json() as { + filings?: { recent?: Record }; + }; + const recent = body?.filings?.recent; + if (!recent?.form?.length) return 0; + + const companyName = await resolveCompanyName(symbol) ?? (db.prepare( + 'SELECT name FROM symbols WHERE symbol = ?', + ).get(symbol.toUpperCase()) as { name: string | null } | undefined)?.name ?? symbol; + + const companyNorm = companyName.toUpperCase().replace(/[,.\s]+/g, ' '); + let storedCount = 0; + + const knownSc = new Set( + ( + db.prepare( + `SELECT accession FROM institution_filings + WHERE symbol = ? AND form IN ('13G','13D') AND accession IS NOT NULL AND accession != ''`, + ).all(symbol.toUpperCase()) as Array<{ accession: string }> + ).map((r) => r.accession), + ); + // Cap SC XML downloads per run (issuer feed can list years of history). + const MAX_SC_DOWNLOADS = 20; + let scDownloads = 0; + + for (let i = 0; i < recent.form.length; i++) { + const form = (recent.form[i] ?? '').toUpperCase(); + const isSc = /^SC\s*(?:13[GD])(?:\/A)?$/.test(form) || /^SCHEDULE\s+13[GD](?:\/A)?$/.test(form); + if (!isSc) continue; + + const accession = recent.accessionNumber?.[i]; + const filedAt = recent.filingDate?.[i] || now; + const primaryDoc = recent.primaryDocument?.[i] || 'primary_doc.xml'; + if (!accession) continue; + if (knownSc.has(accession)) continue; + if (scDownloads >= MAX_SC_DOWNLOADS) break; + + try { + const accNoDashes = accession.replace(/-/g, ''); + // The accession embeds the FILER's CIK in its first 10 digits. + const filerCik = padCik(accession.slice(0, 10)); + const docName = primaryDoc.split('/').pop() ?? primaryDoc; + const xmlUrl = `https://www.sec.gov/Archives/edgar/data/${filerCik}/${accNoDashes}/${docName}`; + const xmlResp = await secFetch(xmlUrl, { + accept: 'application/xml, text/xml, */*', + }); + if (!xmlResp.ok) continue; + scDownloads += 1; + const xmlText = await xmlResp.text(); + + const parsed = parseScFiling(xmlText); + if (!parsed) continue; + + // Verify the filing is about our target company. + const issuerNorm = (parsed.issuerName ?? '').toUpperCase().replace(/[, .\s]+/g, ' '); + if (issuerNorm && !issuerNorm.includes(companyNorm) && !companyNorm.includes(issuerNorm)) { + continue; + } + + let filerSic: string | null = null; + try { + const meta = await edgar.filer_cik_meta(parsed.filerCik); + filerSic = (meta.value as { sic?: string }).sic ?? null; + } catch { /* ignore */ } + + const quarter = quarterFromPeriod(parsed.reportedDate, filedAt); + const insert = db.prepare(` + INSERT OR REPLACE INTO institution_filings + (filer_cik, filer_name, filer_sic, symbol, form, shares, value_usd, reported_quarter, filed_at, accession, fetched_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `); + + insert.run( + parsed.filerCik, parsed.filerName, filerSic, symbol, + parsed.formType, parsed.shares, parsed.value, quarter, filedAt, accession, now, + ); + // SC XML carries issuer CUSIP - critical for automated 13F holder refresh. + if (parsed.cusip && isValidCusip(parsed.cusip)) { + writeCachedCusip(db, symbol, parsed.cusip); + } + storedCount++; + } catch { + continue; + } + } + + return storedCount; +} + +/** + * SC 13G/13D discovery via EFTS full-text search (secondary path). + */ +async function fetchAndStoreSCFilingsByEfts( + db: DatabaseSync, + symbol: string, + companyName: string, + now: string, +): Promise { + const filings = await searchByCompanyName(companyName, ['SC 13G', 'SC 13D']); if (filings.length === 0) return 0; let storedCount = 0; - const now = new Date().toISOString(); for (const filing of filings) { try { @@ -353,9 +674,7 @@ async function fetchAndStoreSCFilings( const accessionNoDashes = filing.accession.replace(/-/g, ''); const indexUrl = `https://www.sec.gov/Archives/edgar/data/${padded}/${accessionNoDashes}/index.json`; - const idxResp = await fetch(indexUrl, { - headers: { 'User-Agent': UA, Accept: 'application/json' }, - }); + const idxResp = await secFetch(indexUrl); if (!idxResp.ok) continue; const indexBody = await idxResp.json() as { @@ -374,8 +693,8 @@ async function fetchAndStoreSCFilings( if (!xmlDoc?.name) continue; const xmlUrl = `https://www.sec.gov/Archives/edgar/data/${padded}/${accessionNoDashes}/${xmlDoc.name}`; - const xmlResp = await fetch(xmlUrl, { - headers: { 'User-Agent': UA, Accept: 'application/xml, text/xml, */*' }, + const xmlResp = await secFetch(xmlUrl, { + accept: 'application/xml, text/xml, */*', }); if (!xmlResp.ok) continue; @@ -445,10 +764,11 @@ function quarterFromPeriod(periodEnding: string, filedAt: string): string { } /** - * Store all of a filing's positions that match `cusip` for `symbol`. - * Shared by the initial fetch and the lint backfill. `INSERT OR REPLACE` plus - * the unique index on (filer_cik, symbol, reported_quarter, form) makes this - * idempotent (re-runs refresh rather than duplicate). + * Store a filing's positions that match `cusip` for `symbol`. + * Mega-managers (e.g. BlackRock) split one issuer across many info-table lines; + * we **aggregate** shares/value into a single row. Unique index is + * (filer_cik, symbol, reported_quarter, form) — per-line inserts would clobber + * each other and keep only the last (often tiny) line. */ async function storeMatchedHoldings( db: DatabaseSync, @@ -467,18 +787,44 @@ async function storeMatchedHoldings( const quarter = quarterFromPeriod(filing.periodEnding, filing.filedAt); const now = new Date().toISOString(); - const insert = db.prepare(` - INSERT OR REPLACE INTO institution_filings + let shares = 0; + let value = 0; + let putCall = ''; + for (const h of matching) { + shares += Number(h.sshPrnamt) || 0; + value += Number(h.value) || 0; + if (h.putCall) putCall = h.putCall; + } + if (shares <= 0 && value <= 0) return 0; + + db.prepare(` + INSERT INTO institution_filings (filer_cik, filer_name, filer_sic, symbol, form, shares, value_usd, reported_quarter, filed_at, accession, fetched_at, put_call) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `); - - let n = 0; - for (const h of matching) { - insert.run(filing.cik, filing.filerName, filerSic, symbol, formType, h.sshPrnamt, h.value, quarter, filing.filedAt, filing.accession, now, h.putCall || null); - n++; - } - return n; + ON CONFLICT(filer_cik, symbol, reported_quarter, form) DO UPDATE SET + shares = excluded.shares, + value_usd = excluded.value_usd, + filer_name = COALESCE(excluded.filer_name, institution_filings.filer_name), + filer_sic = COALESCE(excluded.filer_sic, institution_filings.filer_sic), + accession = excluded.accession, + put_call = excluded.put_call, + filed_at = excluded.filed_at, + fetched_at = excluded.fetched_at + `).run( + filing.cik, + filing.filerName, + filerSic, + symbol, + formType, + shares, + value, + quarter, + filing.filedAt, + filing.accession, + now, + putCall || null, + ); + return 1; } /** @@ -494,26 +840,249 @@ async function fetchAndStoreInstitutionalHolders( db: DatabaseSync, symbol: string, ): Promise { - // Step 1: Resolve CUSIP from a recent filing - const cusip = await resolveCusip(symbol); - if (!cusip) return 0; + const upper = symbol.toUpperCase(); + // Step 1: Resolve CUSIP from cache or a recent filing + const cusip = await resolveCusip(upper, db); + if (!cusip) { + recordDataQuality(db, { + symbol: upper, + kind: 'institution_filings', + discoveredCount: 0, + storedCount: 0, + missingCount: 0, + backfilled: 0, + stale: 1, + status: 'error', + detail: { reason: 'cusip not resolved', at: new Date().toISOString() }, + }); + return 0; + } - // Step 2: Search EFTS for ALL filings holding this CUSIP (paginated) - const filings = await searchByCusip(cusip); - if (filings.length === 0) return 0; - - let storedCount = 0; - for (const filing of filings) { - try { - const holdingsResult = await edgar.form13f_holdings(filing.cik, filing.accession); - const holdings = (holdingsResult.value as { holdings: ParsedHolding[] }).holdings; - const matching = holdings.filter((h) => h.cusip === cusip); - storedCount += await storeMatchedHoldings(db, symbol, filing, matching); - } catch { - continue; + // Step 2: Prefer EFTS CUSIP pagination (complete holder set). + // Step 2b: On EFTS outage / empty, reverse-index via data.sec.gov (known + major filers). + let filings: EftsFiling[] = []; + let eftsOutage: SecEftsOutageError | null = null; + try { + filings = await searchByCusip(cusip); + } catch (e) { + if (e instanceof SecEftsOutageError) { + eftsOutage = e; + } else { + throw e; } } + let storedCount = 0; + let path: 'efts' | 'reverse_13f' | 'efts_empty' = 'efts'; + + if (filings.length > 0) { + // Incremental: skip accessions we already stored. Full EFTS re-download of + // 500+ 13F XMLs never finishes under fair-access pacing, so the UI never + // sees "new" filings. Only fetch unknown accessions, newest first, capped. + // Normalize accession (with/without dashes) so skip matching is reliable. + const normAcc = (a: string) => a.replace(/-/g, '').toUpperCase(); + const knownAccessions = new Set( + ( + db.prepare( + `SELECT accession FROM institution_filings + WHERE symbol = ? AND form = '13F-HR' AND accession IS NOT NULL AND accession != ''`, + ).all(upper) as Array<{ accession: string }> + ).map((r) => normAcc(r.accession)), + ); + const ordered = [...filings].sort((a, b) => { + const tb = Date.parse(b.filedAt) || 0; + const ta = Date.parse(a.filedAt) || 0; + return tb - ta; + }); + // Cold symbols need a bigger first pass; warm symbols only chase new filings. + const maxDownloads = knownAccessions.size >= 50 ? 40 : 100; + let downloaded = 0; + let skippedKnown = 0; + for (const filing of ordered) { + const accKey = filing.accession ? normAcc(filing.accession) : ''; + if (accKey && knownAccessions.has(accKey)) { + skippedKnown += 1; + continue; + } + if (downloaded >= maxDownloads) break; + try { + const holdingsResult = await edgar.form13f_holdings(filing.cik, filing.accession); + const holdings = (holdingsResult.value as { holdings: ParsedHolding[] }).holdings; + const matching = holdings.filter((h) => cusipsEqual(h.cusip, cusip)); + const n = await storeMatchedHoldings(db, upper, filing, matching); + storedCount += n; + downloaded += 1; + if (accKey) knownAccessions.add(accKey); + } catch { + continue; + } + } + // Stash skip stats on a local for detail below + (filings as EftsFiling[] & { _inc?: { skippedKnown: number; downloaded: number; maxDownloads: number } })._inc = { + skippedKnown, + downloaded, + maxDownloads, + }; + + // EFTS CUSIP full-text often never ranks mega-filers (BlackRock 50k-line 13F + // never appears in the first 500 IREN hits). Always supplement with majors + // missing from the latest quarter so coverage is not permanently incomplete. + try { + const { refreshHoldersViaReverse13f } = await import('./reverse13fRefresh.ts'); + const majorRev = await refreshHoldersViaReverse13f(db, upper, cusip, { + maxFilers: 10, + mode: 'missing_majors', + // Need ≥2 quarters so "vs prior Q" works (not just latest snapshot). + maxQuartersPerFiler: 4, + }); + storedCount += majorRev.rowsWritten; + (filings as EftsFiling[] & { _majorRev?: typeof majorRev })._majorRev = majorRev; + } catch (e) { + console.warn( + '[secDataFetcher] missing_majors supplement failed:', + e instanceof Error ? e.message : e, + ); + } + } else { + // EFTS empty or out — reverse path keeps institutional pipeline automated. + path = eftsOutage ? 'reverse_13f' : 'efts_empty'; + const { refreshHoldersViaReverse13f } = await import('./reverse13fRefresh.ts'); + // Prefer majors first when cold / EFTS down so BlackRock is not buried behind + // tiny prior holders that fill the maxFilers budget. + const rev = await refreshHoldersViaReverse13f(db, upper, cusip, { + maxFilers: 12, + mode: eftsOutage ? 'majors_first' : 'balanced', + maxQuartersPerFiler: 4, + }); + storedCount = rev.rowsWritten; + + // Propagate hard rate-limit from reverse path so the queue cools data.sec.gov + const rateHit = rev.errors.find((m) => + /429|403|too many|rate.?limit|access denied|threshold/i.test(m), + ); + if (rateHit && storedCount === 0) { + throw new Error(rateHit); + } + + if (storedCount === 0 && eftsOutage) { + const existing = ( + db.prepare( + `SELECT COUNT(*) AS n FROM institution_filings WHERE symbol=? AND form='13F-HR'`, + ).get(upper) as { n: number } + ).n; + const maxQ = ( + db.prepare( + `SELECT MAX(reported_quarter) AS q FROM institution_filings WHERE symbol=? AND form='13F-HR'`, + ).get(upper) as { q: string | null } + ).q; + if (existing > 0) { + recordDataQuality(db, { + symbol: upper, + kind: 'institution_filings', + discoveredCount: existing, + storedCount: existing, + missingCount: 0, + backfilled: 0, + stale: isQuarterStale(maxQ) ? 1 : 0, + status: isQuarterStale(maxQ) ? 'error' : 'ok', + detail: { + cusip, + reason: 'efts_outage_reverse_empty_kept_existing', + reverse: rev, + message: eftsOutage.message, + maxQuarter: maxQ, + }, + }); + return 0; + } + // No local data and both paths failed — cool if EFTS outage, else soft error + throw eftsOutage; + } + + if (storedCount === 0 && !eftsOutage) { + recordDataQuality(db, { + symbol: upper, + kind: 'institution_filings', + discoveredCount: 0, + storedCount: 0, + missingCount: 0, + backfilled: 0, + stale: 1, + status: 'error', + detail: { reason: 'no efts hits and reverse empty', cusip, reverse: rev }, + }); + return 0; + } + + // Reverse wrote rows — quality record below + const existingAfter = ( + db.prepare('SELECT COUNT(*) AS n FROM institution_filings WHERE symbol=?').get(upper) as { + n: number; + } + ).n; + const maxQ = ( + db.prepare( + 'SELECT MAX(reported_quarter) AS q FROM institution_filings WHERE symbol=? AND form=?', + ).get(upper, '13F-HR') as { q: string | null } + ).q; + recordDataQuality(db, { + symbol: upper, + kind: 'institution_filings', + discoveredCount: rev.filersWithMatch, + storedCount: existingAfter, + missingCount: 0, + backfilled: storedCount, + stale: isQuarterStale(maxQ) ? 1 : 0, + status: existingAfter > 0 ? 'ok' : 'error', + detail: { + cusip, + path, + reverse: rev, + eftsOutage: eftsOutage?.message ?? null, + }, + }); + return storedCount; + } + + const existing = ( + db.prepare('SELECT COUNT(*) AS n FROM institution_filings WHERE symbol=?').get(upper) as { + n: number; + } + ).n; + const inc = (filings as EftsFiling[] & { + _inc?: { skippedKnown: number; downloaded: number; maxDownloads: number }; + _majorRev?: { filersTried: number; filersWithMatch: number; rowsWritten: number; errors: string[] }; + })._inc; + const majorRev = (filings as EftsFiling[] & { + _majorRev?: { filersTried: number; filersWithMatch: number; rowsWritten: number; errors: string[] }; + })._majorRev; + recordDataQuality(db, { + symbol: upper, + kind: 'institution_filings', + discoveredCount: filings.length, + storedCount: existing, + missingCount: 0, + backfilled: storedCount, + stale: isQuarterStale( + ( + db.prepare( + 'SELECT MAX(reported_quarter) AS q FROM institution_filings WHERE symbol=? AND form=?', + ).get(upper, '13F-HR') as { q: string | null } + ).q, + ) + ? 1 + : 0, + status: storedCount > 0 || existing > 0 ? 'ok' : 'error', + detail: { + cusip, + path: 'efts+missing_majors', + eftsFilings: filings.length, + writtenThisRun: storedCount, + incremental: inc ?? null, + missingMajors: majorRev ?? null, + }, + }); + return storedCount; } @@ -575,7 +1144,7 @@ export async function lintInstitutionalHolders( symbol: string, ): Promise { const upper = symbol.toUpperCase(); - const cusip = await resolveCusip(upper); + const cusip = await resolveCusip(upper, db); if (!cusip) { const r: LintResult = { symbol: upper, kind: 'institution_filings', discoveredCount: 0, storedCount: 0, missingCount: 0, backfilled: 0, stale: 1, status: 'error', detail: { reason: 'cusip not resolved' } }; recordDataQuality(db, r); @@ -818,29 +1387,18 @@ export async function lintSCFilings( symbol: string, ): Promise { const upper = symbol.toUpperCase(); - const companyName = await resolveCompanyName(upper); - if (!companyName) { - const r: LintResult = { symbol: upper, kind: 'sc_filings', discoveredCount: 0, storedCount: 0, missingCount: 0, backfilled: 0, stale: 1, status: 'error', detail: { reason: 'company name not resolved' } }; - recordDataQuality(db, r); - return r; - } - - const discovered = await searchByCompanyName(companyName, ['SCHEDULE 13G', 'SCHEDULE 13D']); - const discoveredAccessions = new Set(discovered.map((f) => f.accession)); + const companyName = await resolveCompanyName(upper) ?? (db.prepare( + 'SELECT name FROM symbols WHERE symbol = ?', + ).get(upper) as { name: string | null } | undefined)?.name ?? upper; const storedCount = (db.prepare('SELECT COUNT(DISTINCT filer_cik) AS cnt FROM institution_filings WHERE symbol = ? AND form IN (?, ?)').get(upper, '13G', '13D') as { cnt: number }).cnt; - let backfilled = 0; - if (discovered.length > storedCount) { - const storedFilerCiks = new Set( - (db.prepare('SELECT DISTINCT filer_cik FROM institution_filings WHERE symbol = ? AND form IN (?, ?)').all(upper, '13G', '13D') as Array<{ filer_cik: string }>).map((r) => r.filer_cik), - ); - const missing = discovered.filter((f) => !storedFilerCiks.has(f.cik)); - backfilled += await fetchAndStoreSCFilings(db, upper); // re-fetch will skip dupes via INSERT OR REPLACE - // More precise: count only the newly stored - const newCount = (db.prepare('SELECT COUNT(DISTINCT filer_cik) AS cnt FROM institution_filings WHERE symbol = ? AND form IN (?, ?)').get(upper, '13G', '13D') as { cnt: number }).cnt; - backfilled = newCount - storedCount; - } + // Always run the fetch — the issuer-CIK submissions feed is the reliable + // discovery path (EFTS full-text search misses many SC 13G/13D filings). + const before = storedCount; + const stored = await fetchAndStoreSCFilings(db, upper); // idempotent via INSERT OR REPLACE + const newCount = (db.prepare('SELECT COUNT(DISTINCT filer_cik) AS cnt FROM institution_filings WHERE symbol = ? AND form IN (?, ?)').get(upper, '13G', '13D') as { cnt: number }).cnt; + const backfilled = newCount - before; const latest = (db.prepare('SELECT MAX(reported_quarter) AS q FROM institution_filings WHERE symbol = ? AND form IN (?, ?)').get(upper, '13G', '13D') as { q: string | null }).q; const stale = isQuarterStale(latest, 4) ? 1 : 0; // SC filings filed within 45 days of year end — 4 quarters threshold @@ -848,13 +1406,13 @@ export async function lintSCFilings( const result: LintResult = { symbol: upper, kind: 'sc_filings', - discoveredCount: discoveredAccessions.size, - storedCount: storedCount + backfilled, + discoveredCount: newCount, + storedCount: newCount, missingCount: 0, backfilled, stale, status, - detail: { totalDiscovered: discovered.length, backfilled }, + detail: { stored, backfilled, companyName }, }; recordDataQuality(db, result); return result; @@ -870,18 +1428,65 @@ export async function lintSCFilings( * - SC 13G/13D: Company-name EFTS search → parse → store beneficial ownership * - Form 4: Company CIK → insider transactions */ +export type SecFetchResult = { + filingsStored: number; + scStored: number; + transactionsStored: number; + cik: string | null; + cusipResolved: boolean; + /** Non-empty → adapter should throw so the queue retries (not silent done). */ + hardErrors: string[]; +}; + +/** + * Fetch SEC data for a symbol. Hard errors (no CUSIP for 13F path on equities with + * no existing holders) are listed so the queue can retry instead of marking done. + */ export async function fetchAndStoreSecData( db: DatabaseSync, symbol: string, -): Promise<{ filingsStored: number; scStored: number; transactionsStored: number; cik: string | null }> { +): Promise { const upper = symbol.toUpperCase(); - const cik = await resolveCik(upper); + const hardErrors: string[] = []; + // Seed curated CUSIP early so 13F path can run even when EFTS is blocked. + resolveCusipLocal(db, upper); - const [filingsStored, scStored, transactionsStored] = await Promise.all([ - fetchAndStoreInstitutionalHolders(db, upper), - fetchAndStoreSCFilings(db, upper), - cik ? fetchAndStoreForm4(db, upper, cik) : Promise.resolve(0), - ]); + const cik = await resolveCik(upper, db); + if (!cik) { + recordDataQuality(db, { + symbol: upper, + kind: 'insider_transactions', + discoveredCount: 0, + storedCount: 0, + missingCount: 0, + backfilled: 0, + stale: 1, + status: 'error', + detail: { reason: 'cik not resolved' }, + }); + hardErrors.push('cik not resolved'); + } - return { filingsStored, scStored, transactionsStored, cik }; + // SC first so issuerCusip can seed the CUSIP cache before 13F CUSIP search. + // Rate-limit / EFTS outage errors propagate so AdapterQueue cools the source. + const scStored = await fetchAndStoreSCFilings(db, upper); + const filingsStored = await fetchAndStoreInstitutionalHolders(db, upper); + const transactionsStored = cik ? await fetchAndStoreForm4(db, upper, cik) : 0; + + const cusipResolved = !!readCachedCusip(db, upper); + + // Alerts need ongoing 13F refresh. No CUSIP = blocked pipeline → hard fail so + // the queue retries. Never silent-done. (Curated map covers demand mega-caps.) + if (!cusipResolved) { + hardErrors.push('cusip not resolved - 13F holder refresh blocked'); + } + + return { + filingsStored, + scStored, + transactionsStored, + cik, + cusipResolved, + hardErrors, + }; } diff --git a/app/server/src/services/secHttp.ts b/app/server/src/services/secHttp.ts new file mode 100644 index 0000000..6eaa9bf --- /dev/null +++ b/app/server/src/services/secHttp.ts @@ -0,0 +1,129 @@ +// SEC fair-access HTTP client — thin wrapper over vendorGate('sec'). +// +// EVERY outbound call to sec.gov / data.sec.gov / efts.sec.gov must go through +// this module (or vendorGate('sec') directly). User-Agent always identifies +// the operator (SEC fair-access policy). + +import { + clearVendorRateLimit, + getVendorCooldownState, + isVendorCoolingDown, + noteVendorRateLimit, + resetVendorGateForTests, + vendorCooldownRemainingMs, + vendorFetch, + VendorRateLimitError, + type VendorFetchOptions, +} from './vendorGate.ts'; + +export class SecRateLimitError extends VendorRateLimitError { + constructor(status: number, url: string, cooldownMs: number) { + super( + 'sec', + `SEC rate limit HTTP ${status} for ${url} - cool down ${Math.round(cooldownMs / 1000)}s, do not thrash`, + cooldownMs, + status, + ); + this.name = 'SecRateLimitError'; + } +} + +const OPERATOR_EMAIL = process.env.SEC_OPERATOR_EMAIL ?? 'research@example.com'; +export const SEC_USER_AGENT = `Investor Flow (${OPERATOR_EMAIL})`; + +/** Steady-state min gap (overridable via IFLOW_SEC_MIN_INTERVAL_MS). */ +export const SEC_MIN_INTERVAL_MS = Number(process.env.IFLOW_SEC_MIN_INTERVAL_MS ?? 350); + +const SEC_HOST_RE = /https?:\/\/([^/]*\.)?sec\.gov\b/i; + +export function isSecHttpCoolingDown(now = Date.now()): boolean { + return isVendorCoolingDown('sec', now); +} + +export function secHttpCooldownRemainingMs(now = Date.now()): number { + return vendorCooldownRemainingMs('sec', now); +} + +export function getSecHttpCooldownState(now = Date.now()) { + return getVendorCooldownState('sec', now); +} + +export function resetSecHttpStateForTests(): void { + resetVendorGateForTests(); +} + +export function noteSecRateLimitHit(now = Date.now()): number { + return noteVendorRateLimit('sec', now); +} + +export function clearSecRateLimitHits(): void { + clearVendorRateLimit('sec'); +} + +export type SecFetchOptions = Omit & { + method?: string; + headers?: Record; + accept?: string; + retries?: number; +}; + +/** + * Rate-limited fetch for SEC hosts. Serialized process-wide via vendorGate('sec'). + */ +export async function secFetch(url: string, opts: SecFetchOptions = {}): Promise { + if (!SEC_HOST_RE.test(url)) { + throw new Error(`secFetch: refusing non-SEC URL ${url}`); + } + + try { + return await vendorFetch('sec', url, { + method: opts.method, + headers: { + 'User-Agent': SEC_USER_AGENT, + ...(opts.headers ?? {}), + }, + accept: opts.accept ?? 'application/json', + retries: opts.retries ?? 1, + hostAllowlist: SEC_HOST_RE, + signal: opts.signal, + }); + } catch (e) { + if (e instanceof VendorRateLimitError) { + throw new SecRateLimitError(e.status ?? 429, url, e.cooldownMs); + } + throw e; + } +} + +export async function secFetchJson(url: string, opts?: SecFetchOptions): Promise { + const resp = await secFetch(url, { ...opts, accept: opts?.accept ?? 'application/json' }); + if (!resp.ok) { + throw new Error(`SEC HTTP ${resp.status} ${resp.statusText} for ${url}`); + } + return (await resp.json()) as T; +} + +export async function secFetchText(url: string, opts?: SecFetchOptions): Promise { + const resp = await secFetch(url, { + ...opts, + accept: opts?.accept ?? 'application/xml, text/xml, text/html, */*', + }); + if (!resp.ok) { + throw new Error(`SEC HTTP ${resp.status} ${resp.statusText} for ${url}`); + } + return resp.text(); +} + +/** Source kinds that share the SEC rate budget / cool-down family. */ +export const SEC_SOURCE_FAMILY = [ + 'sec', + 'sec-fetch', + 'sec-sc-fetch', + 'sec-tickers', + 'sec-lint-holders', + 'sec-lint-insiders', +] as const; + +export function isSecSourceFamily(source: string): boolean { + return (SEC_SOURCE_FAMILY as readonly string[]).includes(source) || source.startsWith('sec'); +} diff --git a/app/server/src/services/secTickersIngest.ts b/app/server/src/services/secTickersIngest.ts new file mode 100644 index 0000000..c451986 --- /dev/null +++ b/app/server/src/services/secTickersIngest.ts @@ -0,0 +1,75 @@ +// Investor Flow — SEC company_tickers.json materializer (M22 Symbol Search Index). +// +// Pure merge logic: the SEC bulk file is the authoritative source of issuer identity +// (cik, name, exchange) but NOT of classification (ticker_kind) or descriptive metadata +// (sector/industry/peers). Per ADR-0011's Symbol-Metadata Merge Policy: +// - cik: always overwritten (SEC is the only source of issuer CIK) +// - name/exchange: filled only when absent (don't clobber richer yfinance hydration) +// - ticker_kind: never downgraded on conflict (etf/crypto/index preserved — kind gates module applicability) +// - sector/industry/peers: never touched (absent from the SEC file) +// - rows absent from the SEC file are never purged (SEC universe is a subset, not the whole) +import type { DatabaseSync } from 'node:sqlite'; + +export interface CompanyTickerRow { + cik_str: number; + ticker: string; + title: string; +} + +export interface SecTickersIngestResult { + upserted: number; // rows matched + merged (any change incl. cik fill) + created: number; // rows newly inserted + skipped: number; // rows with no ticker (defensive) +} + +/** Merge the SEC ticker file into the symbols table. Pure DB logic, no network. */ +export function mergeCompanyTickers( + db: DatabaseSync, + tickers: Record | CompanyTickerRow[], +): SecTickersIngestResult { + const insert = db.prepare(` + INSERT INTO symbols (symbol, name, exchange, ticker_kind, cik, updated_at) + VALUES (?, ?, NULL, 'equity', ?, ?) + ON CONFLICT(symbol) DO UPDATE SET + cik = excluded.cik, + name = CASE WHEN symbols.name IS NULL OR symbols.name = '' THEN excluded.name ELSE symbols.name END, + updated_at = excluded.updated_at + `); + + const rows = Array.isArray(tickers) + ? tickers + : Object.values(tickers as Record); + + let upserted = 0; + let created = 0; + let skipped = 0; + const now = new Date().toISOString(); + + db.exec('BEGIN'); + try { + for (const r of rows) { + const symbol = (r.ticker ?? '').trim().toUpperCase(); + const name = (r.title ?? '').trim(); + const cik = String(r.cik_str).padStart(10, '0'); + if (!symbol) { skipped++; continue; } + + const existing = db.prepare('SELECT symbol, cik FROM symbols WHERE symbol = ?').get(symbol) as + | { symbol: string; cik: string | null } + | undefined; + + insert.run(symbol, name, cik, now); + + if (existing) { + upserted++; + } else { + created++; + } + } + db.exec('COMMIT'); + } catch (e) { + db.exec('ROLLBACK'); + throw e; + } + + return { upserted, created, skipped }; +} diff --git a/app/server/src/services/stockFloatService.ts b/app/server/src/services/stockFloatService.ts new file mode 100644 index 0000000..8956d4b --- /dev/null +++ b/app/server/src/services/stockFloatService.ts @@ -0,0 +1,175 @@ +// Investor Flow — stock float persistence service. +// +// Fetches sharesOutstanding / floatShares from Yahoo Finance (defaultKeyStatistics), +// persists to `stock_float` table, and provides ownership % computation helpers. +// +// Design: +// - Float/outstanding are time-varying market data, not static symbol metadata. +// Separate table so it doesn't pollute the symbols schema and supports history. +// - Co-located with existing short-interest pipeline (same YF module). +// - Rate-limited via vendorGate('yfinance') — never bare HTTP. + +import type { DatabaseSync } from 'node:sqlite'; +import { withVendorGate } from './vendorGate.ts'; + +/** TTL for float data before re-fetching (24h default). */ +const FLOAT_TTL_MS = 24 * 60 * 60 * 1000; + +interface YFKeyStats { + sharesOutstanding?: unknown; + floatShares?: unknown; +} + +/** Parse a numeric value from Yahoo's response (handles strings with commas, nulls). */ +function parseNum(v: unknown): number | null { + if (v === null || v === undefined) return null; + if (typeof v === 'number') return Number.isFinite(v) ? v : null; + const s = String(v).replace(/[,]/g, ''); + const n = parseFloat(s); + return Number.isNaN(n) ? null : n; +} + +interface YFInstance { + quoteSummary(symbol: string, opts: { modules: string[] }): Promise>; +} + +/** Lazy-loaded yahoo-finance2 instance. */ +let cachedYf: YFInstance | null = null; + +async function getYf(): Promise { + if (!cachedYf) { + const mod = await import('yahoo-finance2'); + cachedYf = new mod.default({ suppressNotices: ['yahooSurvey'] }) as unknown as YFInstance; + } + return cachedYf!; +} + +/** Fetch float data from Yahoo Finance and persist to stock_float table. */ +export async function fetchAndPersistFloat(db: DatabaseSync, symbol: string): Promise<{ + sharesOutstanding: number | null; + floatShares: number | null; + fetchedAt: string; +}> { + const yf = await getYf(); + + const result = await withVendorGate('yfinance', async () => { + return (await yf.quoteSummary(symbol.toUpperCase(), { + modules: ['defaultKeyStatistics'], + })) as unknown as Record; + }); + + const stats = (result?.defaultKeyStatistics ?? {}) as YFKeyStats; + const outstanding = parseNum(stats.sharesOutstanding); + const floatShares = parseNum(stats.floatShares); + const fetchedAt = new Date().toISOString(); + + // Upsert into stock_float — use today's date as the key for latest snapshot. + const today = new Date().toISOString().slice(0, 10); + db.prepare(` + INSERT INTO stock_float (symbol, shares_outstanding, float_shares, as_of) + VALUES (?, ?, ?, ?) + ON CONFLICT(symbol, as_of) DO UPDATE SET + shares_outstanding = excluded.shares_outstanding, + float_shares = excluded.float_shares, + as_of = excluded.as_of + `).run(symbol.toUpperCase(), outstanding, floatShares, fetchedAt); + + return { sharesOutstanding: outstanding, floatShares: floatShares, fetchedAt }; +} + +/** Options for refreshAllStockFloats. */ +export interface RefreshFloatOptions { + /** Force refresh all symbols regardless of TTL. Default: false (skip fresh entries). */ + forceRefresh?: boolean; +} + +/** Refresh float data for all symbols that are stale (>24h) or missing. */ +export async function refreshAllStockFloats(db: DatabaseSync, opts: RefreshFloatOptions = {}): Promise<{ refreshed: number; skipped: number; errors: string[] }> { + const today = new Date().toISOString(); + const cutoff = opts.forceRefresh ? '1970-01-01T00:00:00.000Z' : new Date(Date.now() - FLOAT_TTL_MS).toISOString(); + + // Get all unique symbols from fund_position_records (tracked funds) and institution_filings. + const symbols: string[] = []; + const seen = new Set(); + + const trackedRows = db.prepare(`SELECT DISTINCT symbol FROM fund_position_records WHERE source IN ('13f', 'capture')`).all() as Array<{ symbol: string }>; + for (const r of trackedRows) { + if (!seen.has(r.symbol)) { seen.add(r.symbol); symbols.push(r.symbol); } + } + + const filingRows = db.prepare(`SELECT DISTINCT symbol FROM institution_filings WHERE shares IS NOT NULL`).all() as Array<{ symbol: string }>; + for (const r of filingRows) { + if (!seen.has(r.symbol)) { seen.add(r.symbol); symbols.push(r.symbol); } + } + + let refreshed = 0; + let skipped = 0; + const errors: string[] = []; + + for (const symbol of symbols) { + try { + // Check if we have fresh data. + const existing = db.prepare( + `SELECT as_of FROM stock_float WHERE symbol = ? ORDER BY as_of DESC LIMIT 1` + ).get(symbol) as { as_of: string } | undefined; + + if (existing && existing.as_of >= cutoff) { + skipped++; + continue; + } + + await fetchAndPersistFloat(db, symbol); + refreshed++; + } catch (err) { + errors.push(`${symbol}: ${(err as Error).message}`); + } + } + + return { refreshed, skipped, errors }; +} + +/** Get the latest float snapshot for a symbol. */ +export function getLatestFloat(db: DatabaseSync, symbol: string): { sharesOutstanding: number | null; floatShares: number | null; asOf: string } | null { + const row = db.prepare( + `SELECT shares_outstanding, float_shares, as_of FROM stock_float WHERE symbol = ? ORDER BY as_of DESC LIMIT 1` + ).get(symbol) as { shares_outstanding: number | null; float_shares: number | null; as_of: string } | undefined; + + if (!row) return null; + return { + sharesOutstanding: row.shares_outstanding, + floatShares: row.float_shares, + asOf: row.as_of, + }; +} + +/** Compute institutional ownership % for a symbol across all tracked funds. */ +export function computeOwnershipPercentages(db: DatabaseSync, symbol: string): { + heldShares: number; + outstanding: number | null; + float: number | null; + pctOutstanding: number | null; + pctFloat: number | null; +} { + // Sum all tracked-fund shares for this symbol (latest record per fund). + const totalHeld = db.prepare(` + SELECT COALESCE(SUM(sub.shares), 0) AS held + FROM ( + SELECT fpr.shares, MAX(fpr.as_of) AS max_as_of + FROM fund_position_records fpr + JOIN tracked_funds tf ON tf.id = fpr.fund_id AND tf.enabled = 1 + WHERE fpr.symbol = ? AND fpr.source IN ('13f', 'capture') + GROUP BY fpr.fund_id + ) sub + `).get(symbol) as { held: number } | undefined; + + const heldShares = totalHeld?.held ?? 0; + const latestFloat = getLatestFloat(db, symbol); + + return { + heldShares, + outstanding: latestFloat?.sharesOutstanding ?? null, + float: latestFloat?.floatShares ?? null, + pctOutstanding: latestFloat?.sharesOutstanding ? heldShares / latestFloat.sharesOutstanding : null, + pctFloat: latestFloat?.floatShares ? heldShares / latestFloat.floatShares : null, + }; +} diff --git a/app/server/src/services/vendorGate.ts b/app/server/src/services/vendorGate.ts new file mode 100644 index 0000000..39901d0 --- /dev/null +++ b/app/server/src/services/vendorGate.ts @@ -0,0 +1,487 @@ +// Process-wide vendor rate gate (ADR-0009). +// +// CONTRACT FOR EVERY EXTERNAL INTEGRATION (existing + future): +// +// 1. Register a family before any traffic: +// registerVendorFamily('polygon', { minIntervalMs: 200, maxInflight: 1, drainJobBudget: 2 }) +// bindSourceKind('polygon', 'polygon') +// 2. All outbound work goes through: +// withVendorGate('polygon', () => client.get(...)) +// vendorFetch('polygon', url, { hostAllowlist: /polygon\.io/i }) +// or extend VendorSourceAdapter (auto-gates fetchOne). +// 3. Never use bare fetch()/SDK calls from adapters or services. +// 4. AdapterQueue refuses source_kinds with no family binding at construction. +// +// Guard test: src/services/__tests__/vendorHttpGuard.test.ts fails CI if bare +// fetch sneaks into adapters/services (except this module + allowlisted paths). + +import { isRateLimitError, rateLimitCooldownMs } from '../queue/sourceRatePolicy.ts'; + +/** Open string type so future vendors register without editing a union. */ +export type VendorFamily = string; + +export class VendorRateLimitError extends Error { + readonly family: VendorFamily; + readonly status: number | null; + readonly cooldownMs: number; + + constructor(family: VendorFamily, message: string, cooldownMs: number, status: number | null = null) { + super(message); + this.name = 'VendorRateLimitError'; + this.family = family; + this.cooldownMs = cooldownMs; + this.status = status; + } +} + +export class UnknownVendorFamilyError extends Error { + constructor(message: string) { + super(message); + this.name = 'UnknownVendorFamilyError'; + } +} + +interface FamilyState { + cooldownUntil: number; + consecutiveHits: number; + lastEndedAt: number; + /** Waiters blocked waiting for an inflight slot (maxInflight gate). */ + waiters: Array<() => void>; + inflight: number; +} + +export interface FamilyPolicy { + /** Min ms between completed calls in this family. */ + minIntervalMs: number; + /** Max concurrent calls (1 = strict single-flight). */ + maxInflight: number; + /** Max jobs of this family per AdapterQueue.drain() cycle. */ + drainJobBudget: number; + /** Optional host allowlist regex source for docs / vendorFetch defaults. */ + hostPattern?: string; +} + +const DEFAULT_POLICY: FamilyPolicy = { + minIntervalMs: 500, + maxInflight: 1, + drainJobBudget: 1, +}; + +/** Registered family policies (built-ins + future registerVendorFamily calls). */ +const POLICIES = new Map(); + +/** Queue source_kind → vendor family. */ +const SOURCE_TO_FAMILY = new Map(); + +/** + * Register (or replace) a vendor family policy. + * Call this before bindSourceKind / starting traffic for a new integration. + */ +export function registerVendorFamily(family: VendorFamily, policy: Partial = {}): void { + if (!family || !/^[a-z][a-z0-9_-]*$/i.test(family)) { + throw new UnknownVendorFamilyError( + `registerVendorFamily: invalid family name '${family}' (use [a-z0-9_-]+)`, + ); + } + const prev = POLICIES.get(family); + POLICIES.set(family, { + minIntervalMs: policy.minIntervalMs ?? prev?.minIntervalMs ?? DEFAULT_POLICY.minIntervalMs, + maxInflight: policy.maxInflight ?? prev?.maxInflight ?? DEFAULT_POLICY.maxInflight, + drainJobBudget: policy.drainJobBudget ?? prev?.drainJobBudget ?? DEFAULT_POLICY.drainJobBudget, + hostPattern: policy.hostPattern ?? prev?.hostPattern, + }); +} + +/** + * Bind a queue source_kind to a registered family. + * Required for AdapterQueue to accept adapters of that kind. + */ +export function bindSourceKind(sourceKind: string, family: VendorFamily): void { + if (!POLICIES.has(family)) { + throw new UnknownVendorFamilyError( + `bindSourceKind('${sourceKind}', '${family}'): family not registered. Call registerVendorFamily('${family}', …) first.`, + ); + } + SOURCE_TO_FAMILY.set(sourceKind, family); +} + +/** Register family + bind one or more source kinds in one call (preferred for new vendors). */ +export function registerVendorIntegration(opts: { + family: VendorFamily; + sourceKinds: string[]; + policy?: Partial; +}): void { + registerVendorFamily(opts.family, opts.policy ?? {}); + for (const sk of opts.sourceKinds) { + bindSourceKind(sk, opts.family); + } +} + +/** True if family was registered (built-in or runtime). */ +export function isVendorFamilyRegistered(family: string): boolean { + return POLICIES.has(family); +} + +/** True if this source_kind has a family binding. */ +export function isSourceBound(sourceKind: string): boolean { + return sourceToFamily(sourceKind) != null; +} + +/** + * Assert every source_kind is bound to a family. + * AdapterQueue calls this on construction so unregistered vendors fail fast. + */ +export function assertSourcesBound(sourceKinds: Iterable): void { + const missing: string[] = []; + for (const sk of sourceKinds) { + if (!sourceToFamily(sk)) missing.push(sk); + } + if (missing.length) { + throw new UnknownVendorFamilyError( + `Adapter source_kind(s) have no vendor family: ${missing.join(', ')}. ` + + `Register with registerVendorIntegration({ family, sourceKinds, policy }) ` + + `before constructing AdapterQueue. See docs/VENDOR_INTEGRATIONS.md.`, + ); + } +} + +function seedBuiltIns(): void { + if (POLICIES.size > 0) return; + + registerVendorIntegration({ + family: 'yfinance', + sourceKinds: ['yfinance', 'yfinance-quote', 'yfinance-eod', 'yfinance-meta', 'yfinance-holdings'], + policy: { minIntervalMs: 400, maxInflight: 1, drainJobBudget: 3, hostPattern: 'yahoo|finance\\.yahoo' }, + }); + registerVendorIntegration({ + family: 'sec', + sourceKinds: ['sec', 'sec-fetch', 'sec-sc-fetch', 'sec-tickers', 'sec-lint-holders', 'sec-lint-insiders'], + // SEC fair access allows ~10 req/s; concurrent fetches are paced by the + // family min-interval + per-source spacing, so 2 inflight is safe and + // clears long-lived backlogs ~4x faster than strict single-flight. + policy: { minIntervalMs: 350, maxInflight: 2, drainJobBudget: 2, hostPattern: 'sec\\.gov' }, + }); + registerVendorIntegration({ + family: 'fred', + sourceKinds: ['fred', 'macro'], + policy: { minIntervalMs: 500, maxInflight: 1, drainJobBudget: 1, hostPattern: 'stlouisfed\\.org' }, + }); + registerVendorIntegration({ + family: 'finra', + sourceKinds: ['finra-bulk', 'finra-si'], + policy: { minIntervalMs: 1000, maxInflight: 1, drainJobBudget: 1, hostPattern: 'finra\\.org' }, + }); + registerVendorIntegration({ + family: 'nasdaq', + sourceKinds: ['nasdaq'], + policy: { minIntervalMs: 800, maxInflight: 1, drainJobBudget: 1, hostPattern: 'nasdaq\\.com' }, + }); + registerVendorIntegration({ + family: 'reddit', + sourceKinds: ['reddit'], + policy: { minIntervalMs: 1500, maxInflight: 1, drainJobBudget: 1, hostPattern: 'reddit\\.com' }, + }); + registerVendorIntegration({ + family: 'x', + sourceKinds: ['x'], + policy: { minIntervalMs: 3000, maxInflight: 1, drainJobBudget: 1 }, + }); + registerVendorIntegration({ + family: 'llm', + sourceKinds: ['llm'], + policy: { minIntervalMs: 0, maxInflight: 2, drainJobBudget: 2 }, + }); +} + +// Seed on module load so existing adapters work without ceremony. +seedBuiltIns(); + +/** All queue source kinds belonging to a family (for family-wide cool-down). */ +export function sourcesForFamily(family: VendorFamily): string[] { + const out: string[] = []; + for (const [sk, f] of SOURCE_TO_FAMILY) { + if (f === family) out.push(sk); + } + return out; +} + +export function sourceToFamily(source: string): VendorFamily | null { + const direct = SOURCE_TO_FAMILY.get(source); + if (direct) return direct; + // Prefix fallback only for registered families (future yfinance-*, sec-*) + if (source.startsWith('yfinance') && POLICIES.has('yfinance')) return 'yfinance'; + if (source.startsWith('sec') && POLICIES.has('sec')) return 'sec'; + if (source.startsWith('finra') && POLICIES.has('finra')) return 'finra'; + return null; +} + +/** + * Resolve family or throw. Prefer this for new code paths so silent + * ungated traffic cannot appear for unregistered vendors. + */ +export function requireSourceFamily(source: string): VendorFamily { + const f = sourceToFamily(source); + if (!f) { + throw new UnknownVendorFamilyError( + `No vendor family for source_kind '${source}'. ` + + `Call registerVendorIntegration({ family, sourceKinds: ['${source}'], policy }) first.`, + ); + } + return f; +} + +export function familyPolicy(family: VendorFamily): FamilyPolicy { + const p = POLICIES.get(family); + if (!p) { + throw new UnknownVendorFamilyError( + `Unknown vendor family '${family}'. Call registerVendorFamily('${family}', …) first.`, + ); + } + return p; +} + +export function familyDrainBudget(family: VendorFamily): number { + return familyPolicy(family).drainJobBudget; +} + +export function listRegisteredFamilies(): VendorFamily[] { + return [...POLICIES.keys()].sort(); +} + +const states = new Map(); + +function state(family: VendorFamily): FamilyState { + let s = states.get(family); + if (!s) { + s = { + cooldownUntil: 0, + consecutiveHits: 0, + lastEndedAt: 0, + waiters: [], + inflight: 0, + }; + states.set(family, s); + } + return s; +} + +export function isVendorCoolingDown(family: VendorFamily, now = Date.now()): boolean { + return now < state(family).cooldownUntil; +} + +export function vendorCooldownRemainingMs(family: VendorFamily, now = Date.now()): number { + return Math.max(0, state(family).cooldownUntil - now); +} + +export function getVendorCooldownState( + family: VendorFamily, + now = Date.now(), +): { + family: VendorFamily; + active: boolean; + remainingMs: number; + consecutiveHits: number; + until: string | null; +} { + const s = state(family); + const remainingMs = Math.max(0, s.cooldownUntil - now); + return { + family, + active: remainingMs > 0, + remainingMs, + consecutiveHits: s.consecutiveHits, + until: remainingMs > 0 ? new Date(s.cooldownUntil).toISOString() : null, + }; +} + +/** Record a rate-limit hit; returns cool-down duration applied (ms). */ +export function noteVendorRateLimit(family: VendorFamily, now = Date.now()): number { + // Ensure family exists so ad-hoc note doesn't create zombie state + if (!POLICIES.has(family)) { + registerVendorFamily(family, DEFAULT_POLICY); + } + const s = state(family); + s.consecutiveHits += 1; + const ms = rateLimitCooldownMs(s.consecutiveHits); + s.cooldownUntil = Math.max(s.cooldownUntil, now + ms); + return ms; +} + +export function clearVendorRateLimit(family: VendorFamily): void { + const s = state(family); + s.consecutiveHits = 0; + s.cooldownUntil = 0; +} + +/** + * Test helper: wipe runtime state (cool-downs, chains). + * Does NOT unregister built-in families (re-seed if maps were cleared). + */ +export function resetVendorGateForTests(): void { + states.clear(); + // If a test called registerVendorFamily for ad-hoc names, leave policies; + // always re-seed built-ins so bindings exist. + if (!POLICIES.has('yfinance')) { + POLICIES.clear(); + SOURCE_TO_FAMILY.clear(); + seedBuiltIns(); + } +} + +/** Full reset including custom registrations (unit tests only). */ +export function resetVendorRegistryForTests(): void { + states.clear(); + POLICIES.clear(); + SOURCE_TO_FAMILY.clear(); + seedBuiltIns(); +} + +function envMinInterval(family: VendorFamily): number { + const key = `IFLOW_${family.toUpperCase().replace(/[^A-Z0-9]/g, '_')}_MIN_INTERVAL_MS`; + const raw = process.env[key]; + if (raw && Number.isFinite(Number(raw))) return Math.max(0, Number(raw)); + return familyPolicy(family).minIntervalMs; +} + +async function waitTurn(family: VendorFamily): Promise { + const s = state(family); + const now = Date.now(); + if (now < s.cooldownUntil) { + const remaining = s.cooldownUntil - now; + throw new VendorRateLimitError( + family, + `${family} rate limit preflight cool-down ${Math.round(remaining / 1000)}s - do not thrash`, + remaining, + ); + } + const minGap = envMinInterval(family); + const sinceLast = now - s.lastEndedAt; + if (s.lastEndedAt > 0 && sinceLast < minGap) { + await new Promise((r) => setTimeout(r, minGap - sinceLast)); + } +} + +/** + * Run an async vendor call under the family gate (pace + inflight-cap + + * cool-down). Up to `maxInflight` calls run concurrently; starts are spaced by + * `minIntervalMs`. On thrown errors that look like rate limits, notes cool-down + * then rethrows. + */ +export async function withVendorGate(family: VendorFamily, fn: () => Promise): Promise { + if (!POLICIES.has(family)) { + throw new UnknownVendorFamilyError( + `withVendorGate('${family}'): family not registered. Call registerVendorFamily first.`, + ); + } + const s = state(family); + const policy = familyPolicy(family); + + await acquireSlot(s, policy.maxInflight); + try { + await waitTurn(family); + const result = await fn(); + s.consecutiveHits = 0; + return result; + } catch (e) { + if (e instanceof VendorRateLimitError) throw e; + const msg = e instanceof Error ? e.message : String(e); + if (isRateLimitError(msg) || /edge:\s*too many|429|throttl/i.test(msg)) { + const ms = noteVendorRateLimit(family); + throw new VendorRateLimitError( + family, + `${family} rate limit: ${msg} - cool down ${Math.round(ms / 1000)}s, do not thrash`, + ms, + ); + } + throw e; + } finally { + s.inflight = Math.max(0, s.inflight - 1); + s.lastEndedAt = Date.now(); + const next = s.waiters.shift(); + if (next) next(); + } +} + +/** + * Block until fewer than `maxInflight` calls are in flight for the family. + * The released waiter takes its slot synchronously inside `next()` (called by + * the releaser's finally), so inflight never dips below the cap between + * release and resume. + */ +function acquireSlot(s: FamilyState, maxInflight: number): Promise { + if (s.inflight < maxInflight) { + s.inflight += 1; + return Promise.resolve(); + } + return new Promise((resolve) => { + s.waiters.push(() => { + s.inflight += 1; + resolve(); + }); + }); +} + +export interface VendorFetchOptions { + method?: string; + headers?: Record; + accept?: string; + retries?: number; + signal?: AbortSignal; + /** Required for safety — callers must declare allowed hosts. */ + hostAllowlist?: RegExp; +} + +function isRateLimitStatus(status: number): boolean { + return status === 429 || status === 503 || status === 403; +} + +/** + * Rate-limited fetch for a vendor family. + * Prefer hostAllowlist so callers cannot mis-route. + */ +export async function vendorFetch( + family: VendorFamily, + url: string, + opts: VendorFetchOptions = {}, +): Promise { + if (opts.hostAllowlist && !opts.hostAllowlist.test(url)) { + throw new Error(`vendorFetch(${family}): URL not allowed: ${url}`); + } + + const retries = opts.retries ?? 0; + + return withVendorGate(family, async () => { + for (let attempt = 0; attempt <= retries; attempt++) { + const headers: Record = { + ...(opts.accept ? { Accept: opts.accept } : {}), + ...(opts.headers ?? {}), + }; + const resp = await fetch(url, { + method: opts.method ?? 'GET', + headers, + signal: opts.signal, + }); + + if (resp.ok || resp.status === 304) return resp; + + if (isRateLimitStatus(resp.status)) { + if (attempt < retries) { + await new Promise((r) => setTimeout(r, 1000 * (attempt + 1))); + continue; + } + const ms = noteVendorRateLimit(family); + throw new VendorRateLimitError( + family, + `${family} rate limit HTTP ${resp.status} for ${url} - cool down ${Math.round(ms / 1000)}s, do not thrash`, + ms, + resp.status, + ); + } + + return resp; + } + + const ms = noteVendorRateLimit(family); + throw new VendorRateLimitError(family, `${family} rate limit for ${url}`, ms); + }); +} diff --git a/app/server/src/sizing/SizingEngine.ts b/app/server/src/sizing/SizingEngine.ts index 550bf5a..d375a5c 100644 --- a/app/server/src/sizing/SizingEngine.ts +++ b/app/server/src/sizing/SizingEngine.ts @@ -163,7 +163,7 @@ export function sizePosition( } const shares = Math.max(0, Math.floor(sharesAfterMacro)); - explanations.push(`The math implies ~${shares} shares given your stop of ${plan.stopPerShare} and ${(plan.riskFraction * 100).toFixed(1)}% risk. Educational analysis, not investment advice.`); + explanations.push(`The math implies ~${shares} shares given your stop of ${plan.stopPerShare} and ${(plan.riskFraction * 100).toFixed(1)}% risk.`); return { shares, diff --git a/app/server/src/sizing/convictionUnlock.ts b/app/server/src/sizing/convictionUnlock.ts index 0f69910..38b56b3 100644 --- a/app/server/src/sizing/convictionUnlock.ts +++ b/app/server/src/sizing/convictionUnlock.ts @@ -59,7 +59,7 @@ export function computeTierStats(db: DatabaseSync, userId: string): TierStats[] byTier[t].total += 1; // Profitability heuristic: a closed trade with a positive realized gain. // Falls back gracefully if no outcome column exists (counts total only). - if (isProfit(db, userId, r as never)) byTier[t].profitable += 1; + if (isProfit(r as { realized_pnl: number | null })) byTier[t].profitable += 1; } return (['A_STAR', 'A', 'B', 'C'] as const).map((tier) => { const s = byTier[tier] ?? { total: 0, profitable: 0 }; @@ -67,15 +67,9 @@ export function computeTierStats(db: DatabaseSync, userId: string): TierStats[] }); } -/** Deterministic profitability probe — returns false when the outcome column - * is absent (no false positives; unlock stays conservative). */ -function isProfit(_db: DatabaseSync, _userId: string, _row: never): boolean { - // Reserved: once a `realized_pnl`/`outcome` column is added to `trades`, - // inspect it here. Today there is no such column, so we conservatively return - // false — unlocking requires explicit operator/user evidence recorded in - // `sizing_unlocks`. This keeps the gate fail-closed until the journal records - // realized PnL (planned in the alerts/thesis slices). - return false; +/** Deterministic profitability probe — checks realized_pnl on the trade row. */ +function isProfit(row: { realized_pnl: number | null }): boolean { + return (row.realized_pnl ?? 0) > 0; } /** Has the user unlocked A_STAR? Reads `sizing_unlocks` (operator-evidenced). */ diff --git a/app/server/src/strategy/PortfolioBacktestEngine.ts b/app/server/src/strategy/PortfolioBacktestEngine.ts new file mode 100644 index 0000000..ba7176a --- /dev/null +++ b/app/server/src/strategy/PortfolioBacktestEngine.ts @@ -0,0 +1,318 @@ +// Investor Flow — PortfolioBacktestEngine +// +// Multi-symbol backtest that simulates a portfolio with periodic or drift-based +// rebalancing. Reuses the existing candle cache (no I/O). +// +// v1 scope: aligned-date intersection, weight-based allocation, periodic + drift +// rebalance, correlation matrix, max DD. No tax/fee/slippage/leverage. + +import type { PriceCandle } from '../cache/CacheRepository.ts'; + +export interface PortfolioAllocation { + symbol: string; + targetPct: number; // e.g. 70 for 70% +} + +export interface PortfolioBacktestInput { + allocations: PortfolioAllocation[]; + candles: Record; + initialEquity?: number; + rebalanceFreq?: 'none' | 'quarterly' | 'annual'; + driftThreshold?: number; // percentage points from target that triggers rebalance +} + +export interface PortfolioEquityPoint { + date: string; + value: number; +} + +export interface PositionResult { + symbol: string; + targetPct: number; + totalReturn: number; + finalValue: number; +} + +export interface RebalanceEvent { + date: string; + type: 'periodic' | 'drift'; +} + +export interface PortfolioBacktestResult { + equityCurve: PortfolioEquityPoint[]; + totalReturn: number; + cagr: number; + maxDrawdown: number; + positions: PositionResult[]; + rebalanceEvents: RebalanceEvent[]; + correlationMatrix: Record>; + sampleSize: number; +} + +export class PortfolioBacktestEngine { + run(input: PortfolioBacktestInput): PortfolioBacktestResult { + const { + allocations, + candles, + initialEquity = 100000, + rebalanceFreq = 'quarterly', + driftThreshold = 5, + } = input; + + // --- Step 1: Align dates (intersection across all symbols) --- + const dateSets: Set[] = allocations.map((a) => { + const cs = candles[a.symbol]; + if (!cs || cs.length === 0) return new Set(); + return new Set(cs.map((c) => new Date(c.ts).getTime())); + }); + + const alignedTimestamps = intersectSortedDates(dateSets); + if (alignedTimestamps.length < 2) { + return emptyResult(allocations, initialEquity); + } + + // --- Step 2: Build price maps symbol → { timestamp → price } --- + const priceMap: Record> = {}; + for (const a of allocations) { + const map = new Map(); + for (const c of candles[a.symbol] ?? []) { + map.set(new Date(c.ts).getTime(), c.c); + } + priceMap[a.symbol] = map; + } + + // --- Step 3: Run simulation --- + const TRADING_DAYS_PER_YEAR = 252; + const freqDays: Record = { + none: Infinity, + quarterly: 63, + annual: 252, + }; + const rebalanceInterval = freqDays[rebalanceFreq] ?? Infinity; + + // State + const positions = allocations.map((a) => ({ + symbol: a.symbol, + targetPct: a.targetPct, + shares: (initialEquity * (a.targetPct / 100)) / (priceMap[a.symbol]?.get(alignedTimestamps[0]) ?? 1), + entryPrice: priceMap[a.symbol]?.get(alignedTimestamps[0]) ?? 0, + })); + + const equityCurve: PortfolioEquityPoint[] = []; + const rebalanceEvents: RebalanceEvent[] = []; + let peakEquity = initialEquity; + let lastRebalanceIdx = 0; + + for (let idx = 0; idx < alignedTimestamps.length; idx++) { + const ts = alignedTimestamps[idx]; + + // Mark positions to market + let portfolioValue = 0; + for (const p of positions) { + const price = priceMap[p.symbol]?.get(ts) ?? 0; + portfolioValue += p.shares * price; + } + + equityCurve.push({ date: new Date(ts).toISOString().slice(0, 10), value: Math.round(portfolioValue * 100) / 100 }); + peakEquity = Math.max(peakEquity, portfolioValue); + + // Check rebalance + if (idx < alignedTimestamps.length - 1 && portfolioValue > 0) { + const rebType = checkRebalance( + idx, lastRebalanceIdx, rebalanceInterval, driftThreshold, + positions, priceMap, ts, portfolioValue, allocations + ); + + if (rebType) { + // Redistribute to target weights + for (const p of positions) { + const alloc = allocations.find((a) => a.symbol === p.symbol); + if (!alloc) continue; + const price = priceMap[p.symbol]?.get(ts) ?? 1; + if (price <= 0) continue; + p.shares = (portfolioValue * (alloc.targetPct / 100)) / price; + } + lastRebalanceIdx = idx; + rebalanceEvents.push({ + date: new Date(ts).toISOString().slice(0, 10), + type: rebType, + }); + } + } + } + + // --- Step 4: Compute metrics --- + const finalValue = equityCurve[equityCurve.length - 1]?.value ?? initialEquity; + const totalReturn = (finalValue - initialEquity) / initialEquity; + + const totalDays = alignedTimestamps.length; + const years = totalDays / TRADING_DAYS_PER_YEAR; + const cagr = years > 0 ? Math.pow(finalValue / initialEquity, 1 / years) - 1 : 0; + + const maxDrawdown = computeMaxDrawdown(equityCurve); + + // Per-position returns + const positionResults: PositionResult[] = positions.map((p, i) => { + const firstPrice = priceMap[p.symbol]?.get(alignedTimestamps[0]) ?? 1; + const lastPrice = priceMap[p.symbol]?.get(alignedTimestamps[alignedTimestamps.length - 1]) ?? firstPrice; + return { + symbol: p.symbol, + targetPct: p.targetPct, + totalReturn: (lastPrice - firstPrice) / firstPrice, + finalValue: p.shares * lastPrice, + }; + }); + + // Correlation matrix + const correlationMatrix = computeCorrelationMatrix(allocations, priceMap, alignedTimestamps); + + return { + equityCurve, + totalReturn: Math.round(totalReturn * 10000) / 10000, + cagr: Math.round(cagr * 10000) / 10000, + maxDrawdown: Math.round(maxDrawdown * 10000) / 10000, + positions: positionResults, + rebalanceEvents, + correlationMatrix, + sampleSize: alignedTimestamps.length, + }; + } +} + +// ─── Helpers ────────────────────────────────────────────────────────────── + +function intersectSortedDates(dateSets: Set[]): number[] { + if (dateSets.length === 0) return []; + // Start with all dates from the first symbol + let intersection = new Set(dateSets[0]); + for (let i = 1; i < dateSets.length; i++) { + const next = new Set(); + for (const d of dateSets[i]) { + if (intersection.has(d)) next.add(d); + } + intersection = next; + } + return Array.from(intersection).sort((a, b) => a - b); +} + +function checkRebalance( + idx: number, + lastRebalanceIdx: number, + rebalanceInterval: number, + driftThreshold: number, + positions: Array<{ symbol: string; targetPct: number; shares: number; entryPrice: number }>, + priceMap: Record>, + ts: number, + portfolioValue: number, + allocations: PortfolioAllocation[], +): RebalanceEvent['type'] | null { + // Periodic + if (idx - lastRebalanceIdx >= rebalanceInterval) return 'periodic'; + + // Drift check + if (driftThreshold < Infinity && portfolioValue > 0) { + for (const p of positions) { + const alloc = allocations.find((a) => a.symbol === p.symbol); + if (!alloc) continue; + const price = priceMap[p.symbol]?.get(ts) ?? 0; + const currentPct = (p.shares * price) / portfolioValue; + if (Math.abs(currentPct - alloc.targetPct / 100) > driftThreshold / 100) { + return 'drift'; + } + } + } + + return null; +} + +function computeMaxDrawdown(curve: PortfolioEquityPoint[]): number { + let peak = -Infinity; + let maxDD = 0; + for (const pt of curve) { + if (pt.value > peak) peak = pt.value; + const dd = (peak - pt.value) / peak; + if (dd > maxDD) maxDD = dd; + } + return maxDD; +} + +function computeCorrelationMatrix( + allocations: PortfolioAllocation[], + priceMap: Record>, + timestamps: number[], +): Record> { + const matrix: Record> = {}; + + // Compute daily returns for each symbol + const returns: Record = {}; + for (const a of allocations) { + const prices: number[] = []; + for (const ts of timestamps) { + prices.push(priceMap[a.symbol]?.get(ts) ?? NaN); + } + const dailyReturns: number[] = []; + for (let i = 1; i < prices.length; i++) { + if (isNaN(prices[i]) || isNaN(prices[i - 1]) || prices[i - 1] === 0) { + dailyReturns.push(NaN); + } else { + dailyReturns.push(prices[i] / prices[i - 1] - 1); + } + } + returns[a.symbol] = dailyReturns; + } + + // Compute pairwise Pearson correlation + for (const a of allocations) { + matrix[a.symbol] = {}; + for (const b of allocations) { + if (a.symbol === b.symbol) { + matrix[a.symbol][b.symbol] = 1; + continue; + } + const ra = returns[a.symbol]; + const rb = returns[b.symbol]; + matrix[a.symbol][b.symbol] = pearsonR(ra, rb); + } + } + + return matrix; +} + +function pearsonR(a: number[], b: number[]): number { + const n = Math.min(a.length, b.length); + let count = 0; + let sumA = 0, sumB = 0, sumAA = 0, sumBB = 0, sumAB = 0; + for (let i = 0; i < n; i++) { + if (isNaN(a[i]) || isNaN(b[i])) continue; + count++; + sumA += a[i]; + sumB += b[i]; + sumAA += a[i] * a[i]; + sumBB += b[i] * b[i]; + sumAB += a[i] * b[i]; + } + if (count < 3) return 0; + const num = count * sumAB - sumA * sumB; + const den = Math.sqrt((count * sumAA - sumA * sumA) * (count * sumBB - sumB * sumB)); + return den === 0 ? 0 : Math.round((num / den) * 10000) / 10000; +} + +function emptyResult(allocations: PortfolioAllocation[], initialEquity: number): PortfolioBacktestResult { + const emptyMatrix: Record> = {}; + for (const a of allocations) { + emptyMatrix[a.symbol] = {}; + for (const b of allocations) { + emptyMatrix[a.symbol][b.symbol] = a.symbol === b.symbol ? 1 : 0; + } + } + return { + equityCurve: [{ date: new Date().toISOString().slice(0, 10), value: initialEquity }], + totalReturn: 0, + cagr: 0, + maxDrawdown: 0, + positions: allocations.map((a) => ({ symbol: a.symbol, targetPct: a.targetPct, totalReturn: 0, finalValue: 0 })), + rebalanceEvents: [], + correlationMatrix: emptyMatrix, + sampleSize: 0, + }; +} diff --git a/app/server/src/trpc/__tests__/router.test.ts b/app/server/src/trpc/__tests__/router.test.ts index 3c10aa6..fa8f53b 100644 --- a/app/server/src/trpc/__tests__/router.test.ts +++ b/app/server/src/trpc/__tests__/router.test.ts @@ -235,9 +235,12 @@ test('onboarding.complete writes complexity/risk/drawdown + starter watchlist + assert.equal(u.risk_tolerance, 'moderate'); assert.equal(u.drawdown_tolerance, -20); const wl = db.prepare('SELECT symbols FROM watchlists WHERE owner_id=?').get(userId) as { symbols: string }; - assert.equal(JSON.parse(wl.symbols).length, 9); + const wlSymbols = JSON.parse(wl.symbols) as string[]; + // focused density pack (beginner experience → never_invested → focused) + assert.equal(wlSymbols.length, 6); + assert.ok(wlSymbols.includes('VTI') && wlSymbols.includes('AAPL')); const demand = await cache.demandSet(); - assert.ok(demand.includes('NVDA') && demand.includes('BTC'), 'starter symbols subscribed to demand set'); + assert.ok(demand.includes('VTI') && demand.includes('AAPL'), 'starter symbols subscribed to demand set'); }); // --- Slice 5: market.candles + market.indicators --- diff --git a/app/server/src/trpc/metric-context-test.ts b/app/server/src/trpc/metric-context-test.ts new file mode 100644 index 0000000..6e821ed --- /dev/null +++ b/app/server/src/trpc/metric-context-test.ts @@ -0,0 +1,2 @@ +// test write to locked directory +export const test = true; diff --git a/app/server/src/trpc/router.ts b/app/server/src/trpc/router.ts index e1ac656..70e3a85 100644 --- a/app/server/src/trpc/router.ts +++ b/app/server/src/trpc/router.ts @@ -7,16 +7,29 @@ import { hashPassword, verifyPassword, createSession, clearCookie, oauthStateCoo import { generateBase32Secret, totp as computeTotp, verifyTotp, otpauthUrl } from '../auth/totp.ts'; import { generateBackupCodes, hashBackupCode } from '../auth/backup-codes.ts'; import { buildAuthorizeUrl, generateState, exchangeCode, type OAuthProvider } from '../auth/oauth.ts'; -import { STARTER_WATCHLIST, defaultDrawdownTolerancePct, defaultRiskTolerance, ONBOARDING_DISCLAIMER, type Complexity } from '../onboarding/starter.ts'; +import { + STARTER_WATCHLIST, + starterPackFor, + resolveWorkspaceDefaults, + defaultDrawdownTolerancePct, + defaultRiskTolerance, + ONBOARDING_DISCLAIMER, + type Complexity, + type Density, + type ExperienceStage, + type Goal, + type Horizon, + type JargonComfort, +} from '../onboarding/starter.ts'; import type { Quote, PriceCandle, SymbolMeta } from '../cache/CacheRepository.ts'; import type { SymbolUniverseData } from '../screener/UniverseEvaluator.ts'; import type { ThesisEvent } from '../thesis/ThesisMonitor.ts'; -import type { OptionsUnlockState } from '../options/ConvexityGate.ts'; import type { XCookieHealth } from '../adapters/XCookieAdapter.ts'; import { emaFromCandles, rsi as rsiFn, relativeVolume, macd as macdFn } from '../analysis/indicators.ts'; -import { listUsers, resetPassword, gdprExport, queueHealth, resetQueueBackoff, NotOwnerError, listUserSessions, listAuditLog, queueSecFetch, setUserModules, disableUser, enableUser, deleteUser, getFinraBaseUrl, setFinraBaseUrl } from '../admin/admin.ts'; +import { listUsers, resetPassword, gdprExport, queueHealth, alertStatus, resetQueueBackoff, NotOwnerError, listUserSessions, listAuditLog, queueSecFetch, setUserModules, disableUser, enableUser, deleteUser, getFinraBaseUrl, setFinraBaseUrl, recordAudit } from '../admin/admin.ts'; import { restartServers, type RestartTarget } from '../lib/restart.ts'; import type { LintResult } from '../services/secDataFetcher.ts'; +import { computeOwnershipPercentages, fetchAndPersistFloat } from '../services/stockFloatService.ts'; import { EdgarAdapter } from '../adapters/EdgarAdapter.ts'; import { OptionsAdapter, parseOptionChainRows } from '../adapters/OptionsAdapter.ts'; import type { OptionChainRow, OptionGreeks } from '../adapters/OptionsAdapter.ts'; @@ -101,7 +114,21 @@ const authRouter = router({ }), me: publicProcedure.query(({ ctx }) => { if (!ctx.userId) return null; - const u = ctx.db.prepare('SELECT id,email,complexity,risk_tolerance,convexity_posture,modules FROM users WHERE id=?').get(ctx.userId) as { id: string; email: string; complexity: string; risk_tolerance: string; convexity_posture: string; modules: string } | undefined; + const u = ctx.db.prepare( + 'SELECT id,email,complexity,risk_tolerance,convexity_posture,modules,experience_stage,goal,horizon,density,jargon_comfort FROM users WHERE id=?', + ).get(ctx.userId) as { + id: string; + email: string; + complexity: string; + risk_tolerance: string; + convexity_posture: string; + modules: string; + experience_stage: string | null; + goal: string | null; + horizon: string | null; + density: string | null; + jargon_comfort: string | null; + } | undefined; const wl = ctx.db.prepare('SELECT 1 FROM watchlists WHERE owner_id=? LIMIT 1').get(ctx.userId); let modules: string[] = ['research', 'settings']; try { modules = JSON.parse(u?.modules ?? '["research","settings"]'); } catch { /* keep default */ } @@ -109,7 +136,23 @@ const authRouter = router({ const adminRow = ctx.db.prepare('SELECT is_admin FROM users WHERE id=?').get(u.id) as { is_admin: number } | undefined; if (adminRow?.is_admin && !modules.includes('admin')) modules.push('admin'); } - return u ? { userId: u.id, email: u.email, complexity: u.complexity, riskTolerance: u.risk_tolerance, convexityPosture: u.convexity_posture, onboarded: !!wl, modules } : null; + const density = (u?.density === 'standard' || u?.density === 'full' || u?.density === 'focused') + ? u.density + : (u?.complexity === 'advanced' ? 'full' : u?.complexity === 'intermediate' ? 'standard' : 'focused'); + return u ? { + userId: u.id, + email: u.email, + complexity: u.complexity, + riskTolerance: u.risk_tolerance, + convexityPosture: u.convexity_posture, + onboarded: !!wl, + modules, + experienceStage: u.experience_stage, + goal: u.goal, + horizon: u.horizon, + density, + jargonComfort: u.jargon_comfort ?? 'plain', + } : null; }), enable2fa: protectedProcedure.input(z.object({})).mutation(async ({ ctx }) => { const userId = ctx.userId as string; @@ -171,12 +214,33 @@ const authRouter = router({ }); const onboardingRouter = router({ - // Public: the starter watchlist + disclaimer shown in the wizard before completing. - starter: publicProcedure.query(() => ({ watchlist: STARTER_WATCHLIST, disclaimer: ONBOARDING_DISCLAIMER })), - // Protected: write complexity/risk/drawdown + first watchlist + optional portfolio; subscribe symbols to demand. + // Public: starter symbols for a density (default focused). Used in workspace setup preview. + starter: publicProcedure + .input(z.object({ + density: z.enum(['focused', 'standard', 'full']).optional(), + experienceStage: z.enum(['never_invested', 'some_stocks', 'active_self_directed']).optional(), + }).optional()) + .query(({ input }) => { + const density: Density = input?.density + ?? (input?.experienceStage === 'active_self_directed' + ? 'full' + : input?.experienceStage === 'some_stocks' + ? 'standard' + : 'focused'); + const pack = starterPackFor(density); + return { watchlist: pack, density, disclaimer: ONBOARDING_DISCLAIMER }; + }), + // Protected: interview → workspace profile + first watchlist + optional portfolio. complete: protectedProcedure .input(z.object({ - complexity: z.enum(['beginner', 'intermediate', 'advanced']), + // Preferred interview path + experienceStage: z.enum(['never_invested', 'some_stocks', 'active_self_directed']).optional(), + goal: z.enum(['grow', 'income', 'protect']).optional(), + horizon: z.enum(['short', 'medium', 'long']).optional(), + density: z.enum(['focused', 'standard', 'full']).optional(), + jargonComfort: z.enum(['plain', 'mixed', 'technical']).optional(), + // Legacy path (still accepted) + complexity: z.enum(['beginner', 'intermediate', 'advanced']).optional(), riskTolerance: z.enum(['conservative', 'moderate', 'aggressive']).optional(), drawdownTolerancePct: z.number().optional(), firstWatchlistSymbols: z.array(z.string()).optional(), @@ -184,30 +248,116 @@ const onboardingRouter = router({ })) .mutation(async ({ ctx, input }) => { const userId = ctx.userId as string; - const complexity = input.complexity as Complexity; - const riskTolerance = input.riskTolerance ?? defaultRiskTolerance(complexity); - const drawdown = input.drawdownTolerancePct ?? defaultDrawdownTolerancePct(complexity); - ctx.db.prepare('UPDATE users SET complexity=?, risk_tolerance=?, drawdown_tolerance=? WHERE id=?').run(complexity, riskTolerance, drawdown, userId); - const symbols = input.firstWatchlistSymbols ?? STARTER_WATCHLIST.map((s) => s.symbol); - const wlId = randomUUID(); - ctx.db.prepare('INSERT INTO watchlists (id, owner_id, name, symbols, created_at, sort_order) VALUES (?,?,?,?,?,?)').run(wlId, userId, 'default', JSON.stringify(symbols), new Date().toISOString(), 0); + const experienceStage: ExperienceStage = input.experienceStage + ?? (input.complexity === 'advanced' + ? 'active_self_directed' + : input.complexity === 'intermediate' + ? 'some_stocks' + : 'never_invested'); + const defaults = resolveWorkspaceDefaults({ + experienceStage, + goal: input.goal as Goal | undefined, + horizon: input.horizon as Horizon | undefined, + density: input.density as Density | undefined, + jargonComfort: input.jargonComfort as JargonComfort | undefined, + }); + const complexity = (input.complexity ?? defaults.complexity) as Complexity; + const riskTolerance = input.riskTolerance ?? defaults.riskTolerance; + const drawdown = input.drawdownTolerancePct ?? defaults.drawdownTolerancePct; + ctx.db.prepare( + `UPDATE users SET complexity=?, risk_tolerance=?, drawdown_tolerance=?, + experience_stage=?, goal=?, horizon=?, density=?, jargon_comfort=? WHERE id=?`, + ).run( + complexity, + riskTolerance, + drawdown, + defaults.experienceStage, + defaults.goal, + defaults.horizon, + defaults.density, + defaults.jargonComfort, + userId, + ); + const pack = defaults.starterPack; + const symbols = input.firstWatchlistSymbols ?? pack.map((s) => s.symbol); + const existingWl = ctx.db.prepare('SELECT id FROM watchlists WHERE owner_id=? LIMIT 1').get(userId) as { id: string } | undefined; + let wlId = existingWl?.id; + if (!wlId) { + wlId = randomUUID(); + ctx.db.prepare('INSERT INTO watchlists (id, owner_id, name, symbols, created_at, sort_order) VALUES (?,?,?,?,?,?)').run( + wlId, userId, 'default', JSON.stringify(symbols), new Date().toISOString(), 0, + ); + } + const allKnown = [...STARTER_WATCHLIST, ...starterPackFor('full')]; for (const sym of symbols) { - const kind = (STARTER_WATCHLIST.find((s) => s.symbol === sym)?.tickerKind ?? 'equity') as 'equity' | 'crypto' | 'etf' | 'index'; - await ctx.cache.subscribe(sym, kind); - queueSecFetch(ctx.db, sym); + const kind = (allKnown.find((s) => s.symbol === sym)?.tickerKind + ?? pack.find((s) => s.symbol === sym)?.tickerKind + ?? 'equity') as 'equity' | 'crypto' | 'etf' | 'index'; + try { await ctx.cache.subscribe(sym, kind); } catch { /* ignore */ } + try { queueSecFetch(ctx.db, sym); } catch { /* ignore */ } } try { const { materializeClassificationWatchlists } = await import('../db/watchlistRepository.ts'); materializeClassificationWatchlists(ctx.db, userId); - } catch { /* ignore — sector data not available yet, will materialize on first listWatchlists */ } + } catch { /* ignore — sector data not available yet */ } if (input.portfolio) { const ins = ctx.db.prepare('INSERT INTO portfolio_holdings (id, owner_id, symbol, qty, avg_cost, acquired_at, status) VALUES (?,?,?,?,?,?,?)'); for (const h of input.portfolio) ins.run(randomUUID(), userId, h.symbol.toUpperCase(), h.qty, h.avgCost, h.acquiredAt, 'open'); } - return { ok: true, watchlistId: wlId }; + return { + ok: true, + watchlistId: wlId, + density: defaults.density, + complexity, + experienceStage: defaults.experienceStage, + }; + }), + // Protected: update workspace profile after onboarding (density, goal, jargon) without re-seeding watchlist. + updateProfile: protectedProcedure + .input(z.object({ + experienceStage: z.enum(['never_invested', 'some_stocks', 'active_self_directed']).optional(), + goal: z.enum(['grow', 'income', 'protect']).optional(), + horizon: z.enum(['short', 'medium', 'long']).optional(), + density: z.enum(['focused', 'standard', 'full']).optional(), + jargonComfort: z.enum(['plain', 'mixed', 'technical']).optional(), + })) + .mutation(({ ctx, input }) => { + const userId = ctx.userId as string; + const row = ctx.db.prepare( + 'SELECT experience_stage, goal, horizon, density, jargon_comfort, complexity FROM users WHERE id=?', + ).get(userId) as { + experience_stage: string | null; + goal: string | null; + horizon: string | null; + density: string | null; + jargon_comfort: string | null; + complexity: string; + } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'User not found.' }); + + const experienceStage = (input.experienceStage + ?? row.experience_stage + ?? 'never_invested') as ExperienceStage; + const density = (input.density ?? row.density ?? 'focused') as Density; + const goal = (input.goal ?? row.goal ?? 'grow') as Goal; + const horizon = (input.horizon ?? row.horizon ?? 'long') as Horizon; + const jargonComfort = (input.jargonComfort ?? row.jargon_comfort ?? 'plain') as JargonComfort; + const complexity = complexityFromExperienceSafe(experienceStage); + + ctx.db.prepare( + `UPDATE users SET experience_stage=?, goal=?, horizon=?, density=?, jargon_comfort=?, complexity=? WHERE id=?`, + ).run(experienceStage, goal, horizon, density, jargonComfort, complexity, userId); + + return { ok: true, experienceStage, goal, horizon, density, jargonComfort, complexity }; }), }); +function complexityFromExperienceSafe(stage: ExperienceStage): Complexity { + if (stage === 'active_self_directed') return 'advanced'; + if (stage === 'some_stocks') return 'intermediate'; + return 'beginner'; +} + const marketRouter = router({ snapshot: publicProcedure .input(z.object({ symbol: z.string().min(1) })) @@ -227,6 +377,55 @@ const marketRouter = router({ }; }), + /** + * Batched snapshot for multiple symbols — reduces N+1 API calls. + * Used by portfolio/watchlist views that need quotes for many symbols at once. + */ + snapshots: publicProcedure + .input(z.object({ symbols: z.array(z.string().min(1)).max(50) })) + .query(async ({ ctx, input }) => { + const symbols = input.symbols.map((s) => s.toUpperCase()); + // Build all cache keys at once + const keys = symbols.flatMap((symbol) => [ + `yfinance:quote:${symbol}`, + `yfinance:candles:${symbol}:1d`, + `yfinance:symbol:${symbol}`, + ]); + + // Batch read all at once + const entries = await ctx.cache.getMany(keys); + const byKey = new Map(entries.map((e) => [e.key, e])); + + // Group results by symbol + const results: Array<{ + symbol: string; + quote: Quote | null; + candles: PriceCandle[] | null; + sector: SymbolMeta | null; + stale: { quote: boolean; candles: boolean; sector: boolean }; + }> = []; + + for (const symbol of symbols) { + const kQuote = `yfinance:quote:${symbol}`; + const kCandles = `yfinance:candles:${symbol}:1d`; + const kSector = `yfinance:symbol:${symbol}`; + + results.push({ + symbol, + quote: byKey.get(kQuote)?.value as Quote | null ?? null, + candles: byKey.get(kCandles)?.value as PriceCandle[] | null ?? null, + sector: byKey.get(kSector)?.value as SymbolMeta | null ?? null, + stale: { + quote: byKey.get(kQuote)?.isStale ?? true, + candles: byKey.get(kCandles)?.isStale ?? true, + sector: byKey.get(kSector)?.isStale ?? true, + }, + }); + } + + return results; + }), + /** * Focused-ticker context: performance vs market, sector/theme proxies, and peers. * Educational relative snapshot (ADR-0007). @@ -248,8 +447,8 @@ const marketRouter = router({ const { classifyRegime } = await import('../macro/MacroRegime.ts'); const symbol = input.symbol.toUpperCase(); - try { await ctx.cache.subscribe(symbol, 'equity'); } catch { /* ignore */ } - try { await ctx.cache.subscribe(BENCHMARK_SYMBOL, 'etf'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand(symbol, 'equity'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand(BENCHMARK_SYMBOL, 'etf'); } catch { /* ignore */ } const metaEntry = await ctx.cache.get(`yfinance:symbol:${symbol}`); const meta = metaEntry.value; @@ -269,7 +468,7 @@ const marketRouter = router({ const sectorMap = biz.sectorEtf ? { etf: biz.sectorEtf, label: biz.sectorLabel ?? biz.sectorEtf } : null; const themeMap = biz.themeEtf ? { etf: biz.themeEtf, label: biz.themeLabel ?? biz.themeEtf } : null; - // Peers: curated override first, then vendor peers, then sector-ETF holdings (never when blocked). + // Peers: curated override first, then vendor peers, then sector-ETF holdings cache (never live Yahoo on request path). let peerSymbols: string[] = biz.peers.length > 0 ? biz.peers.filter((p) => p !== symbol) : Array.isArray(meta?.peers) @@ -278,16 +477,28 @@ const marketRouter = router({ if (peerSymbols.length === 0 && sectorMap && !biz.blockSectorEtfPeers) { try { - const mod = await import('yahoo-finance2'); - const yf = new mod.default(); - const summary = await yf.quoteSummary(sectorMap.etf, { modules: ['topHoldings'] }) as { - topHoldings?: { holdings?: Array<{ symbol?: string }> }; - }; - peerSymbols = (summary?.topHoldings?.holdings ?? []) + const holdEntry = await ctx.cache.get<{ holdings?: Array<{ symbol?: string }> }>( + `yfinance:topHoldings:${sectorMap.etf}`, + ); + const holdings = holdEntry.value?.holdings ?? []; + peerSymbols = holdings .map((h) => (h.symbol ?? '').toUpperCase()) .filter((p) => p && p !== symbol) .slice(0, 8); - } catch { /* live peer pull optional */ } + // Also try kv_cache layout used by sectorHoldings + if (peerSymbols.length === 0) { + const row = ctx.db.prepare('SELECT value FROM kv_cache WHERE key=?').get(`etf:topHoldings:${sectorMap.etf}`) as + | { value: string } + | undefined; + if (row?.value) { + const parsed = JSON.parse(row.value) as { holdings?: Array<{ symbol?: string }> }; + peerSymbols = (parsed.holdings ?? []) + .map((h) => (h.symbol ?? '').toUpperCase()) + .filter((p) => p && p !== symbol) + .slice(0, 8); + } + } + } catch { /* cache peer pull optional */ } } const symbolsToLoad = Array.from(new Set([ @@ -299,7 +510,8 @@ const marketRouter = router({ ])); for (const s of symbolsToLoad) { - try { await ctx.cache.subscribe(s, s === BENCHMARK_SYMBOL || s.length <= 4 && MARKET_ROTATION_UNIVERSE.some((u) => u.symbol === s) ? 'etf' : 'equity'); } catch { /* ignore */ } + const kind = s === BENCHMARK_SYMBOL || MARKET_ROTATION_UNIVERSE.some((u) => u.symbol === s) ? 'etf' : 'equity'; + try { await ctx.cache.ensureInDemand(s, kind); } catch { /* ignore */ } } const candleKeys = symbolsToLoad.map((s) => `yfinance:candles:${s}:1d`); @@ -476,13 +688,14 @@ const marketRouter = router({ }), /** * Market Condition strip — regime + risk flags. - * FRED series are fetched live (key from admin) when not already warm; SPY/VIX from yfinance. + * FRED series are queue-warmed (fred schedule, fred_macro tier) — never fetched + * live on this request path. SPY/VIX from yfinance cache only. * Educational snapshot only (ADR-0007). */ condition: publicProcedure.query(async ({ ctx }) => { const { classifyRegime } = await import('../macro/MacroRegime.ts'); const { totalReturnPct } = await import('../analysis/marketRotationRs.ts'); - const { FredAdapterImpl, FRED_SERIES } = await import('../macro/FredAdapter.ts'); + const { FRED_SERIES } = await import('../macro/FredAdapter.ts'); const factors: Record = { gdpGrowth: null, @@ -494,21 +707,15 @@ const marketRouter = router({ vix: null, spy1M: null, spyEmaSlope: null, + consumerSentiment: null, }; - // Decrypt FRED key (same store as truck sales / manufacturing). - let fredKey = ''; - try { - const row = ctx.db.prepare('SELECT fred_api_key_enc FROM x_credentials WHERE id=?').get('singleton') as { fred_api_key_enc?: string | null } | undefined; - if (row?.fred_api_key_enc) { - const { default: encrypt } = await import('../lib/crypto.ts'); - try { fredKey = encrypt.decrypt(row.fred_api_key_enc); } catch { /* corrupt */ } - } - } catch { /* ignore */ } - type FredObs = Array<{ date: string; value: number }>; - /** FRED observations: kv_cache first, then live API; write-through to kv_cache. */ + /** + * FRED observations: kv_cache only (queue warms it). If cold/stale, queue a + * background warm-up and return null — never open a live FRED call here. + */ async function fredObs(seriesId: string): Promise { const cacheKey = `fred:series:${seriesId}`; try { @@ -524,23 +731,9 @@ const marketRouter = router({ } } catch { /* miss */ } - if (!fredKey) return null; - try { - const adapter = new FredAdapterImpl(fredKey); - const entry = await adapter.series(seriesId); - const series = entry.value as { observations?: FredObs } | null; - const obs = series?.observations; - if (!obs?.length) return null; - const now = new Date().toISOString(); - try { - ctx.db.prepare( - 'INSERT OR REPLACE INTO kv_cache (key, value, observed_at) VALUES (?,?,?)', - ).run(cacheKey, JSON.stringify({ seriesId, observations: obs }), now); - } catch { /* best-effort cache; may race under SQLite */ } - return obs; - } catch { - return null; - } + // Cold or stale: queue warm-up for the scheduler (no live fetch on request path). + try { await ctx.queue.queue(cacheKey); } catch { /* ignore */ } + return null; } function lastValue(obs: FredObs | null): number | null { @@ -558,30 +751,34 @@ const marketRouter = router({ return ((last - yearAgo) / yearAgo) * 100; } - // Parallel FRED pulls (rates, labor, inflation). - const [unrateObs, gs10Obs, gs2Obs, cpiObs] = await Promise.all([ + // Parallel FRED pulls (rates, labor, inflation, consumer sentiment). + const [unrateObs, gs10Obs, gs2Obs, cpiObs, umcObs] = await Promise.all([ fredObs(FRED_SERIES.UNEMPLOYMENT), fredObs(FRED_SERIES.TREASURY_10Y), fredObs(FRED_SERIES.TREASURY_2Y), fredObs(FRED_SERIES.CPI), + fredObs(FRED_SERIES.CONSUMER_SENTIMENT), ]); factors.unemployment = lastValue(unrateObs); factors.treasury10Y = lastValue(gs10Obs); factors.treasury2Y = lastValue(gs2Obs); factors.cpi = yoyPct(cpiObs); // classifier expects inflation %, not CPI index level + factors.consumerSentiment = lastValue(umcObs); // UMCSENT: University of Michigan Consumer Sentiment Index (index level, not YoY) if (factors.treasury10Y !== null && factors.treasury2Y !== null) { factors.curve10y2y = factors.treasury10Y - factors.treasury2Y; } - // SPY trend + VIX from yfinance (subscribe if cold). + // SPY trend + VIX from cache/queue only (ADR-0009: no live Yahoo on request path). + let spyCandles: PriceCandle[] = []; try { - try { await ctx.cache.subscribe('SPY', 'etf'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand('SPY', 'etf'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand('^VIX', 'index'); } catch { /* ignore */ } const spyEntry = await ctx.cache.get('yfinance:candles:SPY:1d'); - const spy = (spyEntry?.value ?? []) as PriceCandle[]; - factors.spy1M = totalReturnPct(spy, 30 * 86_400_000); - if (spy.length >= 60) { - const last = spy[spy.length - 1].c; - const ago = spy[spy.length - 50].c; + spyCandles = (spyEntry?.value ?? []) as PriceCandle[]; + factors.spy1M = totalReturnPct(spyCandles, 30 * 86_400_000); + if (spyCandles.length >= 60) { + const last = spyCandles[spyCandles.length - 1].c; + const ago = spyCandles[spyCandles.length - 50].c; if (ago > 0) factors.spyEmaSlope = ((last - ago) / ago) * 100; } } catch { /* ignore */ } @@ -594,27 +791,102 @@ const marketRouter = router({ const arr = vixC?.value as PriceCandle[] | undefined; if (arr?.length) factors.vix = arr[arr.length - 1].c; } - // Live fallback via yahoo-finance2 if still missing. + // Queue VIX quote if still cold — never open a live Yahoo call here (shared cool-down). if (factors.vix == null) { - try { - const mod = await import('yahoo-finance2'); - const yf = new mod.default(); - const q = await yf.quote('^VIX') as { regularMarketPrice?: number }; - if (typeof q?.regularMarketPrice === 'number') factors.vix = q.regularMarketPrice; - } catch { /* ignore */ } + try { await ctx.queue.queue('yfinance:quote:^VIX'); } catch { /* ignore */ } } } catch { /* ignore */ } - // Yield fallback from Yahoo if FRED key missing/failed (^TNX ≈ 10Y yield). - if (factors.treasury10Y == null) { - try { - const mod = await import('yahoo-finance2'); - const yf = new mod.default(); - const q = await yf.quote('^TNX') as { regularMarketPrice?: number }; - if (typeof q?.regularMarketPrice === 'number') factors.treasury10Y = q.regularMarketPrice; - } catch { /* ignore */ } + // Per-metric context strings (hover tooltips explaining what each value means). + function trailingAvg(obs: Array<{value: number}>, n: number): number | null { + if (!obs || obs.length < n) return null; + const recent = obs.slice(-n); + let sum = 0; + for (const o of recent) sum += o.value; + return sum / n; } + const metricCtx: Record = {}; + + // SPY 1M — current vs trailing ~6mo average return. + if (factors.spy1M != null && spyCandles.length >= 126) { + const recent = spyCandles.slice(-127, -1); // last ~5 months minus today + if (recent.length >= 2) { + const firstPrice = recent[0].c; + const lastPrice = recent[recent.length - 1].c; + if (firstPrice > 0) { + const avgReturn = ((lastPrice - firstPrice) / firstPrice) * 100; + const label = factors.spy1M > avgReturn ? 'above trend' : 'below trend'; + metricCtx.spy1M = label; + } + } + } + + // VIX — regime-based interpretation (no advice language). + if (factors.vix != null) { + const v = factors.vix; + if (v < 15) metricCtx.vix = 'Low volatility — risk-on regime'; + else if (v < 20) metricCtx.vix = 'Normal range'; + else if (v < 30) metricCtx.vix = 'Elevated — historically precedes drawdowns'; + else metricCtx.vix = 'Stressed — risk-off / crisis territory'; + } + + // Treasury 10Y vs rolling ~3mo average. + if (factors.treasury10Y != null && gs10Obs?.length) { + const last = factors.treasury10Y; + const avg3m = trailingAvg(gs10Obs, Math.min(63, gs10Obs.length)); + if (avg3m != null) { + const diff = last - avg3m; + metricCtx.treasury10Y = diff > 0 + ? 'Above 3-month average — headwind for growth if elevated' + : 'Below 3-month average'; + } else { + metricCtx.treasury10Y = 'Headwind for growth if elevated'; + } + } + + // Treasury 2Y vs rolling ~3mo average. + if (factors.treasury2Y != null && gs2Obs?.length) { + const last = factors.treasury2Y; + const avg3m = trailingAvg(gs2Obs, Math.min(63, gs2Obs.length)); + if (avg3m != null) { + const diff = last - avg3m; + metricCtx.treasury2Y = diff > 0 + ? 'Above 3-month average — tighter monetary conditions' + : 'Below 3-month average'; + } else { + metricCtx.treasury2Y = 'Tight monetary conditions'; + } + } + + // Curve sign + magnitude (50bp threshold for inversion signal). + if (factors.curve10y2y != null) { + const cBp = factors.curve10y2y * 100; + if (cBp < -50) metricCtx.curve10y2y = 'Inverted — historically bearish for equities'; + else if (cBp < 0) metricCtx.curve10y2y = 'Slightly inverted — watch for steepening'; + else if (cBp < 50) metricCtx.curve10y2y = 'Positive spread — normal cycle'; + else metricCtx.curve10y2y = 'Steep — recovery or inflation expectations rising'; + } + + // Unemployment vs ~5% recession threshold. + if (factors.unemployment != null) { + const u = factors.unemployment; + if (u < 4.0) metricCtx.unemployment = 'Strong labor market'; + else if (u < 5.0) metricCtx.unemployment = 'Stable — below recession threshold of 5%'; + else if (u < 6.0) metricCtx.unemployment = 'Elevated — monitor for further rises'; + else metricCtx.unemployment = 'High — historically correlates with recessions'; + } + + // Consumer sentiment vs UMCSENT psychological thresholds (80/100). + if (factors.consumerSentiment != null) { + const s = factors.consumerSentiment; + if (s > 100) metricCtx.consumerSentiment = 'Strong confidence'; + else if (s > 80) metricCtx.consumerSentiment = 'Moderate sentiment'; + else if (s > 60) metricCtx.consumerSentiment = 'Weak — below 80 historically precedes drawdowns'; + else metricCtx.consumerSentiment = 'Very low confidence — crisis territory'; + } + + // Compute regime + risk flags using raw factors. const classification = classifyRegime({ gdpGrowth: factors.gdpGrowth ?? undefined, cpi: factors.cpi ?? undefined, @@ -685,6 +957,8 @@ const marketRouter = router({ treasury2Y: factors.treasury2Y, curve10y2y: factors.curve10y2y, unemployment: factors.unemployment, + consumerSentiment: factors.consumerSentiment, + context: metricCtx, }, risks, updatedAt: new Date().toISOString(), @@ -702,7 +976,7 @@ const marketRouter = router({ const symbols = [BENCHMARK_SYMBOL, ...MARKET_ROTATION_UNIVERSE.map((s) => s.symbol)]; for (const sym of symbols) { - try { await ctx.cache.subscribe(sym, 'etf'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand(sym, 'etf'); } catch { /* ignore */ } } const keys = symbols.map((s) => `yfinance:candles:${s}:1d`); @@ -740,7 +1014,7 @@ const marketRouter = router({ if (existing) continue; const alert = createAlert( crypto.randomUUID(), u.id, 'rotation_incipient' as any, undefined, - `Relative leadership may be shifting (${signal}). Leading: ${summary.leadingGroup}. Lagging: ${summary.laggingGroup}. Spread ~${spread.toFixed(1)} pp vs SPY. Educational notice — not an allocation instruction.`, + `Relative leadership may be shifting (${signal}). Leading: ${summary.leadingGroup}. Lagging: ${summary.laggingGroup}. Spread ~${spread.toFixed(1)} pp vs SPY.`, signal, { spread, signal, leadingGroup: summary.leadingGroup, laggingGroup: summary.laggingGroup }, ); @@ -772,10 +1046,93 @@ const marketRouter = router({ buildSectorRsMap, summarizeRotation, } = await import('../analysis/marketRotationRs.ts'); + const { resolveBusinessContext } = await import('../analysis/tickerContext.ts'); + const { listHoldings } = await import('../db/portfolioRepository.ts'); - const symbols = [BENCHMARK_SYMBOL, ...MARKET_ROTATION_UNIVERSE.map((s) => s.symbol)]; + const userId = ctx.userId ?? 'anonymous'; + + // Load custom symbols from DB + const customRows = ctx.db.prepare( + 'SELECT symbol, name, grp FROM rotation_custom_symbols WHERE owner_id=? ORDER BY symbol', + ).all(userId) as Array<{ symbol: string; name: string | null; grp: string }>; + const customDefs: import('../analysis/marketRotationRs.ts').SectorDef[] = customRows.map((r) => ({ + symbol: r.symbol, + name: r.name ?? r.symbol, + group: r.grp, + kind: 'custom' as const, + })); + + // Auto-group: ETFs that open portfolio holdings map into (sector + theme). + // Many of these already exist in MARKET_ROTATION_UNIVERSE (XLK, XLF, …). We must not + // drop them on dedupe — track portfolio linkage separately so the Auto filter works. + const holdings = listHoldings(ctx.db, userId); + type AutoMeta = { name: string; group: string; holdings: string[] }; + const portfolioEtfMeta = new Map(); + const addPortfolioEtf = (etf: string, name: string, group: string, holdingSym: string) => { + const key = etf.toUpperCase(); + const prev = portfolioEtfMeta.get(key); + if (prev) { + if (!prev.holdings.includes(holdingSym)) prev.holdings.push(holdingSym); + return; + } + portfolioEtfMeta.set(key, { name, group, holdings: [holdingSym] }); + }; + for (const h of holdings) { + const holdingSym = h.symbol.toUpperCase(); + const meta = ctx.db.prepare('SELECT sector, industry FROM symbols WHERE symbol=?').get(holdingSym) as + | { sector: string | null; industry: string | null } + | undefined; + // Always resolve — symbol overrides (IREN, CIFR, …) work even without a symbols row. + const biz = resolveBusinessContext({ + symbol: holdingSym, + sector: meta?.sector ?? null, + industry: meta?.industry ?? null, + description: null, + }); + if (biz.sectorEtf) { + addPortfolioEtf( + biz.sectorEtf, + biz.sectorLabel ?? biz.sectorEtf, + biz.sectorLabel ?? 'Portfolio', + holdingSym, + ); + } + if (biz.themeEtf) { + addPortfolioEtf( + biz.themeEtf, + biz.themeLabel ?? biz.themeEtf, + biz.themeLabel ?? 'Portfolio theme', + holdingSym, + ); + } + } + // Portfolio-only ETFs (e.g. IGV) that are not already in the market universe + const autoOnlyDefs: import('../analysis/marketRotationRs.ts').SectorDef[] = []; + for (const [etf, meta] of portfolioEtfMeta) { + const inUniverse = MARKET_ROTATION_UNIVERSE.some((u) => u.symbol === etf) + || customDefs.some((c) => c.symbol === etf); + if (!inUniverse) { + autoOnlyDefs.push({ + symbol: etf, + name: meta.name, + group: meta.group, + kind: 'auto', + }); + } + } + + // Combine and deduplicate (universe first so sector/style kinds stay stable) + const allDefs = [...MARKET_ROTATION_UNIVERSE, ...customDefs, ...autoOnlyDefs]; + const seen = new Set(); + const dedupedDefs = allDefs.filter((d) => { + if (seen.has(d.symbol)) return false; + seen.add(d.symbol); + return true; + }); + + const symbols = [BENCHMARK_SYMBOL, ...dedupedDefs.map((s) => s.symbol)]; for (const sym of symbols) { - try { await ctx.cache.subscribe(sym, 'etf'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand(sym, 'etf'); } catch { /* ignore */ } } const keys = symbols.map((s) => `yfinance:candles:${s}:1d`); @@ -788,11 +1145,31 @@ const marketRouter = router({ if (candles.length > 0) dataPoints += 1; }); + // Load live quotes for oneDay overrides — when available, use intraday + // change_pct (current price vs previous close) instead of candle dayChangePct, + // which reflects overnight gaps because yfinance 1d candles carry 13:30Z bar + // timestamps. Other horizons stay candle-based. + const quoteRows = ctx.db.prepare( + `SELECT symbol, change_pct FROM quotes WHERE symbol IN (${symbols.map(() => '?').join(',')})`, + ).all(...symbols) as Array<{ symbol: string; change_pct: number | null }>; + const liveQuotes = new Map(quoteRows.map((r) => [r.symbol, r.change_pct ?? null])); + const rows = buildSectorRsMap( - MARKET_ROTATION_UNIVERSE, + dedupedDefs, map, map[BENCHMARK_SYMBOL] ?? [], ); + + // Override oneDay with live quote change_pct when available (true intraday vs prev close). + for (const r of rows) { + const qPct = liveQuotes.get(r.symbol); + if (qPct != null && Number.isFinite(qPct)) { /* already in percent units from yfinance */ + r.abs.oneDay = qPct; + const benchPct = liveQuotes.get(BENCHMARK_SYMBOL); + r.rs.oneDay = benchPct != null ? qPct - benchPct : qPct; + } + } + const summary = summarizeRotation(rows); const state = ctx.db.prepare('SELECT signal_since FROM rotation_state WHERE id=?').get('singleton') as { signal_since: string | null } | undefined; @@ -800,30 +1177,42 @@ const marketRouter = router({ const daysSince = signalSince ? Math.round((Date.now() - new Date(signalSince).getTime()) / 86_400_000) : null; // Backward-compatible sector cards (group = leadership, not abs-return inflow). - const sectors = rows.map((r) => ({ - symbol: r.symbol, - name: r.name, - group: r.leadership === 'leading' ? 'inflow' as const - : r.leadership === 'lagging' ? 'outflow' as const - : 'inflow' as const, // inline treated as neutral; UI will use leadership field - subGroup: r.group, - kind: r.kind, - leadership: r.leadership, - earlyWatch: r.earlyWatch, - rank1M: r.rank1M, - rank1W: r.rank1W, - relVol: r.relVol, - oneWeek: r.abs.oneWeek, - oneMonth: r.abs.oneMonth, - threeMonth: r.abs.threeMonth, - sixMonth: r.abs.sixMonth, - oneYear: r.abs.oneYear, - rsOneWeek: r.rs.oneWeek, - rsOneMonth: r.rs.oneMonth, - rsThreeMonth: r.rs.threeMonth, - rsSixMonth: r.rs.sixMonth, - rsOneYear: r.rs.oneYear, - })); + // portfolioLinked: true when this ETF is a sector/theme proxy for an open holding. + // Auto filter uses portfolioLinked so XLK/XLF/etc. still appear under Auto even though + // they also live in the main GICS universe (kind remains sector/thematic/…). + const sectors = rows.map((r) => { + const port = portfolioEtfMeta.get(r.symbol); + // Keep universe display names (Technology, Financials); use portfolio label only for auto-only ETFs. + const displayName = r.kind === 'auto' && port?.name ? port.name : r.name; + return { + symbol: r.symbol, + name: displayName, + group: r.leadership === 'leading' ? 'inflow' as const + : r.leadership === 'lagging' ? 'outflow' as const + : 'inflow' as const, // inline treated as neutral; UI will use leadership field + subGroup: r.group, + kind: r.kind, + portfolioLinked: !!port, + portfolioHoldings: port?.holdings ?? [], + leadership: r.leadership, + earlyWatch: r.earlyWatch, + rank1M: r.rank1M, + rank1W: r.rank1W, + relVol: r.relVol, + oneWeek: r.abs.oneWeek, + oneMonth: r.abs.oneMonth, + threeMonth: r.abs.threeMonth, + sixMonth: r.abs.sixMonth, + oneYear: r.abs.oneYear, + oneDay: r.abs.oneDay, + rsOneWeek: r.rs.oneWeek, + rsOneMonth: r.rs.oneMonth, + rsThreeMonth: r.rs.threeMonth, + rsSixMonth: r.rs.sixMonth, + rsOneYear: r.rs.oneYear, + rsOneDay: r.rs.oneDay, + }; + }); const signal = { strength: summary.strength, @@ -887,7 +1276,7 @@ const marketRouter = router({ .query(async ({ ctx, input }) => { const { buildSeasonalitySnapshot, upcomingSimpleEvents } = await import('../analysis/seasonality.ts'); const symbol = (input?.symbol ?? 'SPY').toUpperCase(); - try { await ctx.cache.subscribe(symbol, symbol === 'SPY' ? 'etf' : 'equity'); } catch { /* ignore */ } + try { await ctx.cache.ensureInDemand(symbol, symbol === 'SPY' ? 'etf' : 'equity'); } catch { /* ignore */ } const entry = await ctx.cache.get(`yfinance:candles:${symbol}:1d`); const candles = (entry?.value ?? []) as PriceCandle[]; const snapshot = buildSeasonalitySnapshot( @@ -963,11 +1352,14 @@ const marketRouter = router({ holdings = cached.rows; source = 'cache'; } else if (cached) { - // Stale composition is fine for a peek panel; do not block on Yahoo under rate limits. + // Stale composition is fine for a peek panel; background queue refreshes. holdings = cached.rows; source = 'cache'; + if (!ctx.queue.isSourceCoolingDown('yfinance')) { + try { await ctx.queue.queue(`yfinance:topHoldings:${symbol}`); } catch { /* ignore */ } + } } else { - // Cold path: static first (instant), then optional short Yahoo upgrade. + // Cold path: static fallback first (authoritative when present). const fallback = fromStatic(); if (fallback.length > 0) { holdings = fallback; @@ -975,31 +1367,9 @@ const marketRouter = router({ writeCachedComposition(fallback); } - try { - const mod = await import('yahoo-finance2'); - const yf = new mod.default(); - const summary = await Promise.race([ - yf.quoteSummary(symbol, { modules: ['topHoldings'] }) as Promise<{ - topHoldings?: { holdings?: HoldingRow[] }; - }>, - new Promise((_, reject) => { - setTimeout(() => reject(new Error('yahoo-topHoldings-timeout')), 2500); - }), - ]); - const live = (summary?.topHoldings?.holdings ?? []) - .filter((h) => h?.symbol) - .map((h) => ({ - symbol: String(h.symbol).toUpperCase(), - holdingName: h.holdingName ?? String(h.symbol), - holdingPercent: Number(h.holdingPercent) || 0, - })); - if (live.length > 0) { - holdings = live; - source = 'live'; - writeCachedComposition(live); - } - } catch { - /* rate limit / timeout / network — keep static or empty */ + // Queue background upgrade — never live Yahoo on the request path (ADR-0009). + if (!ctx.queue.isSourceCoolingDown('yfinance')) { + try { await ctx.queue.queue(`yfinance:topHoldings:${symbol}`); } catch { /* ignore */ } } } @@ -1007,15 +1377,15 @@ const marketRouter = router({ return { symbol, holdings: [], source: 'empty' as const }; } - const symbols = holdings.map((h) => h.symbol.toUpperCase()); + // Cap at top 15 for the peek panel (matches UI) and for quote fan-in. + const top = holdings.slice(0, 15); + const symbols = top.map((h) => h.symbol.toUpperCase()); - // Local quotes only — never open another Yahoo batch from this endpoint. + // Local quotes first — never open live Yahoo on the request path (ADR-0009). const quoteMap = new Map(); try { const placeholders = symbols.map(() => '?').join(','); @@ -1032,24 +1402,45 @@ const marketRouter = router({ price: r.price, change: r.change, changePercent: r.change_pct, - volume: null, - marketCap: null, }); } } catch { /* local quotes optional */ } - const result = holdings.map((h) => { + // Candle close fallback when quote row is missing (still local, no vendor call). + const candleClose = new Map(); + try { + for (const sym of symbols) { + if (quoteMap.get(sym)?.price != null) continue; + const row = ctx.db.prepare( + `SELECT c FROM price_candles WHERE symbol=? AND timeframe='1d' ORDER BY ts DESC LIMIT 1`, + ).get(sym) as { c: number } | undefined; + if (row && Number.isFinite(row.c)) candleClose.set(sym, row.c); + } + } catch { /* ignore */ } + + // Background-queue missing US-style tickers so the next open has live marks. + // Skip exotic listings (dots / long suffixes) to avoid poisoning the queue. + if (!ctx.queue.isSourceCoolingDown('yfinance')) { + for (const sym of symbols) { + if (quoteMap.has(sym) && quoteMap.get(sym)!.price != null) continue; + if (sym.includes('.') || sym.length > 5) continue; + try { await ctx.cache.ensureInDemand(sym, 'equity'); } catch { /* ignore */ } + } + } + + const result = top.map((h) => { const sym = h.symbol.toUpperCase(); const q = quoteMap.get(sym); + const price = q?.price ?? candleClose.get(sym) ?? null; return { symbol: sym, name: h.holdingName, weight: h.holdingPercent, - price: q?.price ?? null, + price, change: q?.change ?? null, changePercent: q?.changePercent ?? null, - volume: q?.volume ?? null, - marketCap: q?.marketCap ?? null, + volume: null as number | null, + marketCap: null as number | null, returns1W: null as number | null, returns1M: null as number | null, returns3M: null as number | null, @@ -1060,6 +1451,36 @@ const marketRouter = router({ return { symbol, holdings: result, source }; }), + + /** Add a custom ETF to the user's rotation tracking. */ + addCustomEtf: publicProcedure + .input(z.object({ symbol: z.string().toUpperCase(), name: z.string().optional(), group: z.string().optional() })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId ?? 'anonymous'; + const existing = ctx.db.prepare('SELECT id FROM rotation_custom_symbols WHERE owner_id=? AND symbol=?').get(userId, input.symbol) as { id: string } | undefined; + if (existing) return { added: false }; + ctx.db.prepare( + 'INSERT INTO rotation_custom_symbols (id, owner_id, symbol, name, grp, created_at) VALUES (?,?,?,?,?,?)' + ).run(crypto.randomUUID(), userId, input.symbol, input.name ?? input.symbol, input.group ?? 'Custom', new Date().toISOString()); + try { await ctx.cache.ensureInDemand(input.symbol, 'etf'); } catch { /* ignore */ } + return { added: true }; + }), + + /** Remove a custom ETF from the user's rotation tracking. */ + removeCustomEtf: publicProcedure + .input(z.object({ symbol: z.string().toUpperCase() })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId ?? 'anonymous'; + const r = ctx.db.prepare('DELETE FROM rotation_custom_symbols WHERE owner_id=? AND symbol=?').run(userId, input.symbol); + return { removed: r.changes > 0 }; + }), + + /** List the user's custom rotation ETFs. */ + listCustomEtfs: publicProcedure.query(async ({ ctx }) => { + const userId = ctx.userId ?? 'anonymous'; + const rows = ctx.db.prepare('SELECT symbol, name, grp, created_at FROM rotation_custom_symbols WHERE owner_id=? ORDER BY symbol').all(userId) as Array<{ symbol: string; name: string | null; grp: string; created_at: string }>; + return rows.map((r) => ({ ...r, name: r.name ?? r.symbol })); + }), }); function parseCandlesFromChart(raw: Record): PriceCandle[] { @@ -1140,6 +1561,9 @@ const adminRouter = router({ queueHealth: adminProcedure.query(({ ctx }) => queueHealth(ctx.db)), + /** Alert-system observability: producer cadence + source freshness. */ + alertStatus: adminProcedure.query(({ ctx }) => alertStatus(ctx.db)), + resetBackoff: adminProcedure .input(z.object({ sourceKind: z.string().regex(/^[a-z0-9_]+$/i) })) .mutation(({ ctx, input }) => resetQueueBackoff(ctx.db, ctx.userId, input.sourceKind)), @@ -1164,7 +1588,88 @@ const adminRouter = router({ catch (e) { throw new TRPCError({ code: 'BAD_REQUEST', message: e instanceof Error ? e.message : 'Failed to queue fetch.' }); } }), + /** Run GEX/VEX integrity audit (and optional as-of replay) for a symbol. */ + dealerMapIntegrity: adminProcedure + .input(z.object({ + symbol: z.string().min(1).max(12), + /** When set, rebuild surface from chains with ts ≤ asOf (backtest). */ + asOf: z.string().min(10).optional(), + recompute: z.boolean().optional().default(true), + })) + .mutation(async ({ ctx, input }) => { + const symbol = input.symbol.trim().toUpperCase(); + const { auditDealerMap, integritySummary } = await import('../analysis/dealerMapIntegrity.ts'); + const { + recomputeDealerMapFromCacheDetailed, + writeDealerMapDataQuality, + } = await import('../analysis/dealerMapService.ts'); + const { replayDealerMapAt } = await import('../analysis/dealerMapReplay.ts'); + + if (input.asOf) { + const replay = replayDealerMapAt(ctx.db, symbol, input.asOf); + writeDealerMapDataQuality(ctx.db, symbol, replay.integrity); + recordAudit(ctx.db, ctx.userId ?? 'admin', 'dealer.integrity', symbol, { + mode: 'replay', + asOf: input.asOf, + status: replay.integrity.status, + ok: replay.ok, + }); + return { + mode: 'replay' as const, + symbol, + asOf: input.asOf, + ok: replay.ok, + summary: replay.summary, + integrity: replay.integrity, + integrityText: integritySummary(replay.integrity), + king: replay.map?.levels.king ?? null, + spot: replay.spot, + spotSource: replay.spotSource, + surfaceRowCount: replay.surfaceRowCount, + snapshotCompare: replay.snapshotCompare, + }; + } + + let map = null as Awaited>['map']; + let integrity = auditDealerMap(null, null); + let keptPriorMap = false; + let message: string | undefined; + if (input.recompute) { + const detail = recomputeDealerMapFromCacheDetailed(ctx.db, symbol, { write: true }); + map = detail.map; + integrity = detail.integrity ?? auditDealerMap(detail.map, detail.surface); + keptPriorMap = detail.keptPriorMap; + message = detail.message; + if (detail.integrity) writeDealerMapDataQuality(ctx.db, symbol, detail.integrity); + } + recordAudit(ctx.db, ctx.userId ?? 'admin', 'dealer.integrity', symbol, { + mode: 'live', + status: integrity.status, + keptPriorMap, + }); + return { + mode: 'live' as const, + symbol, + ok: integrity.status !== 'incomplete', + summary: message ?? integritySummary(integrity), + integrity, + integrityText: integritySummary(integrity), + king: map?.levels.king ?? null, + spot: map?.spot ?? null, + keptPriorMap, + cells: map?.cells.length ?? 0, + quality: map?.quality ?? null, + }; + }), + queueStatus: adminProcedure.query(({ ctx }) => ctx.queue.health()), + /** Force-clear a vendor cool-down (operator recovery after rate-limit thrash). */ + clearSourceCooldown: adminProcedure + .input(z.object({ sourceKind: z.string().min(1) })) + .mutation(({ ctx, input }) => { + ctx.queue.clearSourceCooldownManual(input.sourceKind); + return { ok: true, sourceKind: input.sourceKind }; + }), queuePause: adminProcedure.mutation(({ ctx }) => { ctx.queue.setPaused(true); @@ -1247,22 +1752,12 @@ const adminRouter = router({ queueLintAll: adminProcedure .input(z.object({ kind: z.enum(['sec-lint-holders', 'sec-lint-insiders']) })) .mutation(async ({ ctx, input }) => { - const { lintInstitutionalHolders, lintInsiderTransactions } = await import('../services/secDataFetcher.ts'); const symbols = (ctx.db.prepare('SELECT symbol FROM symbol_demand WHERE in_demand=1 ORDER BY symbol').all() as Array<{ symbol: string }>).map((r) => r.symbol); - const results: LintResult[] = []; + const kindPath = input.kind === 'sec-lint-holders' ? 'holders' : 'insiders'; for (const sym of symbols) { - try { - if (input.kind === 'sec-lint-holders') { - results.push(await lintInstitutionalHolders(ctx.db, sym)); - } else { - results.push(await lintInsiderTransactions(ctx.db, sym)); - } - } catch (e) { - const msg = e instanceof Error ? e.message : String(e); - results.push({ symbol: sym.toUpperCase(), kind: input.kind === 'sec-lint-holders' ? 'institution_filings' : 'insider_transactions', discoveredCount: 0, storedCount: 0, missingCount: 0, backfilled: 0, stale: 1, status: 'error', detail: { reason: msg } }); - } + try { await ctx.queue.queue(`${input.kind}:${kindPath}:${sym}`); } catch { /* ignore */ } } - return { total: symbols.length, results }; + return { total: symbols.length, enqueued: symbols.length }; }), // --------------------------------------------------------------------------- @@ -1476,6 +1971,26 @@ const adminRouter = router({ finraConfigUpdate: adminProcedure .input(z.object({ baseUrl: z.string().url() })) .mutation(({ ctx, input }) => setFinraBaseUrl(ctx.db, input.baseUrl)), + + /** Trigger FINRA 30-day backfill. Skips existing dates, rate-limited 500ms. */ + finraBackfill: adminProcedure + .input(z.object({ days: z.number().int().min(1).max(200).optional().default(30) })) + .mutation(async ({ ctx, input }) => { + const { backfillFinra } = await import('../services/FinraIngestService.ts'); + const result = await backfillFinra(ctx.db, input.days, getFinraBaseUrl(ctx.db)); + recordAudit(ctx.db, ctx.userId!, 'finra.backfill', null, { days: input.days, filesStored: result.filesStored, errors: result.errors.length }); + return result; + }), + + /** Trigger FINRA bi-monthly short interest backfill (~24 months default). */ + finraSiBackfill: adminProcedure + .input(z.object({ months: z.number().int().min(1).max(36).optional().default(24) })) + .mutation(async ({ ctx, input }) => { + const { backfillFinraSi } = await import('../services/FinraIngestService.ts'); + const result = await backfillFinraSi(ctx.db, input.months); + recordAudit(ctx.db, ctx.userId!, 'finra-si.backfill', null, { months: input.months, filesStored: result.filesStored, errors: result.errors.length }); + return result; + }), }); @@ -1629,6 +2144,30 @@ const alertsRouter = router({ if (!deleted) throw new TRPCError({ code: 'NOT_FOUND', message: 'Subscription not found.' }); return { ok: true }; }), + + /** Per-type toggle list: which alert types the user receives, plus labels. */ + listTypes: protectedProcedure + .query(async ({ ctx }) => { + const userId = ctx.userId as string; + const { listAlertTypeToggles } = await import('../db/alertSubscriptionRepository.ts'); + return listAlertTypeToggles(ctx.db, userId); + }), + + /** Toggle a whole alert type on/off for the current user (authoritative). */ + toggleType: protectedProcedure + .input(z.object({ alertType: z.string(), enabled: z.boolean() })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId as string; + const { toggleAlertType } = await import('../db/alertSubscriptionRepository.ts'); + try { + return toggleAlertType(ctx.db, userId, input.alertType, input.enabled); + } catch (e) { + throw new TRPCError({ + code: 'BAD_REQUEST', + message: e instanceof Error ? e.message : 'Failed to toggle alert type.', + }); + } + }), }); // ─── Institutional Flow Router (Slice 7 / M4 + M5) ──────────────────────── @@ -1714,34 +2253,44 @@ const institutionalRouter = router({ }> = []; for (const [cik, cikRows] of byCik) { - // Sort by filed_at descending - const sorted = [...cikRows].sort((a, b) => b.filed_at.localeCompare(a.filed_at)); - if (sorted.length < 2) continue; + // One position per reported quarter (latest filed wins) — not filed_at pair only. + // Prior bug: only compared the two newest rows by file date, skipped intermediate + // quarters, and sometimes spanned non-adjacent periods (false mega-deltas). + const byQuarter = new Map(); + const sortedByFiled = [...cikRows].sort((a, b) => + String(b.filed_at).localeCompare(String(a.filed_at)), + ); + for (const r of sortedByFiled) { + if (!byQuarter.has(r.reported_quarter)) byQuarter.set(r.reported_quarter, r); + } + const quartersAsc = [...byQuarter.keys()].sort((a, b) => a.localeCompare(b)); + for (let i = 1; i < quartersAsc.length; i++) { + const prevQ = quartersAsc[i - 1]; + const currQ = quartersAsc[i]; + const prev = byQuarter.get(prevQ)!; + const curr = byQuarter.get(currQ)!; + const prevShares = prev.shares ?? 0; + const currShares = curr.shares ?? 0; + const delta = currShares - prevShares; + if (delta === 0) continue; - const curr = sorted[0]; - const prev = sorted[1]; - const prevShares = prev.shares ?? 0; - const currShares = curr.shares ?? 0; - const delta = currShares - prevShares; + let classification: string; + if (prevShares === 0 && currShares > 0) classification = 'new position'; + else if (prevShares > 0 && currShares === 0) classification = 'exited'; + else if (delta > 0) classification = 'added to position'; + else classification = 'reduced position'; - if (delta === 0) continue; - - let classification: string; - if (prevShares === 0 && currShares > 0) classification = 'new position'; - else if (prevShares > 0 && currShares === 0) classification = 'exited'; - else if (delta > 0) classification = 'added to position'; - else classification = 'reduced position'; - - allFlow.push({ - filerCik: cik, - filerName: curr.filer_name, - form: curr.form, - prevShares, - currShares, - delta, - classification, - reportedQuarter: curr.reported_quarter, - }); + allFlow.push({ + filerCik: cik, + filerName: curr.filer_name, + form: curr.form, + prevShares, + currShares, + delta, + classification, + reportedQuarter: currQ, + }); + } } allFlow.sort((a, b) => Math.abs(b.delta) - Math.abs(a.delta)); @@ -1761,7 +2310,10 @@ const institutionalRouter = router({ putCall: r.put_call ?? '', })); - return { symbol, flow: allFlow, quarters, filings }; + // Compute institutional ownership % across all tracked funds. + const ownership = computeOwnershipPercentages(ctx.db, symbol); + + return { symbol, flow: allFlow, quarters, filings, ownership }; }), /** Get insider activity stream for a symbol (M5 quarterly price strip). */ @@ -1891,120 +2443,104 @@ const institutionalRouter = router({ return { symbol, events }; }), - /** Analyst ratings (upgrades/downgrades + consensus) from yahoo-finance2. */ + /** Analyst ratings (upgrades/downgrades + consensus). Cache first; live only when Yahoo not cooling down. */ analystRatings: publicProcedure .input(z.object({ symbol: z.string().min(1) })) .query(async ({ ctx, input }) => { const symbol = input.symbol.toUpperCase(); const { fetchAndStoreAnalystRatings, getAnalystRatings } = await import('../services/analystRatingsService.ts'); - // Try cache first const cached = getAnalystRatings(ctx.db, symbol); if (cached && !cached.stale) return { symbol, ratings: cached.ratings, consensus: cached.consensus }; - // Fetch fresh - const result = await fetchAndStoreAnalystRatings(ctx.db, symbol); + // Respect shared yfinance cool-down — never stampede Edge from this panel. + if (ctx.queue.isSourceCoolingDown('yfinance')) { + if (cached) return { symbol, ratings: cached.ratings, consensus: cached.consensus, stale: true }; + return { symbol, ratings: [], consensus: null, error: 'Yahoo cooling down — ratings will refresh when rate limit clears' }; + } + const result = await fetchAndStoreAnalystRatings(ctx.db, symbol, { + onRateLimit: () => { try { ctx.queue.applySourceCooldown('yfinance', 'analystRatings 429'); } catch { /* ignore */ } }, + }); if ('error' in result) { - // If we have cached data, serve it stale rather than erroring if (cached) return { symbol, ratings: cached.ratings, consensus: cached.consensus, stale: true }; return { symbol, ratings: [], consensus: null, error: result.error }; } - return { symbol, ratings: result.ratings, consensus: result.consensus }; - }), + return { symbol, ratings: result.ratings, consensus: result.consensus }; + }), - /** Short interest reconciled across Yahoo Finance + NASDAQ API + FINRA. - * Yahoo covers all US-listed tickers (basic: shares short, % float, short ratio). - * NASDAQ covers NASDAQ-listed only but adds days-to-cover + 24mo history. - * FINRA covers all exchanges via consolidated twice-monthly files. - * Reconciliation hierarchy (most → least authoritative for each field): - * sharesShort: FINRA > NASDAQ > Yahoo - * daysToCover: NASDAQ (specific) > FINRA (computed) > Yahoo (ratio) - * % of Float: Yahoo only (has floatShares) - * Discrepancy detection: flags >10% difference between top-2 available sources. */ + /** Short interest: FINRA bi-monthly (outstanding, regulatory gold) + + * FINRA daily volume (weekly activity chart) + NASDAQ/Yahoo (cross-check). + * All three outstanding sources compared for discrepancy (apples-to-apples). */ shortInterest: publicProcedure .input(z.object({ symbol: z.string().min(1) })) .query(async ({ ctx, input }) => { const symbol = input.symbol.toUpperCase(); - const [yfEntry, ndqEntry, frEntry] = await Promise.all([ + const [yfEntry, ndqEntry, frVolEntry, frSiEntry] = await Promise.all([ ctx.cache.get>(`yfinance:shortinterest:${symbol}`), ctx.cache.get>(`nasdaq:nasdaqShortinterest:${symbol}`), ctx.cache.get>(`finra-bulk:finraShortinterest:${symbol}`), + ctx.cache.get>(`finra-si:finraSiBiweekly:${symbol}`), ]); let yf = yfEntry.value; let ndq = ndqEntry.value; - const fr = frEntry.value; + let frVol = frVolEntry.value; + let frSi = frSiEntry.value; - // Direct fetch when cache is empty — the drain queue can be clogged with - // hundreds of pending jobs, making stale-while-revalidate too slow. - if (!yf && !ndq) { - const { YFinanceAdapter } = await import('../adapters/YFinanceAdapter.ts'); - const { NasdaqAdapter } = await import('../adapters/NasdaqAdapter.ts'); - const results = await Promise.allSettled([ - (async () => { - const ya = new YFinanceAdapter(); - const r = await ya.fetchOne(`yfinance:shortinterest:${symbol}`); - await ctx.cache.set(`yfinance:shortinterest:${symbol}`, r.value, r.ttlClass, r.provenance); - return r.value as Record; - })(), - (async () => { - const na = new NasdaqAdapter(); - const r = await na.fetchOne(`nasdaq:nasdaqShortinterest:${symbol}`); - await ctx.cache.set(`nasdaq:nasdaqShortinterest:${symbol}`, r.value, r.ttlClass, r.provenance); - return r.value as Record; - })(), - ]); - if (results[0].status === 'fulfilled') yf = results[0].value; - if (results[1].status === 'fulfilled') ndq = results[1].value; + // Background refresh via shared queue only — no live Yahoo/Nasdaq on request path. + if (!yf) { + try { await ctx.queue.queue(`yfinance:shortinterest:${symbol}`); } catch { /* ignore */ } } - - // Collect sharesShort from all sources for discrepancy detection - const sharesSources: Array<{ source: string; sharesShort: number | null }> = [ - { source: 'finra', sharesShort: (fr?.shortVolume as number | undefined) ?? null }, - { source: 'nasdaq', sharesShort: ndq?.sharesShort as number | null }, - { source: 'yfinance', sharesShort: yf?.sharesShort as number | null }, - ]; - const availableSources = sharesSources.filter((s) => s.sharesShort !== null); - - let discrepancyPct: number | null = null; - let discrepancyBetween: string | null = null; - if (availableSources.length >= 2) { - const sorted = [...availableSources].sort((a, b) => (b.sharesShort ?? 0) - (a.sharesShort ?? 0)); - const hi = sorted[0].sharesShort!; - const lo = sorted[sorted.length - 1].sharesShort!; - if (lo > 0) { - const pct = Math.abs(hi - lo) / lo; - if (pct > 0.10) { - discrepancyPct = Math.round(pct * 100); - discrepancyBetween = `${sorted[0].source} vs ${sorted[sorted.length - 1].source}`; + if (!ndq) { + try { await ctx.queue.queue(`nasdaq:nasdaqShortinterest:${symbol}`); } catch { /* ignore */ } + } + if (!frVol) { + try { + const row = ctx.db.prepare('SELECT settlement_date,exchange,short_volume,short_exempt,total_volume,days_to_cover FROM finra_short_interest WHERE symbol=? ORDER BY settlement_date DESC LIMIT 1').get(symbol) as Record | undefined; + if (row) { + const v = { + shortVolume: row.short_volume, shortExempt: row.short_exempt, totalVolume: row.total_volume, + settlementDate: row.settlement_date, exchange: row.exchange, daysToCover: row.days_to_cover, + }; + await ctx.cache.set(`finra-bulk:finraShortinterest:${symbol}`, v, 'short_interest', { + fetchedAt: new Date().toISOString(), sourceKind: 'finra-bulk', rawSourceId: symbol, + }); + frVol = v as Record; } - } + } catch { /* ignore */ } + } + if (!frSi) { + try { + const row = ctx.db.prepare('SELECT current_short_position,previous_short_position,avg_daily_volume,days_to_cover,change_percent,settlement_date,ingested_at FROM finra_short_interest_biweekly WHERE symbol=? ORDER BY settlement_date DESC LIMIT 1').get(symbol) as Record | undefined; + if (row) { + const v = { + currentShortPosition: row.current_short_position, previousShortPosition: row.previous_short_position, + avgDailyVolume: row.avg_daily_volume, daysToCover: row.days_to_cover, + changePercent: row.change_percent, settlementDate: row.settlement_date, + }; + await ctx.cache.set(`finra-si:finraSiBiweekly:${symbol}`, v, 'short_interest', { + fetchedAt: new Date().toISOString(), sourceKind: 'finra-si', rawSourceId: symbol, + }); + frSi = v as Record; + } + } catch { /* ignore */ } } - const sharesShort = fr?.shortVolume ?? ndq?.sharesShort ?? yf?.sharesShort ?? null; - const daysToCover = ndq?.daysToCover ?? fr?.daysToCover ?? yf?.shortRatio ?? null; - const settlementDate = fr?.settlementDate ?? ndq?.settlementDate ?? yf?.dateShortInterest ?? null; + const comp = [ {s:'finra',v:(frSi?.currentShortPosition as number|undefined)??null}, {s:'nasdaq',v:ndq?.sharesShort as number|null}, {s:'yfinance',v:yf?.sharesShort as number|null} ].filter(x=>x.v!=null&&x.v>0); + let discPct: number|null=null, discBetw: string|null=null; + if (comp.length>=2){ const s=[...comp].sort((a,b)=>(b.v??0)-(a.v??0)); const hi=s[0].v!, lo=s[s.length-1].v!; const p=Math.abs(hi-lo)/lo; if(p>0.10){ discPct=Math.round(p*100); discBetw=`${s[0].s} vs ${s[s.length-1].s}`; } } + + const sharesShort = frSi?.currentShortPosition ?? ndq?.sharesShort ?? yf?.sharesShort ?? null; + const daysToCover = frSi?.daysToCover ?? ndq?.daysToCover ?? yf?.shortRatio ?? null; + const settlementDate = frSi?.settlementDate ?? ndq?.settlementDate ?? yf?.dateShortInterest ?? null; const history = ndq?.history ?? null; - return { - symbol, - sharesShort, - sharesShortPriorMonth: yf?.sharesShortPriorMonth ?? null, - sharesPercentSharesOut: yf?.sharesPercentSharesOut ?? null, - shortRatio: yf?.shortRatio ?? null, - shortPercentOfFloat: yf?.shortPercentOfFloat ?? null, - dateShortInterest: yf?.dateShortInterest ?? null, - floatShares: yf?.floatShares ?? null, - sharesOutstanding: yf?.sharesOutstanding ?? null, - daysToCover, - settlementDate, - history, - discrepancyPct, - discrepancyBetween, - sources: { - yfinance: { available: !!yf, stale: yfEntry.isStale }, - nasdaq: { available: !!ndq, stale: ndqEntry.isStale }, - finra: { available: !!fr, stale: frEntry.isStale }, - }, - stale: yfEntry.isStale && ndqEntry.isStale, // FINRA is schedule-only, not per-symbol - }; + const biRows = ctx.db.prepare('SELECT settlement_date, current_short_position, previous_short_position, days_to_cover, change_percent FROM finra_short_interest_biweekly WHERE symbol=? AND settlement_date >= date(\'now\',\'-24 months\') ORDER BY settlement_date ASC').all(symbol) as Array<{settlement_date:string;current_short_position:number;previous_short_position:number|null;days_to_cover:number|null;change_percent:number|null}>; + const finraBiweeklyHistory = biRows.map(r=>({settlementDate:r.settlement_date,currentShortPosition:r.current_short_position,previousShortPosition:r.previous_short_position,daysToCover:r.days_to_cover,changePercent:r.change_percent})); + + const volRows = ctx.db.prepare('SELECT settlement_date,short_volume,total_volume FROM finra_short_interest WHERE symbol=? AND settlement_date >= date(\'now\',\'-180 days\') ORDER BY settlement_date ASC').all(symbol) as Array<{settlement_date:string;short_volume:number;total_volume:number}>; + const wm=new Map(); + for (const r of volRows){ const d=new Date(r.settlement_date+'T00:00:00Z'); const mon=d.getUTCDay()===0?6:d.getUTCDay()-1; const we=new Date(d); we.setUTCDate(d.getUTCDate()+mon); const k=we.toISOString().slice(0,10); const e=wm.get(k)||{sv:0,tv:0,e:k}; e.sv+=r.short_volume; e.tv+=r.total_volume; wm.set(k,e); } + const finraWeekly=[...wm.values()].sort((a,b)=>a.e.localeCompare(b.e)).map(w=>({weekEnding:w.e,shortVolume:w.sv,totalVolume:w.tv,shortPct:w.tv>0?Math.round(w.sv/w.tv*10000)/100:0})); + + return { symbol, sharesShort, sharesShortPriorMonth:yf?.sharesShortPriorMonth??null, sharesPercentSharesOut:yf?.sharesPercentSharesOut??null, shortRatio:yf?.shortRatio??null, shortPercentOfFloat:yf?.shortPercentOfFloat??null, dateShortInterest:yf?.dateShortInterest??null, floatShares:yf?.floatShares??null, sharesOutstanding:yf?.sharesOutstanding??null, daysToCover, settlementDate, history, finraBiweeklyHistory, finraWeekly, discrepancyPct:discPct, discrepancyBetween:discBetw, sources:{yfinance:{available:!!yf,stale:yfEntry.isStale},nasdaq:{available:!!ndq,stale:ndqEntry.isStale},finra:{available:!!frSi,stale:frSiEntry.isStale}}, stale:yfEntry.isStale&&ndqEntry.isStale&&frSiEntry.isStale }; }), }); @@ -2224,8 +2760,8 @@ const watchlistRouter = router({ const added = addSymbol(ctx.db, userId, input.symbol, input.notes, input.watchlistName); if (added) { materializeClassificationWatchlists(ctx.db, userId); - await ctx.cache.subscribe(input.symbol, 'equity'); - queueSecFetch(ctx.db, input.symbol); + try { await ctx.cache.subscribe(input.symbol, 'equity'); } catch { /* ignore */ } + try { queueSecFetch(ctx.db, input.symbol); } catch { /* ignore */ } } return { added }; }), @@ -2240,7 +2776,10 @@ const watchlistRouter = router({ const userId = ctx.userId ?? 'anonymous'; const { removeSymbol, materializeClassificationWatchlists } = await import('../db/watchlistRepository.ts'); const removed = removeSymbol(ctx.db, userId, input.symbol, input.watchlistName); - if (removed) materializeClassificationWatchlists(ctx.db, userId); + if (removed) { + materializeClassificationWatchlists(ctx.db, userId); + try { await ctx.cache.unsubscribe(input.symbol); } catch { /* ignore */ } + } return { removed }; }), @@ -2271,22 +2810,55 @@ const portfolioRouter = router({ }), /** Add or accumulate a holding. */ - addHolding: protectedProcedure + addHolding: publicProcedure .input(z.object({ symbol: z.string().toUpperCase(), shares: z.number().positive(), avgCost: z.number().min(0) })) .mutation(async ({ ctx, input }) => { - const userId = ctx.userId as string; + const userId = ctx.userId ?? 'anonymous'; const { addHolding } = await import('../db/portfolioRepository.ts'); const created = addHolding(ctx.db, userId, input.symbol, input.shares, input.avgCost); + try { + const { materializePortfolioWatchlist } = await import('../db/watchlistRepository.ts'); + materializePortfolioWatchlist(ctx.db, userId); + } catch { /* ignore */ } + if (created) { + try { await ctx.cache.subscribe(input.symbol, 'equity'); } catch { /* ignore */ } + try { queueSecFetch(ctx.db, input.symbol); } catch { /* ignore */ } + } return { created }; }), + /** Update an existing holding's shares and/or avg cost. */ + updateHolding: publicProcedure + .input(z.object({ + symbol: z.string().toUpperCase(), + shares: z.number().positive().optional(), + avgCost: z.number().min(0).optional(), + })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId ?? 'anonymous'; + const { updateHolding } = await import('../db/portfolioRepository.ts'); + const updated = updateHolding(ctx.db, userId, input.symbol, { shares: input.shares, avgCost: input.avgCost }); + try { + const { materializePortfolioWatchlist } = await import('../db/watchlistRepository.ts'); + materializePortfolioWatchlist(ctx.db, userId); + } catch { /* ignore */ } + return { updated }; + }), + /** Remove (close) a holding. */ - removeHolding: protectedProcedure + removeHolding: publicProcedure .input(z.object({ symbol: z.string().toUpperCase() })) .mutation(async ({ ctx, input }) => { - const userId = ctx.userId as string; + const userId = ctx.userId ?? 'anonymous'; const { removeHolding } = await import('../db/portfolioRepository.ts'); const removed = removeHolding(ctx.db, userId, input.symbol); + try { + const { materializePortfolioWatchlist } = await import('../db/watchlistRepository.ts'); + materializePortfolioWatchlist(ctx.db, userId); + } catch { /* ignore */ } + if (removed) { + await ctx.cache.unsubscribe(input.symbol); + } return { removed }; }), @@ -2417,6 +2989,1065 @@ const optionsRouter = router({ }), }); +// ─── Dealer Flow (GEX/VEX map) — cache-only reads, schedule-on-miss (ADR-0009) ─── +const dealerMapRouter = router({ + /** Full map + Layer-0 educator. Never hits Yahoo on this path. */ + get: protectedProcedure + .input(z.object({ + symbol: z.string().min(1), + forceSchedule: z.boolean().optional(), + maxExpiries: z.number().int().min(1).max(8).optional(), + /** classic = call+/put−; dealer_inventory = Heatseeker-style sign flip */ + convention: z.enum(['classic_call_pos_put_neg', 'dealer_inventory']).optional(), + })) + .query(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + return getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + { + forceSchedule: input.forceSchedule, + maxExpiries: input.maxExpiries, + convention: input.convention, + }, + ); + }), + + /** Compact levels for Research strip / chips. */ + levels: protectedProcedure + .input(z.object({ symbol: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + if (!result.map) { + return { + symbol: input.symbol.toUpperCase(), + status: result.status, + scheduled: result.scheduled, + levels: null as null, + spot: null as number | null, + delayNote: null as string | null, + providerId: result.providerId, + }; + } + return { + symbol: result.map.symbol, + status: result.status, + scheduled: result.scheduled, + levels: result.map.levels, + spot: result.map.spot, + delayNote: result.map.delayNote, + providerId: result.map.providerId, + regimeAtSpot: result.map.regimeAtSpot, + scenario: result.map.scenario, + patterns: result.map.patterns.map((p) => p.id), + }; + }), + + /** Scenario + Layer-0 educator only. */ + scenario: protectedProcedure + .input(z.object({ symbol: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + return { + symbol: input.symbol.toUpperCase(), + status: result.status, + scenario: result.map?.scenario ?? 'insufficient_data', + regimeAtSpot: result.map?.regimeAtSpot ?? 'mixed', + patterns: result.map?.patterns ?? [], + educator: result.educator, + delayNote: result.map?.delayNote ?? result.message ?? null, + providerId: result.providerId, + }; + }), + + /** Velocity summary when prior snapshot exists. */ + velocity: protectedProcedure + .input(z.object({ symbol: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + return { + symbol: input.symbol.toUpperCase(), + status: result.status, + velocity: result.map?.velocity ?? null, + message: result.map?.velocity + ? null + : 'No prior-day snapshot yet - velocity appears after the first daily recompute.', + }; + }), + + /** L1 beginner explain via user's OpenAI-compatible endpoint (or L0 fallback). */ + explain: protectedProcedure + .input(z.object({ symbol: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + try { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const { explainDealerMap } = await import('../analysis/dealerMapExplain.ts'); + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + if (!result.map) { + return { + ok: false, + source: 'error' as const, + headline: 'No map available', + sections: { + whatMapShows: result.message ?? 'Map not ready yet.', + keyLevels: [] as string[], + scenarioInPlainEnglish: '', + questionsToAsk: [] as string[], + caveats: [result.message ?? ''], + dataFreshness: '', + }, + error: result.message, + }; + } + return await explainDealerMap(ctx.db, ctx.userId!, result.map); + } catch (e) { + // Never surface an uncaught 500 for AI explain - degrade to structured error. + console.error('[dealerMap.explain]', e); + const msg = e instanceof Error ? e.message : String(e); + return { + ok: false, + source: 'error' as const, + headline: 'Interpretation unavailable', + sections: { + whatMapShows: 'The AI explanation step failed. The map itself is unaffected.', + keyLevels: [] as string[], + scenarioInPlainEnglish: '', + questionsToAsk: [ + 'Retry in a moment?', + 'Is your OpenAI-compatible endpoint reachable from Settings → Test?', + ], + caveats: [msg], + dataFreshness: '', + }, + error: msg, + }; + } + }), +}); + +// ─── Dealer Flow Study Desk (educational practice + auto-grade) ──────────── +const HIST_MIN_N = 5; +/** Mentor structure rates only influence rank when enough decided grades exist. */ +const MENTOR_HIST_MIN_N = 20; + +function studyScorecardForUser(db: Context['db'], userId: string) { + const rows = db.prepare( + `SELECT hypothesis_id, grade, score FROM dealer_study_setups WHERE owner_id=?`, + ).all(userId) as Array<{ hypothesis_id: string; grade: string | null; score: number | null }>; + + type Agg = { + hypothesisId: string; + n: number; + wins: number; + losses: number; + expired: number; + incomplete: number; + scoreSum: number; + scoreN: number; + }; + const map = new Map(); + for (const r of rows) { + const a = map.get(r.hypothesis_id) ?? { + hypothesisId: r.hypothesis_id, + n: 0, + wins: 0, + losses: 0, + expired: 0, + incomplete: 0, + scoreSum: 0, + scoreN: 0, + }; + a.n++; + if (r.grade === 'win') a.wins++; + else if (r.grade === 'loss') a.losses++; + else if (r.grade === 'expired') a.expired++; + else a.incomplete++; + if (typeof r.score === 'number') { + a.scoreSum += r.score; + a.scoreN++; + } + map.set(r.hypothesis_id, a); + } + + return [...map.values()].map((a) => { + const decided = a.wins + a.losses; + return { + hypothesisId: a.hypothesisId, + n: a.n, + wins: a.wins, + losses: a.losses, + expired: a.expired, + incomplete: a.incomplete, + decidedN: decided, + winRate: decided > 0 ? a.wins / decided : null, + avgScore: a.scoreN > 0 ? a.scoreSum / a.scoreN : null, + }; + }).sort((x, y) => y.n - x.n); +} + +/** Path-match rates by mentor claim_type (all handles combined for structure type). */ +function mentorClaimScorecard(db: Context['db'], userId: string) { + const rows = db.prepare( + `SELECT claim_type, grade, score FROM mentor_calls + WHERE owner_id=? AND status IN ('confirmed','graded')`, + ).all(userId) as Array<{ claim_type: string; grade: string | null; score: number | null }>; + + type Agg = { + claimType: string; + n: number; + wins: number; + losses: number; + scoreSum: number; + scoreN: number; + }; + const map = new Map(); + for (const r of rows) { + const a = map.get(r.claim_type) ?? { + claimType: r.claim_type, + n: 0, + wins: 0, + losses: 0, + scoreSum: 0, + scoreN: 0, + }; + a.n++; + if (r.grade === 'win') a.wins++; + else if (r.grade === 'loss') a.losses++; + if (typeof r.score === 'number') { + a.scoreSum += r.score; + a.scoreN++; + } + map.set(r.claim_type, a); + } + return [...map.values()].map((a) => { + const decided = a.wins + a.losses; + return { + claimType: a.claimType, + n: a.n, + decidedN: decided, + winRate: decided > 0 ? a.wins / decided : null, + avgScore: a.scoreN > 0 ? a.scoreSum / a.scoreN : null, + }; + }); +} + +const dealerStudyRouter = router({ + /** Propose study setups from current cache-only map (+ hist / optional mentor rank). */ + propose: protectedProcedure + .input(z.object({ + symbol: z.string().min(1), + /** When true, blend mentor structure path-match rates into rank (default false). */ + useMentorHistory: z.boolean().optional(), + })) + .query(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const { proposeStudySetups } = await import('../analysis/dealerStudyEngine.ts'); + const { HYPOTHESIS_TO_CLAIM } = await import('../analysis/mentorCallExtract.ts'); + const useMentor = input.useMentorHistory === true; + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + if (!result.map) { + return { + symbol: input.symbol.toUpperCase(), + proposals: [] as Array[number] & { + histWinRate: number | null; + histN: number; + histAvgScore: number | null; + mentorWinRate: number | null; + mentorN: number; + mentorClaimType: string | null; + rankScore: number; + }>, + message: result.message ?? 'Map not ready yet.', + status: result.status, + useMentorHistory: useMentor, + }; + } + const card = studyScorecardForUser(ctx.db, ctx.userId!); + const byH = new Map(card.map((c) => [c.hypothesisId, c])); + const mentorByClaim = new Map( + mentorClaimScorecard(ctx.db, ctx.userId!).map((c) => [c.claimType, c]), + ); + + const proposals = proposeStudySetups(result.map).map((p) => { + const h = byH.get(p.hypothesisId); + const decidedN = h?.decidedN ?? 0; + const histWinRate = decidedN >= HIST_MIN_N ? h!.winRate : null; + const histAvgScore = decidedN >= HIST_MIN_N ? h!.avgScore : null; + + const claimType = HYPOTHESIS_TO_CLAIM[p.hypothesisId] ?? null; + const m = claimType ? mentorByClaim.get(claimType) : undefined; + const mentorDecided = m?.decidedN ?? 0; + // Always surface mentor stats when available; only blend into rank when toggled + N large enough. + const mentorWinRate = mentorDecided > 0 ? m!.winRate : null; + const mentorN = mentorDecided; + const mentorForRank = + useMentor && mentorDecided >= MENTOR_HIST_MIN_N ? mentorWinRate : null; + + let rankScore = p.confidence; + if (histWinRate != null && mentorForRank != null) { + rankScore = p.confidence * 0.5 + histWinRate * 0.3 + mentorForRank * 0.2; + } else if (histWinRate != null) { + rankScore = p.confidence * 0.6 + histWinRate * 0.4; + } else if (mentorForRank != null) { + rankScore = p.confidence * 0.7 + mentorForRank * 0.3; + } + + return { + ...p, + histWinRate, + histN: decidedN, + histAvgScore, + mentorWinRate, + mentorN, + mentorClaimType: claimType, + rankScore, + }; + }).sort((a, b) => b.rankScore - a.rankScore); + return { + symbol: result.map.symbol, + proposals, + message: null as string | null, + status: result.status, + useMentorHistory: useMentor, + }; + }), + + /** Log a study setup for later auto-grade. */ + log: protectedProcedure + .input(z.object({ + symbol: z.string().min(1), + hypothesisId: z.string().min(1), + title: z.string().min(1), + bias: z.enum(['educational_long', 'educational_short', 'educational_range']), + mapReading: z.string(), + studyQuestion: z.string(), + entry: z.number(), + invalidation: z.number(), + target: z.number(), + horizonDays: z.number().int().min(1).max(30), + confidence: z.number().optional(), + disclaimer: z.string().optional(), + })) + .mutation(async ({ ctx, input }) => { + const { getDealerMap } = await import('../analysis/dealerMapService.ts'); + const result = await getDealerMap( + { db: ctx.db, cache: ctx.cache, queue: ctx.queue }, + input.symbol, + ); + const id = randomUUID(); + const loggedAt = new Date().toISOString(); + const snapshot = result.map + ? JSON.stringify({ + symbol: result.map.symbol, + spot: result.map.spot, + asOf: result.map.asOf, + regimeAtSpot: result.map.regimeAtSpot, + scenario: result.map.scenario, + levels: result.map.levels, + patterns: result.map.patterns, + }) + : '{}'; + const disclaimer = + input.disclaimer ?? + 'Educational observation only.'; + ctx.db.prepare(` + INSERT INTO dealer_study_setups ( + id, owner_id, symbol, hypothesis_id, title, bias, map_reading, study_question, + entry, invalidation, target, horizon_days, confidence, map_snapshot, + status, logged_at, disclaimer + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?, 'open', ?, ?) + `).run( + id, + ctx.userId!, + input.symbol.toUpperCase(), + input.hypothesisId, + input.title, + input.bias, + input.mapReading, + input.studyQuestion, + input.entry, + input.invalidation, + input.target, + input.horizonDays, + input.confidence ?? null, + snapshot, + loggedAt, + disclaimer, + ); + // Ensure daily candles are in demand for later grading (ADR-0009). + try { + await ctx.cache.ensureInDemand(input.symbol.toUpperCase(), 'equity'); + } catch { /* optional */ } + return { ok: true as const, id, loggedAt }; + }), + + list: protectedProcedure + .input(z.object({ + symbol: z.string().optional(), + limit: z.number().int().min(1).max(100).optional(), + }).optional()) + .query(async ({ ctx, input }) => { + const limit = input?.limit ?? 30; + const symbol = input?.symbol?.toUpperCase(); + const rows = symbol + ? ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE owner_id=? AND symbol=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, symbol, limit) + : ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE owner_id=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, limit); + return rows as Array>; + }), + + grade: protectedProcedure + .input(z.object({ id: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const { gradeStudySetup } = await import('../analysis/dealerStudyEngine.ts'); + const row = ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE id=? AND owner_id=?`, + ).get(input.id, ctx.userId!) as { + id: string; + symbol: string; + bias: 'educational_long' | 'educational_short' | 'educational_range'; + entry: number; + invalidation: number; + target: number; + horizon_days: number; + logged_at: string; + } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Study setup not found.' }); + + const candles = ctx.db.prepare( + `SELECT ts,o,h,l,c FROM price_candles WHERE symbol=? AND timeframe='1d' ORDER BY ts ASC`, + ).all(row.symbol) as Array<{ ts: string; o: number; h: number; l: number; c: number }>; + + const result = gradeStudySetup( + { + bias: row.bias, + entry: row.entry, + invalidation: row.invalidation, + target: row.target, + horizonDays: row.horizon_days, + loggedAt: row.logged_at, + }, + candles.map((b) => ({ ts: b.ts, o: b.o, h: b.h, l: b.l, c: b.c })), + ); + + const status = result.grade === 'incomplete' ? 'open' : 'graded'; + ctx.db.prepare(` + UPDATE dealer_study_setups + SET status=?, grade=?, score=?, grade_summary=?, grade_path=?, graded_at=? + WHERE id=? AND owner_id=? + `).run( + status, + result.grade, + result.score, + result.summary, + JSON.stringify(result.pathNotes), + result.gradedAt, + row.id, + ctx.userId!, + ); + + return { ok: true as const, ...result, status }; + }), + + gradeDue: protectedProcedure.mutation(async ({ ctx }) => { + const { gradeStudySetup } = await import('../analysis/dealerStudyEngine.ts'); + const open = ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE owner_id=? AND (status='open' OR grade='incomplete')`, + ).all(ctx.userId!) as Array<{ + id: string; + symbol: string; + bias: 'educational_long' | 'educational_short' | 'educational_range'; + entry: number; + invalidation: number; + target: number; + horizon_days: number; + logged_at: string; + }>; + + let graded = 0; + let incomplete = 0; + for (const row of open) { + const candles = (ctx.db.prepare( + `SELECT ts,o,h,l,c FROM price_candles WHERE symbol=? AND timeframe='1d' ORDER BY ts ASC`, + ).all(row.symbol) as Array<{ ts: string; o: number; h: number; l: number; c: number }>); + const result = gradeStudySetup( + { + bias: row.bias, + entry: row.entry, + invalidation: row.invalidation, + target: row.target, + horizonDays: row.horizon_days, + loggedAt: row.logged_at, + }, + candles, + ); + const status = result.grade === 'incomplete' ? 'open' : 'graded'; + if (status === 'graded') graded++; + else incomplete++; + ctx.db.prepare(` + UPDATE dealer_study_setups + SET status=?, grade=?, score=?, grade_summary=?, grade_path=?, graded_at=? + WHERE id=? AND owner_id=? + `).run( + status, + result.grade, + result.score, + result.summary, + JSON.stringify(result.pathNotes), + result.gradedAt, + row.id, + ctx.userId!, + ); + } + return { ok: true as const, considered: open.length, graded, incomplete }; + }), + + scorecard: protectedProcedure.query(async ({ ctx }) => { + return studyScorecardForUser(ctx.db, ctx.userId!).map(({ decidedN: _d, ...rest }) => rest); + }), + + /** Copy a study into journal as a planned trade draft (not an order). */ + promoteToJournal: protectedProcedure + .input(z.object({ studyId: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const row = ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE id=? AND owner_id=?`, + ).get(input.studyId, ctx.userId!) as { + id: string; + symbol: string; + hypothesis_id: string; + title: string; + bias: string; + map_reading: string; + study_question: string; + entry: number; + invalidation: number; + target: number; + horizon_days: number; + } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Study setup not found.' }); + + const tradeId = randomUUID(); + const date = new Date().toISOString().slice(0, 10); + const targets = JSON.stringify([ + { price: row.target, reason: `Study target (${row.hypothesis_id})` }, + ]); + const justifications = JSON.stringify({ + entryWhy: row.map_reading, + stopWhy: `Study invalidation for ${row.hypothesis_id}`, + targetWhy: row.study_question, + studySetupId: row.id, + educational: true, + note: 'Draft from Study Desk.', + }); + const env = + row.bias === 'educational_range' + ? 'ranging' + : row.bias === 'educational_long' + ? 'trending' + : 'transitioning'; + + ctx.db.prepare(` + INSERT INTO trades ( + id, owner_id, trade_id, symbol, date, status, environment, tier, + confluence_score, risk_pct, position_size, entry_price, stop_loss, + targets, justifications, checklist_completed + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,0) + `).run( + tradeId, + ctx.userId!, + tradeId, + row.symbol, + date, + 'planned', + env, + 'B', + null, + null, + null, + row.entry, + row.invalidation, + targets, + justifications, + ); + + return { + ok: true as const, + tradeId, + message: + 'Journal draft created (planned). Review in Plan/Journal - not an order to trade real money.', + }; + }), + + /** Owner-scoped CSV of study rows for offline review. */ + exportCsv: protectedProcedure + .input(z.object({ + symbol: z.string().optional(), + limit: z.number().int().min(1).max(2000).optional(), + }).optional()) + .query(async ({ ctx, input }) => { + const limit = input?.limit ?? 500; + const symbol = input?.symbol?.toUpperCase(); + const rows = symbol + ? ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE owner_id=? AND symbol=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, symbol, limit) + : ctx.db.prepare( + `SELECT * FROM dealer_study_setups WHERE owner_id=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, limit); + + const cols = [ + 'id', 'symbol', 'hypothesis_id', 'title', 'bias', 'entry', 'invalidation', 'target', + 'horizon_days', 'confidence', 'status', 'grade', 'score', 'grade_summary', 'logged_at', 'graded_at', + ]; + const escape = (v: unknown) => { + if (v == null) return ''; + const s = String(v); + if (/[",\n]/.test(s)) return `"${s.replace(/"/g, '""')}"`; + return s; + }; + const lines = [cols.join(',')]; + for (const r of rows as Array>) { + lines.push(cols.map((c) => escape(r[c])).join(',')); + } + return { + filename: `dealer-study-${symbol ?? 'all'}-${new Date().toISOString().slice(0, 10)}.csv`, + csv: lines.join('\n'), + rowCount: rows.length, + }; + }), +}); + +// ─── Mentor call ledger (local path-match grades; privacy-first) ─────────── +const mentorLedgerRouter = router({ + /** Scan local harvest raw/ for draft mentor calls (no cloud LLM). */ + importFromHarvest: protectedProcedure + .input(z.object({ + handle: z.string().optional(), + maxFiles: z.number().int().min(1).max(500).optional(), + }).optional()) + .mutation(async ({ ctx, input }) => { + const { readdirSync, readFileSync, existsSync, statSync } = await import('node:fs'); + const { join } = await import('node:path'); + const { extractMentorCallFromMarkdown } = await import('../analysis/mentorCallExtract.ts'); + const rawRoot = join(process.cwd(), 'data', 'dealer-flow-raw', 'by-handle'); + if (!existsSync(rawRoot)) { + return { ok: true as const, imported: 0, skipped: 0, message: 'No harvest folder yet. Run npm run dealer-flow:harvest first.' }; + } + + const handles = input?.handle + ? [input.handle.replace(/^@/, '')] + : readdirSync(rawRoot).filter((h) => h !== '_template' && !h.startsWith('.')); + const maxFiles = input?.maxFiles ?? 200; + let imported = 0; + let skipped = 0; + let filesSeen = 0; + const now = new Date().toISOString(); + + for (const handle of handles) { + const dir = join(rawRoot, handle); + if (!existsSync(dir) || !statSync(dir).isDirectory()) continue; + ctx.db.prepare( + `INSERT INTO mentor_sources (handle, enabled, notes, updated_at) VALUES (?,?,?,?) + ON CONFLICT(handle) DO UPDATE SET updated_at=excluded.updated_at`, + ).run(handle, 1, 'Imported from local harvest', now); + + const files = readdirSync(dir).filter((f) => f.endsWith('.md')).slice(0, maxFiles); + for (const f of files) { + if (filesSeen >= maxFiles) break; + filesSeen++; + const path = join(dir, f); + let md = ''; + try { + md = readFileSync(path, 'utf8'); + } catch { + skipped++; + continue; + } + const draft = extractMentorCallFromMarkdown(md, { handle, rawPath: path }); + if (!draft || !draft.symbol || draft.bias === 'unclear') { + skipped++; + continue; + } + const id = randomUUID(); + try { + const run = ctx.db.prepare(` + INSERT INTO mentor_calls ( + id, owner_id, handle, post_id, posted_at, symbol, bias, claim_type, + entry, invalidation, target, horizon_days, excerpt, extract_confidence, + status, logged_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?, 'draft', ?) + ON CONFLICT(owner_id, handle, post_id) DO NOTHING + `).run( + id, + ctx.userId!, + draft.handle, + draft.postId, + draft.postedAt, + draft.symbol, + draft.bias, + draft.claimType, + draft.entry, + draft.invalidation, + draft.target, + draft.horizonDays, + draft.excerpt, + draft.extractConfidence, + now, + ) as { changes?: number }; + if ((run.changes ?? 0) > 0) imported++; + else skipped++; + } catch { + skipped++; + } + } + } + return { ok: true as const, imported, skipped, filesSeen, message: null as string | null }; + }), + + list: protectedProcedure + .input(z.object({ + handle: z.string().optional(), + status: z.string().optional(), + limit: z.number().int().min(1).max(200).optional(), + }).optional()) + .query(async ({ ctx, input }) => { + const limit = input?.limit ?? 50; + const handle = input?.handle?.replace(/^@/, ''); + const status = input?.status; + if (handle && status) { + return ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE owner_id=? AND handle=? AND status=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, handle, status, limit); + } + if (handle) { + return ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE owner_id=? AND handle=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, handle, limit); + } + return ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE owner_id=? ORDER BY logged_at DESC LIMIT ?`, + ).all(ctx.userId!, limit); + }), + + /** Confirm draft so it can be graded. */ + confirm: protectedProcedure + .input(z.object({ + id: z.string().min(1), + entry: z.number().optional(), + invalidation: z.number().optional(), + target: z.number().optional(), + bias: z.enum(['educational_long', 'educational_short', 'educational_range']).optional(), + horizonDays: z.number().int().min(1).max(30).optional(), + symbol: z.string().optional(), + })) + .mutation(async ({ ctx, input }) => { + const row = ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE id=? AND owner_id=?`, + ).get(input.id, ctx.userId!) as Record | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Mentor call not found.' }); + ctx.db.prepare(` + UPDATE mentor_calls SET + status='confirmed', + entry=COALESCE(?, entry), + invalidation=COALESCE(?, invalidation), + target=COALESCE(?, target), + bias=COALESCE(?, bias), + horizon_days=COALESCE(?, horizon_days), + symbol=COALESCE(?, symbol) + WHERE id=? AND owner_id=? + `).run( + input.entry ?? null, + input.invalidation ?? null, + input.target ?? null, + input.bias ?? null, + input.horizonDays ?? null, + input.symbol?.toUpperCase() ?? null, + input.id, + ctx.userId!, + ); + return { ok: true as const }; + }), + + discard: protectedProcedure + .input(z.object({ id: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + ctx.db.prepare( + `UPDATE mentor_calls SET status='discarded' WHERE id=? AND owner_id=?`, + ).run(input.id, ctx.userId!); + return { ok: true as const }; + }), + + grade: protectedProcedure + .input(z.object({ id: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const { gradeStudySetup } = await import('../analysis/dealerStudyEngine.ts'); + const row = ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE id=? AND owner_id=?`, + ).get(input.id, ctx.userId!) as { + id: string; + symbol: string | null; + bias: 'educational_long' | 'educational_short' | 'educational_range'; + entry: number; + invalidation: number; + target: number; + horizon_days: number; + posted_at: string | null; + logged_at: string; + status: string; + } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Mentor call not found.' }); + if (row.status === 'discarded') { + throw new TRPCError({ code: 'BAD_REQUEST', message: 'Call was discarded.' }); + } + if (!row.symbol) { + throw new TRPCError({ code: 'BAD_REQUEST', message: 'Symbol required before grade.' }); + } + + const loggedAt = row.posted_at && row.posted_at.length >= 10 + ? (row.posted_at.includes('T') ? row.posted_at : `${row.posted_at}T00:00:00.000Z`) + : row.logged_at; + + try { + await ctx.cache.ensureInDemand(row.symbol, 'equity'); + } catch { /* optional */ } + + const candles = ctx.db.prepare( + `SELECT ts,o,h,l,c FROM price_candles WHERE symbol=? AND timeframe='1d' ORDER BY ts ASC`, + ).all(row.symbol) as Array<{ ts: string; o: number; h: number; l: number; c: number }>; + + const result = gradeStudySetup( + { + bias: row.bias, + entry: row.entry, + invalidation: row.invalidation, + target: row.target, + horizonDays: row.horizon_days, + loggedAt, + }, + candles, + ); + const status = result.grade === 'incomplete' ? 'confirmed' : 'graded'; + ctx.db.prepare(` + UPDATE mentor_calls + SET status=?, grade=?, score=?, grade_summary=?, grade_path=?, graded_at=? + WHERE id=? AND owner_id=? + `).run( + status, + result.grade, + result.score, + result.summary, + JSON.stringify(result.pathNotes), + result.gradedAt, + row.id, + ctx.userId!, + ); + return { ok: true as const, ...result, status }; + }), + + gradeDue: protectedProcedure.mutation(async ({ ctx }) => { + const { gradeStudySetup } = await import('../analysis/dealerStudyEngine.ts'); + const open = ctx.db.prepare( + `SELECT * FROM mentor_calls WHERE owner_id=? AND status IN ('confirmed','graded') AND (grade IS NULL OR grade='incomplete')`, + ).all(ctx.userId!) as Array<{ + id: string; + symbol: string | null; + bias: 'educational_long' | 'educational_short' | 'educational_range'; + entry: number; + invalidation: number; + target: number; + horizon_days: number; + posted_at: string | null; + logged_at: string; + }>; + let graded = 0; + let incomplete = 0; + for (const row of open) { + if (!row.symbol) { + incomplete++; + continue; + } + const loggedAt = row.posted_at && row.posted_at.length >= 10 + ? (row.posted_at.includes('T') ? row.posted_at : `${row.posted_at.slice(0, 10)}T00:00:00.000Z`) + : row.logged_at; + const candles = ctx.db.prepare( + `SELECT ts,o,h,l,c FROM price_candles WHERE symbol=? AND timeframe='1d' ORDER BY ts ASC`, + ).all(row.symbol) as Array<{ ts: string; o: number; h: number; l: number; c: number }>; + const result = gradeStudySetup( + { + bias: row.bias, + entry: row.entry, + invalidation: row.invalidation, + target: row.target, + horizonDays: row.horizon_days, + loggedAt, + }, + candles, + ); + const status = result.grade === 'incomplete' ? 'confirmed' : 'graded'; + if (status === 'graded') graded++; + else incomplete++; + ctx.db.prepare(` + UPDATE mentor_calls + SET status=?, grade=?, score=?, grade_summary=?, grade_path=?, graded_at=? + WHERE id=? AND owner_id=? + `).run( + status, + result.grade, + result.score, + result.summary, + JSON.stringify(result.pathNotes), + result.gradedAt, + row.id, + ctx.userId!, + ); + } + return { ok: true as const, considered: open.length, graded, incomplete }; + }), + + scorecard: protectedProcedure + .input(z.object({ handle: z.string().optional() }).optional()) + .query(async ({ ctx, input }) => { + const handle = input?.handle?.replace(/^@/, ''); + const rows = handle + ? ctx.db.prepare( + `SELECT handle, claim_type, grade, score FROM mentor_calls WHERE owner_id=? AND handle=?`, + ).all(ctx.userId!, handle) + : ctx.db.prepare( + `SELECT handle, claim_type, grade, score FROM mentor_calls WHERE owner_id=?`, + ).all(ctx.userId!); + + type Agg = { + key: string; + handle: string; + claimType: string; + n: number; + wins: number; + losses: number; + expired: number; + incomplete: number; + scoreSum: number; + scoreN: number; + }; + const map = new Map(); + for (const r of rows as Array<{ handle: string; claim_type: string; grade: string | null; score: number | null }>) { + const key = `${r.handle}::${r.claim_type}`; + const a = map.get(key) ?? { + key, + handle: r.handle, + claimType: r.claim_type, + n: 0, + wins: 0, + losses: 0, + expired: 0, + incomplete: 0, + scoreSum: 0, + scoreN: 0, + }; + a.n++; + if (r.grade === 'win') a.wins++; + else if (r.grade === 'loss') a.losses++; + else if (r.grade === 'expired') a.expired++; + else a.incomplete++; + if (typeof r.score === 'number') { + a.scoreSum += r.score; + a.scoreN++; + } + map.set(key, a); + } + return [...map.values()].map((a) => { + const decided = a.wins + a.losses; + return { + handle: a.handle, + claimType: a.claimType, + n: a.n, + wins: a.wins, + losses: a.losses, + expired: a.expired, + incomplete: a.incomplete, + winRate: decided > 0 ? a.wins / decided : null, + avgScore: a.scoreN > 0 ? a.scoreSum / a.scoreN : null, + }; + }).sort((x, y) => y.n - x.n); + }), +}); + +// ─── Per-user OpenAI-compatible LLM endpoint ─────────────────────────────── +const userLlmRouter = router({ + status: protectedProcedure.query(async ({ ctx }) => { + const { getUserLlmStatus } = await import('../llm/userLlmEndpoint.ts'); + return getUserLlmStatus(ctx.db, ctx.userId!); + }), + + upsertEndpoint: protectedProcedure + .input(z.object({ + baseUrl: z.string().min(1), + apiKey: z.string().optional(), + model: z.string().optional(), + keepExistingKey: z.boolean().optional(), + })) + .mutation(async ({ ctx, input }) => { + const { upsertUserLlmEndpoint } = await import('../llm/userLlmEndpoint.ts'); + return upsertUserLlmEndpoint(ctx.db, ctx.userId!, { + baseUrl: input.baseUrl, + apiKey: input.apiKey, + model: input.model, + keepExistingKey: input.keepExistingKey ?? true, + }); + }), + + clear: protectedProcedure.mutation(async ({ ctx }) => { + const { clearUserLlmEndpoint } = await import('../llm/userLlmEndpoint.ts'); + clearUserLlmEndpoint(ctx.db, ctx.userId!); + return { ok: true as const }; + }), + + test: protectedProcedure + .input(z.object({ + baseUrl: z.string().min(1).optional(), + apiKey: z.string().optional(), + model: z.string().optional(), + }).optional()) + .mutation(async ({ ctx, input }) => { + const { loadUserLlmSecret, getUserLlmStatus } = await import('../llm/userLlmEndpoint.ts'); + const { testEndpoint } = await import('../llm/openaiCompatible.ts'); + const saved = loadUserLlmSecret(ctx.db, ctx.userId!); + const baseUrl = input?.baseUrl?.trim() || saved?.baseUrl; + if (!baseUrl) return { ok: false as const, error: 'No base URL configured' }; + const apiKey = input?.apiKey !== undefined && input.apiKey !== '' + ? input.apiKey + : saved?.apiKey ?? null; + const model = input?.model?.trim() || saved?.model || 'gpt-4o-mini'; + const result = await testEndpoint({ baseUrl, apiKey, model }); + if (result.ok) { + // touch status read for UI + getUserLlmStatus(ctx.db, ctx.userId!); + } + return result; + }), +}); + const reportsRouter = router({ /** Generate a research note report. */ generate: protectedProcedure @@ -2504,6 +4135,193 @@ const strategyRouter = router({ ctx.db.prepare('INSERT INTO strategies (id, owner_id, name, components, unlocked, created_at) VALUES (?,?,?,?,?,?)').run(id, userId, input.name, JSON.stringify(input.components), 0, new Date().toISOString()); return { id }; }), + + listPresets: protectedProcedure.query(async ({ ctx }) => { + return ctx.db.prepare('SELECT * FROM strategy_presets ORDER BY created_at ASC').all(); + }), + + get: protectedProcedure + .input(z.object({ id: z.string() })) + .query(async ({ ctx, input }) => { + const userId = ctx.userId as string; + const row = ctx.db.prepare('SELECT * FROM strategies WHERE id=? AND owner_id=?').get(input.id, userId); + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Strategy not found.' }); + return row; + }), + + getPreset: protectedProcedure + .input(z.object({ id: z.string() })) + .query(async ({ ctx, input }) => { + const preset = ctx.db.prepare('SELECT * FROM strategy_presets WHERE id=?').get(input.id); + if (!preset) throw new TRPCError({ code: 'NOT_FOUND', message: 'Preset not found.' }); + return preset; + }), + + forkPreset: protectedProcedure + .input(z.object({ presetId: z.string() })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId as string; + const preset = ctx.db.prepare('SELECT * FROM strategy_presets WHERE id=?').get(input.presetId) as Record | undefined; + if (!preset) throw new TRPCError({ code: 'NOT_FOUND', message: 'Preset not found.' }); + const id = randomUUID(); + const now = new Date().toISOString(); + ctx.db.prepare(` + INSERT INTO strategies (id, owner_id, name, preset_id, scope, regime_gate, setup, components, exit_profile, risk_policy, is_public, created_at) + VALUES (?,?,?,?,?,?,?,?,?,?,0,?) + `).run(id, userId, preset.name, preset.id, preset.scope, 'any', preset.entry_rules, preset.entry_rules, preset.exit_profile, preset.risk_policy, now); + return { id, name: preset.name }; + }), + + suggestTickers: protectedProcedure + .input(z.object({ presetId: z.string() })) + .query(async ({ ctx, input }) => { + const preset = ctx.db.prepare('SELECT * FROM strategy_presets WHERE id=?').get(input.presetId) as Record | undefined; + if (!preset) throw new TRPCError({ code: 'NOT_FOUND', message: 'Preset not found.' }); + + // Load available data for live suggestions + const rotationData = ctx.db.prepare(` + SELECT symbol, kind, rs_1m, rs_1w, leadership FROM rotation_rank_snapshots + WHERE date = (SELECT MAX(date) FROM rotation_rank_snapshots) + ORDER BY rs_1m DESC + `).all() as Array<{ symbol: string; kind: string; rs_1m: number | null; rs_1w: number | null; leadership: string }>; + + const symbolsData = ctx.db.prepare('SELECT symbol, name, sector, ticker_kind FROM symbols').all() as Array<{ symbol: string; name: string | null; sector: string | null; ticker_kind: string }>; + const symbolsMap = new Map(symbolsData.map(s => [s.symbol, s])); + + const dividendData = ctx.db.prepare('SELECT symbol, dividend_yield, payout_ratio, growth_streak_years FROM dividend_fundamentals').all() as Array<{ symbol: string; dividend_yield: number | null; payout_ratio: number | null; growth_streak_years: number | null }>; + const dividendMap = new Map(dividendData.map(d => [d.symbol, d])); + + // Helper: get symbol name from map or return null + const nameOf = (s: string): string | null => symbolsMap.get(s)?.name ?? null; + + // Helper: get tickers in leading rotation sectors + const leadingSectors = new Set(rotationData.filter(r => r.leadership === 'leading').map(r => r.symbol)); + const rotatedSymbols = rotationData.map(r => r.symbol); + + // Helper: find equities in a sector from symbols table (limited to the 102 we track) + const stocksInSectors = (sectors: string[]): string[] => + symbolsData.filter(s => s.ticker_kind === 'equity' && s.sector && sectors.some(sec => s.sector!.includes(sec))).map(s => s.symbol); + + interface Suggestion { symbol: string; name: string | null; role: string; allocationPct: number; reason: string; source: 'live' | 'default'; } + + let suggestions: Suggestion[] = []; + + switch (input.presetId) { + case 'dca_accumulation': { + // Top ETFs + top large-cap stocks from rotation leaders + const etfs = symbolsData.filter(s => s.ticker_kind === 'etf').slice(0, 3); + suggestions = etfs.map((e, i) => ({ + symbol: e.symbol, name: e.name, role: 'Core holding', + allocationPct: [60, 25, 15][i] ?? 10, + reason: 'Broad market ETF - low cost diversification', + source: 'live' as const, + })); + if (suggestions.length === 0) { + suggestions = [ + { symbol: 'VOO', name: 'S&P 500 ETF', role: 'Core holding', allocationPct: 60, reason: 'Broad market low-cost ETF', source: 'default' }, + { symbol: 'IVV', name: 'S&P 500 ETF (iShares)', role: 'Core holding', allocationPct: 40, reason: 'Alternative broad market ETF', source: 'default' }, + ]; + } + break; + } + case 'dividend_growth': { + // Live: query dividend_fundamentals for yield >= 2%, payout <= 60% + const candidates = dividendData.filter(d => + d.dividend_yield != null && d.dividend_yield >= 2 && + d.payout_ratio != null && d.payout_ratio <= 60 + ).sort((a, b) => (b.growth_streak_years ?? 0) - (a.growth_streak_years ?? 0)).slice(0, 4); + if (candidates.length > 0) { + const pcts = [40, 25, 20, 15]; + suggestions = candidates.map((c, i) => ({ + symbol: c.symbol, name: nameOf(c.symbol), role: 'Dividend growth', + allocationPct: pcts[i] ?? 10, + reason: c.growth_streak_years ? `${c.growth_streak_years}+ year dividend growth streak` : 'Stable dividend payer', + source: 'live' as const, + })); + } else { + suggestions = [ + { symbol: 'SCHD', name: 'Schwab US Dividend Equity ETF', role: 'Dividend growth', allocationPct: 40, reason: 'Strong dividend growth history', source: 'default' }, + { symbol: 'JNJ', name: 'Johnson & Johnson', role: 'Dividend growth', allocationPct: 20, reason: '60+ years of dividend growth', source: 'default' }, + { symbol: 'KO', name: 'Coca-Cola', role: 'Dividend growth', allocationPct: 20, reason: '60+ years of dividend growth', source: 'default' }, + { symbol: 'PG', name: 'Procter & Gamble', role: 'Dividend growth', allocationPct: 20, reason: '130+ years of dividends', source: 'default' }, + ]; + } + break; + } + case 'core_satellite': { + // Core: fixed ETFs. Satellite: top rotation ETFs + top stock in leading sector. + const coreEtfs = symbolsData.filter(s => ['VOO', 'VTI', 'IVV', 'VXUS'].includes(s.symbol)); + suggestions = coreEtfs.length > 0 ? coreEtfs.slice(0, 2).map((e, i) => ({ + symbol: e.symbol, name: e.name, role: 'Core', allocationPct: [50, 20][i] ?? 15, + reason: i === 0 ? 'Broad market core holding' : 'International diversification', source: 'live' as const, + })) : [ + { symbol: 'VOO', name: 'S&P 500 ETF', role: 'Core', allocationPct: 50, reason: 'Broad market core holding', source: 'default' }, + { symbol: 'VXUS', name: 'Total International Stock ETF', role: 'Core', allocationPct: 20, reason: 'International diversification', source: 'default' }, + ]; + // Satellite: top ETF from rotation leaders + const topRotated = rotationData.filter(r => r.leadership === 'leading' || r.rs_1m != null).slice(0, 2); + suggestions.push(...topRotated.map((r, i) => ({ + symbol: r.symbol, name: nameOf(r.symbol) ?? r.symbol, role: i === 0 ? 'Satellite - Growth' : 'Satellite - Thematic', + allocationPct: [15, 15][i] ?? 10, + reason: r.rs_1m ? `Sector momentum: 1M RS ${r.rs_1m > 0 ? '+' : ''}${r.rs_1m.toFixed(1)}pp` : 'Rotation leader', + source: 'live' as const, + }))); + if (suggestions.length < 4) { + suggestions.push({ symbol: 'QQQ', name: 'Nasdaq 100 ETF', role: 'Satellite - Growth', allocationPct: 15, reason: 'Tech-heavy growth exposure', source: 'default' }); + } + break; + } + case 'barbell_strategy': { + // Safe core + bond ETF + top growth stocks from rotation + suggestions = [ + { symbol: 'VOO', name: nameOf('VOO') ?? 'S&P 500 ETF', role: 'Safe core', allocationPct: 50, reason: 'Broad market core', source: 'live' as const }, + { symbol: 'BND', name: nameOf('BND') ?? 'Total Bond Market ETF', role: 'Safe bonds', allocationPct: 30, reason: 'Bond diversification', source: 'live' as const }, + ]; + // Growth picks: top equities from symbols in leading sectors, or fallback + const growthPicks = symbolsData.filter(s => + s.ticker_kind === 'equity' && s.sector && leadingSectors.size > 0 && + Array.from(leadingSectors).some(ls => s.sector!.includes(ls.replace(' ', '/'))) + ).slice(0, 2); + if (growthPicks.length >= 2) { + suggestions.push({ symbol: growthPicks[0].symbol, name: growthPicks[0].name, role: 'Growth bet', allocationPct: 10, reason: 'Leading sector momentum', source: 'live' }); + suggestions.push({ symbol: growthPicks[1].symbol, name: growthPicks[1].name, role: 'Growth bet', allocationPct: 10, reason: 'Sector rotation leader', source: 'live' }); + } else { + suggestions.push({ symbol: 'NVDA', name: 'NVIDIA Corporation', role: 'Growth bet', allocationPct: 10, reason: 'High-growth tech leader', source: 'default' }); + suggestions.push({ symbol: 'AMZN', name: 'Amazon.com Inc', role: 'Growth bet', allocationPct: 10, reason: 'High-growth consumer/cloud leader', source: 'default' }); + } + break; + } + case 'covered_call_wheel': { + // Large-cap liquid stocks with active options + const liquidStocks = symbolsData.filter(s => s.ticker_kind === 'equity' && ['AAPL', 'MSFT', 'NVDA', 'AMZN', 'GOOGL', 'META', 'TSLA'].includes(s.symbol)).slice(0, 4); + if (liquidStocks.length >= 4) { + const pcts = [30, 30, 20, 20]; + suggestions = liquidStocks.map((s, i) => ({ + symbol: s.symbol, name: s.name, role: 'Covered call', + allocationPct: pcts[i] ?? 15, + reason: 'High liquidity, strong options market', source: 'live' as const, + })); + } else { + suggestions = [ + { symbol: 'AAPL', name: 'Apple Inc.', role: 'Covered call', allocationPct: 30, reason: 'High liquidity, strong options market', source: 'default' }, + { symbol: 'MSFT', name: 'Microsoft Corporation', role: 'Covered call', allocationPct: 30, reason: 'High liquidity, stable price', source: 'default' }, + { symbol: 'NVDA', name: 'NVIDIA Corporation', role: 'Covered call', allocationPct: 20, reason: 'Active options chain', source: 'default' }, + { symbol: 'AMZN', name: 'Amazon.com Inc', role: 'Covered call', allocationPct: 20, reason: 'High liquidity options market', source: 'default' }, + ]; + } + break; + } + default: { + // Fallback: use example_tickers from preset + const examples = preset.example_tickers as string | null; + if (examples) { + try { suggestions = JSON.parse(examples); } catch { suggestions = []; } + } + } + } + + return suggestions; + }), }); const backtestRouter = router({ @@ -2538,6 +4356,28 @@ const backtestRouter = router({ const engine = new BacktestEngine(); return engine.evaluateLatest(strategy, candles as any[]); }), + + runPortfolio: protectedProcedure + .input(z.object({ + allocations: z.array(z.object({ symbol: z.string(), targetPct: z.number() })), + rebalanceFreq: z.enum(['none', 'quarterly', 'annual']).default('quarterly'), + driftThreshold: z.number().default(5), + })) + .query(async ({ ctx, input }) => { + const { PortfolioBacktestEngine } = await import('../strategy/PortfolioBacktestEngine.ts'); + const candles: Record = {}; + for (const a of input.allocations) { + const entry = await ctx.cache.get(`yfinance:candles:${a.symbol}:1d`); + candles[a.symbol] = entry.value ?? []; + } + const engine = new PortfolioBacktestEngine(); + return engine.run({ + allocations: input.allocations, + candles, + rebalanceFreq: input.rebalanceFreq, + driftThreshold: input.driftThreshold, + }); + }), }); // ─── Sector Crosslink Router (Slice 14) ───────────────────────────────────── @@ -2565,31 +4405,6 @@ const sectorCrosslinkRouter = router({ }), }); -// ─── Options Convexity Router (Slice 19) ──────────────────────────────────── - -const optionsConvexityRouter = router({ - unlock: protectedProcedure - .input(z.object({ fromState: z.number().int().min(0).max(4), toState: z.number().int().min(0).max(4) })) - .mutation(async ({ ctx, input }) => { - const { canElevate } = await import('../options/ConvexityGate.ts'); - const userId = ctx.userId as string; - const row = ctx.db.prepare('SELECT state, understanding FROM options_unlock WHERE user_id=?').get(userId) as { state: number; understanding: number } | undefined; - const currentState = (row?.state ?? 0); - const hasUnderstanding = (row?.understanding ?? 0) === 1; - const result = canElevate(currentState as OptionsUnlockState, input.toState as OptionsUnlockState, hasUnderstanding, true, input.toState >= 3, input.toState === 4); - if (!result.allowed) throw new TRPCError({ code: 'BAD_REQUEST', message: result.reason }); - ctx.db.prepare('INSERT OR REPLACE INTO options_unlock (user_id, state, last_unlock_at, understanding) VALUES (?,?,?,?)').run(userId, input.toState, new Date().toISOString(), row?.understanding ?? 0); - return { state: input.toState, reason: result.reason }; - }), - - getPayoff: protectedProcedure - .input(z.object({ strike: z.number(), premium: z.number(), right: z.enum(['call', 'put']), underlyingPrice: z.number() })) - .query(async ({ ctx, input }) => { - const { computePayoffDiagram } = await import('../options/ConvexityGate.ts'); - return computePayoffDiagram(input.strike, input.premium, input.right, input.underlyingPrice); - }), -}); - // ─── Derisking Router (Slice 23) ──────────────────────────────────────────── const deriskingRouter = router({ @@ -2604,13 +4419,21 @@ const deriskingRouter = router({ thesisStatus: z.enum(['intact', 'weakening', 'broken']).optional(), currentRegime: z.enum(['trending-up', 'trending-down', 'range-bound']).optional(), portfolioCorrelation: z.number().optional(), - optionsUnlockState: z.number().optional(), profitTargets: z.array(z.number()).optional(), + dividendYield: z.number().optional(), + payoutRatio: z.number().optional(), })) .query(async ({ ctx, input }) => { const { suggestDerisking } = await import('../derisking/DeriskingEngine.ts'); return { suggestions: suggestDerisking(input) }; }), + + dividendHealth: protectedProcedure + .input(z.object({ symbol: z.string() })) + .query(async ({ ctx, input }) => { + const entry = await ctx.cache.get>(`yfinance:dividendFundamentals:${input.symbol}`); + return { data: entry.value }; + }), }); // ─── Macro Router (Slice 20) ──────────────────────────────────────────────── @@ -2960,22 +4783,21 @@ const sizingRouter = router({ return { ...result, halted, - footer: 'Educational analysis, not investment advice. Verify the underlying data; you are responsible for your own decisions.', + footer: 'Educational observation only.', }; }), }); -// ─── Risk Posture Router (P0: wire-sizing-risk-trpc-m20 / M20) ──────────────── +// ─── Risk Posture Router ──────────────────────────────────────────────────── const riskRouter = router({ - /** Full RiskPosture from portfolio + account equity inputs. */ + /** Simplified risk posture: holdings, drawdown vs tolerance, basic metrics. */ posture: protectedProcedure .input(z.object({ equity: z.number().positive(), peakEquity: z.number().positive().optional(), drawdownTolerancePct: z.number().positive().max(100).optional(), regime: z.enum(['trending-up', 'trending-down', 'range-bound']).optional(), - /** Optional plan-level stops/targets keyed by symbol for asymmetry math. */ plans: z.array(z.object({ symbol: z.string(), stopPrice: z.number().optional(), @@ -2985,8 +4807,12 @@ const riskRouter = router({ })) .query(async ({ ctx, input }) => { const { assessRisk, ADR_0007_FOOTER } = await import('../risk/RiskEngine.ts'); - const { isHalted, getHaltRecord, triggerHalt } = await import('../risk/haltCircuitBreaker.ts'); const { listHoldings } = await import('../db/portfolioRepository.ts'); + const { listOptionLegs } = await import('../db/portfolioOptionRepository.ts'); + const { + assessOptionRiskContribution, + mergeOptionCapitalIntoClusters, + } = await import('../risk/optionRiskContribution.ts'); const userId = ctx.userId as string; const user = ctx.db.prepare( @@ -2995,7 +4821,6 @@ const riskRouter = router({ const complexity = (user?.complexity ?? 'beginner') as 'beginner' | 'intermediate' | 'advanced'; - // Onboarding stores drawdown as a signed percent (e.g. -20); RiskEngine uses magnitude. const drawdownTolerancePct = input.drawdownTolerancePct ?? (typeof user?.drawdown_tolerance === 'number' ? Math.abs(user.drawdown_tolerance) @@ -3006,6 +4831,7 @@ const riskRouter = router({ ); const holdings = listHoldings(ctx.db, userId); + const optionLegs = listOptionLegs(ctx.db, userId); const portfolio = holdings.map((h) => { const plan = planBySymbol.get(h.symbol.toUpperCase()); return { @@ -3018,13 +4844,6 @@ const riskRouter = router({ }; }); - const { listOptionLegs } = await import('../db/portfolioOptionRepository.ts'); - const { - assessOptionRiskContribution, - mergeOptionCapitalIntoClusters, - } = await import('../risk/optionRiskContribution.ts'); - const optionLegs = listOptionLegs(ctx.db, userId); - const equity = input.equity; const peakEquity = input.peakEquity ?? equity; const defaultCap = complexity === 'beginner' ? equity * 0.25 @@ -3032,6 +4851,12 @@ const riskRouter = router({ : null; const clusterCaps = defaultCap !== null ? { uncategorized: defaultCap } as Record : null; + const optionRisk = assessOptionRiskContribution( + optionLegs, + holdings.map((h) => ({ symbol: h.symbol, shares: h.shares })), + equity, + ); + const posture = assessRisk({ portfolio, account: { equity, drawdownTolerancePct, complexity }, @@ -3040,67 +4865,38 @@ const riskRouter = router({ sizingContext: { clusterCaps }, }); - const optionRisk = assessOptionRiskContribution( - optionLegs, - holdings.map((h) => ({ symbol: h.symbol, shares: h.shares })), - equity, - ); + const foldUncategorized = complexity === 'beginner'; const clusterExposure = mergeOptionCapitalIntoClusters( posture.clusterExposure, optionRisk.capitalByUnderlying, - true, + foldUncategorized, ); + const equityExposureUsd = portfolio.reduce((s, p) => s + p.shares * p.avgCost, 0); + const totalExposureUsd = equityExposureUsd + optionRisk.capitalCommittedUsd; const recommendedActions = [ ...posture.recommendedActions, ...optionRisk.recommendations, ]; - // Persist gentle-halt when math says breached (24h cooldown). - if (posture.halted && !isHalted(ctx.db, userId)) { - triggerHalt(ctx.db, userId, 'max_drawdown_tolerance_breach'); - } - - const haltRecord = getHaltRecord(ctx.db, userId); - const haltActive = isHalted(ctx.db, userId); - return { ...posture, clusterExposure, recommendedActions, - halted: posture.halted || haltActive, - halt: haltRecord && haltActive - ? { - haltedUntil: haltRecord.halted_until, - triggeredBy: haltRecord.triggered_by, - ts: haltRecord.ts, - } - : null, + halted: false, + halt: null, holdingsCount: holdings.length, - totalExposureUsd: portfolio.reduce((s, p) => s + p.shares * p.avgCost, 0), - options: { - legsCount: optionRisk.legsCount, + optionLegsCount: optionRisk.legsCount, + totalExposureUsd, + optionRisk: { premiumAtRiskUsd: optionRisk.premiumAtRiskUsd, cashReservedUsd: optionRisk.cashReservedUsd, creditReceivedUsd: optionRisk.creditReceivedUsd, capitalCommittedUsd: optionRisk.capitalCommittedUsd, uncoveredShortCallCount: optionRisk.uncoveredShortCallCount, - byUnderlying: optionRisk.byUnderlying, }, footer: ADR_0007_FOOTER, }; }), - - haltStatus: protectedProcedure.query(async ({ ctx }) => { - const { isHalted, getHaltRecord } = await import('../risk/haltCircuitBreaker.ts'); - const userId = ctx.userId as string; - const active = isHalted(ctx.db, userId); - const record = getHaltRecord(ctx.db, userId); - return { - active, - haltedUntil: active && record ? record.halted_until : null, - triggeredBy: active && record ? record.triggered_by : null, - }; - }), }); // ─── Emotion Logger Router (Slice: emotion-logger-storage) ────────────────────── @@ -3158,15 +4954,438 @@ const emotionLoggerRouter = router({ }), }); +// ─── Trades Router (Phase 5) ───────────────────────────────────────────── + +const tradesRouter = router({ + list: protectedProcedure.query(async ({ ctx }) => { + return ctx.db.prepare('SELECT * FROM trades WHERE owner_id=? ORDER BY date DESC').all(ctx.userId as string); + }), + + get: protectedProcedure + .input(z.object({ id: z.string() })) + .query(async ({ ctx, input }) => { + const row = ctx.db.prepare('SELECT * FROM trades WHERE id=? AND owner_id=?').get(input.id, ctx.userId as string); + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Trade not found.' }); + return row; + }), + + save: protectedProcedure + .input(z.object({ + id: z.string(), + symbol: z.string(), + date: z.string(), + status: z.string(), + environment: z.string().optional(), + tier: z.string(), + strategyId: z.string().optional(), + confluenceScore: z.number().optional(), + riskPct: z.number().optional(), + positionSize: z.number().optional(), + entryPrice: z.number().optional(), + stopLoss: z.number().optional(), + targets: z.string().optional(), + justifications: z.string().optional(), + checklistCompleted: z.number().optional(), + allocationPlanId: z.string().optional(), + allocationRole: z.string().optional(), + tradeId: z.string().optional(), + })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId as string; + const tradeKey = input.tradeId ?? input.id; + const existing = ctx.db.prepare('SELECT id FROM trades WHERE id=? AND owner_id=?').get(input.id, userId) as Record | undefined; + if (existing) { + ctx.db.prepare(` + UPDATE trades SET symbol=?,date=?,status=?,environment=?,tier=?,strategy_id=?, + confluence_score=?,risk_pct=?,position_size=?,entry_price=?,stop_loss=?, + targets=?,justifications=?,checklist_completed=?,allocation_plan_id=?,allocation_role=?,trade_id=? + WHERE id=? AND owner_id=? + `).run( + input.symbol.toUpperCase(), input.date, input.status, input.environment ?? null, input.tier, + input.strategyId ?? null, input.confluenceScore ?? null, input.riskPct ?? null, + input.positionSize ?? null, input.entryPrice ?? null, input.stopLoss ?? null, + input.targets ?? '[]', input.justifications ?? '{}', input.checklistCompleted ?? 0, + input.allocationPlanId ?? null, input.allocationRole ?? null, tradeKey, input.id, userId + ); + } else { + ctx.db.prepare(` + INSERT INTO trades (id,owner_id,trade_id,symbol,date,status,environment,tier,strategy_id, + confluence_score,risk_pct,position_size,entry_price,stop_loss,targets,justifications,checklist_completed,allocation_plan_id,allocation_role) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + `).run( + input.id, userId, tradeKey, input.symbol.toUpperCase(), input.date, input.status, input.environment ?? null, + input.tier, input.strategyId ?? null, input.confluenceScore ?? null, input.riskPct ?? null, + input.positionSize ?? null, input.entryPrice ?? null, input.stopLoss ?? null, + input.targets ?? '[]', input.justifications ?? '{}', input.checklistCompleted ?? 0, + input.allocationPlanId ?? null, input.allocationRole ?? null + ); + } + return { ok: true, id: input.id }; + }), + + close: protectedProcedure + .input(z.object({ + id: z.string(), + realizedPnl: z.number(), + reflection: z.string().optional(), + })) + .mutation(async ({ ctx, input }) => { + const userId = ctx.userId as string; + const row = ctx.db.prepare('SELECT justifications FROM trades WHERE id=? AND owner_id=?').get(input.id, userId) as { justifications: string | null } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Trade not found.' }); + let just: Record = {}; + try { just = JSON.parse(row.justifications || '{}'); } catch { /* ignore */ } + if (input.reflection?.trim()) just.closeReflection = input.reflection.trim(); + ctx.db.prepare('UPDATE trades SET status=?, realized_pnl=?, justifications=? WHERE id=? AND owner_id=?') + .run('closed', input.realizedPnl, JSON.stringify(just), input.id, userId); + return { ok: true }; + }), + + stats: protectedProcedure.query(async ({ ctx }) => { + const userId = ctx.userId as string; + const closed = ctx.db.prepare('SELECT COUNT(*) c FROM trades WHERE owner_id=? AND status=?').get(userId, 'closed') as { c: number } | undefined; + const profitable = ctx.db.prepare('SELECT COUNT(*) c FROM trades WHERE owner_id=? AND status=? AND realized_pnl>0').get(userId, 'closed') as { c: number } | undefined; + return { totalClosed: closed?.c ?? 0, profitable: profitable?.c ?? 0 }; + }), +}); + +const thesesRouter = router({ + list: protectedProcedure.query(({ ctx }) => { + return ctx.db.prepare( + 'SELECT * FROM theses WHERE user_id=? ORDER BY updated_at DESC', + ).all(ctx.userId as string); + }), + + get: protectedProcedure + .input(z.object({ id: z.string() })) + .query(({ ctx, input }) => { + const row = ctx.db.prepare('SELECT * FROM theses WHERE id=? AND user_id=?').get(input.id, ctx.userId as string); + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Thesis not found.' }); + return row; + }), + + create: protectedProcedure + .input(z.object({ + symbol: z.string().min(1), + statement: z.string().min(1), + invalidationCriteria: z.array(z.string()).default([]), + status: z.enum(['intact', 'weakening', 'broken']).default('intact'), + })) + .mutation(({ ctx, input }) => { + const id = randomUUID(); + const now = new Date().toISOString(); + ctx.db.prepare( + `INSERT INTO theses (id, user_id, symbol, statement, invalidation_criteria, status, created_at, updated_at) + VALUES (?,?,?,?,?,?,?,?)`, + ).run( + id, + ctx.userId as string, + input.symbol.toUpperCase(), + input.statement, + JSON.stringify(input.invalidationCriteria), + input.status, + now, + now, + ); + return { id, symbol: input.symbol.toUpperCase() }; + }), + + update: protectedProcedure + .input(z.object({ + id: z.string(), + statement: z.string().min(1).optional(), + invalidationCriteria: z.array(z.string()).optional(), + status: z.enum(['intact', 'weakening', 'broken']).optional(), + })) + .mutation(({ ctx, input }) => { + const userId = ctx.userId as string; + const row = ctx.db.prepare('SELECT * FROM theses WHERE id=? AND user_id=?').get(input.id, userId) as { + statement: string; + invalidation_criteria: string; + status: string; + } | undefined; + if (!row) throw new TRPCError({ code: 'NOT_FOUND', message: 'Thesis not found.' }); + const statement = input.statement ?? row.statement; + const criteria = input.invalidationCriteria + ? JSON.stringify(input.invalidationCriteria) + : row.invalidation_criteria; + const status = input.status ?? row.status; + ctx.db.prepare( + 'UPDATE theses SET statement=?, invalidation_criteria=?, status=?, updated_at=? WHERE id=? AND user_id=?', + ).run(statement, criteria, status, new Date().toISOString(), input.id, userId); + return { ok: true }; + }), + + delete: protectedProcedure + .input(z.object({ id: z.string() })) + .mutation(({ ctx, input }) => { + const info = ctx.db.prepare('DELETE FROM theses WHERE id=? AND user_id=?').run(input.id, ctx.userId as string); + if (info.changes === 0) throw new TRPCError({ code: 'NOT_FOUND', message: 'Thesis not found.' }); + return { ok: true }; + }), +}); + +// ─── Fund Router (M21 Mirror Portfolio) ───────────────────────────────────── + +const symbolsRouter = router({ + /** Autocomplete search over the symbol index (M22). Local-first (ADR-0009). */ + search: publicProcedure + .input(z.object({ + q: z.string().trim().min(1).max(60), + limit: z.number().int().min(1).max(20).optional(), + })) + .query(async ({ ctx, input }) => { + const limit = input.limit ?? 10; + const q = input.q.toUpperCase(); + const db = ctx.db; + + // Prefix match on symbol first (exact > prefix, shortest first), then name contains. + const rows = db.prepare( + `SELECT symbol, name, sector, industry, exchange, ticker_kind, cik, + CASE WHEN symbol = ? THEN 0 ELSE 1 END AS rank + FROM symbols + WHERE symbol = ? OR symbol LIKE ? + ORDER BY rank ASC, LENGTH(symbol) ASC, symbol ASC + LIMIT ?`, + ).all(q, q, `${q}%`, limit) as Array>; + + let results = rows.map((r) => ({ + symbol: r.symbol, + name: r.name ?? null, + sector: r.sector ?? null, + industry: r.industry ?? null, + exchange: r.exchange ?? null, + ticker_kind: r.ticker_kind, + cik: r.cik ?? null, + })); + + if (results.length < limit) { + const seen = new Set(results.map((r) => r.symbol)); + const nameRows = db.prepare( + `SELECT symbol, name, sector, industry, exchange, ticker_kind, cik + FROM symbols + WHERE name IS NOT NULL AND name != '' AND name LIKE ? + ORDER BY LENGTH(name) ASC + LIMIT ?`, + ).all(`%${input.q}%`, limit) as Array>; + const fresh = nameRows + .filter((r) => !seen.has(r.symbol as string)) + .map((r) => ({ + symbol: r.symbol, + name: r.name ?? null, + sector: r.sector ?? null, + industry: r.industry ?? null, + exchange: r.exchange ?? null, + ticker_kind: r.ticker_kind, + cik: r.cik ?? null, + })); + results = [...results, ...fresh.slice(0, limit - results.length)]; + } + + return { q: input.q, results }; + }), + + /** Two-tier fund-holdings strip for a symbol (M22): tracked funds w/ weight, then institution count. */ + holders: publicProcedure + .input(z.object({ symbol: z.string().trim().toUpperCase().min(1).max(20) })) + .query(async ({ ctx, input }) => { + const db = ctx.db; + const symbol = input.symbol; + + // Tier 1 — tracked funds holding the symbol (most recent record per fund), with weight in their disclosed book. + const trackedRows = db.prepare( + `SELECT tf.id AS fund_id, tf.fund_name, fpr.symbol, fpr.shares, fpr.value_usd, fpr.as_of, fpr.source + FROM fund_position_records fpr + JOIN tracked_funds tf ON tf.id = fpr.fund_id AND tf.enabled = 1 + JOIN ( + SELECT fund_id, MAX(as_of) AS max_as_of + FROM fund_position_records + WHERE symbol = ? + GROUP BY fund_id + ) latest ON latest.fund_id = fpr.fund_id AND latest.max_as_of = fpr.as_of + WHERE fpr.symbol = ? + ORDER BY tf.fund_name ASC`, + ).all(symbol, symbol) as Array>; + + // Weight = position value / total disclosed book value at the fund's live book (per fund). + const fundIds = [...new Set(trackedRows.map((r) => r.fund_id as string))]; + const bookValueByFund = new Map(); + for (const fid of fundIds) { + const { liveBook } = await import('../db/fundRepository.ts'); + const book = liveBook(db, fid); + const total = book.reduce((s, p) => s + (p.value_usd ?? 0), 0); + bookValueByFund.set(fid, total > 0 ? total : 0); + } + + const tracked = trackedRows.map((r) => ({ + fundId: r.fund_id, + fundName: r.fund_name, + shares: r.shares ?? null, + valueUsd: r.value_usd ?? null, + asOf: r.as_of, + source: r.source, + weightPct: bookValueByFund.get(r.fund_id) + ? ((r.value_usd ?? 0) / bookValueByFund.get(r.fund_id)!) * 100 + : null, + })); + + // Tier 2 — how many distinct 13F filers reported holding the symbol. + const instRow = db.prepare( + `SELECT COUNT(DISTINCT filer_cik) AS n FROM institution_filings WHERE symbol = ?`, + ).get(symbol) as { n: number } | undefined; + + return { + symbol, + tracked, + institutionCount: instRow?.n ?? 0, + }; + }), +}); + +const fundsRouter = router({ + /** List operator-curated tracked funds (v1: Alpine Fox). */ + list: publicProcedure + .query(async ({ ctx }) => { + const { listTrackedFunds } = await import('../db/fundRepository.ts'); + return listTrackedFunds(ctx.db, { includeDisabled: true }); + }), + + /** Get a single tracked fund by id. */ + get: publicProcedure + .input(z.object({ id: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { getTrackedFund } = await import('../db/fundRepository.ts'); + return getTrackedFund(ctx.db, input.id); + }), + + /** Live Book for a fund — most recent record per symbol, source-labeled. */ + liveBook: publicProcedure + .input(z.object({ fundId: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { liveBook } = await import('../db/fundRepository.ts'); + return liveBook(ctx.db, input.fundId); + }), + + /** Full append-only position timeline for a fund. */ + records: publicProcedure + .input(z.object({ fundId: z.string().min(1) })) + .query(async ({ ctx, input }) => { + const { listRecords } = await import('../db/fundRepository.ts'); + return listRecords(ctx.db, input.fundId); + }), + + /** Admin CRUD — operator-curated registry. */ + adminCreate: adminProcedure + .input(z.object({ + ci_key: z.string().min(1), + fund_name: z.string().min(1), + manager_name: z.string().min(1), + x_handle: z.string().optional(), + paywall_status: z.enum(['unknown', 'open', 'paywalled']).optional(), + enabled: z.boolean().optional(), + })) + .mutation(async ({ ctx, input }) => { + const { upsertTrackedFund } = await import('../db/fundRepository.ts'); + return upsertTrackedFund(ctx.db, input); + }), + + adminSetEnabled: adminProcedure + .input(z.object({ id: z.string().min(1), enabled: z.boolean() })) + .mutation(async ({ ctx, input }) => { + const { setFundEnabled } = await import('../db/fundRepository.ts'); + return { updated: setFundEnabled(ctx.db, input.id, input.enabled) }; + }), + + adminDelete: adminProcedure + .input(z.object({ id: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const { deleteTrackedFund } = await import('../db/fundRepository.ts'); + return { deleted: deleteTrackedFund(ctx.db, input.id) }; + }), + /** Fetch the fund's full 13F history + refresh Live-Book records. */ + adminSync13f: protectedProcedure + .input(z.object({ fundId: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const { fetchFund13F } = await import('../mirror/fund13fFetcher.ts'); + return fetchFund13F(ctx.db, input.fundId); + }), + + /** Materialize position captures from the fund manager's X posts (M21). */ + adminIngestCaptures: protectedProcedure + .input(z.object({ fundId: z.string().min(1) })) + .mutation(async ({ ctx, input }) => { + const { ingestFundCaptures } = await import('../services/captureIngest.ts'); + return ingestFundCaptures(ctx.db, input.fundId); + }), +}); + +// ─── Mirror Router (M21) — math, not advice ──────────────────────────────── + +const mirrorRouter = router({ + /** + * Mirror diff: target vs the user's actual book. Pure engine output; all + * strings are mechanical (ADR-0010). Base defaults to live portfolio equity. + */ + diff: protectedProcedure + .input(z.object({ + fundId: z.string().min(1), + base: z.number().positive().optional(), // if omitted, use live equity + locked: z.boolean().optional(), + minPositionUsd: z.number().optional(), + minWeightPct: z.number().optional(), + tickSize: z.number().optional(), + })) + .query(async ({ ctx, input }) => { + const userId = ctx.userId ?? 'anonymous'; + const { liveBook } = await import('../db/fundRepository.ts'); + const { listHoldings } = await import('../db/portfolioRepository.ts'); + const { mirrorBook } = await import('../mirror/mirrorEngine.ts'); + + const book = liveBook(ctx.db, input.fundId); + const holdings = listHoldings(ctx.db, userId); + const symbols = Array.from(new Set([...book.map((p) => p.symbol), ...holdings.map((h) => h.symbol)])); + const entries = await ctx.cache.getMany(symbols.map((s) => `yfinance:quote:${s}`)); + const prices: Record = {}; + symbols.forEach((s, i) => { + const q = (entries[i]?.value ?? null) as { regularMarketPrice?: number; price?: number } | null; + prices[s] = q?.regularMarketPrice ?? q?.price; + }); + + // Actual portfolio equity (sum of market value) for re-baseline default. + let equity = 0; + for (const h of holdings) { + const px = prices[h.symbol]; + if (typeof px === 'number') equity += px * h.qty; + } + const base = input.base ?? (equity > 0 ? equity : 200_000); + + return mirrorBook( + book, + holdings.map((h) => ({ symbol: h.symbol, qty: h.shares, avg_cost: h.avg_cost })), + prices, + { + base, + locked: input.locked ?? false, + minPositionUsd: input.minPositionUsd, + minWeightPct: input.minWeightPct, + tickSize: input.tickSize, + }, + ); + }), +}); + export const appRouter = router({ auth: authRouter, onboarding: onboardingRouter, market: marketRouter, dashboard: dashboardRouter, admin: adminRouter, alerts: alertsRouter, edgar: edgarRouter, institutional: institutionalRouter, - watchlists: watchlistRouter, portfolio: portfolioRouter, options: optionsRouter, reports: reportsRouter, + watchlists: watchlistRouter, portfolio: portfolioRouter, options: optionsRouter, dealerMap: dealerMapRouter, dealerStudy: dealerStudyRouter, mentorLedger: mentorLedgerRouter, userLlm: userLlmRouter, reports: reportsRouter, screener: screenerRouter, strategies: strategyRouter, backtest: backtestRouter, - sectorCrosslink: sectorCrosslinkRouter, optionsConvexity: optionsConvexityRouter, + sectorCrosslink: sectorCrosslinkRouter, derisking: deriskingRouter, macro: macroRouter, thesisMonitor: thesisMonitorRouter, x: xRouter, reddit: redditRouter, emotionLogger: emotionLoggerRouter, - sizing: sizingRouter, risk: riskRouter, + sizing: sizingRouter, risk: riskRouter, trades: tradesRouter, theses: thesesRouter, + funds: fundsRouter, + mirror: mirrorRouter, + symbols: symbolsRouter, }); export type AppRouter = typeof appRouter; diff --git a/app/src/__tests__/primary-rule-lint.test.ts b/app/src/__tests__/primary-rule-lint.test.ts index a261441..02bb1b8 100644 --- a/app/src/__tests__/primary-rule-lint.test.ts +++ b/app/src/__tests__/primary-rule-lint.test.ts @@ -19,6 +19,8 @@ function imperativeHits(text: string): string[] { while ((m = re.exec(text)) !== null) { const ctx = text.slice(Math.max(0, m.index - 14), m.index + w.length + 14).replace(/\s+/g, " ").trim(); if (ALLOWED_PHRASES.some((a) => ctx.toLowerCase().includes(a))) continue; + // TypeScript field names in API types (strongBuy/buy/sell counts) are not UI copy. + if (/\b(buy|sell)\s*[?:]/.test(ctx) && /number|string|boolean/.test(ctx)) continue; hits.push(`"${w}" near: …${ctx}…`); } } @@ -41,6 +43,9 @@ test("panel/page/client source pass the Primary-Rule lint", () => { "../lib/strings.ts", "../lib/trpc.ts", "../stores/active-symbol-store.ts", + "../components/dealer-flow/DealerFlowView.tsx", + "../components/dealer-flow/DealerLevelsStrip.tsx", + "../app/dealer-flow/page.tsx", ]; for (const f of files) { const src = readFileSync(join(__dirname, f), "utf8"); diff --git a/app/src/__tests__/volumeProfile.test.ts b/app/src/__tests__/volumeProfile.test.ts new file mode 100644 index 0000000..ccc27b8 --- /dev/null +++ b/app/src/__tests__/volumeProfile.test.ts @@ -0,0 +1,53 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { buildVolumeProfile } from "../lib/volumeProfile.ts"; + +describe("buildVolumeProfile", () => { + it("returns empty for no candles", () => { + const r = buildVolumeProfile([]); + assert.equal(r.bins.length, 0); + assert.equal(r.totalVolume, 0); + assert.equal(r.pocMid, null); + }); + + it("returns empty for zero volume", () => { + const r = buildVolumeProfile([{ h: 10, l: 9, v: 0 }]); + assert.equal(r.bins.length, 0); + }); + + it("preserves total volume across bins", () => { + const candles = [ + { h: 12, l: 10, v: 100 }, + { h: 11, l: 9, v: 200 }, + { h: 14, l: 13, v: 50 }, + ]; + const r = buildVolumeProfile(candles, { bins: 10 }); + const sum = r.bins.reduce((a, b) => a + b.volume, 0); + assert.ok(Math.abs(sum - 350) < 1e-6, `sum=${sum}`); + assert.equal(r.totalVolume, 350); + }); + + it("poc is the max volume bin mid", () => { + // All volume at the high end + const candles = [ + { h: 20, l: 19, v: 1000 }, + { h: 11, l: 10, v: 10 }, + ]; + const r = buildVolumeProfile(candles, { bins: 8 }); + assert.ok(r.pocMid != null); + const maxBin = r.bins.reduce((a, b) => (b.volume > a.volume ? b : a)); + assert.equal(r.pocMid, maxBin.mid); + assert.ok(r.pocMid! >= 15, `poc should sit high, got ${r.pocMid}`); + }); + + it("sets vah/val around value area", () => { + const candles = Array.from({ length: 20 }, (_, i) => ({ + h: 100 + i * 0.5, + l: 99 + i * 0.5, + v: i === 10 ? 500 : 20, + })); + const r = buildVolumeProfile(candles, { bins: 16, valueAreaPct: 0.7 }); + assert.ok(r.vah != null && r.val != null); + assert.ok(r.vah! >= r.val!); + }); +}); diff --git a/app/src/app/admin/page.tsx b/app/src/app/admin/page.tsx index 0e756be..dd1da5a 100644 --- a/app/src/app/admin/page.tsx +++ b/app/src/app/admin/page.tsx @@ -2,7 +2,7 @@ import { useEffect, useState } from "react"; import { AdminLayout } from "@/components/AdminLayout"; import { Card, CardHeader, Badge, DataTable, type Column } from "@/components/ui"; -import { api } from "@/lib/trpc"; +import { api, type AlertStatusRow } from "@/lib/trpc"; interface UserRow { id: string; @@ -147,7 +147,6 @@ function QueueCard({ queue }: { queue: QueueItem[] }) { ); } - function ActivityCard({ logs }: { logs: AuditEntry[] }) { const formatTime = (ts: string): string => { try { @@ -176,10 +175,77 @@ function ActivityCard({ logs }: { logs: AuditEntry[] }) { ); } +function AlertStatusCard({ status }: { status: AlertStatusRow }) { + const fmtAge = (sec: number | null): string => { + if (sec === null) return "never"; + if (sec < 90) return `${sec}s`; + if (sec < 3600) return `${Math.floor(sec / 60)}m`; + if (sec < 86400) return `${Math.floor(sec / 3600)}h`; + return `${Math.floor(sec / 86400)}d`; + }; + + return ( + + {status.producers.length} producers} /> +
+ {status.producers.length === 0 ? ( +

No producer runs recorded yet

+ ) : ( +
+ {status.producers.map((p) => ( +
+ {p.producer} + + {p.lastRunAt ? new Date(p.lastRunAt).toLocaleTimeString() : "never"} · {p.lastEventsFired} fired + +
+ ))} +
+ )} +
+
+

VIX quote age

+

{fmtAge(status.freshness.vixLevelAgeSec)}

+
+
+

Oldest quote age

+

{fmtAge(status.freshness.oldestQuoteAgeSec)}

+
+
+

Latest SC filing

+

+ {status.freshness.latestScFilingAt ? status.freshness.latestScFilingAt.slice(0, 10) : "none"} +

+
+
+

Latest insider

+

+ {status.freshness.latestInsiderAt ? status.freshness.latestInsiderAt.slice(0, 10) : "none"} +

+
+
+ {status.subscriptionsByType.length > 0 && ( +
+

Enabled subscriptions by type

+
+ {status.subscriptionsByType.map((s) => ( + + {s.alertType}:{s.enabledCount} + + ))} +
+
+ )} +
+ + ); +} + export default function AdminPage() { const [users, setUsers] = useState([]); const [queue, setQueue] = useState([]); const [logs, setLogs] = useState([]); + const [alertStatus, setAlertStatus] = useState(null); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); @@ -192,6 +258,7 @@ export default function AdminPage() { api.admin.usersList(), api.admin.queueHealth(), api.admin.auditLog(10), + api.admin.alertStatus(), ]); if (cancelled) return; @@ -199,6 +266,7 @@ export default function AdminPage() { const usersResult = results[0]; const queueResult = results[1]; const logsResult = results[2]; + const alertResult = results[3]; if (usersResult.status === "fulfilled") setUsers(usersResult.value); else setError((prev) => prev ? `${prev} | Users fetch failed` : "Failed to load users"); @@ -208,6 +276,8 @@ export default function AdminPage() { if (logsResult.status === "fulfilled") setLogs(logsResult.value); else setError((prev) => prev ? `${prev} | Audit log fetch failed` : "Failed to load audit log"); + + if (alertResult.status === "fulfilled") setAlertStatus(alertResult.value); } catch (e) { if (!cancelled) setError(e instanceof Error ? e.message : "Failed to load admin data"); } finally { @@ -231,6 +301,12 @@ export default function AdminPage() { + + {alertStatus && ( +
+ +
+ )} ); diff --git a/app/src/app/admin/queue/page.tsx b/app/src/app/admin/queue/page.tsx index 33e7b46..e3a993c 100644 --- a/app/src/app/admin/queue/page.tsx +++ b/app/src/app/admin/queue/page.tsx @@ -91,6 +91,26 @@ export default function QueuePage() { const [symbol, setSymbol] = useState(""); const [triggering, setTriggering] = useState(false); + // Dealer map integrity / replay + const [dealerSymbol, setDealerSymbol] = useState("IREN"); + const [dealerAsOf, setDealerAsOf] = useState(""); + const [dealerBusy, setDealerBusy] = useState(false); + const [dealerResult, setDealerResult] = useState<{ + ok: boolean; + summary: string; + integrityText: string; + integrity: { + status: string; + hardFailed: string[]; + softFailed: string[]; + checks: Array<{ id: string; level: string; ok: boolean; detail: string }>; + metrics: Record; + }; + king: { strike: number; expiry: string; gex: number } | null; + spot: number | null; + mode: string; + } | null>(null); + const [resettingSource, setResettingSource] = useState(null); const [resetResult, setResetResult] = useState>({}); @@ -98,6 +118,9 @@ export default function QueuePage() { const [newScheduleInterval, setNewScheduleInterval] = useState("86400000"); const [addingSchedule, setAddingSchedule] = useState(false); + // Data quality filter + const [hideErrors, setHideErrors] = useState(true); + // B4: Data quality state interface DataQualityRow { symbol: string; @@ -175,13 +198,12 @@ export default function QueuePage() { const handleRunLintAll = useCallback(async (kind: 'sec-lint-holders' | 'sec-lint-insiders') => { setLintLoading(kind === 'sec-lint-holders' ? 'holders' : 'insiders'); try { - await api.admin.queueLintAll({ kind }); - setSuccess(`Lint ${kind === 'sec-lint-holders' ? 'holders' : 'insiders'} completed`); + const result = await api.admin.queueLintAll({ kind }); + setSuccess(`Lint ${kind === 'sec-lint-holders' ? 'holders' : 'insiders'}: ${result.enqueued} jobs enqueued`); } catch (e) { setError(e instanceof Error ? e.message : "Lint failed"); } finally { setLintLoading(null); - await loadData(); } }, []); @@ -202,8 +224,15 @@ export default function QueuePage() { setError(null); setSuccess(null); try { - await api.admin.queueSecFetch(symbol.trim().toUpperCase()); - setSuccess(`Queued SEC fetch for ${symbol.toUpperCase()}`); + const sym = symbol.trim().toUpperCase(); + await api.admin.queueSecFetch(sym); + // Also enqueue holders lint so CUSIP/13F gaps re-check promptly + try { + await api.admin.queueLint(sym, "sec-lint-holders"); + } catch { + /* optional */ + } + setSuccess(`Queued SEC fetch + holders lint for ${sym}`); setSymbol(""); await loadData(); } catch (e) { @@ -213,6 +242,48 @@ export default function QueuePage() { } }; + const handleDealerIntegrity = async (mode: "live" | "replay") => { + if (!dealerSymbol.trim()) return; + setDealerBusy(true); + setError(null); + setSuccess(null); + setDealerResult(null); + try { + const input: { symbol: string; asOf?: string; recompute?: boolean } = { + symbol: dealerSymbol.trim().toUpperCase(), + recompute: true, + }; + if (mode === "replay") { + if (!dealerAsOf.trim()) { + setError("As-of ISO required for replay (e.g. 2026-08-07T13:14:00.000Z)"); + setDealerBusy(false); + return; + } + input.asOf = dealerAsOf.trim(); + } + const r = await api.admin.dealerMapIntegrity(input); + setDealerResult({ + ok: r.ok, + summary: r.summary, + integrityText: r.integrityText, + integrity: r.integrity as typeof dealerResult extends null ? never : NonNullable["integrity"], + king: r.king, + spot: r.spot, + mode: r.mode, + }); + setSuccess( + r.ok + ? `Dealer integrity ${r.mode}: ${r.integrity.status} for ${r.symbol}` + : `Dealer integrity ${r.mode}: ${r.integrity.status} — see details below`, + ); + await loadData(); + } catch (e) { + setError(e instanceof Error ? e.message : "Dealer integrity failed"); + } finally { + setDealerBusy(false); + } + }; + const handleResetBackoff = async (sourceKind: string) => { setResettingSource(sourceKind); try { @@ -541,6 +612,89 @@ export default function QueuePage() { )} + {/* Dealer map integrity / GEX-VEX backtest */} +
+
+

Dealer map integrity

+

+ Live recompute + completeness gates, or as-of replay from cached chains. Delay is OK; + missing pieces are not. +

+
+
+
+ + + + +
+ {dealerResult && ( +
+

+ {dealerResult.mode} · {dealerResult.integrity.status} + {dealerResult.spot != null ? ` · spot ${dealerResult.spot}` : ""} + {dealerResult.king + ? ` · king ${dealerResult.king.strike} ${dealerResult.king.expiry}` + : ""} +

+

{dealerResult.integrityText}

+

{dealerResult.summary}

+ {dealerResult.integrity.checks.filter((c) => !c.ok).length > 0 && ( +
    + {dealerResult.integrity.checks + .filter((c) => !c.ok) + .slice(0, 8) + .map((c) => ( +
  • + [{c.level}] {c.detail} +
    • + ))} +
+ )} +

+ metrics {JSON.stringify(dealerResult.integrity.metrics)} +

+
+ )} +
+
+ {/* Schedule Configuration */}
@@ -681,6 +835,23 @@ export default function QueuePage() {
+ {dataQuality.length > 0 && ( +
+ + {dataQuality.filter(r => r.status === 'ok').length} ok + {' / '} + {dataQuality.filter(r => r.status === 'gaps_found').length} gaps + {' / '} + {dataQuality.filter(r => r.status === 'stale').length} stale + {' / '} + {dataQuality.filter(r => r.status === 'error').length} errors + + +
+ )}
{dataQuality.length === 0 ? (
No data quality records yet. Run a lint to populate.
@@ -699,7 +870,7 @@ export default function QueuePage() { - {dataQuality.map((row, idx) => ( + {dataQuality.filter(r => !hideErrors || r.status !== 'error').map((row, idx) => ( {row.symbol} {row.kind === 'institution_filings' ? '13F holders' : 'Form 4'} diff --git a/app/src/app/admin/x-accounts/page.tsx b/app/src/app/admin/x-accounts/page.tsx index 438f30d..1deaf49 100644 --- a/app/src/app/admin/x-accounts/page.tsx +++ b/app/src/app/admin/x-accounts/page.tsx @@ -236,8 +236,8 @@ export default function XAccountsPage() {

Cached X posts are kept locally so the feed loads fast. Old posts accumulate over time — prune them periodically to free up space.

{pruneResult !== null && ( -
- {pruneResult.error ? pruneResult.error : `Deleted ${pruneResult.deleted} cached post(s) older than 30 days.`} +
+ {'error' in pruneResult ? pruneResult.error : `Deleted ${pruneResult.deleted} cached post(s) older than 30 days.`}
)} diff --git a/app/src/app/alerts/page.tsx b/app/src/app/alerts/page.tsx index 1108551..439e1fd 100644 --- a/app/src/app/alerts/page.tsx +++ b/app/src/app/alerts/page.tsx @@ -1,8 +1,9 @@ "use client"; import { useEffect, useState } from "react"; +import { ExternalLink as ExternalLinkIcon } from "lucide-react"; import { LayoutShell } from "@/components/LayoutShell"; import { FeatureGate } from "@/components/FeatureGate"; -import { api, type AlertEventRow, type AlertSubscriptionRow } from "@/lib/trpc"; +import { api, type AlertEventRow, type AlertSubscriptionRow, type AlertTypeToggleRow } from "@/lib/trpc"; const SEVERITY_COLORS: Record = { info: "text-blue-400 bg-blue-950/30 border-blue-500/30", @@ -14,6 +15,7 @@ function AlertsPageInner() { const [tab, setTab] = useState<"events" | "subscriptions">("events"); const [events, setEvents] = useState([]); const [subscriptions, setSubscriptions] = useState([]); + const [typeToggles, setTypeToggles] = useState([]); const [alertType, setAlertType] = useState("informed_buy"); const [symbol, setSymbol] = useState(""); const [loading, setLoading] = useState(true); @@ -32,8 +34,15 @@ function AlertsPageInner() { } catch { /* ignore */ } }; + const loadTypeToggles = async () => { + try { + const data = await api.alerts.listTypes(); + setTypeToggles(data); + } catch { /* ignore */ } + }; + useEffect(() => { - Promise.all([loadEvents(), loadSubscriptions()]).finally(() => setLoading(false)); + Promise.all([loadEvents(), loadSubscriptions(), loadTypeToggles()]).finally(() => setLoading(false)); }, []); const acknowledge = async (id: string) => { @@ -58,6 +67,14 @@ function AlertsPageInner() { loadSubscriptions(); }; + const toggleType = async (type: string, enabled: boolean) => { + try { + const updated = await api.alerts.toggleType(type, !enabled); + setTypeToggles(updated); + } catch { /* ignore */ } + loadSubscriptions(); + }; + const deleteSubscription = async (id: string) => { await api.alerts.deleteSubscription(id); loadSubscriptions(); @@ -67,14 +84,22 @@ function AlertsPageInner() { return ( -
-

Alerts

+
+

Alerts

-
- -
@@ -86,7 +111,11 @@ function AlertsPageInner() { ) : tab === "events" ? (
{unacknowledged.length > 0 && ( - )} @@ -95,19 +124,34 @@ function AlertsPageInner() { ) : (
{events.map((e) => ( -
-
+
+
-
+
{e.severity} {e.title} - {e.symbol && {e.symbol}} + {e.symbol && {e.symbol}}

{e.description}

+ {typeof e.payload?.url === "string" && ( + + View filing on SEC.gov + + + )}

{new Date(e.createdAt).toLocaleString()}

{!e.acknowledged && ( - )} @@ -119,10 +163,38 @@ function AlertsPageInner() {
) : (
-
+
+

Alert Types

+

Toggle the alert types you want to receive.

+
+ {typeToggles.map((t) => ( +
+
+

{t.label}

+

{t.description}

+
+
+
+ +

New Subscription

-
- setAlertType(e.target.value)} + className="rounded-lg border border-line bg-surface-sunken px-3 py-2.5 sm:py-2 text-base sm:text-sm text-fg w-full sm:w-auto" + > @@ -134,9 +206,21 @@ function AlertsPageInner() { + - setSymbol(e.target.value)} placeholder="Symbol (optional)" className="rounded-lg border border-border bg-surface-dark px-3 py-2 text-sm text-fg w-32" /> -
diff --git a/app/src/app/api/[...path]/route.ts b/app/src/app/api/[...path]/route.ts new file mode 100644 index 0000000..a779ba4 --- /dev/null +++ b/app/src/app/api/[...path]/route.ts @@ -0,0 +1,139 @@ +/** + * Same-origin /api/* proxy → backend :3001. + * + * Next.js rewrites hard-timeout around 30s and surface a plain + * "Internal Server Error" (HTTP 500). Reasoning models (Ornith) often need + * 60–180s for dealerMap.explain / userLlm.test, so we proxy here with a long + * AbortSignal instead of relying on rewrites for /api. + */ + +import type { NextRequest } from "next/server"; + +export const dynamic = "force-dynamic"; +/** Vercel / long-running hosts; local node ignores this. */ +export const maxDuration = 300; + +const BACKEND = (process.env.IFLOW_BACKEND_URL ?? "http://localhost:3001").replace(/\/+$/, ""); +/** Must exceed TRPC_LLM_TIMEOUT_MS (180s) on the client. */ +const PROXY_TIMEOUT_MS = 200_000; + +const HOP_BY_HOP = new Set([ + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailers", + "transfer-encoding", + "upgrade", + "host", + "content-length", +]); + +async function proxy(req: NextRequest, path: string[]): Promise { + const suffix = path.map(encodeURIComponent).join("/"); + const incoming = new URL(req.url); + const target = `${BACKEND}/api/${suffix}${incoming.search}`; + + const headers = new Headers(); + req.headers.forEach((value, key) => { + if (HOP_BY_HOP.has(key.toLowerCase())) return; + headers.set(key, value); + }); + + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), PROXY_TIMEOUT_MS); + + try { + const init: RequestInit = { + method: req.method, + headers, + signal: controller.signal, + redirect: "manual", + }; + + if (req.method !== "GET" && req.method !== "HEAD") { + init.body = await req.arrayBuffer(); + } + + const upstream = await fetch(target, init); + const outHeaders = new Headers(); + + upstream.headers.forEach((value, key) => { + const k = key.toLowerCase(); + if (HOP_BY_HOP.has(k)) return; + if (k === "set-cookie") return; // handled below (multi-value) + outHeaders.set(key, value); + }); + + const getSetCookie = ( + upstream.headers as Headers & { getSetCookie?: () => string[] } + ).getSetCookie; + if (typeof getSetCookie === "function") { + for (const c of getSetCookie.call(upstream.headers)) { + outHeaders.append("set-cookie", c); + } + } else { + const single = upstream.headers.get("set-cookie"); + if (single) outHeaders.append("set-cookie", single); + } + + return new Response(upstream.body, { + status: upstream.status, + statusText: upstream.statusText, + headers: outHeaders, + }); + } catch (e) { + const aborted = e instanceof Error && e.name === "AbortError"; + const msg = aborted + ? `API proxy timed out after ${Math.round(PROXY_TIMEOUT_MS / 1000)}s (backend still may complete).` + : e instanceof Error + ? e.message + : String(e); + console.error("[api proxy]", target, msg); + return Response.json( + { + error: { + message: msg, + code: aborted ? "TIMEOUT" : "BAD_GATEWAY", + data: { httpStatus: aborted ? 504 : 502 }, + }, + }, + { status: aborted ? 504 : 502 }, + ); + } finally { + clearTimeout(timer); + } +} + +type RouteCtx = { params: Promise<{ path: string[] }> }; + +export async function GET(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} + +export async function POST(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} + +export async function PUT(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} + +export async function PATCH(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} + +export async function DELETE(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} + +export async function OPTIONS(req: NextRequest, ctx: RouteCtx) { + const { path } = await ctx.params; + return proxy(req, path); +} diff --git a/app/src/app/dealer-flow/page.tsx b/app/src/app/dealer-flow/page.tsx new file mode 100644 index 0000000..78736ee --- /dev/null +++ b/app/src/app/dealer-flow/page.tsx @@ -0,0 +1,24 @@ +"use client"; + +import { Suspense } from "react"; +import { LayoutShell } from "@/components/LayoutShell"; +import { FeatureGate } from "@/components/FeatureGate"; +import { DealerFlowView } from "@/components/dealer-flow/DealerFlowView"; + +function DealerFlowPageInner() { + return ( + + Loading…

}> + + + + ); +} + +export default function DealerFlowPage() { + return ( + + + + ); +} diff --git a/app/src/app/execution/page.tsx b/app/src/app/execution/page.tsx deleted file mode 100644 index d025bca..0000000 --- a/app/src/app/execution/page.tsx +++ /dev/null @@ -1,388 +0,0 @@ -'use client'; -export const dynamic = 'force-dynamic'; - -import { useState, useRef } from 'react'; -import { useExecutionStore } from '@/stores/execution-store'; -import { EmotionLogger } from '@/components/EmotionLogger'; -import type { ExecutionPlaybook, EmotionLogState, MistakeStage, ScaleEntry } from '@/types'; -import { LayoutShell } from '@/components/LayoutShell'; -import { FeatureGate } from '@/components/FeatureGate'; - -const PLAYBOOK_LABELS: Record = { - mean_reversion: 'Mean Reversion (MR)', - range_extremity: 'Range Extreme (RE)', -}; - -const EXECUTION_PATTERNS = Object.keys(PLAYBOOK_LABELS) as ExecutionPlaybook[]; - -function ExecutionPageInner() { - const store = useExecutionStore(); - const plan = store.executionPlaybook; - - if (!plan) { - return ( - -
-
-

Trade Execution

-

Live trade tracking with execution library

-
- -

No active execution plan. Create a trade plan first to begin tracking.

-
- - ); - } - - // --- Post-trade review state (must be inside component for hooks) --- - const strengthRef = useRef(null); - const [newStrength, setNewStrength] = useState(''); - const mistakeRef = useRef(null); - const [newMistake, setNewMistake] = useState(''); - - const addStrength = () => { - if (newStrength.trim()) { - store.addStrength(newStrength.trim()); - setNewStrength(''); - } - }; - - const addMistake = () => { - if (newMistake.trim()) { - store.addMistake(newMistake.trim(), 'analysis'); - setNewMistake(''); - } - }; - - // --- Execution detail view --- - return ( - -
- {/* Status Bar */} -
-
-
-

Trade Execution

- - {PLAYBOOK_LABELS[plan]} - - — {PLAYBOOK_LABELS[plan]} — -
-
- {store.checklistCompleted ? ( - ✓ Checklist Complete - ) : ( - ☐ Checklist Incomplete - )} - {/* Back to Plan List button removed - not implemented in store */} -
-
-
-
-

Trade Execution

-

- {PLAYBOOK_LABELS[plan]} — Live tracking and post-trade review -

-
- - {/* Back to Plan List button removed - not implemented in store */} -
- - {/* Checklist Status */} -
-
-
-

Execution Checklist

-

- {store.checklistCompleted ? '✓ All setup steps completed' : '☐ Complete live checklist before entry'} -

-
- -
-
- -
- - {/* Execution Pattern Library */} -
-

Execution Pattern

-

Select execution strategy pattern

- -
- {EXECUTION_PATTERNS.map(pattern => ( - - ))} -
-
- - {/* Scale-In Entry Table */} -
-

Scale-In Entry Table

-

Track partial fills, sizes, and strategies

- - {/* Add Entry Form */} - { - store.addScaleEntry(entry); - }} /> - - {/* Scale Table */} -
- - - - - - - - - - - - {store.scaleEntries.length === 0 ? ( - - - - ) : ( - store.scaleEntries.map((entry, idx) => ( - - - - - - - - )) - )} - -
#PriceSize %StrategyActions
No scale entries recorded
{idx + 1} - store.updateScaleEntry(idx, { price: parseFloat(e.target.value) || 0 })} - className="input text-xs w-28" /> - - store.updateScaleEntry(idx, { sizePercent: parseInt(e.target.value) || 0 })} - className="input text-xs w-20" /> - - - - -
-
- - {/* Scale Metrics */} -
- - -
-
- - {/* Emotion Logger — integrated component with localStorage persistence */} - - - {/* Execution Timing */} -
-

Execution Timing

-

Track actual vs planned entry timing

- -
- - -
- - -
-
-
- - {/* Post-Trade Review */} -
-

Post-Trade Review

-

Reflect on execution quality

- -
-
- -
- - -
-
- - {!store.followedPlan && ( -
- -