investor-flow/docs/TECH_DESIGN.md

Investor Flow — Technical Design (as built)

Audit date: 2026-07-23
Companion: FUNCTIONAL_DESIGN.md
Canonical code roots: app/src (Next frontend), app/server/src (Node backend)

---

1. Stack

Layer	Actual technology	Notes vs older docs
Frontend	Next.js 16.2, React 19, Tailwind v4, Recharts, Zustand, Radix UI	Not Bun SPA
Backend runtime	Node ≥22 (dev on Node 26), native TS via --experimental-strip-types	DESIGN said Bun; runtime glue is Node + node:sqlite
API	tRPC v11 HTTP, path /api/trpc/*	Frontend uses hand-rolled fetch client in app/src/lib/trpc.ts (not @trpc/client)
DB	SQLite file (app/server/data/investor-flow.db)	Single file multi-tenant
Market data	yahoo-finance2 v3 class API
SEC	EdgarAdapter + SecFetchAdapter + SecLintAdapter + secDataFetcher	Filer CIK from accession prefix (critical fix)
Social	XCookieAdapter (encrypted ct0/auth_token), RedditAdapter
Macro	FredAdapter	Key stored admin-side
Auth	Sessions + TOTP + OAuth (GitHub/Google)	Password hash via local crypto helpers
Tests	node --test --experimental-strip-types	Backend ~503 tests; 500 pass, 2 fail, 1 skip as of audit
Deploy	docker-compose.yml (frontend + backend + volume)	Present; operator-run

---

2. Process topology

Browser (Next :3000)
  │  fetch /api/trpc/<proc>  (credentials include)
  ▼
Next rewrites / proxy  →  Backend (:3001)
  │
  ├─ appRouter (tRPC)
  ├─ CacheRepository + AdapterQueue
  ├─ Source adapters (yfinance, sec-*, x, reddit, fred)
  └─ SQLite

• Dev helpers: restart-servers.sh, admin serverRestart.
• Queue drain loop + 30s schedule loop inside backend process.

---

3. Multi-tenant data tiers (still valid)

Tier	Content	Isolation
A	quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro	Shared, no owner
B	threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_*	Shared infrastructure
C	watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters	owner_id / user_id
D	users, sessions, admin_audit, x_credentials (singleton), llm_*	System

Demand set: symbol_demand refcount drives which symbols the queue refreshes.

Stale-while-revalidate: CacheRepository returns cached rows and schedules refresh when past TTL.

3.1 Rate-limit-first data plane (ADR-0009)

All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields stale/static UI, not stampede:

Rule	Mechanism
Request path never stampede	Serve SQLite / kv_cache / static fallback first; no live Yahoo on tRPC (queue only)
One outbound owner	AdapterQueue only (background drain); UI schedules via CacheRepository.get
Steady throttle	Per-source min-interval (sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS)
429 cool-down	Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood
Demand-bounded work	Watchlist/portfolio subscribe + ensureInDemand / pinSystemSymbol (rotation universe)
TTL-aware tiers	yfinance-quote (5m), yfinance-eod (6h candles), yfinance-meta (daily), yfinance-holdings (weekly)
Incremental candles	Warm symbols re-fetch ~14d lookback, not full 10y every tick
Poison quarantine	Delisted / not-found symbols fail permanent; not requeued; demand cleared
Per-kind drain budgets	Quotes cannot starve symbol meta / candles forever
Observability	queue.health(): cooldowns, pendingByKind, demand size, SPY candle lag, dataPlaneHealthy

Anti-patterns (do not reintroduce): N parallel Yahoo charts on a click; live quoteSummary without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling subscribe on every page view (inflates refcount - use ensureInDemand); dealerMap.get calling Yahoo directly; frontend looping expiries to paint Dealer Flow.

Dealer Flow data plane (2026-08)

• Engine: pure dealerExposureEngine on NormalizedOptionSurface only (no vendor imports).
• Default provider: Yahoo via composeYFinanceWithOptions + OPTIONS_CHAIN_PROVIDER (default yfinance).
• Paid switch later: implement SourceFetch for tradier/polygon, register in sourceRatePolicy, set env - engine unchanged.
• Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily dealer_map_snapshots for velocity.
• Integrity: pure dealerMapIntegrity hard/soft checks (missing expiries/OI/greeks fail; delay does not); data_quality kind dealer_map; replay via dealerMapReplay + scripts/dealer-map-replay.ts (as-of chain ts).
• SEC institutional (alert-critical): SC-first fetch seeds sec:cusip:SYMBOL; offline curated CUSIP registry (cusipRegistry) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to reverse 13F (reverse13fRefresh: prior holders + tracked funds + major managers via data.sec.gov); SecFetchAdapter throws on hard resolve failure so queue retries (no silent done); requeueUnhealthySecSymbols caps heal requeues per tick.
• Dealer GEX sign convention: maps are stored as classic OI GEX (classic_call_pos_put_neg: call +, put −). Request path can re-express as dealer_inventory (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via withExposureConvention / dealerMap.get({ convention }). UI toggle: Classic | Dealer (HS).
• Vendor rate-limit enforcement (hard requirement, all sources + future): process-wide vendorGate with open registration (registerVendorIntegration). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind source_kind before AdapterQueue construction (throws otherwise). Prefer VendorSourceAdapter / defineVendorAdapter so fetchOne is auto-gated. HTTP via vendorFetch / secHttp; SDKs via withVendorGate. CI guard bans bare fetch( in adapters/services. See docs/VENDOR_INTEGRATIONS.md.
• LLM: per-user OpenAI-compatible base_url + encrypted key + model (userLlm.*); not OpenAI-only.
• Dealer Flow plain-English notes: in-app dealerFlowExplainNotes.ts only (L0/L1). Optional offline scripts harvest X handles and distill into that file and write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
• Study Desk: pure dealerStudyEngine propose/grade; table dealer_study_setups; tRPC dealerStudy.* (propose, log, list, grade, gradeDue, scorecard). Grades use price_candles 1d barrier logic (target before invalidation). Complementary to strategy backtest.* - not the same surface.

Slow-changing composition (ETF top holdings) uses kv_cache + etfHoldingsFallback.ts + queued yfinance:topHoldings:* refresh.

---

4. Backend module map

app/server/src/
  index.ts                 HTTP server entry
  trpc/router.ts           ~2k LOC — all routers
  trpc/context.ts          session, cookies, db, cache, queue injection
  db/schema.sql            DDL (note: duplicate strategies blocks — debt)
   db/*Repository.ts        watchlist, portfolio, emotion logs, alertSubscription
  cache/CacheRepository.ts content-addressed cache API
  queue/AdapterQueue.ts    schedule, pause, retry, source cool-downs (ADR-0009)
  queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
  adapters/                YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
  analysis/                indicators, rotation, seasonality, etfHoldingsFallback, tickerContext, dealerExposureEngine, dealerMapService, dealerMapExplain, dealerFlowExplainNotes, dealerStudyEngine
  options/                 bsm, types (NormalizedOptionSurface), OptionsChainRouter (paid-ready provider seam)
  llm/                     openaiCompatible client, userLlmEndpoint
  admin/                   operator functions + CLI
  auth/                    totp, oauth, backup codes
   alerts/AlertEngine.ts
   alerts/producers/        types.ts, index.ts (registry), insiderProducer.ts, new13daProducer.ts
   services/emailAlertService.ts  nodemailer SMTP delivery + rate limit
   risk/RiskEngine.ts + haltCircuitBreaker.ts   ← pure; NOT on tRPC
  sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix  ← pure; NOT on tRPC
  strategy/BacktestEngine.ts
  screener/UniverseEvaluator.ts + SectorCrosslink.ts
  options/ConvexityGate.ts
  derisking/DeriskingEngine.ts
  thesis/ThesisMonitor.ts
  macro/FredAdapter.ts + MacroRegime.ts
  reports/ReportRunner.ts
  onboarding/starter.ts
  services/secDataFetcher.ts
  x/backfill.ts

4.1 tRPC surface (mounted)

Router	Procedures (summary)
auth	signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback
onboarding	starter, complete
market	snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings
dashboard	rollup
admin	users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport, smtpConfig, smtpConfigUpdate, smtpConfigTest
alerts	list, acknowledge, acknowledgeAll, unackedCount, createSubscription, listSubscriptions, updateSubscription, deleteSubscription
institutional	flow, insiderStream, ownershipHistory, buyEvents
edgar	filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx
watchlists	list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol
portfolio	holdings, addHolding, removeHolding
options	chain, greeks
dealerMap	get, levels, scenario, velocity, explain — cache-only reads; schedule options chains via queue (ADR-0009)
dealerStudy	propose (hist + optional mentor rank), log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv
mentorLedger	importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard — local mentor path-match; claim types map to study hypotheses for Phase-3 blend
userLlm	status, upsertEndpoint, clear, test — per-user OpenAI-compatible base_url + encrypted key + model
reports	generate
screener	filter, strategy
strategies	list, create
backtest	run, evaluateLatest
sectorCrosslink	confirm
optionsConvexity	unlock, getPayoff
derisking	suggest
macro	series, calendar, regimeClassify, commentary, regimeHistory
thesisMonitor	assess, timeline
x	feed, cashtag_search, timeline, accountsForSymbol
reddit	subreddit, search
emotionLogger	add, getByTrade, delete

4.2 Sizing + risk (mounted 2026-07-18)

Router	Procedures
sizing	compute — pure sizePosition + portfolio/unlock/halt context
risk	posture — pure assessRisk + portfolio load + halt persist; haltStatus

Still deferred: journal create path that throws HaltedError when plans are server-persisted (persist-trade-plan-execution-loop).

---

5. Frontend architecture

app/src/
  app/                     Next App Router pages
  components/              Panels + ui kit + layout
  stores/                  Zustand (several persist to localStorage)
  lib/trpc.ts              Partial API client (subset of routers)
  lib/chart-theme.ts       CSS-variable themed Recharts
  lib/strings.ts           Primary-rule sensitive copy

5.1 Routes

Path	Role
/	Workbench overview
/chart-lab	Charts
/institutional	Institutional + filings
/filings	Filings only
/options-dd	Options DD
/market-outlook	Macro / rotation / notes
/trade-plan, /execution, /trade-closure, /daily-focus	Process (local)
/alerts	Alert events + subscriptions
/settings	Auth + onboarding
/mobile	Thin companion
/admin, /users, /queue, /audit-logs, /x-accounts, /smtp	Admin

5.2 Client coverage gap

lib/trpc.ts exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.

Not exposed in client (but on server): screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under market.commentary), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).

---

6. Caching & adapters

Source kind	Role	Typical freshness
yfinance	quote, candles, sector, options	minutes / EOD
sec / sec-fetch	filings, 13F, Form 4 bulk	daily schedule default
sec-lint-*	gap detection / backfill	on demand admin
x	timelines / cashtags	demand + prune
reddit	subreddit/search	on demand
fred / macro	series + commentary inputs	slower
llm	summaries (tables present)	sparse product use

AdapterQueue features (built): pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.

---

7. AuthZ model

Gate	Behavior
publicProcedure	No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens)
protectedProcedure	Session + users.status === 'active'
adminProcedure	is_admin flag

Watchlist list currently falls back to userId ?? 'anonymous' — convenient for local dev, weak isolation if exposed beyond localhost.

---

8. Testing

Suite	Command	Audit result
Backend	cd app/server && npm test	504 pass / 0 fail / 1 skip (after P0 2026-07-18)
Frontend lite	Primary-Rule lint + options pure tests	Present

Known red: MacroRegime commentary tests (generateMacroCommentary short/long-term + no macro-trade recommendation).

---

9. Schema debt

1. strategies defined twice in schema.sql (legacy regime_gate/setup/risk_policy vs components/unlocked). SQLite keeps first-created shape depending on migration history — dangerous drift.
2. Parallel filter tables: screener_filters vs saved_filters.
3. Process tables (trades, trade_executions, theses, emotion_logs) exist ahead of UI persistence.

---

10. LLM / provenance (as built vs design)

Designed	As built
Full LLMGateway with classifyPayload gate	Tables llm_providers, llm_dispatch_audit, llm_summaries; no complete gateway module under src/
Ornith default is_local=true	Env-driven URL in docker-compose (LLM_PROVIDER_URL)
Sensitive thesis data never leaves host	Thesis content mostly not yet flowing through LLM product paths

ADR-0006/0008 still govern intent; implementation is incomplete.

---

11. Deployment

# Local dev (typical)
cd app/server && npm run dev   # :3001
cd app && npm run dev          # :3000

# Compose
docker compose up --build

Secrets: .env.example documents session secret, SEC operator email, X cookie fields, Reddit, FRED, LLM URL.

---

12. Documentation map (after this audit)

Doc	Role
docs/FUNCTIONAL_DESIGN.md	What users can do / pending
docs/TECH_DESIGN.md	This file — how it is built
CONTEXT.md	Ubiquitous language (glossary only)
docs/adr/*	Durable decisions
Obsidian investor-flow.md	Session orientation wiki

Deprecated as status sources: root HANDOFF.md (2026-06-30 orchestrator snapshot), slice DECOMPOSITION checkbox state.