investor-flow/docs/TECH_DESIGN.md

Investor Flow — Technical Design (as built)

Audit date: 2026-07-23
Companion: FUNCTIONAL_DESIGN.md
Canonical code roots: app/src (Next frontend), app/server/src (Node backend)

---

1. Stack

Layer	Actual technology	Notes vs older docs
Frontend	Next.js 16.2, React 19, Tailwind v4, Recharts, Zustand, Radix UI	Not Bun SPA
Backend runtime	Node ≥22 (dev on Node 26), native TS via --experimental-strip-types	DESIGN said Bun; runtime glue is Node + node:sqlite
API	tRPC v11 HTTP, path /api/trpc/*	Frontend uses hand-rolled fetch client in app/src/lib/trpc.ts (not @trpc/client)
DB	SQLite file (app/server/data/investor-flow.db)	Single file multi-tenant
Market data	yahoo-finance2 v3 class API
SEC	EdgarAdapter + SecFetchAdapter + SecLintAdapter + secDataFetcher	Filer CIK from accession prefix (critical fix)
Social	XCookieAdapter (encrypted ct0/auth_token), RedditAdapter
Macro	FredAdapter	Key stored admin-side
Auth	Sessions + TOTP + OAuth (GitHub/Google)	Password hash via local crypto helpers
Tests	node --test --experimental-strip-types	Backend ~503 tests; 500 pass, 2 fail, 1 skip as of audit
Deploy	docker-compose.yml (frontend + backend + volume)	Present; operator-run

---

2. Process topology

Browser (Next :3000)
  │  fetch /api/trpc/<proc>  (credentials include)
  ▼
Next rewrites / proxy  →  Backend (:3001)
  │
  ├─ appRouter (tRPC)
  ├─ CacheRepository + AdapterQueue
  ├─ Source adapters (yfinance, sec-*, x, reddit, fred)
  └─ SQLite

• Dev helpers: restart-servers.sh, admin serverRestart.
• Queue drain loop + 30s schedule loop inside backend process.

---

3. Multi-tenant data tiers (still valid)

Tier	Content	Isolation
A	quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro	Shared, no owner
B	threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_*	Shared infrastructure
C	watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters	owner_id / user_id
D	users, sessions, admin_audit, x_credentials (singleton), llm_*	System

Demand set: symbol_demand refcount drives which symbols the queue refreshes.

Stale-while-revalidate: CacheRepository returns cached rows and schedules refresh when past TTL.

3.1 Rate-limit-first data plane (ADR-0009)

All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields stale/static UI, not stampede:

Rule	Mechanism
Request path never stampede	Serve SQLite / kv_cache / static fallback first; short timeout if live is unavoidable
One outbound owner	AdapterQueue only (background drain); UI schedules via CacheRepository.get
Steady throttle	Per-source min-interval (sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS)
429 cool-down	Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood
Demand-bounded work	Only symbol_demand symbols get scheduled refresh
Observability	queue.health().sourceCooldowns

Anti-patterns (do not reintroduce): N parallel Yahoo charts on a click; live quoteSummary without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge.

Slow-changing composition (ETF top holdings) uses kv_cache + etfHoldingsFallback.ts. Live upgrade is best-effort when the source is not cooling down.

---

4. Backend module map

app/server/src/
  index.ts                 HTTP server entry
  trpc/router.ts           ~2k LOC — all routers
  trpc/context.ts          session, cookies, db, cache, queue injection
  db/schema.sql            DDL (note: duplicate strategies blocks — debt)
   db/*Repository.ts        watchlist, portfolio, emotion logs, alertSubscription
  cache/CacheRepository.ts content-addressed cache API
  queue/AdapterQueue.ts    schedule, pause, retry, source cool-downs (ADR-0009)
  queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
  adapters/                YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
  analysis/                indicators, rotation, seasonality, etfHoldingsFallback, tickerContext
  admin/                   operator functions + CLI
  auth/                    totp, oauth, backup codes
   alerts/AlertEngine.ts
   alerts/producers/        types.ts, index.ts (registry), insiderProducer.ts, new13daProducer.ts
   services/emailAlertService.ts  nodemailer SMTP delivery + rate limit
   risk/RiskEngine.ts + haltCircuitBreaker.ts   ← pure; NOT on tRPC
  sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix  ← pure; NOT on tRPC
  strategy/BacktestEngine.ts
  screener/UniverseEvaluator.ts + SectorCrosslink.ts
  options/ConvexityGate.ts
  derisking/DeriskingEngine.ts
  thesis/ThesisMonitor.ts
  macro/FredAdapter.ts + MacroRegime.ts
  reports/ReportRunner.ts
  onboarding/starter.ts
  services/secDataFetcher.ts
  x/backfill.ts

4.1 tRPC surface (mounted)

Router	Procedures (summary)
auth	signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback
onboarding	starter, complete
market	snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings
dashboard	rollup
admin	users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport, smtpConfig, smtpConfigUpdate, smtpConfigTest
alerts	list, acknowledge, acknowledgeAll, unackedCount, createSubscription, listSubscriptions, updateSubscription, deleteSubscription
institutional	flow, insiderStream, ownershipHistory, buyEvents
edgar	filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx
watchlists	list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol
portfolio	holdings, addHolding, removeHolding
options	chain, greeks
reports	generate
screener	filter, strategy
strategies	list, create
backtest	run, evaluateLatest
sectorCrosslink	confirm
optionsConvexity	unlock, getPayoff
derisking	suggest
macro	series, calendar, regimeClassify, commentary, regimeHistory
thesisMonitor	assess, timeline
x	feed, cashtag_search, timeline, accountsForSymbol
reddit	subreddit, search
emotionLogger	add, getByTrade, delete

4.2 Sizing + risk (mounted 2026-07-18)

Router	Procedures
sizing	compute — pure sizePosition + portfolio/unlock/halt context
risk	posture — pure assessRisk + portfolio load + halt persist; haltStatus

Still deferred: journal create path that throws HaltedError when plans are server-persisted (persist-trade-plan-execution-loop).

---

5. Frontend architecture

app/src/
  app/                     Next App Router pages
  components/              Panels + ui kit + layout
  stores/                  Zustand (several persist to localStorage)
  lib/trpc.ts              Partial API client (subset of routers)
  lib/chart-theme.ts       CSS-variable themed Recharts
  lib/strings.ts           Primary-rule sensitive copy

5.1 Routes

Path	Role
/	Workbench overview
/chart-lab	Charts
/institutional	Institutional + filings
/filings	Filings only
/options-dd	Options DD
/market-outlook	Macro / rotation / notes
/trade-plan, /execution, /trade-closure, /daily-focus	Process (local)
/alerts	Alert events + subscriptions
/settings	Auth + onboarding
/mobile	Thin companion
/admin, /users, /queue, /audit-logs, /x-accounts, /smtp	Admin

5.2 Client coverage gap

lib/trpc.ts exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.

Not exposed in client (but on server): screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under market.commentary), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).

---

6. Caching & adapters

Source kind	Role	Typical freshness
yfinance	quote, candles, sector, options	minutes / EOD
sec / sec-fetch	filings, 13F, Form 4 bulk	daily schedule default
sec-lint-*	gap detection / backfill	on demand admin
x	timelines / cashtags	demand + prune
reddit	subreddit/search	on demand
fred / macro	series + commentary inputs	slower
llm	summaries (tables present)	sparse product use

AdapterQueue features (built): pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.

---

7. AuthZ model

Gate	Behavior
publicProcedure	No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens)
protectedProcedure	Session + users.status === 'active'
adminProcedure	is_admin flag

Watchlist list currently falls back to userId ?? 'anonymous' — convenient for local dev, weak isolation if exposed beyond localhost.

---

8. Testing

Suite	Command	Audit result
Backend	cd app/server && npm test	504 pass / 0 fail / 1 skip (after P0 2026-07-18)
Frontend lite	Primary-Rule lint + options pure tests	Present

Known red: MacroRegime commentary tests (generateMacroCommentary short/long-term + no macro-trade recommendation).

---

9. Schema debt

1. strategies defined twice in schema.sql (legacy regime_gate/setup/risk_policy vs components/unlocked). SQLite keeps first-created shape depending on migration history — dangerous drift.
2. Parallel filter tables: screener_filters vs saved_filters.
3. Process tables (trades, trade_executions, theses, emotion_logs) exist ahead of UI persistence.

---

10. LLM / provenance (as built vs design)

Designed	As built
Full LLMGateway with classifyPayload gate	Tables llm_providers, llm_dispatch_audit, llm_summaries; no complete gateway module under src/
Ornith default is_local=true	Env-driven URL in docker-compose (LLM_PROVIDER_URL)
Sensitive thesis data never leaves host	Thesis content mostly not yet flowing through LLM product paths

ADR-0006/0008 still govern intent; implementation is incomplete.

---

11. Deployment

# Local dev (typical)
cd app/server && npm run dev   # :3001
cd app && npm run dev          # :3000

# Compose
docker compose up --build

Secrets: .env.example documents session secret, SEC operator email, X cookie fields, Reddit, FRED, LLM URL.

---

12. Documentation map (after this audit)

Doc	Role
docs/FUNCTIONAL_DESIGN.md	What users can do / pending
docs/TECH_DESIGN.md	This file — how it is built
CONTEXT.md	Ubiquitous language (glossary only)
docs/adr/*	Durable decisions
Obsidian investor-flow.md	Session orientation wiki
.automaton/tasks/*	Work items; many complete/ slices are historical

Deprecated as status sources: root HANDOFF.md (2026-06-30 orchestrator snapshot), Automaton WORKFLOW “21 tasks” pool list, slice DECOMPOSITION checkbox state.