investor-flow/docs/TECH_DESIGN.md

# Investor Flow — Technical Design (as built)

**Audit date:** 2026-07-23  
**Companion:** [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md)  
**Canonical code roots:** `app/src` (Next frontend), `app/server/src` (Node backend)

---

## 1. Stack

| Layer | Actual technology | Notes vs older docs |
|-------|-------------------|---------------------|
| Frontend | Next.js **16.2**, React **19**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA |
| Backend runtime | **Node ≥22** (dev on Node 26), native TS via `--experimental-strip-types` | DESIGN said Bun; runtime glue is Node + `node:sqlite` |
| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`) |
| DB | SQLite file (`app/server/data/investor-flow.db`) | Single file multi-tenant |
| Market data | `yahoo-finance2` **v3** class API | |
| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + `secDataFetcher` | Filer CIK from accession prefix (critical fix) |
| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | |
| Macro | FredAdapter | Key stored admin-side |
| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers |
| Tests | `node --test --experimental-strip-types` | Backend ~503 tests; **500 pass, 2 fail, 1 skip** as of audit |
| Deploy | `docker-compose.yml` (frontend + backend + volume) | Present; operator-run |

---

## 2. Process topology

```
Browser (Next :3000)
  │  fetch /api/trpc/<proc>  (credentials include)
  ▼
Next rewrites / proxy  →  Backend (:3001)
  │
  ├─ appRouter (tRPC)
  ├─ CacheRepository + AdapterQueue
  ├─ Source adapters (yfinance, sec-*, x, reddit, fred)
  └─ SQLite
```

- Dev helpers: `restart-servers.sh`, admin `serverRestart`.
- Queue drain loop + 30s schedule loop inside backend process.

---

## 3. Multi-tenant data tiers (still valid)

| Tier | Content | Isolation |
|------|---------|-----------|
| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro | Shared, no owner |
| B | threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_* | Shared infrastructure |
| C | watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters | `owner_id` / `user_id` |
| D | users, sessions, admin_audit, x_credentials (singleton), llm_* | System |

**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes.

**Stale-while-revalidate:** CacheRepository returns cached rows and schedules refresh when past TTL.

### 3.1 Rate-limit-first data plane (ADR-0009)

All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:

| Rule | Mechanism |
|------|-----------|
| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; **no live Yahoo on tRPC** (queue only) |
| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` (rotation universe) |
| TTL-aware tiers | `yfinance-quote` (5m), `yfinance-eod` (6h candles), `yfinance-meta` (daily), `yfinance-holdings` (weekly) |
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
| Observability | `queue.health()`: vendorGate+queue_state cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |

**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow.

### Dealer Flow data plane (2026-08)

- Engine: pure `dealerExposureEngine` on `NormalizedOptionSurface` only (no vendor imports).
- Default provider: Yahoo via `composeYFinanceWithOptions` + `OPTIONS_CHAIN_PROVIDER` (default `yfinance`).
- Paid switch later: implement SourceFetch for `tradier`/`polygon`, register in `sourceRatePolicy`, set env - engine unchanged.
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
- SEC institutional (alert-critical): SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`); `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done); `requeueUnhealthySecSymbols` caps heal requeues per tick.
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. First-paint UI is GEX | VEX only; Method drawer has Customer book | Dealer inventory.
- **Vendor rate-limit enforcement (hard requirement, all sources + future):** process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
- LLM: per-user OpenAI-compatible `base_url` + encrypted key + model (`userLlm.*`); not OpenAI-only.
- Dealer Flow plain-English notes: in-app `dealerFlowExplainNotes.ts` only (L0/L1). Optional offline scripts harvest X handles and distill into that file **and** write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
- Study Desk: pure `dealerStudyEngine` propose/grade; table `dealer_study_setups`; tRPC `dealerStudy.*` (propose, log, list, grade, gradeDue, scorecard). Grades use `price_candles` 1d barrier logic (target before invalidation). Complementary to strategy `backtest.*` - not the same surface.

Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts` + queued `yfinance:topHoldings:*` refresh.

---

## 4. Backend module map

```
app/server/src/
  index.ts                 HTTP server entry
  trpc/router.ts           ~2k LOC — all routers
  trpc/context.ts          session, cookies, db, cache, queue injection
  db/schema.sql            DDL (note: duplicate strategies blocks — debt)
   db/*Repository.ts        watchlist, portfolio, emotion logs, alertSubscription
  cache/CacheRepository.ts content-addressed cache API
  queue/AdapterQueue.ts    schedule, pause, retry, source cool-downs (ADR-0009)
  queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
  adapters/                YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
  analysis/                indicators, rotation, seasonality, etfHoldingsFallback, tickerContext, dealerExposureEngine, dealerMapService, dealerMapExplain, dealerFlowExplainNotes, dealerStudyEngine
  options/                 bsm, types (NormalizedOptionSurface), OptionsChainRouter (paid-ready provider seam)
  llm/                     openaiCompatible client, userLlmEndpoint
  admin/                   operator functions + CLI
  auth/                    totp, oauth, backup codes
   alerts/AlertEngine.ts
   alerts/producers/        types.ts, index.ts (registry), insiderProducer.ts, new13daProducer.ts
   services/emailAlertService.ts  nodemailer SMTP delivery + rate limit
   risk/RiskEngine.ts + haltCircuitBreaker.ts   ← pure; NOT on tRPC
  sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix  ← pure; NOT on tRPC
  strategy/BacktestEngine.ts
  screener/UniverseEvaluator.ts + SectorCrosslink.ts
  options/ConvexityGate.ts
  derisking/DeriskingEngine.ts
  thesis/ThesisMonitor.ts
  macro/FredAdapter.ts + MacroRegime.ts
  reports/ReportRunner.ts
  onboarding/starter.ts
  services/secDataFetcher.ts
  x/backfill.ts
```

### 4.1 tRPC surface (mounted)

| Router | Procedures (summary) |
|--------|----------------------|
| `auth` | signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback |
| `onboarding` | starter, complete |
| `market` | snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings |
| `dashboard` | rollup |
| `admin` | users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport, smtpConfig, smtpConfigUpdate, smtpConfigTest |
| `alerts` | list, acknowledge, acknowledgeAll, unackedCount, createSubscription, listSubscriptions, updateSubscription, deleteSubscription |
| `institutional` | flow, insiderStream, ownershipHistory, buyEvents |
| `edgar` | filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx |
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol |
| `portfolio` | holdings, addHolding, removeHolding |
| `options` | chain, greeks |
| `dealerMap` | get, levels, scenario, velocity, explain — cache-only reads; schedule options chains via queue (ADR-0009) |
| `dealerStudy` | propose (hist + optional mentor rank), log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard — local mentor path-match; claim types map to study hypotheses for Phase-3 blend |
| `userLlm` | status, upsertEndpoint, clear, test — per-user OpenAI-compatible base_url + encrypted key + model |
| `reports` | generate |
| `screener` | filter, strategy |
| `strategies` | list, create |
| `backtest` | run, evaluateLatest |
| `sectorCrosslink` | confirm |
| `optionsConvexity` | unlock, getPayoff |
| `derisking` | suggest |
| `macro` | series, calendar, regimeClassify, commentary, regimeHistory |
| `thesisMonitor` | assess, timeline |
| `x` | feed, cashtag_search, timeline, accountsForSymbol |
| `reddit` | subreddit, search |
| `emotionLogger` | add, getByTrade, delete |

### 4.2 Sizing + risk (mounted 2026-07-18)

| Router | Procedures |
|--------|------------|
| `sizing` | `compute` — pure `sizePosition` + portfolio/unlock/halt context |
| `risk` | `posture` — pure `assessRisk` + portfolio load + halt persist; `haltStatus` |

Still deferred: journal create path that throws `HaltedError` when plans are server-persisted (`persist-trade-plan-execution-loop`).

---

## 5. Frontend architecture

```
app/src/
  app/                     Next App Router pages
  components/              Panels + ui kit + layout
  stores/                  Zustand (several persist to localStorage)
  lib/trpc.ts              Partial API client (subset of routers)
  lib/chart-theme.ts       CSS-variable themed Recharts
  lib/strings.ts           Primary-rule sensitive copy
```

### 5.1 Routes

| Path | Role |
|------|------|
| `/` | Workbench overview |
| `/chart-lab` | Charts |
| `/institutional` | Institutional + filings |
| `/filings` | Filings only |
| `/options-dd` | Options DD |
| `/market-outlook` | Macro / rotation / notes |
| `/trade-plan`, `/execution`, `/trade-closure`, `/daily-focus` | Process (local) |
| `/alerts` | Alert events + subscriptions |
| `/settings` | Auth + onboarding |
| `/mobile` | Thin companion |
| `/admin`, `/users`, `/queue`, `/audit-logs`, `/x-accounts`, `/smtp` | Admin |

### 5.2 Client coverage gap

`lib/trpc.ts` exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.

**Not exposed in client (but on server):** screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under `market.commentary`), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).

---

## 6. Caching & adapters

| Source kind | Role | Typical freshness |
|-------------|------|-------------------|
| yfinance | quote, candles, sector, options | minutes / EOD |
| sec / sec-fetch | filings, 13F, Form 4 bulk | daily schedule default |
| sec-lint-* | gap detection / backfill | on demand admin |
| x | timelines / cashtags | demand + prune |
| reddit | subreddit/search | on demand |
| fred / macro | series + commentary inputs | slower |
| llm | summaries (tables present) | sparse product use |

**AdapterQueue features (built):** pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.

---

## 7. AuthZ model

| Gate | Behavior |
|------|----------|
| `publicProcedure` | No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens) |
| `protectedProcedure` | Session + `users.status === 'active'` |
| `adminProcedure` | `is_admin` flag |

Watchlist list currently falls back to `userId ?? 'anonymous'` — convenient for local dev, weak isolation if exposed beyond localhost.

---

## 8. Testing

| Suite | Command | Audit result |
|-------|---------|--------------|
| Backend | `cd app/server && npm test` | **504 pass / 0 fail / 1 skip** (after P0 2026-07-18) |
| Frontend lite | Primary-Rule lint + options pure tests | Present |

Known red: `MacroRegime` commentary tests (`generateMacroCommentary` short/long-term + no macro-trade recommendation).

---

## 9. Schema debt

1. **`strategies` defined twice** in `schema.sql` (legacy `regime_gate/setup/risk_policy` vs `components/unlocked`). SQLite keeps first-created shape depending on migration history — dangerous drift.
2. Parallel filter tables: `screener_filters` vs `saved_filters`.
3. Process tables (`trades`, `trade_executions`, `theses`, `emotion_logs`) exist ahead of UI persistence.

---

## 10. LLM / provenance (as built vs design)

| Designed | As built |
|----------|----------|
| Full LLMGateway with classifyPayload gate | Tables `llm_providers`, `llm_dispatch_audit`, `llm_summaries`; no complete gateway module under `src/` |
| Ornith default `is_local=true` | Env-driven URL in docker-compose (`LLM_PROVIDER_URL`) |
| Sensitive thesis data never leaves host | Thesis content mostly not yet flowing through LLM product paths |

ADR-0006/0008 still govern intent; implementation is incomplete.

---

## 11. Deployment

Local dev (typical):

```bash
cd app/server && npm run dev   # :3001
cd app && npm run dev          # :3000
```

Production is two Node containers (not Bun). Unraid is the intended host; git + Gitea Actions already live at `unraid.local:3003/transnet/investor-flow`.

```bash
# Build-from-git (laptop or Unraid checkout)
docker compose --env-file .env up -d --build
```

| Piece | Path |
|---|---|
| Backend image | `app/server/Dockerfile` (Node 22, `node --experimental-strip-types`) |
| Frontend image | `app/Dockerfile` (Next standalone, proxies `/api` via `IFLOW_BACKEND_URL`) |
| Unraid operator guide | `docs/DEPLOY_UNRAID.md` |
| Gitea CI | `.github/workflows/ci.yml` |
| Gitea CD (registry push) | `.github/workflows/cd.yml` |
| act_runner | `deploy/unraid/act-runner-compose.yml` |
| Cron CD (no runner) | `deploy/unraid/user-scripts/sync-and-up.sh` |

Secrets: `.env.example` and `deploy/unraid/.env.example`. Production refuses to start without `IFLOW_SESSION_SECRET`. Persist SQLite on a cache/single-disk path (`IFLOW_DATA_DIR`), not `/mnt/user`.

---

## 12. Documentation map (after this audit)

| Doc | Role |
|-----|------|
| `docs/FUNCTIONAL_DESIGN.md` | What users can do / pending |
| `docs/TECH_DESIGN.md` | This file — how it is built |
| `CONTEXT.md` | Ubiquitous language (glossary only) |
| `docs/adr/*` | Durable decisions |
| Obsidian `investor-flow.md` | Session orientation wiki |

Deprecated as status sources: root `HANDOFF.md` (2026-06-30 orchestrator snapshot), slice DECOMPOSITION checkbox state.