Investor Flow Build
ac94acf9e3
CI / Test & Type-Check (push) Canceled after 0s
Snapshot of in-progress module work across multiple slices: - Dealer Flow: dealerExposureEngine, dealerMapService, dealerMapExplain, dealerMapIntegrity, dealerMapReplay, dealerStudyEngine, hanStyleLevels - Mirror Portfolio (M21): fundRepository, captureIngest, mirrorAlertProducers, fund holdings strip, live book, position capture ingest - Options: BSM, NormalizedOptionSurface types, OptionsChainRouter, ConvexityGate, option legs panel - Alert producers: vixLevel, rotation, thesis, unlock, portfolioRisk, mirror (fund_capture, fund_13f, mirror_diff) - FINRA short interest adapter + queue integration - SEC company tickers adapter + ingest (symbol search index seed) - Vendor gate (rate-limit-first data plane, ADR-0009) - CUSIP registry, reverse 13F refresh, stock float service - LRU cache, portfolio backtest engine - Frontend: dealer-flow, funds, journal, lab, monitor, plan, portfolio, reports, screener, strategies, theses, guided-start, exits, more pages - Volume profile, workspace profile, visibility-aware poll - ADRs 0010 (mirror math not advice), 0011 (symbol search index) - VENDOR_INTEGRATIONS.md, END_USER_TEST.md - .gitignore: exclude DBs, .DS_Store, local config, agent scratch
295 lines
16 KiB
Markdown
295 lines
16 KiB
Markdown
# Investor Flow — Technical Design (as built)
|
||
|
||
**Audit date:** 2026-07-23
|
||
**Companion:** [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md)
|
||
**Canonical code roots:** `app/src` (Next frontend), `app/server/src` (Node backend)
|
||
|
||
---
|
||
|
||
## 1. Stack
|
||
|
||
| Layer | Actual technology | Notes vs older docs |
|
||
|-------|-------------------|---------------------|
|
||
| Frontend | Next.js **16.2**, React **19**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA |
|
||
| Backend runtime | **Node ≥22** (dev on Node 26), native TS via `--experimental-strip-types` | DESIGN said Bun; runtime glue is Node + `node:sqlite` |
|
||
| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`) |
|
||
| DB | SQLite file (`app/server/data/investor-flow.db`) | Single file multi-tenant |
|
||
| Market data | `yahoo-finance2` **v3** class API | |
|
||
| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + `secDataFetcher` | Filer CIK from accession prefix (critical fix) |
|
||
| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | |
|
||
| Macro | FredAdapter | Key stored admin-side |
|
||
| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers |
|
||
| Tests | `node --test --experimental-strip-types` | Backend ~503 tests; **500 pass, 2 fail, 1 skip** as of audit |
|
||
| Deploy | `docker-compose.yml` (frontend + backend + volume) | Present; operator-run |
|
||
|
||
---
|
||
|
||
## 2. Process topology
|
||
|
||
```
|
||
Browser (Next :3000)
|
||
│ fetch /api/trpc/<proc> (credentials include)
|
||
▼
|
||
Next rewrites / proxy → Backend (:3001)
|
||
│
|
||
├─ appRouter (tRPC)
|
||
├─ CacheRepository + AdapterQueue
|
||
├─ Source adapters (yfinance, sec-*, x, reddit, fred)
|
||
└─ SQLite
|
||
```
|
||
|
||
- Dev helpers: `restart-servers.sh`, admin `serverRestart`.
|
||
- Queue drain loop + 30s schedule loop inside backend process.
|
||
|
||
---
|
||
|
||
## 3. Multi-tenant data tiers (still valid)
|
||
|
||
| Tier | Content | Isolation |
|
||
|------|---------|-----------|
|
||
| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro | Shared, no owner |
|
||
| B | threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_* | Shared infrastructure |
|
||
| C | watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters | `owner_id` / `user_id` |
|
||
| D | users, sessions, admin_audit, x_credentials (singleton), llm_* | System |
|
||
|
||
**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes.
|
||
|
||
**Stale-while-revalidate:** CacheRepository returns cached rows and schedules refresh when past TTL.
|
||
|
||
### 3.1 Rate-limit-first data plane (ADR-0009)
|
||
|
||
All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:
|
||
|
||
| Rule | Mechanism |
|
||
|------|-----------|
|
||
| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; **no live Yahoo on tRPC** (queue only) |
|
||
| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
|
||
| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
|
||
| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
|
||
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` (rotation universe) |
|
||
| TTL-aware tiers | `yfinance-quote` (5m), `yfinance-eod` (6h candles), `yfinance-meta` (daily), `yfinance-holdings` (weekly) |
|
||
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
|
||
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
|
||
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
|
||
| Observability | `queue.health()`: cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||
|
||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow.
|
||
|
||
### Dealer Flow data plane (2026-08)
|
||
|
||
- Engine: pure `dealerExposureEngine` on `NormalizedOptionSurface` only (no vendor imports).
|
||
- Default provider: Yahoo via `composeYFinanceWithOptions` + `OPTIONS_CHAIN_PROVIDER` (default `yfinance`).
|
||
- Paid switch later: implement SourceFetch for `tradier`/`polygon`, register in `sourceRatePolicy`, set env - engine unchanged.
|
||
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
|
||
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
|
||
- SEC institutional (alert-critical): SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`); `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done); `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. UI toggle: Classic | Dealer (HS).
|
||
- **Vendor rate-limit enforcement (hard requirement, all sources + future):** process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||
- LLM: per-user OpenAI-compatible `base_url` + encrypted key + model (`userLlm.*`); not OpenAI-only.
|
||
- Dealer Flow plain-English notes: in-app `dealerFlowExplainNotes.ts` only (L0/L1). Optional offline scripts harvest X handles and distill into that file **and** write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
|
||
- Study Desk: pure `dealerStudyEngine` propose/grade; table `dealer_study_setups`; tRPC `dealerStudy.*` (propose, log, list, grade, gradeDue, scorecard). Grades use `price_candles` 1d barrier logic (target before invalidation). Complementary to strategy `backtest.*` - not the same surface.
|
||
|
||
Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts` + queued `yfinance:topHoldings:*` refresh.
|
||
|
||
---
|
||
|
||
## 4. Backend module map
|
||
|
||
```
|
||
app/server/src/
|
||
index.ts HTTP server entry
|
||
trpc/router.ts ~2k LOC — all routers
|
||
trpc/context.ts session, cookies, db, cache, queue injection
|
||
db/schema.sql DDL (note: duplicate strategies blocks — debt)
|
||
db/*Repository.ts watchlist, portfolio, emotion logs, alertSubscription
|
||
cache/CacheRepository.ts content-addressed cache API
|
||
queue/AdapterQueue.ts schedule, pause, retry, source cool-downs (ADR-0009)
|
||
queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
|
||
adapters/ YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
|
||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback, tickerContext, dealerExposureEngine, dealerMapService, dealerMapExplain, dealerFlowExplainNotes, dealerStudyEngine
|
||
options/ bsm, types (NormalizedOptionSurface), OptionsChainRouter (paid-ready provider seam)
|
||
llm/ openaiCompatible client, userLlmEndpoint
|
||
admin/ operator functions + CLI
|
||
auth/ totp, oauth, backup codes
|
||
alerts/AlertEngine.ts
|
||
alerts/producers/ types.ts, index.ts (registry), insiderProducer.ts, new13daProducer.ts
|
||
services/emailAlertService.ts nodemailer SMTP delivery + rate limit
|
||
risk/RiskEngine.ts + haltCircuitBreaker.ts ← pure; NOT on tRPC
|
||
sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix ← pure; NOT on tRPC
|
||
strategy/BacktestEngine.ts
|
||
screener/UniverseEvaluator.ts + SectorCrosslink.ts
|
||
options/ConvexityGate.ts
|
||
derisking/DeriskingEngine.ts
|
||
thesis/ThesisMonitor.ts
|
||
macro/FredAdapter.ts + MacroRegime.ts
|
||
reports/ReportRunner.ts
|
||
onboarding/starter.ts
|
||
services/secDataFetcher.ts
|
||
x/backfill.ts
|
||
```
|
||
|
||
### 4.1 tRPC surface (mounted)
|
||
|
||
| Router | Procedures (summary) |
|
||
|--------|----------------------|
|
||
| `auth` | signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback |
|
||
| `onboarding` | starter, complete |
|
||
| `market` | snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings |
|
||
| `dashboard` | rollup |
|
||
| `admin` | users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport, smtpConfig, smtpConfigUpdate, smtpConfigTest |
|
||
| `alerts` | list, acknowledge, acknowledgeAll, unackedCount, createSubscription, listSubscriptions, updateSubscription, deleteSubscription |
|
||
| `institutional` | flow, insiderStream, ownershipHistory, buyEvents |
|
||
| `edgar` | filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx |
|
||
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol |
|
||
| `portfolio` | holdings, addHolding, removeHolding |
|
||
| `options` | chain, greeks |
|
||
| `dealerMap` | get, levels, scenario, velocity, explain — cache-only reads; schedule options chains via queue (ADR-0009) |
|
||
| `dealerStudy` | propose (hist + optional mentor rank), log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
|
||
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard — local mentor path-match; claim types map to study hypotheses for Phase-3 blend |
|
||
| `userLlm` | status, upsertEndpoint, clear, test — per-user OpenAI-compatible base_url + encrypted key + model |
|
||
| `reports` | generate |
|
||
| `screener` | filter, strategy |
|
||
| `strategies` | list, create |
|
||
| `backtest` | run, evaluateLatest |
|
||
| `sectorCrosslink` | confirm |
|
||
| `optionsConvexity` | unlock, getPayoff |
|
||
| `derisking` | suggest |
|
||
| `macro` | series, calendar, regimeClassify, commentary, regimeHistory |
|
||
| `thesisMonitor` | assess, timeline |
|
||
| `x` | feed, cashtag_search, timeline, accountsForSymbol |
|
||
| `reddit` | subreddit, search |
|
||
| `emotionLogger` | add, getByTrade, delete |
|
||
|
||
### 4.2 Sizing + risk (mounted 2026-07-18)
|
||
|
||
| Router | Procedures |
|
||
|--------|------------|
|
||
| `sizing` | `compute` — pure `sizePosition` + portfolio/unlock/halt context |
|
||
| `risk` | `posture` — pure `assessRisk` + portfolio load + halt persist; `haltStatus` |
|
||
|
||
Still deferred: journal create path that throws `HaltedError` when plans are server-persisted (`persist-trade-plan-execution-loop`).
|
||
|
||
---
|
||
|
||
## 5. Frontend architecture
|
||
|
||
```
|
||
app/src/
|
||
app/ Next App Router pages
|
||
components/ Panels + ui kit + layout
|
||
stores/ Zustand (several persist to localStorage)
|
||
lib/trpc.ts Partial API client (subset of routers)
|
||
lib/chart-theme.ts CSS-variable themed Recharts
|
||
lib/strings.ts Primary-rule sensitive copy
|
||
```
|
||
|
||
### 5.1 Routes
|
||
|
||
| Path | Role |
|
||
|------|------|
|
||
| `/` | Workbench overview |
|
||
| `/chart-lab` | Charts |
|
||
| `/institutional` | Institutional + filings |
|
||
| `/filings` | Filings only |
|
||
| `/options-dd` | Options DD |
|
||
| `/market-outlook` | Macro / rotation / notes |
|
||
| `/trade-plan`, `/execution`, `/trade-closure`, `/daily-focus` | Process (local) |
|
||
| `/alerts` | Alert events + subscriptions |
|
||
| `/settings` | Auth + onboarding |
|
||
| `/mobile` | Thin companion |
|
||
| `/admin`, `/users`, `/queue`, `/audit-logs`, `/x-accounts`, `/smtp` | Admin |
|
||
|
||
### 5.2 Client coverage gap
|
||
|
||
`lib/trpc.ts` exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.
|
||
|
||
**Not exposed in client (but on server):** screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under `market.commentary`), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).
|
||
|
||
---
|
||
|
||
## 6. Caching & adapters
|
||
|
||
| Source kind | Role | Typical freshness |
|
||
|-------------|------|-------------------|
|
||
| yfinance | quote, candles, sector, options | minutes / EOD |
|
||
| sec / sec-fetch | filings, 13F, Form 4 bulk | daily schedule default |
|
||
| sec-lint-* | gap detection / backfill | on demand admin |
|
||
| x | timelines / cashtags | demand + prune |
|
||
| reddit | subreddit/search | on demand |
|
||
| fred / macro | series + commentary inputs | slower |
|
||
| llm | summaries (tables present) | sparse product use |
|
||
|
||
**AdapterQueue features (built):** pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.
|
||
|
||
---
|
||
|
||
## 7. AuthZ model
|
||
|
||
| Gate | Behavior |
|
||
|------|----------|
|
||
| `publicProcedure` | No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens) |
|
||
| `protectedProcedure` | Session + `users.status === 'active'` |
|
||
| `adminProcedure` | `is_admin` flag |
|
||
|
||
Watchlist list currently falls back to `userId ?? 'anonymous'` — convenient for local dev, weak isolation if exposed beyond localhost.
|
||
|
||
---
|
||
|
||
## 8. Testing
|
||
|
||
| Suite | Command | Audit result |
|
||
|-------|---------|--------------|
|
||
| Backend | `cd app/server && npm test` | **504 pass / 0 fail / 1 skip** (after P0 2026-07-18) |
|
||
| Frontend lite | Primary-Rule lint + options pure tests | Present |
|
||
|
||
Known red: `MacroRegime` commentary tests (`generateMacroCommentary` short/long-term + no macro-trade recommendation).
|
||
|
||
---
|
||
|
||
## 9. Schema debt
|
||
|
||
1. **`strategies` defined twice** in `schema.sql` (legacy `regime_gate/setup/risk_policy` vs `components/unlocked`). SQLite keeps first-created shape depending on migration history — dangerous drift.
|
||
2. Parallel filter tables: `screener_filters` vs `saved_filters`.
|
||
3. Process tables (`trades`, `trade_executions`, `theses`, `emotion_logs`) exist ahead of UI persistence.
|
||
|
||
---
|
||
|
||
## 10. LLM / provenance (as built vs design)
|
||
|
||
| Designed | As built |
|
||
|----------|----------|
|
||
| Full LLMGateway with classifyPayload gate | Tables `llm_providers`, `llm_dispatch_audit`, `llm_summaries`; no complete gateway module under `src/` |
|
||
| Ornith default `is_local=true` | Env-driven URL in docker-compose (`LLM_PROVIDER_URL`) |
|
||
| Sensitive thesis data never leaves host | Thesis content mostly not yet flowing through LLM product paths |
|
||
|
||
ADR-0006/0008 still govern intent; implementation is incomplete.
|
||
|
||
---
|
||
|
||
## 11. Deployment
|
||
|
||
```bash
|
||
# Local dev (typical)
|
||
cd app/server && npm run dev # :3001
|
||
cd app && npm run dev # :3000
|
||
|
||
# Compose
|
||
docker compose up --build
|
||
```
|
||
|
||
Secrets: `.env.example` documents session secret, SEC operator email, X cookie fields, Reddit, FRED, LLM URL.
|
||
|
||
---
|
||
|
||
## 12. Documentation map (after this audit)
|
||
|
||
| Doc | Role |
|
||
|-----|------|
|
||
| `docs/FUNCTIONAL_DESIGN.md` | What users can do / pending |
|
||
| `docs/TECH_DESIGN.md` | This file — how it is built |
|
||
| `CONTEXT.md` | Ubiquitous language (glossary only) |
|
||
| `docs/adr/*` | Durable decisions |
|
||
| Obsidian `investor-flow.md` | Session orientation wiki |
|
||
|
||
Deprecated as status sources: root `HANDOFF.md` (2026-06-30 orchestrator snapshot), slice DECOMPOSITION checkbox state.
|