feat: welcome desk auth and operator password reset
First visit lands on /welcome instead of burying signup in Settings. Add a host CLI to reset passwords without a session, plus Settings change-password. Refresh as-built design docs and the Unraid operator guide.
This commit is contained in:
Investor Flow Build
+89
-25
@@ -1,22 +1,49 @@
|
||||
# Deploy Investor Flow on Unraid
|
||||
|
||||
Two containers: **backend** (`:3001`, SQLite + queue) and **frontend** (`:3000`, Next.js, proxies `/api` to the backend). Git and Actions already live on this Unraid box at `http://unraid.local:3003/transnet/investor-flow`.
|
||||
Two containers: **backend** (`:3001`, SQLite + queue) and **frontend** (`:3000`, Next.js, proxies `/api` to the backend). Git, Actions, and the container registry live on this Unraid box at `http://unraid.local:3003/transnet/investor-flow` (HTTP registry also at `10.37.0.86:3003`).
|
||||
|
||||
The current Gitea repo has Actions enabled but **no runner registered**, so every CI run is cancelled. Follow "One-time: Gitea runner" below if you want push-to-main to test and publish images.
|
||||
This is the operator guide. As-built product/tech status lives in [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md) and [`TECH_DESIGN.md`](./TECH_DESIGN.md).
|
||||
|
||||
## What you get
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | Build-from-git stack (works on Unraid or a laptop) |
|
||||
| `docker-compose.yml` | Build-from-git stack (laptop or Unraid checkout) |
|
||||
| `deploy/unraid/compose.pull.yml` | Pull prebuilt images from the Gitea registry |
|
||||
| `deploy/unraid/.env.example` | Secrets + ports + data dir |
|
||||
| `deploy/unraid/act-runner-compose.yml` | Gitea act_runner so CI/CD actually runs |
|
||||
| `deploy/unraid/user-scripts/sync-and-up.sh` | Cron CD: `git pull` + `compose up --build` |
|
||||
| `.github/workflows/ci.yml` | Tests on every push / PR |
|
||||
| `.github/workflows/cd.yml` | Build + push images on `main` |
|
||||
| `deploy/unraid/.env.example` | Secrets + ports + data dir + image names |
|
||||
| `deploy/unraid/act-runner-compose.yml` | Gitea act_runner (label `ubuntu-latest`) |
|
||||
| `deploy/unraid/user-scripts/sync-and-up.sh` | Cron fallback: `git pull` + `compose up --build` |
|
||||
| `.github/workflows/ci.yml` | Tests on every push / PR; **build + push images on `main`** |
|
||||
| `.github/workflows/cd.yml` | Manual image rebuild (`workflow_dispatch`) without a new commit |
|
||||
|
||||
## Unraid layout (recommended)
|
||||
## The path a commit takes
|
||||
|
||||
```
|
||||
write code → git push origin main
|
||||
│
|
||||
▼
|
||||
Gitea Actions (act_runner)
|
||||
│
|
||||
┌──────────┴──────────┐
|
||||
▼ ▼
|
||||
Job: test Job: images
|
||||
frontend tests (only if test passed
|
||||
backend tests and the event is a
|
||||
backend typecheck push to main)
|
||||
│
|
||||
▼
|
||||
Gitea registry tags
|
||||
…-backend:{sha,latest}
|
||||
…-frontend:{sha,latest}
|
||||
│
|
||||
▼
|
||||
Unraid pulls (or rebuilds)
|
||||
and recreates the two containers
|
||||
```
|
||||
|
||||
Publishing images is **not** the same as rolling the running stack. After CI is green you still pull (mode A) or rebuild (mode B) on Unraid.
|
||||
|
||||
## Unraid layout (required)
|
||||
|
||||
Use a **cache (or single-disk) path** for SQLite. `/mnt/user/...` is FUSE and can corrupt WAL files.
|
||||
|
||||
@@ -43,9 +70,9 @@ nano /mnt/cache/appdata/investor-flow/.env
|
||||
|
||||
Required in `.env`:
|
||||
|
||||
- `IFLOW_SESSION_SECRET` — `openssl rand -hex 32`
|
||||
- `SEC_OPERATOR_EMAIL` — real address (SEC + Yahoo user-agent)
|
||||
- `LLM_PROVIDER_URL` — Ornith LAN URL, e.g. `http://10.37.0.165:30081/v1` (not `localhost`)
|
||||
- `IFLOW_SESSION_SECRET` - `openssl rand -hex 32`
|
||||
- `SEC_OPERATOR_EMAIL` - real address (SEC + Yahoo user-agent)
|
||||
- `LLM_PROVIDER_URL` - Ornith LAN URL, e.g. `http://10.37.0.165:30081/v1` (not `localhost`)
|
||||
- `IFLOW_DATA_DIR=/mnt/cache/appdata/investor-flow/data`
|
||||
|
||||
If you copy today's Mac DB into `data/`, also set `IFLOW_CRYPTO_KEY` to the same value the Mac used. If the Mac never set one, encrypted X/FRED rows used the built-in dev key; re-enter those keys in Admin after a fresh Unraid DB instead of guessing.
|
||||
@@ -58,7 +85,7 @@ scp app/server/data/investor-flow.db* \
|
||||
root@unraid.local:/mnt/cache/appdata/investor-flow/data/
|
||||
```
|
||||
|
||||
Build and start:
|
||||
### First start (build on Unraid)
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
@@ -69,9 +96,11 @@ Open `http://unraid.local:3000`. Backend health: `http://unraid.local:3001/healt
|
||||
|
||||
Compose Manager plugin: create a stack whose project directory is the repo checkout and whose env file is `/mnt/cache/appdata/investor-flow/.env`.
|
||||
|
||||
Reverse proxy (SWAG / NPM): point the host at **frontend :3000 only**. The browser talks same-origin `/api`; the Next container reaches the backend on the compose network.
|
||||
Reverse proxy (SWAG / NPM): point the host at **frontend :3000 only**. The browser talks same-origin `/api`; the Next container reaches the backend on the compose network (`IFLOW_BACKEND_URL=http://backend:3001`).
|
||||
|
||||
## One-time: Gitea runner (unlocks CI + image CD)
|
||||
## Gitea runner (CI + image publish)
|
||||
|
||||
One-time. Without a runner labeled `ubuntu-latest`, Gitea marks every workflow **cancelled**.
|
||||
|
||||
1. Gitea → Site Administration → Actions → Runners → **Create new runner**. Copy the token.
|
||||
2. Unraid Docker: add insecure registry `10.37.0.86:3003` (Gitea HTTP) so pulls/pushes work. Restart Docker.
|
||||
@@ -86,18 +115,50 @@ docker compose -f deploy/unraid/act-runner-compose.yml up -d
|
||||
|
||||
4. Confirm the runner is **Idle** on the Gitea runners page.
|
||||
5. Repo → Settings → Actions → Secrets: `REGISTRY_TOKEN` = a Gitea token with `write:package` and `write:repository`.
|
||||
6. Push to `main`. CI should run tests; CD should push:
|
||||
6. Optional repo variable `REGISTRY` (default `10.37.0.86:3003`).
|
||||
7. Push to `main`. CI should run tests; on success it should push:
|
||||
|
||||
- `10.37.0.86:3003/transnet/investor-flow-backend:latest`
|
||||
- `10.37.0.86:3003/transnet/investor-flow-frontend:latest`
|
||||
- `10.37.0.86:3003/transnet/investor-flow-backend:latest` (and `:${sha}`)
|
||||
- `10.37.0.86:3003/transnet/investor-flow-frontend:latest` (and `:${sha}`)
|
||||
|
||||
Then switch `.env` image names to those registry tags and use `deploy/unraid/compose.pull.yml` so Unraid no longer builds on the array.
|
||||
Manual republish of the current `main` SHA (no new commit): Gitea → Actions → **CD** → Run workflow.
|
||||
|
||||
## CD that works before a runner exists
|
||||
## Two ways to run the stack after CI
|
||||
|
||||
Unraid → User Scripts → new script, paste `deploy/unraid/user-scripts/sync-and-up.sh`, schedule every 10 minutes (or after you push). It fast-forwards `main` and rebuilds only when HEAD moved.
|
||||
### A. Pull prebuilt images (preferred)
|
||||
|
||||
A push to Gitea is then: write code → `git push origin main` → cron on Unraid rebuilds containers.
|
||||
Point `.env` at the registry tags:
|
||||
|
||||
```
|
||||
IFLOW_IMAGE_BACKEND=10.37.0.86:3003/transnet/investor-flow-backend:latest
|
||||
IFLOW_IMAGE_FRONTEND=10.37.0.86:3003/transnet/investor-flow-frontend:latest
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
docker compose -f deploy/unraid/compose.pull.yml \
|
||||
--env-file /mnt/cache/appdata/investor-flow/.env pull
|
||||
docker compose -f deploy/unraid/compose.pull.yml \
|
||||
--env-file /mnt/cache/appdata/investor-flow/.env up -d
|
||||
```
|
||||
|
||||
`compose.pull.yml` reads `GITEA_REGISTRY` (default `unraid.local:3003`) and `IFLOW_TAG` (default `latest`). Keep those consistent with what CI pushed.
|
||||
|
||||
### B. Build from the git checkout
|
||||
|
||||
Leave image names as `investor-flow-backend:local` / `investor-flow-frontend:local` and:
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
git fetch --prune origin main && git merge --ff-only origin/main
|
||||
docker compose --env-file /mnt/cache/appdata/investor-flow/.env up -d --build
|
||||
```
|
||||
|
||||
Or schedule `deploy/unraid/user-scripts/sync-and-up.sh` in Unraid User Scripts (every 10 minutes is enough). It fast-forwards `main` and rebuilds only when HEAD moved, then waits for both `/health` endpoints.
|
||||
|
||||
Do not mix A and B on the same compose project unless you know which tags `.env` pins. After you switch to A, Unraid no longer compiles Next/Node on the array.
|
||||
|
||||
## Day-2 operations
|
||||
|
||||
@@ -105,14 +166,17 @@ A push to Gitea is then: write code → `git push origin main` → cron on Unrai
|
||||
|---|---|
|
||||
| Logs | `docker compose logs -f --tail=200 backend frontend` |
|
||||
| Restart | `docker compose up -d` |
|
||||
| Update (pull mode) | `docker compose -f deploy/unraid/compose.pull.yml pull && docker compose -f deploy/unraid/compose.pull.yml up -d` |
|
||||
| Update (pull mode) | `compose.pull.yml pull` then `up -d` |
|
||||
| Update (build mode) | `git merge --ff-only origin/main` then `up -d --build` |
|
||||
| Backup DB | copy `/mnt/cache/appdata/investor-flow/data/investor-flow.db*` (stop backend first, or use SQLite backup) |
|
||||
| Reset a password | No email reset. On the host: `docker compose exec backend node --experimental-strip-types src/cli/reset-password.ts --list` then `--email you@x --password 'newpass'` (add `--clear-2fa` if TOTP blocks login). Locally: `cd app/server && npm run reset-password -- --email you@x --password 'newpass'`. Sign in, then change it in Settings. |
|
||||
| Ornith | keep `LLM_PROVIDER_URL` on the LAN IP; user LLM endpoint in Settings can stay `http://10.37.0.165:30081/v1` |
|
||||
| Queue / SMTP / X / FRED | Admin UI at `/admin/*` on the frontend |
|
||||
|
||||
The backend process must stay up for dealer-map snapshots and the queue. `restart: unless-stopped` plus Unraid array autostart covers that.
|
||||
The backend process must stay up for dealer-map snapshots, confluence evaluation, alert producers, and the adapter queue. `restart: unless-stopped` plus Unraid array autostart covers that.
|
||||
|
||||
## What this is not
|
||||
|
||||
- Not Vercel. Next standalone + the Node backend both run on Unraid.
|
||||
- `/reports` is still the thin HTML stub. A daily GEX briefing job is separate.
|
||||
- CI does not SSH into Unraid or call `docker compose` for you. It publishes images; you (or User Scripts / Compose Manager) roll them.
|
||||
- Do not publish `IFLOW_SESSION_SECRET`, `IFLOW_CRYPTO_KEY`, or Gitea tokens.
|
||||
|
||||
+192
-87
@@ -1,9 +1,19 @@
|
||||
# Investor Flow — Functional Design (as built)
|
||||
# Investor Flow - Functional Design (as built)
|
||||
|
||||
**Audit date:** 2026-07-23
|
||||
**Audit date:** 2026-08-18
|
||||
**Source of truth for status:** this file + code under `app/`. Older slice SPECs and HANDOFF.md are historical.
|
||||
**Product intent:** Multi-tenant **investment management** workbench (portfolio, risk, monitor, research context) that helps users become better at market awareness and risk management *while managing*. Not a course app. Legal posture for copy/outputs: educational publisher (ADR-0007) - process and math, never buy/sell advice.
|
||||
|
||||
**What changed since the 2026-07-23 audit (high level):**
|
||||
- Process loop is server-persisted (`/plan`, `/theses`, `/journal`).
|
||||
- Per-user **module access** (research / execution / analytics / settings / admin) plus workspace density.
|
||||
- **Confluence** (M22) and **Price Corridor** live at `/confluence`.
|
||||
- **Tracked Funds / Mirror** (M21) live at `/funds` and `/funds/[id]`.
|
||||
- Symbol Search Index (ADR-0011), short interest (Yahoo + NASDAQ + FINRA), analyst ratings, classification watchlists.
|
||||
- Alert producers that were stubs are now registered (rotation, regime, thesis, unlock, portfolio risk, VIX, 13F, confluence, mirror).
|
||||
- Dealer Flow heatmap + integrity + Study Desk remain live; Han-style day script on the map.
|
||||
- App ships to Unraid as two containers via Gitea Actions (tests, then registry images). Operator guide: [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md).
|
||||
|
||||
---
|
||||
|
||||
## Status legend
|
||||
@@ -24,139 +34,221 @@
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Email/password signup | **Live** | New users get `status=pending_approval`; no session until admin approves |
|
||||
| Login + session cookie | **Live** | Non-active users rejected at login and `protectedProcedure` |
|
||||
| TOTP 2FA + backup codes | **Live** | Settings page |
|
||||
| Email/password signup | **Live** | First-user path is `/welcome`. New users get `status=pending_approval`; no session until admin approves. Pending-approval panel lives on the same desk; no session cookie issued while pending. |
|
||||
| Login + session cookie | **Live** | Non-active users rejected at login and `protectedProcedure`. TOTP step surfaces when required (`Two-factor code required or invalid.`) with a 6-digit input that retries `auth.login(email, password, totp)`. Header CTAs (Sign in / Create account) visible when logged out. |
|
||||
| Forgot / reset password | **Live (operator)** | No email reset. Welcome desk explains the path. Operator CLI `src/cli/reset-password.ts` (no session). Logged-in admin can reset another user under `/admin/users`. Signed-in users change their own password in Settings. |
|
||||
| First-visit redirect | **Live** | Unsigned users without the `iflow-welcome-seen` localStorage flag are redirected from `/` to `/welcome`. Skip sets the flag and goes to `/`. |
|
||||
| TOTP 2FA + backup codes | **Live** | Settings page (enabled / confirmed). Login desk handles the interactive code step. |
|
||||
| GitHub / Google OAuth | **Backend-ready** | Router complete; env-dependent; OAuth users default `active` |
|
||||
| Onboarding wizard | **Live** | Workspace interview (experience, goal, horizon, jargon) → density + risk defaults + starter pack; optional portfolio; queues SEC fetch |
|
||||
| Onboarding wizard | **Live** | Workspace interview shown on `/welcome` after first successful login if `!onboarded`. Steps: experience / goal / horizon / jargon comfort, then `api.onboarding.complete`. Dispatches `workspace-profile-changed`, pushes to `/portfolio`. |
|
||||
| Workspace density | **Live** | `focused` \| `standard` \| `full` filters nav, symbol Overview panels, strategy templates; editable in Settings |
|
||||
| Per-user module access | **Live** | Admin assigns `research`, `execution`, `analytics`, `settings`, `admin`. Default for new users: research + settings. `FeatureGate` redirects if the module is off. |
|
||||
| Admin approve/reject | **Live** | `/admin/users` |
|
||||
| Settings / profile | **Live** | `/settings` |
|
||||
| Settings / profile | **Live (signed-in only)** | `/settings` shows account card, 2FA enrollment/confirmation, workspace density, and user-configured OpenAI-compatible LLM endpoint. Guest users are redirected to `/welcome?mode=signin`. |
|
||||
|
||||
Two independent gates apply to almost every destination:
|
||||
|
||||
1. **Module access** (who is allowed to use a family of screens).
|
||||
2. **Workspace density** (how much chrome a given user wants to see).
|
||||
|
||||
### 1.2 Research workbench (desktop)
|
||||
|
||||
| Surface | Route | Status | Data path |
|
||||
|---------|-------|--------|-----------|
|
||||
| Symbol overview | `/` | **Live** | `market.snapshot`, company meta, X feed, collapsible filings + options |
|
||||
| Multiple watchlists | sidebar | **Live** | Create/delete/rename/reorder, dropdown selector in sidebar, per-watchlist symbols |
|
||||
| Charts | `/chart-lab` | **Live** | candles, indicators, institutional buy markers; equity volume-by-price profile (visible range, from OHLC bars) |
|
||||
| Symbol overview | `/` | **Live** | `market.snapshot`, ticker context, company meta, fund-holdings strip, dealer levels strip, density-gated analyst ratings / short interest / X feed / filings / options |
|
||||
| Multiple watchlists | sidebar | **Live** | Create/delete/rename/reorder/move symbol; dropdown selector; local-first autocomplete (`symbols.search`); hybrid add (known match vs "not in SEC registry - add anyway?") |
|
||||
| Classification watchlists | sidebar | **Live** | Lists can be `user` or derived (`sector` / `thematic` / `style` / `region`) with `class_key` / `class_label` |
|
||||
| Charts | `/chart-lab` | **Live** | candles, indicators (incl. EMA 9/21), institutional buy markers; equity volume-by-price profile (visible range, from OHLC bars) |
|
||||
| Institutional dashboard | `/institutional` | **Live** | dashboard rollup, flow, insider stream, filings; Q-o-Q / M-o-M |
|
||||
| Filings | `/filings` | **Live** | EDGAR index + 13F + Form 4 detail |
|
||||
| Options DD | `/options-dd` | **Live** | chain + greeks (read-only teaching surface) |
|
||||
| Dealer Flow | `/dealer-flow` | **Live (MVP)** | Heatmap-first IA: GEX/VEX is the only map toggle (convention lives in Method); reading / method / study in drawers; point-in-time book clock; **integrity gates** (incomplete vs degraded vs complete); IV hygiene; keep last good map; as-of ET; `dealer-map-replay` CLI + Admin → Queue **Dealer map integrity** buttons; Layer-0 + optional L1 |
|
||||
| Dealer Flow | `/dealer-flow` | **Live** | Heatmap-first IA: GEX/VEX is the only map toggle (convention lives in Method); reading / method / study in drawers; point-in-time book clock; integrity gates (incomplete vs degraded vs complete); IV hygiene; keep last good map; as-of ET; Han-style day script; `dealer-map-replay` CLI + Admin → Queue **Dealer map integrity** buttons; Layer-0 + optional L1 |
|
||||
| Study Desk | (drawer on `/dealer-flow`) | **Live** | Educational setups from map; hist rank; log; auto-grade; scorecard; promote to journal draft; CSV export |
|
||||
| Mentor ledger | (Study Desk tab) | **Live (MVP)** | Local import from harvest → confirm drafts → path-match grade + mentor scorecard (not buy signals) |
|
||||
| Market outlook | `/market-outlook` | **Live (Phase 1–3 core)** | Beginner language. Condition strip; stronger/weaker-vs-market map; seasonality + calendar; rank snapshots; auto leadership check; truck/factory; macro notes. True ETF flow/COT still later. |
|
||||
| Social / X feed | on overview | **Live** | DB-backed 30d history + admin X credentials/accounts |
|
||||
| Command palette + nav | shell | **Live** | ⌘K, g-key shortcuts, theme presets |
|
||||
| Signal Confluence | `/confluence` | **Live** | Picture quality + slot evidence grid + reliability scorecard + signal timeline + recent entry/exit zones (ADR-0012) |
|
||||
| Price Corridor | `/confluence` (tab) | **Live** | Valuation corridor snapshot, corridor watchlist, corridor-method backtest ledger |
|
||||
| Market outlook | `/market-outlook` | **Live** | Beginner language. Condition strip; stronger/weaker-vs-market map; seasonality + calendar; rank snapshots; auto leadership check; truck/factory; macro notes. True ETF *fund flow* still later. COT is consumed as a confluence slot, not a dedicated outlook panel. |
|
||||
| Social / X feed | on overview (density `full`) | **Live** | DB-backed 30d history + admin X credentials/accounts |
|
||||
| Symbol search | header + watchlist | **Live** | `symbols.search` against local `symbols` table (SEC `company_tickers.json` seed + yfinance hydration). No live vendor search on the request path (ADR-0009 / ADR-0011). |
|
||||
| Command palette + nav | shell | **Live** | ⌘K, g-key shortcuts, theme presets. Palette covers core pages; some newer routes (Confluence, Funds, Dealer Flow) are sidebar-only. |
|
||||
|
||||
### 1.3 Process / journal workflow
|
||||
### 1.3 Book (portfolio + risk + funds)
|
||||
|
||||
| Surface | Route | Status | Notes |
|
||||
|---------|-------|--------|-------|
|
||||
| Portfolio book | `/portfolio` | **Live** | Unified holdings + live marks + add/update/remove (`HoldingsBookView`). Optional guided setup via `?strategyId=`. `/monitor` redirects here. |
|
||||
| Option legs book | `/portfolio`, `/risk` | **Live (MVP)** | `OptionLegsPanel`; `portfolio.optionLegs` / add / remove; posture folds premium-at-risk + CSP cash reserve into exposure/flags |
|
||||
| Risk posture + sizing | `/risk` | **Live** | `risk.posture` + `sizing.compute` + RiskPosturePanel. Drawdown / cluster / option-risk commentary. **Halt persist is not on this path today** (`risk.posture` returns `halted: false`); `halt_state` table + drawdown alert producer still exist. |
|
||||
| Tracked funds index | `/funds` | **Live** | List / enable / add / delete tracked funds; 13F sync; capture ingest |
|
||||
| Fund live book + mirror | `/funds/[id]` | **Live** | Live book (13F + X captures/claims) + mirror diff vs user holdings (ADR-0010: replication math, not advice) |
|
||||
|
||||
### 1.4 Process / journal workflow
|
||||
|
||||
| Surface | Route | Status | Gap |
|
||||
|---------|-------|--------|-----|
|
||||
| Guided start | `/guided-start` | **Live** | Density-filtered strategy presets + short quiz; can fork a preset into the user's strategies |
|
||||
| Decision plan | `/plan` | **Live** | Server `trades.save`; optional thesis create |
|
||||
| Theses | `/theses` | **Live** | CRUD on `theses` table |
|
||||
| Journal / close | `/journal` | **Live** | `trades.list` / `close` + reflection |
|
||||
| Journal / close | `/journal` | **Live** | `trades.list` / `close` + reflection + stats |
|
||||
| Strategies | `/strategies` | **Live** | Preset catalog + fork; custom authoring still light |
|
||||
| Strategy lab | `/lab` | **Live** | Backtest equity curve + evaluate latest; portfolio backtest API exists |
|
||||
| Position considerations | `/exits` | **Live** | `derisking.suggest` + dividend health + close-position helper; EmotionLogger mounted here |
|
||||
| Screener | `/screener` | **Live** | Filter + strategy screen over watchlist |
|
||||
| Strategy lab | `/lab` | **Live** | Backtest equity curve + evaluate latest |
|
||||
| Reports | `/reports` | **Live** | HTML research notes |
|
||||
| Daily focus / goals | `/daily-focus` | **UI-local** | localStorage goals |
|
||||
| Emotion logger | API | **Partial** | API live; journal reflection covers close notes |
|
||||
| Reports | `/reports` | **Live** | HTML research notes (`reports.generate`) |
|
||||
| Daily focus / goals | `/daily-focus` | **UI-local** | localStorage goals; density `full` |
|
||||
| Emotion logger | `/exits` + API | **Partial** | API live; used on Position considerations; journal close uses a reflection field instead |
|
||||
| Sizing math | `/risk` (calculator) | **Live** | `sizing.compute` + UI cascade (math only, ADR-0007) |
|
||||
| Risk posture (M20) | `/risk` | **Live** | `risk.posture` / `risk.haltStatus` + RiskPosturePanel; halt persisted on drawdown breach |
|
||||
| Derisking suggestions | (no screen) | **Backend-ready** | `derisking.suggest` |
|
||||
| Thesis monitor | (no screen) | **Stub** | `thesisMonitor.assess` / `timeline` (empty events / empty timeline) |
|
||||
| Derisking suggestions | `/exits` | **Live** | Productized from `derisking.suggest` |
|
||||
| Thesis monitor | (no screen) | **Stub** | `thesisMonitor.assess` / `timeline` (empty events / empty timeline). Thesis *CRUD* is live on `/theses`. |
|
||||
|
||||
### 1.4 Discovery & strategy lab
|
||||
Legacy routes `/trade-plan`, `/execution`, and `/trade-closure` are gone. `/mobile` and `/monitor` redirect to `/portfolio`.
|
||||
|
||||
### 1.5 Discovery & strategy lab
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Filter screener (M15a) | **Live** | `/screener` + `screener.filter` over watchlist quotes |
|
||||
| Filter screener (M15a) | **Live** | `/screener` + `screener.filter` over watchlist |
|
||||
| Strategy screener (M15b) | **Live** | `/screener` strategy mode when density ≥ standard |
|
||||
| Strategy authoring | **Partial** | presets + fork; custom authoring still light |
|
||||
| Backtest | **Live** | `/lab` equity curve + evaluate latest |
|
||||
| Backtest | **Live** | `/lab` equity curve + evaluate latest; `backtest.runPortfolio` for allocation paths |
|
||||
| Sector confirmation cross-link | **Stub** | `sectorCrosslink.confirm` with empty universe in practice |
|
||||
| Screener saved filters | **Schema only** | tables exist; incomplete product loop |
|
||||
| Screener saved filters | **Schema only** | `screener_filters` / `saved_filters` tables exist; incomplete product loop |
|
||||
|
||||
### 1.5 Options convexity sleeve (M17)
|
||||
### 1.6 Confluence signal engine (M22)
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Options DD (read-only) | **Live** | M3 teaching panel |
|
||||
| Unlock ladder + payoff API | **Backend-ready** | `optionsConvexity.unlock` / `getPayoff` |
|
||||
| Option legs book + risk contribution | **Live (MVP)** | `OptionLegsPanel` on `/portfolio` + `/risk`; `portfolio.optionLegs` / add / remove; posture folds premium-at-risk + CSP cash reserve into exposure/flags |
|
||||
| Portfolio sleeve workflow (M17 full) | **Missing** | Unlock UX + sleeve risk budget not enforced in product path |
|
||||
| Slot catalog (34 slots, 6 families) | **Live** | technical, institutional, macro, seasonal, flows, sentiment. ADR-0007 explain text (evidence, never directives). |
|
||||
| Rack evaluation + picture quality | **Live** | Redundancy-aware evidence; quality labels (strong/moderate/weak bullish/bearish, mixed, sparse) |
|
||||
| System racks | **Live** | Seeded on boot: Full Confluence, Technical Momentum, Macro + Flows + Sentiment |
|
||||
| User racks | **Live** | `confluence.saveRack` |
|
||||
| Closed-loop signal history | **Live** | Fires logged; 4-week follow-through resolver; reliability weights |
|
||||
| Entry/exit zone rules | **Live** | Weekly walk-forward derivation on GICS sector ETFs + SPY; last 3 entry + 3 exit windows on the page |
|
||||
| Incremental replay | **Live** | Hourly tick + boot fill of ~3y lookback, budgeted |
|
||||
| Confluence change alerts | **Live** | `confluence_change` producer (picture improved / deteriorated) |
|
||||
| CFTC COT adapter | **Live (data)** | Used by `cotPositioning` slot; no standalone COT screen |
|
||||
| 15-symbol learning universe | **Live** | Pinned into demand set on startup (PLTR, NVDA, AMD, AAPL, MSFT, SMH, XOM, JPM, UNH, COST, AMZN, CAT, LMT, LIN, NEE + SPY) |
|
||||
|
||||
### 1.6 Macro (M18)
|
||||
### 1.7 Mirror portfolio (M21)
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| FRED series / key admin | **Live** | Admin X Accounts page also manages FRED key |
|
||||
| Truck sales + manufacturing PMI charts | **Live** | Market routes |
|
||||
| Sector rotation heatmap-style panel | **Live** | Relative-strength style rotation, not full designed phase library UI |
|
||||
| Tracked fund CRUD | **Live** | CIK + manager + optional X handle; enable/disable |
|
||||
| 13F sync into fund records | **Live** | `funds.adminSync13f` |
|
||||
| X capture / trade-claim ingest | **Live** | `funds.adminIngestCaptures` from cached X posts |
|
||||
| Live book | **Live** | Most recent record per symbol; 13F vs capture vs claim labeled; evidence URL |
|
||||
| Mirror diff | **Live** | User-entered capital base (default $200k) + optional floor; share/value delta to match disclosed weights (ADR-0010) |
|
||||
| Mirror alerts | **Live** | `fund_capture`, `fund_13f`, `mirror_diff` producers |
|
||||
| Manager Form 4 as fund book | **Missing (by design)** | Personal insider activity is not part of the disclosed book |
|
||||
|
||||
### 1.8 Options convexity sleeve (M17)
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Options DD (read-only) | **Live** | Teaching panel |
|
||||
| Option legs book + risk contribution | **Live (MVP)** | On `/portfolio` + `/risk` |
|
||||
| Unlock ladder + payoff API | **Removed** | `optionsConvexity` router is gone. `ConvexityGate` is a stub (unlock system removed; strategies are not gated by a sleeve ladder). |
|
||||
| Sleeve risk budget enforcement | **Missing** | Combined stock+options ceiling is not enforced as a product path |
|
||||
|
||||
### 1.9 Macro (M18)
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| FRED series / key admin | **Live** | Admin X Accounts page also manages FRED key; series warmed by queue schedule (`fred`, daily) |
|
||||
| Truck sales + manufacturing PMI charts | **Live** | Market Outlook |
|
||||
| Sector rotation heatmap-style panel | **Live** | Relative-strength vs SPY, not measured fund flows |
|
||||
| Custom rotation ETFs | **Live** | `market.addCustomEtf` / remove / list |
|
||||
| Regime classify / history | **Backend-ready** | `macro.regimeClassify`, `regimeHistory` |
|
||||
| Portfolio-impact commentary | **Live** (partial) | `macro.commentary` via Market Outlook; **2 unit tests failing** on wording assertions |
|
||||
| Full economic calendar UI | **Stub** | `macro.calendar` |
|
||||
| Portfolio-impact commentary | **Live** | `macro.commentary` via Market Outlook |
|
||||
| Full economic calendar UI | **Stub** | `macro.calendar` returns cached events (often empty) |
|
||||
|
||||
### 1.7 Alerts & mobile
|
||||
### 1.10 Alerts
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Alert list / ack / unacked count | **Live** | `/alerts` page with event history + subscription management |
|
||||
| Alert subscriptions (create/list/update/delete) | **Live** | Per-user with ticker-level / list-level / global inheritance |
|
||||
| Alert producers: informed_buy, informed_sell | **Live** | Insider tx comparison state, batched every 10 min |
|
||||
| Alert producer: new_13da | **Live** | New 13D/13G filing detection, batched every 10 min |
|
||||
| Alert producers: 7 more types (rotation, regime, conviction, thesis, cluster, drawdown, asymmetry) | **Stub** | Engine types exist; producers not wired |
|
||||
| Email delivery (SMTP) | **Live** | Nodemailer, admin SMTP config page, rate limit 1/hr per type/symbol/user |
|
||||
| Alert list / ack / unacked / clear-all | **Live** | `/alerts` + notification bell |
|
||||
| Alert subscriptions + per-type toggles | **Live** | Ticker / list / global inheritance; `alerts.listTypes` / `toggleType` |
|
||||
| Email delivery (SMTP) | **Live** | Nodemailer, `/admin/smtp`, outbox table, 60s drain, rate limit 1/hr per type/symbol/user |
|
||||
| SSE push | **Missing** | Design called for SSE; poll/list only |
|
||||
| Mobile companion `/mobile` | **Redirect** | Redirects to Portfolio; main shell is phone-optimized companion (Job A) |
|
||||
| Mobile shell (iPhone Air) | **Live** | Fixed `dvh` chrome, safe-area insets, bottom tabs (Portfolio · Risk · Research · Alerts · More), slim header + search overlay; no desktop layout flash |
|
||||
|
||||
**Producers (registered at backend boot):**
|
||||
|
||||
| Type | Frequency | Status |
|
||||
|------|-----------|--------|
|
||||
| `informed_buy`, `informed_sell` | batched (10 min) | **Live** |
|
||||
| `new_13da` | batched | **Live** |
|
||||
| `new_13f` | batched | **Live** |
|
||||
| `vix_level` | realtime (30s loop) | **Live** |
|
||||
| `rotation_incipient`, `regime_shift` | batched | **Live** |
|
||||
| `conviction_unlock` | batched | **Live** |
|
||||
| `thesis_broken`, `thesis_weakening` | batched | **Live** (depends on thesis rows + monitor input; monitor events still thin) |
|
||||
| `cluster_breach`, `drawdown_halt`, `asymmetry_warning` | batched | **Live** |
|
||||
| `confluence_change` | batched | **Live** |
|
||||
| `fund_capture`, `fund_13f`, `mirror_diff` | batched | **Live** (lazy register) |
|
||||
|
||||
Admin default: every catalog type is seeded ON for `is_admin=1` (idempotent).
|
||||
|
||||
### 1.11 Mobile companion
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Dedicated `/mobile` app | **Redirect** | Redirects to Portfolio; main shell is the phone surface |
|
||||
| Mobile shell (iPhone Air) | **Live** | Fixed `dvh` chrome, safe-area insets, bottom tabs (Portfolio · Risk · Research · Alerts · More). CSS-first breakpoints so mobile never flashes desktop chrome. |
|
||||
| More sheet | **Live** | Density- and module-filtered secondary destinations |
|
||||
| Mobile lists sheet | **Live** | Header list icon opens portfolio + watchlist bottom sheet for symbol switch |
|
||||
| Portfolio on phone | **Live** | Card list + 2×2 snapshot under `md`; wide table from `md` up |
|
||||
| Rotation / institutional on phone | **Live** | Card rows with key horizons; full tables from `md` up with sticky first column |
|
||||
| Mobile polling hygiene | **Live** | Snapshot / alerts / market condition pause while document is hidden |
|
||||
| Portfolio book | **Live** | Unified holdings + live marks + add/remove (`HoldingsBookView`). `/monitor` redirects here. |
|
||||
|
||||
### 1.8 Admin / operator
|
||||
### 1.12 Admin / operator
|
||||
|
||||
| Capability | Status |
|
||||
|------------|--------|
|
||||
| Users, sessions, reset password | **Live** |
|
||||
| Users, sessions, reset password, enable/disable/delete | **Live** |
|
||||
| Per-user module assignment | **Live** |
|
||||
| Pending approval queue | **Live** |
|
||||
| Adapter queue health, pause/resume, retry, schedules, error stacks | **Live** | Cooldown column ticks 429 pauses; blank is not-paused. Unhealthy Yahoo shows backlog/notes. T0/focused quotes drain first. |
|
||||
| Adapter queue health, global pause/resume, **per-source pause/stop/start**, retry, schedules, error stacks | **Live** | Cooldown column ticks 429 pauses; blank is not-paused. Unhealthy Yahoo shows backlog/notes. T0/focused quotes drain first. |
|
||||
| SEC queue fetch + lint holders/insiders + data quality | **Live** |
|
||||
| Dealer map integrity (live or replay) | **Live** |
|
||||
| Alert producer run status | **Live** | `admin.alertStatus` |
|
||||
| X credentials, accounts, prune | **Live** |
|
||||
| FRED key | **Live** |
|
||||
| SMTP config (host, port, auth, test) | **Live** | Admin page at `/admin/smtp` |
|
||||
| FINRA download URL | **Live** | Admin-configurable; `finra-bulk` schedule not auto-seeded (historical 403) |
|
||||
| SMTP config (host, port, auth, test) | **Live** | `/admin/smtp` |
|
||||
| Audit log | **Live** |
|
||||
| Server restart control | **Live** |
|
||||
| GDPR export | **Backend-ready** (admin API) |
|
||||
| Server restart control | **Live** | Meaningful on a laptop process; less so inside Unraid (`restart: unless-stopped` is the operator path) |
|
||||
| GDPR export | **Backend-ready** | admin API only |
|
||||
|
||||
### 1.9 Reports
|
||||
### 1.13 Reports
|
||||
|
||||
| Capability | Status |
|
||||
|------------|--------|
|
||||
| HTML research note generator | **Live** | `/reports` + `reports.generate` |
|
||||
| HTML research note generator | **Live** | `/reports` + `reports.generate` (symbol, portfolio, watchlist, risk posture, rotation) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Functional architecture (intended jobs)
|
||||
|
||||
```
|
||||
Job A — Monitor (mobile + alerts)
|
||||
alerts.list/ack · portfolio.holdings · rotation chip
|
||||
Job A - Monitor (mobile + alerts)
|
||||
alerts.list/ack · portfolio.holdings · rotation chip · VIX / 13D / insider
|
||||
|
||||
Job B — Research (desktop workbench)
|
||||
Job B - Research (desktop workbench)
|
||||
watchlist → active symbol → market + SEC + institutional + options + social
|
||||
market outlook (macro + rotation)
|
||||
dealer flow map + study desk
|
||||
confluence picture + corridor
|
||||
|
||||
Job C — Process (planned investor loop)
|
||||
thesis + confluence + sizing → plan → execute → emotion → close → unlock
|
||||
[mostly UI-local today; engines exist but not productized]
|
||||
Job C - Process (planned investor loop)
|
||||
thesis + confluence + sizing → plan → execute → emotion → close
|
||||
[plan / theses / journal persist server-side; daily focus still UI-local]
|
||||
|
||||
Job D — Discover (planned)
|
||||
Job D - Discover
|
||||
filter/strategy screener → open in workbench
|
||||
[API only]
|
||||
guided-start presets → fork → optional portfolio setup
|
||||
|
||||
Job E - Mirror (M21)
|
||||
tracked fund live book → mirror diff vs user book (math, not advice)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -165,13 +257,18 @@ Job D — Discover (planned)
|
||||
|
||||
Unchanged from domain model / ADRs:
|
||||
|
||||
1. **Primary Rule (ADR-0007)** — education, not advice; Primary-Rule lint test on curated strings.
|
||||
2. **Analyst Voice (ADR-0005)** — Alfred/Druckenmiller blend where LLM copy is used.
|
||||
3. **P6 plain English / P7 teach mechanics** — in-context cockpit labels (not a curriculum product).
|
||||
3b. **Adaptive workspace density** — experience interview sets focused/standard/full; user can change anytime.
|
||||
1. **Primary Rule (ADR-0007)** - education, not advice; Primary-Rule lint test on curated strings.
|
||||
2. **Analyst Voice (ADR-0005)** - Alfred/Druckenmiller blend where LLM copy is used.
|
||||
3. **P6 plain English / P7 teach mechanics** - in-context cockpit labels (not a curriculum product).
|
||||
3b. **Adaptive workspace density** - experience interview sets focused/standard/full; user can change anytime.
|
||||
3c. **Module access** - operator decides which families a user can open; density then filters within those families.
|
||||
4. **Local-first multi-tenant (ADR-0001)** + shared cache dedupe (ADR-0004).
|
||||
5. **LLM data provenance (ADR-0006)** — sensitive data stays on local-class providers; full gateway module is thin/incomplete vs original design.
|
||||
6. **Anti-gamification** — no confetti / outcome celebration.
|
||||
5. **LLM data provenance (ADR-0006)** - sensitive data stays on local-class providers; full gateway module is still thin vs original design. Per-user OpenAI-compatible endpoint is live for Dealer Flow explain.
|
||||
6. **Anti-gamification** - no confetti / outcome celebration.
|
||||
7. **Rate-limit-first data plane (ADR-0009)** - stale UI beats vendor stampede.
|
||||
8. **Mirror math, not advice (ADR-0010)** - user states the replication goal; app computes the delta.
|
||||
9. **Symbol Search Index (ADR-0011)** - local-first autocomplete; SEC seed owns issuer identity.
|
||||
10. **Confluence is evidence (ADR-0012)** - picture-quality labels, never directives.
|
||||
|
||||
---
|
||||
|
||||
@@ -182,32 +279,40 @@ Unchanged from domain model / ADRs:
|
||||
- Mobile-first full research workbench
|
||||
- Undefined-risk options strategies for beginners
|
||||
- Training on user portfolio/thesis content via external LLMs
|
||||
- Treating a tracked fund's personal Form 4 as the fund's disclosed book
|
||||
|
||||
---
|
||||
|
||||
## 5. Gap priority (product value)
|
||||
|
||||
| P | Gap | Why it matters |
|
||||
|---|-----|----------------|
|
||||
| P0 | Fix failing MacroRegime commentary tests | **Done 2026-07-18** — suite green (504 pass / 1 skip) |
|
||||
| P0 | Expose SizingEngine + RiskEngine on tRPC + M20 UI | **Done 2026-07-18** — `/risk` + `sizing`/`risk` routers |
|
||||
| P1 | Persist trade plan / thesis / execution / closure server-side | Today process screens are single-browser toys |
|
||||
| P1 | Frontend client coverage for backend-only routers | APIs unreachable from SPA client |
|
||||
| P1 | Wire emotion logger UI to `emotionLogger` API | Schema + API exist; dual paths confuse |
|
||||
| P2 | Strategy Lab + backtest UI | Engines exist; no learning surface |
|
||||
| P2 | Screener UI (filter + strategy) | Discovery job unfinished |
|
||||
| P2 | Derisking + thesis monitor product surfaces | Process depth |
|
||||
| P2 | Options convexity sleeve UI | M17 incomplete |
|
||||
| P3 | Reports UI | Export pedagogy |
|
||||
| P3 | Wire remaining 7 alert producers | rotation, regime, conviction, thesis, cluster, drawdown, asymmetry |
|
||||
| P3 | SSE alerts / continuous event producers | Ops polish |
|
||||
| P3 | Full LLM Gateway as designed | Provenance enforcement completeness |
|
||||
| P | Gap | Why it matters | Status |
|
||||
|---|-----|----------------|--------|
|
||||
| P0 | Fix failing MacroRegime commentary tests | Suite health | **Done** (2026-07-18) |
|
||||
| P0 | Expose SizingEngine + RiskEngine on tRPC + M20 UI | Risk job | **Done** (`/risk`) |
|
||||
| P1 | Persist trade plan / thesis / journal server-side | Process loop | **Done** (`/plan`, `/theses`, `/journal`) |
|
||||
| P1 | Frontend client coverage for backend-only routers | APIs unreachable from SPA | **Mostly done** - `lib/trpc.ts` now covers the product routers; leftovers are thin (gdprExport, some macro helpers) |
|
||||
| P1 | Wire emotion logger UI to `emotionLogger` API | Dual paths confuse | **Partial** - live on `/exits`; journal uses reflection |
|
||||
| P2 | Strategy Lab + backtest UI | Learning surface | **Done** (`/lab`, `/strategies`) |
|
||||
| P2 | Screener UI (filter + strategy) | Discovery job | **Done** (`/screener`) |
|
||||
| P2 | Derisking + thesis monitor product surfaces | Process depth | Derisking **done** (`/exits`); thesis monitor still **stub** |
|
||||
| P2 | Options convexity sleeve UI | M17 incomplete | Legs **live**; unlock ladder **removed**; budget not enforced |
|
||||
| P2 | Confluence + corridor | Evidence picture | **Done** (M22) |
|
||||
| P2 | Mirror portfolio | Fund-first job | **Done** (M21) |
|
||||
| P3 | Reports UI | Export pedagogy | **Done** (`/reports`) |
|
||||
| P3 | Wire remaining alert producers | Awareness loop | **Done** (see §1.10) |
|
||||
| P3 | SSE alerts / continuous event producers | Ops polish | **Open** |
|
||||
| P3 | Full LLM Gateway as designed | Provenance enforcement completeness | **Open** |
|
||||
| P3 | Persist risk halt onto `risk.posture` | Circuit breaker product path | **Open** (table + producer exist; posture hardcodes `halted: false`) |
|
||||
| P3 | Persist daily-focus goals | Multi-device process | **Open** |
|
||||
| P3 | Saved screener filters product loop | Repeatable discovery | **Open** |
|
||||
|
||||
---
|
||||
|
||||
## 6. Related docs
|
||||
|
||||
- Tech design (modules, stack, API map): [`TECH_DESIGN.md`](./TECH_DESIGN.md)
|
||||
- Tech design (modules, stack, API map, CI/CD): [`TECH_DESIGN.md`](./TECH_DESIGN.md)
|
||||
- Unraid operator / CI/CD process: [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md)
|
||||
- Vendor integration rules: [`VENDOR_INTEGRATIONS.md`](./VENDOR_INTEGRATIONS.md)
|
||||
- Domain glossary: [`../CONTEXT.md`](../CONTEXT.md)
|
||||
- ADRs: [`adr/`](./adr/)
|
||||
- Living wiki page: Obsidian `investor-flow.md` (synced with this audit)
|
||||
- Living wiki page: Obsidian `investor-flow.md` (orientation; this file wins for status)
|
||||
|
||||
+357
-141
@@ -1,7 +1,8 @@
|
||||
# Investor Flow — Technical Design (as built)
|
||||
# Investor Flow - Technical Design (as built)
|
||||
|
||||
**Audit date:** 2026-07-23
|
||||
**Audit date:** 2026-08-18
|
||||
**Companion:** [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md)
|
||||
**Operator deploy:** [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md)
|
||||
**Canonical code roots:** `app/src` (Next frontend), `app/server/src` (Node backend)
|
||||
|
||||
---
|
||||
@@ -10,36 +11,76 @@
|
||||
|
||||
| Layer | Actual technology | Notes vs older docs |
|
||||
|-------|-------------------|---------------------|
|
||||
| Frontend | Next.js **16.2**, React **19**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA |
|
||||
| Frontend | Next.js **16.2.9**, React **19.2**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA. `output: "standalone"` for the frontend image. |
|
||||
| Backend runtime | **Node ≥22** (dev on Node 26), native TS via `--experimental-strip-types` | DESIGN said Bun; runtime glue is Node + `node:sqlite` |
|
||||
| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`) |
|
||||
| DB | SQLite file (`app/server/data/investor-flow.db`) | Single file multi-tenant |
|
||||
| Market data | `yahoo-finance2` **v3** class API | |
|
||||
| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + `secDataFetcher` | Filer CIK from accession prefix (critical fix) |
|
||||
| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | |
|
||||
| Macro | FredAdapter | Key stored admin-side |
|
||||
| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers |
|
||||
| Tests | `node --test --experimental-strip-types` | Backend ~503 tests; **500 pass, 2 fail, 1 skip** as of audit |
|
||||
| Deploy | `docker-compose.yml` (frontend + backend + volume) | Present; operator-run |
|
||||
| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses a hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`). Next **route handler** proxies `/api` with a 200s timeout (rewrites time out ~30s and break Ornith). |
|
||||
| DB | SQLite file (`IFLOW_DB_PATH`, default `app/server/data/investor-flow.db`) | Single file multi-tenant. On Unraid this is a bind-mounted cache-disk path. |
|
||||
| Market data | `yahoo-finance2` **v3** class API | Quotes, candles, options, short interest (Yahoo slice) |
|
||||
| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + SecCompanyTickersAdapter + `secDataFetcher` | Filer CIK from accession prefix. Daily `company_tickers.json` seed. |
|
||||
| Short interest | NasdaqAdapter + FinraShortInterestAdapter + FinraBulkAdapter | Three-way merge with discrepancy flags. `finra-bulk` not auto-scheduled. |
|
||||
| Macro / COT | FredAdapter + CotAdapter | FRED key admin-side. COT from CFTC zip (confluence slot). |
|
||||
| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | X is queued. Reddit tRPC calls the adapter directly and is not in the boot adapter map. |
|
||||
| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers. Production refuses to start without `IFLOW_SESSION_SECRET`. |
|
||||
| Tests | `node --test --experimental-strip-types` | Backend **903 pass / 0 fail / 1 skip**; frontend **58 pass / 0 fail** (2026-08-18). |
|
||||
| Deploy | Two Node containers on Unraid | Gitea Actions: test, then push images to the Gitea registry. See §11. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Process topology
|
||||
|
||||
### 2.1 Runtime (laptop or Unraid)
|
||||
|
||||
```
|
||||
Browser (Next :3000)
|
||||
│ fetch /api/trpc/<proc> (credentials include)
|
||||
Browser
|
||||
│ fetch /api/trpc/<proc> (credentials include, same-origin)
|
||||
▼
|
||||
Next rewrites / proxy → Backend (:3001)
|
||||
Next frontend (:3000)
|
||||
│ app/src/app/api/[...path]/route.ts (200s proxy)
|
||||
▼
|
||||
Backend (:3001)
|
||||
│
|
||||
├─ appRouter (tRPC)
|
||||
├─ CacheRepository + AdapterQueue
|
||||
├─ Source adapters (yfinance, sec-*, x, reddit, fred)
|
||||
├─ CacheRepository + AdapterQueue + vendorGate
|
||||
├─ Source adapters (yfinance+options, nasdaq, finra-*, sec-*, x, fred, cot)
|
||||
├─ Background loops (drain, schedules, alerts, SMTP outbox, confluence, housekeeping)
|
||||
└─ SQLite
|
||||
```
|
||||
|
||||
- Dev helpers: `restart-servers.sh`, admin `serverRestart`.
|
||||
- Queue drain loop + 30s schedule loop inside backend process.
|
||||
- Dev helpers: `restart-servers.sh`, admin `serverRestart` (laptop-oriented).
|
||||
- Queue drain default: every 2s (`IFLOW_DRAIN_MS`).
|
||||
- Schedule loop: every 30s.
|
||||
- Realtime/per-fetch alerts: every 30s (VIX).
|
||||
- Batched alerts: every 10 min.
|
||||
- SMTP outbox drain: every 60s.
|
||||
- Confluence tick: hourly (+ boot fill after 3 min).
|
||||
- Queue housekeeping: daily (`clearDone` + prune `queue_errors`).
|
||||
|
||||
### 2.2 Production host
|
||||
|
||||
Unraid is the intended host. Git + Gitea Actions live at `unraid.local:3003/transnet/investor-flow` (HTTP registry also at `10.37.0.86:3003`).
|
||||
|
||||
```
|
||||
git push origin main
|
||||
│
|
||||
▼
|
||||
Gitea repo + act_runner (label ubuntu-latest)
|
||||
│
|
||||
├─ ci.yml / test frontend tests, backend tests, backend typecheck
|
||||
└─ ci.yml / images (main only, after test) build + push
|
||||
│
|
||||
▼
|
||||
Gitea registry
|
||||
transnet/investor-flow-backend:{sha,latest}
|
||||
transnet/investor-flow-frontend:{sha,latest}
|
||||
│
|
||||
▼
|
||||
Unraid compose (backend :3001 + frontend :3000)
|
||||
data volume: IFLOW_DATA_DIR (cache/single-disk, not /mnt/user FUSE)
|
||||
```
|
||||
|
||||
`cd.yml` is **manual** (`workflow_dispatch`) - rebuild and push images without a new commit.
|
||||
|
||||
A git-pull + `compose up --build` cron (`deploy/unraid/user-scripts/sync-and-up.sh`) remains as a fallback when you want Unraid to build from source instead of pulling registry tags.
|
||||
|
||||
---
|
||||
|
||||
@@ -47,18 +88,18 @@ Next rewrites / proxy → Backend (:3001)
|
||||
|
||||
| Tier | Content | Isolation |
|
||||
|------|---------|-----------|
|
||||
| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro | Shared, no owner |
|
||||
| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro, symbols, dealer maps, short interest, COT, FRED | Shared, no owner |
|
||||
| B | threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_* | Shared infrastructure |
|
||||
| C | watchlists, portfolio_holdings, trades, strategies, alerts, theses, emotion_logs, reports, saved filters | `owner_id` / `user_id` |
|
||||
| D | users, sessions, admin_audit, x_credentials (singleton), llm_* | System |
|
||||
| C | watchlists, portfolio_holdings, portfolio_option_legs, trades, strategies, alerts, theses, emotion_logs, reports, saved filters, confluence user racks, corridor watchlist | `owner_id` / `user_id` |
|
||||
| D | users, sessions, admin_audit, x_credentials (singleton), smtp_config, llm_*, tracked_funds (operator-curated), fund_position_records | System / shared fund facts |
|
||||
|
||||
**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes.
|
||||
**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes. System pins (rotation universe, SPY, VIX, confluence universe) use `system_pin` so they are not refcount-inflated.
|
||||
|
||||
**Stale-while-revalidate:** CacheRepository returns cached rows and schedules refresh when past TTL.
|
||||
|
||||
### 3.1 Rate-limit-first data plane (ADR-0009)
|
||||
|
||||
All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:
|
||||
All vendors (Yahoo, X, FRED, SEC, Reddit, NASDAQ, FINRA, CFTC) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:
|
||||
|
||||
| Rule | Mechanism |
|
||||
|------|-----------|
|
||||
@@ -66,28 +107,44 @@ All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is
|
||||
| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
|
||||
| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
|
||||
| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
|
||||
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` (rotation universe) |
|
||||
| TTL-aware tiers | `yfinance-quote` (5m), `yfinance-eod` (6h candles), `yfinance-meta` (daily), `yfinance-holdings` (weekly) |
|
||||
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` |
|
||||
| TTL-aware tiers | quote-portfolio 1m, quote-priority 5m, quote-watched 10m, EOD candles 6h, meta daily, holdings weekly |
|
||||
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
|
||||
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
|
||||
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
|
||||
| Observability | `queue.health()`: vendorGate+queue_state cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||||
| Per-source admin control | pause / resume / stop (clear pending) / start |
|
||||
|
||||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow.
|
||||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow; Next **rewrites** for `/api` (30s hard timeout).
|
||||
|
||||
### Dealer Flow data plane (2026-08)
|
||||
### 3.2 Dealer Flow data plane
|
||||
|
||||
- Engine: pure `dealerExposureEngine` on `NormalizedOptionSurface` only (no vendor imports).
|
||||
- Default provider: Yahoo via `composeYFinanceWithOptions` + `OPTIONS_CHAIN_PROVIDER` (default `yfinance`).
|
||||
- Paid switch later: implement SourceFetch for `tradier`/`polygon`, register in `sourceRatePolicy`, set env - engine unchanged.
|
||||
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
|
||||
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
|
||||
- SEC institutional (alert-critical): SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`); `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done); `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. First-paint UI is GEX | VEX only; Method drawer has Customer book | Dealer inventory.
|
||||
- **Vendor rate-limit enforcement (hard requirement, all sources + future):** process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||||
- LLM: per-user OpenAI-compatible `base_url` + encrypted key + model (`userLlm.*`); not OpenAI-only.
|
||||
- Dealer Flow plain-English notes: in-app `dealerFlowExplainNotes.ts` only (L0/L1). Optional offline scripts harvest X handles and distill into that file **and** write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
|
||||
- Study Desk: pure `dealerStudyEngine` propose/grade; table `dealer_study_setups`; tRPC `dealerStudy.*` (propose, log, list, grade, gradeDue, scorecard). Grades use `price_candles` 1d barrier logic (target before invalidation). Complementary to strategy `backtest.*` - not the same surface.
|
||||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip) via `withExposureConvention` / `dealerMap.get({ convention })`. First-paint UI is GEX | VEX only; Method drawer has Customer book | Dealer inventory.
|
||||
- Han-style day script (`hanStyleLevels`) is derived from the active convention and returned on `dealerMap.get`.
|
||||
|
||||
### 3.3 SEC / institutional data plane
|
||||
|
||||
- SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS.
|
||||
- 13F prefers EFTS CUSIP pagination; on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`).
|
||||
- `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done).
|
||||
- `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||||
- Daily `sec-tickers` materializes `company_tickers.json` into `symbols` (issuer CIK, name, exchange). Merge policy: SEC never overwrites yfinance `sector` / `industry` / `peers` / `ticker_kind` and never purges non-SEC rows (ADR-0011).
|
||||
|
||||
### 3.4 Vendor gate (hard requirement)
|
||||
|
||||
Process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm, cftc. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||||
|
||||
### 3.5 Confluence data plane (ADR-0012)
|
||||
|
||||
- Evaluators resolve candles through `CandleProvider` (cache + freshest quote folded into a partial daily bar).
|
||||
- Hourly evaluation of *today* + incremental as-of replay (~3y, budgeted).
|
||||
- Weekly walk-forward derivation of entry/exit zone rules on 11 GICS sector ETFs + SPY.
|
||||
- Reliability weights (once a slot has ≥2 resolved fires) scale evidence inside `evaluateRack`.
|
||||
|
||||
Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts` + queued `yfinance:topHoldings:*` refresh.
|
||||
|
||||
@@ -97,77 +154,110 @@ Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallb
|
||||
|
||||
```
|
||||
app/server/src/
|
||||
index.ts HTTP server entry
|
||||
trpc/router.ts ~2k LOC — all routers
|
||||
index.ts HTTP server + boot: adapters, schedules, pins, loops
|
||||
trpc/router.ts all routers
|
||||
trpc/context.ts session, cookies, db, cache, queue injection
|
||||
db/schema.sql DDL (note: duplicate strategies blocks — debt)
|
||||
db/*Repository.ts watchlist, portfolio, emotion logs, alertSubscription
|
||||
db/schema.sql DDL
|
||||
db/*Repository.ts watchlist, portfolio, option legs, emotion logs,
|
||||
alertSubscription, fund, confluence, corridor
|
||||
cache/CacheRepository.ts content-addressed cache API
|
||||
queue/AdapterQueue.ts schedule, pause, retry, source cool-downs (ADR-0009)
|
||||
queue/sourceRatePolicy.ts 429 detect, cool-down ladders, min-intervals
|
||||
adapters/ YFinance, Options, Edgar, SecFetch, SecLint, X, Reddit
|
||||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback, tickerContext, dealerExposureEngine, dealerMapService, dealerMapExplain, dealerFlowExplainNotes, dealerStudyEngine
|
||||
options/ bsm, types (NormalizedOptionSurface), OptionsChainRouter (paid-ready provider seam)
|
||||
queue/AdapterQueue.ts schedule, pause/stop per source, retry, cool-downs
|
||||
queue/sourceRatePolicy.ts 429 detect, cool-down ladders, SCHEDULE_INTERVALS
|
||||
adapters/ YFinance, Options, Nasdaq, Finra*, Cot, Edgar,
|
||||
SecFetch, SecCompanyTickers, SecLint, X, Reddit
|
||||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback,
|
||||
tickerContext, dealerExposureEngine, dealerMapService,
|
||||
dealerMapExplain, dealerFlowExplainNotes,
|
||||
dealerStudyEngine, hanStyleLevels, volumeByPrice
|
||||
options/ bsm, types, OptionsChainRouter, ConvexityGate (stub)
|
||||
llm/ openaiCompatible client, userLlmEndpoint
|
||||
admin/ operator functions + CLI
|
||||
auth/ totp, oauth, backup codes
|
||||
alerts/AlertEngine.ts
|
||||
alerts/producers/ types.ts, index.ts (registry), insiderProducer.ts, new13daProducer.ts
|
||||
services/emailAlertService.ts nodemailer SMTP delivery + rate limit
|
||||
risk/RiskEngine.ts + haltCircuitBreaker.ts ← pure; NOT on tRPC
|
||||
sizing/SizingEngine.ts + convictionUnlock + twoAxisMatrix ← pure; NOT on tRPC
|
||||
strategy/BacktestEngine.ts
|
||||
screener/UniverseEvaluator.ts + SectorCrosslink.ts
|
||||
options/ConvexityGate.ts
|
||||
derisking/DeriskingEngine.ts
|
||||
thesis/ThesisMonitor.ts
|
||||
macro/FredAdapter.ts + MacroRegime.ts
|
||||
reports/ReportRunner.ts
|
||||
onboarding/starter.ts
|
||||
services/secDataFetcher.ts
|
||||
x/backfill.ts
|
||||
alerts/AlertEngine.ts
|
||||
alerts/producers/ insider, 13D/13F, VIX, rotation, unlock, thesis,
|
||||
portfolio risk, confluence, mirror
|
||||
services/ emailAlertService, vendorGate, secHttp, secDataFetcher,
|
||||
reverse13fRefresh, cusipRegistry, captureIngest,
|
||||
analystRatingsService, FinraIngestService, stockFloat
|
||||
confluence/ slots, library, rack, engine, seed, evaluators,
|
||||
candleProvider, zones, corridor*, backtest
|
||||
mirror/ captureParser, fund13fFetcher, mirrorEngine
|
||||
risk/ RiskEngine, haltCircuitBreaker, optionRiskContribution
|
||||
sizing/ SizingEngine, convictionUnlock, twoAxisMatrix
|
||||
strategy/ BacktestEngine, PortfolioBacktestEngine
|
||||
screener/ UniverseEvaluator, SectorCrosslink
|
||||
derisking/ DeriskingEngine
|
||||
thesis/ ThesisMonitor
|
||||
macro/ FredAdapter, MacroRegime
|
||||
reports/ ReportRunner
|
||||
onboarding/ starter
|
||||
x/ backfill
|
||||
```
|
||||
|
||||
Boot adapter map (`index.ts`): `yfinance` (composed with options), `nasdaq`, `finra-bulk`, `finra-si`, `sec-fetch`, `sec-sc-fetch`, `sec-tickers`, `sec-lint-holders`, `sec-lint-insiders`, `cot`. X and FRED register only when credentials/keys exist.
|
||||
|
||||
### 4.1 tRPC surface (mounted)
|
||||
|
||||
| Router | Procedures (summary) |
|
||||
|--------|----------------------|
|
||||
| `auth` | signup, login, logout, me, enable2fa, confirm2fa, oauthStart, oauthCallback |
|
||||
| `onboarding` | starter, complete |
|
||||
| `market` | snapshot, candles, indicators, truckSales, manufacturingPmi, rotation, rotationCheckForAlert, sectorHoldings |
|
||||
| `auth` | signup, login, logout, me, changePassword, enable2fa, confirm2fa, oauthStart, oauthCallback |
|
||||
| `onboarding` | starter, complete, updateProfile |
|
||||
| `market` | snapshot, snapshots, tickerContext, candles, indicators, truckSales, manufacturingPmi, condition, rotation, add/remove/listCustomEtf, rotationCheckForAlert, seasonality, sectorHoldings |
|
||||
| `dashboard` | rollup |
|
||||
| `admin` | users, sessions, passwords, queue*, lint, data quality, X creds/accounts/prune, FRED key, pending/approve/reject, audit, serverRestart, gdprExport, smtpConfig, smtpConfigUpdate, smtpConfigTest |
|
||||
| `alerts` | list, acknowledge, acknowledgeAll, unackedCount, createSubscription, listSubscriptions, updateSubscription, deleteSubscription |
|
||||
| `institutional` | flow, insiderStream, ownershipHistory, buyEvents |
|
||||
| `admin` | users + modules + enable/disable/delete, sessions, passwords, queue* (incl. per-source pause/stop), lint, data quality, dealerMapIntegrity, alertStatus, X creds/accounts/prune, FRED key, FINRA URL, pending/approve/reject, audit, serverRestart, gdprExport, smtp* |
|
||||
| `alerts` | list, acknowledge, acknowledgeAll, clearAll, unackedCount, create/list/update/delete subscription, listTypes, toggleType |
|
||||
| `institutional` | flow, insiderStream, ownershipHistory, buyEvents, analystRatings, shortInterest |
|
||||
| `edgar` | filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx |
|
||||
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, addSymbol, removeSymbol |
|
||||
| `portfolio` | holdings, addHolding, removeHolding |
|
||||
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, moveSymbol, addSymbol, removeSymbol |
|
||||
| `portfolio` | holdings, addHolding, updateHolding, removeHolding, optionLegs, addOptionLeg, removeOptionLeg |
|
||||
| `options` | chain, greeks |
|
||||
| `dealerMap` | get, levels, scenario, velocity, explain — cache-only reads; schedule options chains via queue (ADR-0009) |
|
||||
| `dealerStudy` | propose (hist + optional mentor rank), log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
|
||||
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard — local mentor path-match; claim types map to study hypotheses for Phase-3 blend |
|
||||
| `userLlm` | status, upsertEndpoint, clear, test — per-user OpenAI-compatible base_url + encrypted key + model |
|
||||
| `dealerMap` | get, levels, scenario, velocity, explain |
|
||||
| `dealerStudy` | propose, log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
|
||||
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard |
|
||||
| `userLlm` | status, upsertEndpoint, clear, test |
|
||||
| `reports` | generate |
|
||||
| `screener` | filter, strategy |
|
||||
| `strategies` | list, create |
|
||||
| `backtest` | run, evaluateLatest |
|
||||
| `strategies` | list, create, get, listPresets, getPreset, forkPreset, suggestTickers |
|
||||
| `backtest` | run, evaluateLatest, runPortfolio |
|
||||
| `sectorCrosslink` | confirm |
|
||||
| `optionsConvexity` | unlock, getPayoff |
|
||||
| `derisking` | suggest |
|
||||
| `derisking` | suggest, dividendHealth |
|
||||
| `macro` | series, calendar, regimeClassify, commentary, regimeHistory |
|
||||
| `thesisMonitor` | assess, timeline |
|
||||
| `thesisMonitor` | assess (by symbol), timeline (empty events) |
|
||||
| `x` | feed, cashtag_search, timeline, accountsForSymbol |
|
||||
| `reddit` | subreddit, search |
|
||||
| `reddit` | subreddit, search (not queued) |
|
||||
| `emotionLogger` | add, getByTrade, delete |
|
||||
| `sizing` | compute |
|
||||
| `risk` | posture (no `haltStatus`; halt fields currently hardcoded false/null) |
|
||||
| `trades` | list, get, save, close, stats |
|
||||
| `theses` | list, get, create, update, delete |
|
||||
| `symbols` | search, holders |
|
||||
| `funds` | list, get, liveBook, adminCreate, adminDelete, adminSetEnabled, adminSync13f, adminIngestCaptures |
|
||||
| `mirror` | diff |
|
||||
| `confluence` | slots, racks, evaluation, backtest, scorecard, signalHistory, recentZones, saveRack, runEvaluationNow, corridorSnapshot, corridorWatchlist, corridorWatchlistAdd/Remove, corridorBacktest, corridorBacktestRun |
|
||||
|
||||
### 4.2 Sizing + risk (mounted 2026-07-18)
|
||||
**Removed since 2026-07-23:** `optionsConvexity.unlock` / `getPayoff`.
|
||||
|
||||
| Router | Procedures |
|
||||
|--------|------------|
|
||||
| `sizing` | `compute` — pure `sizePosition` + portfolio/unlock/halt context |
|
||||
| `risk` | `posture` — pure `assessRisk` + portfolio load + halt persist; `haltStatus` |
|
||||
### 4.2 Default schedule intervals
|
||||
|
||||
Still deferred: journal create path that throws `HaltedError` when plans are server-persisted (`persist-trade-plan-execution-loop`).
|
||||
From `sourceRatePolicy.SCHEDULE_INTERVALS`:
|
||||
|
||||
| Kind | Interval |
|
||||
|------|----------|
|
||||
| `yfinance-quote-portfolio` | 1 min |
|
||||
| `yfinance-quote-priority` | 5 min |
|
||||
| `yfinance-quote-watched` | 10 min |
|
||||
| `yfinance-eod` | 6 h |
|
||||
| `yfinance-meta` | 1 d |
|
||||
| `yfinance-holdings` | 7 d |
|
||||
| `sec-fetch` | 1 d |
|
||||
| `sec-sc-fetch` | 6 h |
|
||||
| `sec-tickers` | 1 d |
|
||||
| `sec-lint-holders` / `sec-lint-insiders` | 7 d |
|
||||
| `x` | 1 h |
|
||||
| `finra-si` | 14 d |
|
||||
| `fred` | 1 d |
|
||||
|
||||
`finra-bulk` is deleted on every boot if present (historical 403).
|
||||
|
||||
---
|
||||
|
||||
@@ -175,10 +265,12 @@ Still deferred: journal create path that throws `HaltedError` when plans are ser
|
||||
|
||||
```
|
||||
app/src/
|
||||
app/ Next App Router pages
|
||||
components/ Panels + ui kit + layout
|
||||
stores/ Zustand (several persist to localStorage)
|
||||
lib/trpc.ts Partial API client (subset of routers)
|
||||
app/ Next App Router pages + /health + /api proxy
|
||||
components/ Panels + ui kit + layout + dealer-flow + confluence
|
||||
stores/ Zustand (theme, goals, active symbol/watchlist, outlook sections)
|
||||
lib/trpc.ts Hand-rolled client (covers the product routers)
|
||||
lib/workspace-profile.ts Density + nav visibility
|
||||
lib/useFeatureAccess.ts Module access
|
||||
lib/chart-theme.ts CSS-variable themed Recharts
|
||||
lib/strings.ts Primary-rule sensitive copy
|
||||
```
|
||||
@@ -187,23 +279,47 @@ app/src/
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `/` | Workbench overview |
|
||||
| `/` | Symbol workbench (density-gated sections) |
|
||||
| `/chart-lab` | Charts |
|
||||
| `/institutional` | Institutional + filings |
|
||||
| `/filings` | Filings only |
|
||||
| `/options-dd` | Options DD |
|
||||
| `/dealer-flow` | Dealer map + Study Desk |
|
||||
| `/confluence` | Signal Confluence + Price Corridor |
|
||||
| `/market-outlook` | Macro / rotation / notes |
|
||||
| `/trade-plan`, `/execution`, `/trade-closure`, `/daily-focus` | Process (local) |
|
||||
| `/alerts` | Alert events + subscriptions |
|
||||
| `/settings` | Auth + onboarding |
|
||||
| `/mobile` | Thin companion |
|
||||
| `/admin`, `/users`, `/queue`, `/audit-logs`, `/x-accounts`, `/smtp` | Admin |
|
||||
| `/portfolio` | Book (holdings + option legs; guided setup via `?strategyId=`) |
|
||||
| `/risk` | Risk posture + sizing math |
|
||||
| `/funds`, `/funds/[id]` | Tracked funds + mirror |
|
||||
| `/guided-start` | Preset picker + quiz |
|
||||
| `/plan` | Decision plan (server) |
|
||||
| `/theses` | Thesis CRUD (server) |
|
||||
| `/journal` | Close + reflection (server) |
|
||||
| `/strategies` | Preset catalog + fork |
|
||||
| `/lab` | Backtest |
|
||||
| `/screener` | Filter + strategy screen |
|
||||
| `/exits` | Derisking considerations |
|
||||
| `/reports` | HTML research notes |
|
||||
| `/daily-focus` | Goals (localStorage) |
|
||||
| `/alerts` | Events + subscriptions |
|
||||
| `/welcome` | First-user desk (signup, signin, pending approval panel, TOTP step, interview). Standalone layout — no `LayoutShell`, no SymbolHeader. |
|
||||
| `/settings` | Account + density + user LLM (signed-in only; guest redirects to `/welcome?mode=signin`) |
|
||||
| `/more` | Mobile secondary destinations |
|
||||
| `/mobile`, `/monitor` | Redirect → `/portfolio` |
|
||||
| `/admin`, `/admin/users`, `/admin/queue`, `/admin/audit-logs`, `/admin/x-accounts`, `/admin/smtp` | Admin |
|
||||
| `/health` | Frontend liveness `{ ok, service: "investor-flow-web" }` |
|
||||
|
||||
### 5.2 Client coverage gap
|
||||
### 5.2 Client coverage
|
||||
|
||||
`lib/trpc.ts` exposes: market, edgar, auth, onboarding, watchlists, options, portfolio, dashboard, institutional, emotionLogger, alerts, x, admin.
|
||||
`lib/trpc.ts` now exposes the product surface listed in §4.1 (including confluence, funds, mirror, symbols, trades, theses, sizing, risk, strategies, backtest, screener, reports, derisking, dealerMap/Study, mentorLedger, userLlm).
|
||||
|
||||
**Not exposed in client (but on server):** screener, strategies, backtest, sectorCrosslink, optionsConvexity, derisking, macro (except commentary aliased under `market.commentary`), reports, thesisMonitor, reddit, some admin helpers (e.g. gdprExport).
|
||||
**Still server-only / thin on the client:** `admin.gdprExport`, `macro.series` / `calendar` / `regimeHistory` (commentary is aliased as `market.commentary`), `thesisMonitor`, `reddit` (mounted, no product screen).
|
||||
|
||||
### 5.3 Shell
|
||||
|
||||
- Desktop (`lg+`): sidebar + watchlist + main.
|
||||
- Tablet (`md`–`lg`): compact watchlist bar + main.
|
||||
- Phone (`<md`): header + scroll main + bottom tabs. CSS-first so there is no desktop-layout flash.
|
||||
- Nested `<main data-app-scroll>` is the only scroller (`100dvh`). Nav clicks force that pane to the top.
|
||||
|
||||
---
|
||||
|
||||
@@ -211,15 +327,20 @@ app/src/
|
||||
|
||||
| Source kind | Role | Typical freshness |
|
||||
|-------------|------|-------------------|
|
||||
| yfinance | quote, candles, sector, options | minutes / EOD |
|
||||
| sec / sec-fetch | filings, 13F, Form 4 bulk | daily schedule default |
|
||||
| sec-lint-* | gap detection / backfill | on demand admin |
|
||||
| x | timelines / cashtags | demand + prune |
|
||||
| reddit | subreddit/search | on demand |
|
||||
| fred / macro | series + commentary inputs | slower |
|
||||
| llm | summaries (tables present) | sparse product use |
|
||||
| yfinance (tiered) | quote, candles, sector, options, short-interest Yahoo slice | 1–10 min quotes / EOD candles |
|
||||
| nasdaq | days-to-cover + 24mo short-interest history | on demand + queue |
|
||||
| finra-si | FINRA short interest | ~14 d |
|
||||
| finra-bulk | bulk ingest (optional; not auto-scheduled) | operator |
|
||||
| sec / sec-fetch / sec-sc-fetch | filings, 13F, Form 4, SC 13D/G | daily / 6 h SC |
|
||||
| sec-tickers | issuer CIK seed | daily |
|
||||
| sec-lint-* | gap detection / backfill | weekly / on demand |
|
||||
| x | timelines / cashtags | hourly + demand |
|
||||
| reddit | subreddit/search | on demand, not queued |
|
||||
| fred | series + commentary inputs | daily |
|
||||
| cot | CFTC TFF positioning | confluence consumers |
|
||||
| llm | summaries / dealer explain | sparse; 180s client timeout |
|
||||
|
||||
**AdapterQueue features (built):** pause/resume, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery.
|
||||
**AdapterQueue features (built):** global and per-source pause/resume/stop/start, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery, demand hygiene, daily housekeeping.
|
||||
|
||||
---
|
||||
|
||||
@@ -227,30 +348,52 @@ app/src/
|
||||
|
||||
| Gate | Behavior |
|
||||
|------|----------|
|
||||
| `publicProcedure` | No session required (many market/edgar/watchlist reads still public — intentional for local demo; tighten later if multi-tenant hardens) |
|
||||
| `publicProcedure` | No session required (many market/edgar/watchlist reads still public - intentional for local demo; tighten later if multi-tenant hardens) |
|
||||
| `protectedProcedure` | Session + `users.status === 'active'` |
|
||||
| `adminProcedure` | `is_admin` flag |
|
||||
| Module access | `users.modules` JSON. Default `["research","settings"]`. Admin flag injects `admin`. Frontend `FeatureGate` + sidebar filter. |
|
||||
| Density | `users.density` (`focused` / `standard` / `full`) filters nav items and overview sections. Independent of modules. |
|
||||
|
||||
Watchlist list currently falls back to `userId ?? 'anonymous'` — convenient for local dev, weak isolation if exposed beyond localhost.
|
||||
Watchlist list currently falls back to `userId ?? 'anonymous'` - convenient for local dev, weak isolation if exposed beyond localhost.
|
||||
|
||||
Production: `trpc/context.ts` throws if `IFLOW_SESSION_SECRET` is unset and `NODE_ENV !== 'development'`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing
|
||||
|
||||
| Suite | Command | Audit result |
|
||||
|-------|---------|--------------|
|
||||
| Backend | `cd app/server && npm test` | **504 pass / 0 fail / 1 skip** (after P0 2026-07-18) |
|
||||
| Frontend lite | Primary-Rule lint + options pure tests | Present |
|
||||
| Suite | Command | Audit result (2026-08-18) |
|
||||
|-------|---------|---------------------------|
|
||||
| Backend | `cd app/server && npm test` | **903 pass / 0 fail / 1 skip** |
|
||||
| Backend types | `cd app/server && npm run typecheck` | Run in CI |
|
||||
| Frontend unit | `cd app && node --test --experimental-strip-types "src/**/*.test.ts"` | **58 pass / 0 fail** |
|
||||
| CI | `.github/workflows/ci.yml` | Frontend tests + backend tests + backend typecheck, then image push on `main` |
|
||||
|
||||
Known red: `MacroRegime` commentary tests (`generateMacroCommentary` short/long-term + no macro-trade recommendation).
|
||||
Node may warn that frontend test files are typeless in `package.json` (Next app is not `"type": "module"`). Do not add `"type": "module"` to the Next package; it would change how Next loads config.
|
||||
|
||||
---
|
||||
|
||||
## 9. Schema debt
|
||||
## 9. Schema notes
|
||||
|
||||
1. **`strategies` defined twice** in `schema.sql` (legacy `regime_gate/setup/risk_policy` vs `components/unlocked`). SQLite keeps first-created shape depending on migration history — dangerous drift.
|
||||
2. Parallel filter tables: `screener_filters` vs `saved_filters`.
|
||||
3. Process tables (`trades`, `trade_executions`, `theses`, `emotion_logs`) exist ahead of UI persistence.
|
||||
Single `strategies` table (the 2026-07 duplicate-definition debt is gone). New / notable tables since that audit:
|
||||
|
||||
- `symbols` (issuer CIK + search index)
|
||||
- `portfolio_option_legs`
|
||||
- `dealer_map_snapshots`, `dealer_study_setups`, `mentor_sources`, `mentor_calls`
|
||||
- `user_llm_endpoints`
|
||||
- `finra_short_interest`, `finra_short_interest_biweekly`, `finra_config`, `stock_float`
|
||||
- `tracked_funds`, `fund_position_records`
|
||||
- `confluence_racks`, `confluence_evaluations`, `confluence_signal_history`, `confluence_zone_rules`
|
||||
- `corridor_snapshots`, `corridor_watchlist`, `corridor_backtest`
|
||||
- `producer_run_log`, `notification_outbox`, `smtp_config`
|
||||
- `halt_state` (exists; not currently folded into `risk.posture`)
|
||||
|
||||
Remaining debt:
|
||||
|
||||
1. Parallel filter tables: `screener_filters` vs `saved_filters` (no product loop).
|
||||
2. `options_unlock` table leftover after the unlock ladder was removed.
|
||||
3. `thesisMonitor` does not yet populate events from filings/insiders.
|
||||
4. `risk.posture` does not read/write `halt_state`.
|
||||
|
||||
---
|
||||
|
||||
@@ -259,51 +402,124 @@ Known red: `MacroRegime` commentary tests (`generateMacroCommentary` short/long-
|
||||
| Designed | As built |
|
||||
|----------|----------|
|
||||
| Full LLMGateway with classifyPayload gate | Tables `llm_providers`, `llm_dispatch_audit`, `llm_summaries`; no complete gateway module under `src/` |
|
||||
| Ornith default `is_local=true` | Env-driven URL in docker-compose (`LLM_PROVIDER_URL`) |
|
||||
| Sensitive thesis data never leaves host | Thesis content mostly not yet flowing through LLM product paths |
|
||||
| Ornith default `is_local=true` | Env `LLM_PROVIDER_URL` / `ORNITH_LLM_URL` on the backend container (LAN IP, not `localhost`) |
|
||||
| Per-user endpoint | **Live** - `userLlm.*` + Settings UI; encrypted key; used for `dealerMap.explain` |
|
||||
| Sensitive thesis data never leaves host | Thesis content is not flowing through LLM product paths |
|
||||
|
||||
ADR-0006/0008 still govern intent; implementation is incomplete.
|
||||
ADR-0006/0008 still govern intent; the full gateway is incomplete. Dealer Flow Layer-0 notes are in-app (`dealerFlowExplainNotes.ts`). Optional offline harvest/distill scripts write that file **and** a personal Obsidian copy - the app never reads Obsidian at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 11. Deployment
|
||||
## 11. Deployment and CI/CD
|
||||
|
||||
Local dev (typical):
|
||||
This is the as-built operator path. Day-to-day commands live in [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md).
|
||||
|
||||
### 11.1 Images
|
||||
|
||||
| Piece | Path | Runtime |
|
||||
|---|---|---|
|
||||
| Backend image | `app/server/Dockerfile` | Node 22, `tini`, `node --experimental-strip-types src/index.ts`, data at `/app/data` |
|
||||
| Frontend image | `app/Dockerfile` | Next standalone, `IFLOW_BACKEND_URL=http://backend:3001` |
|
||||
| Build-from-git compose | `docker-compose.yml` | Used on a laptop or an Unraid checkout |
|
||||
| Pull-prebuilt compose | `deploy/unraid/compose.pull.yml` | After CI has published tags |
|
||||
| Gitea runner | `deploy/unraid/act-runner-compose.yml` | `gitea/act_runner:0.2.13`, Docker socket, label `ubuntu-latest` |
|
||||
| Cron fallback | `deploy/unraid/user-scripts/sync-and-up.sh` | `git fetch` + ff-only merge + `compose up --build` when HEAD moved |
|
||||
|
||||
Health:
|
||||
|
||||
- Backend `GET /health` → `{ ok, queue: queue.health() }`
|
||||
- Frontend `GET /health` → `{ ok, service: "investor-flow-web" }`
|
||||
|
||||
Compose `depends_on: backend.service_healthy` so the web container does not start until tRPC is up.
|
||||
|
||||
### 11.2 Gitea Actions
|
||||
|
||||
**`ci.yml` (automatic)**
|
||||
|
||||
| Trigger | Jobs |
|
||||
|---------|------|
|
||||
| push / PR to `main`, `workflow_dispatch` | `test` |
|
||||
| push to `main` after `test` succeeds | `images` |
|
||||
|
||||
`test`: Node 22, `npm ci` in `app/` and `app/server/`, frontend unit tests, backend `npm test`, backend `npm run typecheck`.
|
||||
|
||||
`images`: Docker Buildx against the insecure HTTP registry, login with `REGISTRY_TOKEN` (or `github.token`), push:
|
||||
|
||||
- `${REGISTRY}/transnet/investor-flow-backend:${{ github.sha }}` and `:latest`
|
||||
- `${REGISTRY}/transnet/investor-flow-frontend:${{ github.sha }}` and `:latest`
|
||||
- registry cache tags `:buildcache`
|
||||
|
||||
Default `REGISTRY` is `10.37.0.86:3003` (override with a Gitea Actions variable).
|
||||
|
||||
**`cd.yml` (manual)**
|
||||
|
||||
Same image build/push without the test gate. Use when you need to republish images from the current `main` SHA.
|
||||
|
||||
### 11.3 How a change reaches Unraid
|
||||
|
||||
1. Commit and `git push origin main` to Gitea.
|
||||
2. act_runner picks up the workflow (label `ubuntu-latest`).
|
||||
3. Tests run. On failure, images are **not** published.
|
||||
4. On success, new `:latest` and `:${sha}` tags land in the Gitea registry.
|
||||
5. Unraid must **pull and recreate** containers. Publishing is not the same as rolling the running stack.
|
||||
|
||||
Two supported run modes:
|
||||
|
||||
**A. Pull prebuilt (preferred after CI is green)**
|
||||
|
||||
```bash
|
||||
# .env image names point at the registry (see deploy/unraid/.env.example)
|
||||
docker compose -f deploy/unraid/compose.pull.yml pull
|
||||
docker compose -f deploy/unraid/compose.pull.yml up -d
|
||||
```
|
||||
|
||||
**B. Build on Unraid from the git checkout**
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
git pull --ff-only
|
||||
docker compose --env-file /mnt/cache/appdata/investor-flow/.env up -d --build
|
||||
```
|
||||
|
||||
Mode B is what `sync-and-up.sh` automates. Mode A is what the registry tags are for. Do not mix them on the same project name without knowing which image tags `.env` pins.
|
||||
|
||||
### 11.4 Secrets and data
|
||||
|
||||
Required in production `.env`:
|
||||
|
||||
- `IFLOW_SESSION_SECRET` - `openssl rand -hex 32`
|
||||
- `SEC_OPERATOR_EMAIL` - real address (SEC + Yahoo user-agent)
|
||||
- `IFLOW_DATA_DIR` - cache or single-disk path, **not** `/mnt/user` (FUSE can corrupt SQLite WAL)
|
||||
|
||||
Recommended:
|
||||
|
||||
- `IFLOW_CRYPTO_KEY` - required to decrypt existing X/FRED rows if you copy a Mac DB
|
||||
- `LLM_PROVIDER_URL` / `ORNITH_LLM_URL` - Ornith LAN URL (not `localhost` inside the container)
|
||||
- `YF_OPERATOR_EMAIL`, `FRED_API_KEY`, OAuth client ids (optional)
|
||||
|
||||
SQLite lives at `/app/data/investor-flow.db` inside the backend container. Persist that directory. Stop the backend (or use the SQLite backup API) before copying `investor-flow.db*`.
|
||||
|
||||
### 11.5 Local dev (unchanged)
|
||||
|
||||
```bash
|
||||
cd app/server && npm run dev # :3001
|
||||
cd app && npm run dev # :3000
|
||||
```
|
||||
|
||||
Production is two Node containers (not Bun). Unraid is the intended host; git + Gitea Actions already live at `unraid.local:3003/transnet/investor-flow`.
|
||||
|
||||
```bash
|
||||
# Build-from-git (laptop or Unraid checkout)
|
||||
docker compose --env-file .env up -d --build
|
||||
```
|
||||
|
||||
| Piece | Path |
|
||||
|---|---|
|
||||
| Backend image | `app/server/Dockerfile` (Node 22, `node --experimental-strip-types`) |
|
||||
| Frontend image | `app/Dockerfile` (Next standalone, proxies `/api` via `IFLOW_BACKEND_URL`) |
|
||||
| Unraid operator guide | `docs/DEPLOY_UNRAID.md` |
|
||||
| Gitea CI | `.github/workflows/ci.yml` |
|
||||
| Gitea CD (registry push) | `.github/workflows/cd.yml` |
|
||||
| act_runner | `deploy/unraid/act-runner-compose.yml` |
|
||||
| Cron CD (no runner) | `deploy/unraid/user-scripts/sync-and-up.sh` |
|
||||
|
||||
Secrets: `.env.example` and `deploy/unraid/.env.example`. Production refuses to start without `IFLOW_SESSION_SECRET`. Persist SQLite on a cache/single-disk path (`IFLOW_DATA_DIR`), not `/mnt/user`.
|
||||
Or `docker compose --env-file .env up -d --build` from the repo root.
|
||||
|
||||
---
|
||||
|
||||
## 12. Documentation map (after this audit)
|
||||
## 12. Documentation map
|
||||
|
||||
| Doc | Role |
|
||||
|-----|------|
|
||||
| `docs/FUNCTIONAL_DESIGN.md` | What users can do / pending |
|
||||
| `docs/TECH_DESIGN.md` | This file — how it is built |
|
||||
| `docs/TECH_DESIGN.md` | This file - how it is built |
|
||||
| `docs/DEPLOY_UNRAID.md` | Operator process: Unraid + Gitea CI/CD |
|
||||
| `docs/VENDOR_INTEGRATIONS.md` | How to add a vendor without bypassing the gate |
|
||||
| `CONTEXT.md` | Ubiquitous language (glossary only) |
|
||||
| `docs/adr/*` | Durable decisions |
|
||||
| Obsidian `investor-flow.md` | Session orientation wiki |
|
||||
| `docs/adr/*` | Durable decisions (0001–0012) |
|
||||
| Obsidian `investor-flow.md` | Session orientation wiki (status here wins) |
|
||||
|
||||
Deprecated as status sources: root `HANDOFF.md` (2026-06-30 orchestrator snapshot), slice DECOMPOSITION checkbox state.
|
||||
|
||||
Reference in New Issue
Block a user