Investor Flow Build
7649eaf399
First visit lands on /welcome instead of burying signup in Settings. Add a host CLI to reset passwords without a session, plus Settings change-password. Refresh as-built design docs and the Unraid operator guide.
526 lines
28 KiB
Markdown
526 lines
28 KiB
Markdown
# Investor Flow - Technical Design (as built)
|
||
|
||
**Audit date:** 2026-08-18
|
||
**Companion:** [`FUNCTIONAL_DESIGN.md`](./FUNCTIONAL_DESIGN.md)
|
||
**Operator deploy:** [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md)
|
||
**Canonical code roots:** `app/src` (Next frontend), `app/server/src` (Node backend)
|
||
|
||
---
|
||
|
||
## 1. Stack
|
||
|
||
| Layer | Actual technology | Notes vs older docs |
|
||
|-------|-------------------|---------------------|
|
||
| Frontend | Next.js **16.2.9**, React **19.2**, Tailwind **v4**, Recharts, Zustand, Radix UI | Not Bun SPA. `output: "standalone"` for the frontend image. |
|
||
| Backend runtime | **Node ≥22** (dev on Node 26), native TS via `--experimental-strip-types` | DESIGN said Bun; runtime glue is Node + `node:sqlite` |
|
||
| API | **tRPC v11** HTTP, path `/api/trpc/*` | Frontend uses a hand-rolled fetch client in `app/src/lib/trpc.ts` (not `@trpc/client`). Next **route handler** proxies `/api` with a 200s timeout (rewrites time out ~30s and break Ornith). |
|
||
| DB | SQLite file (`IFLOW_DB_PATH`, default `app/server/data/investor-flow.db`) | Single file multi-tenant. On Unraid this is a bind-mounted cache-disk path. |
|
||
| Market data | `yahoo-finance2` **v3** class API | Quotes, candles, options, short interest (Yahoo slice) |
|
||
| SEC | EdgarAdapter + SecFetchAdapter + SecLintAdapter + SecCompanyTickersAdapter + `secDataFetcher` | Filer CIK from accession prefix. Daily `company_tickers.json` seed. |
|
||
| Short interest | NasdaqAdapter + FinraShortInterestAdapter + FinraBulkAdapter | Three-way merge with discrepancy flags. `finra-bulk` not auto-scheduled. |
|
||
| Macro / COT | FredAdapter + CotAdapter | FRED key admin-side. COT from CFTC zip (confluence slot). |
|
||
| Social | XCookieAdapter (encrypted ct0/auth_token), RedditAdapter | X is queued. Reddit tRPC calls the adapter directly and is not in the boot adapter map. |
|
||
| Auth | Sessions + TOTP + OAuth (GitHub/Google) | Password hash via local crypto helpers. Production refuses to start without `IFLOW_SESSION_SECRET`. |
|
||
| Tests | `node --test --experimental-strip-types` | Backend **903 pass / 0 fail / 1 skip**; frontend **58 pass / 0 fail** (2026-08-18). |
|
||
| Deploy | Two Node containers on Unraid | Gitea Actions: test, then push images to the Gitea registry. See §11. |
|
||
|
||
---
|
||
|
||
## 2. Process topology
|
||
|
||
### 2.1 Runtime (laptop or Unraid)
|
||
|
||
```
|
||
Browser
|
||
│ fetch /api/trpc/<proc> (credentials include, same-origin)
|
||
▼
|
||
Next frontend (:3000)
|
||
│ app/src/app/api/[...path]/route.ts (200s proxy)
|
||
▼
|
||
Backend (:3001)
|
||
│
|
||
├─ appRouter (tRPC)
|
||
├─ CacheRepository + AdapterQueue + vendorGate
|
||
├─ Source adapters (yfinance+options, nasdaq, finra-*, sec-*, x, fred, cot)
|
||
├─ Background loops (drain, schedules, alerts, SMTP outbox, confluence, housekeeping)
|
||
└─ SQLite
|
||
```
|
||
|
||
- Dev helpers: `restart-servers.sh`, admin `serverRestart` (laptop-oriented).
|
||
- Queue drain default: every 2s (`IFLOW_DRAIN_MS`).
|
||
- Schedule loop: every 30s.
|
||
- Realtime/per-fetch alerts: every 30s (VIX).
|
||
- Batched alerts: every 10 min.
|
||
- SMTP outbox drain: every 60s.
|
||
- Confluence tick: hourly (+ boot fill after 3 min).
|
||
- Queue housekeeping: daily (`clearDone` + prune `queue_errors`).
|
||
|
||
### 2.2 Production host
|
||
|
||
Unraid is the intended host. Git + Gitea Actions live at `unraid.local:3003/transnet/investor-flow` (HTTP registry also at `10.37.0.86:3003`).
|
||
|
||
```
|
||
git push origin main
|
||
│
|
||
▼
|
||
Gitea repo + act_runner (label ubuntu-latest)
|
||
│
|
||
├─ ci.yml / test frontend tests, backend tests, backend typecheck
|
||
└─ ci.yml / images (main only, after test) build + push
|
||
│
|
||
▼
|
||
Gitea registry
|
||
transnet/investor-flow-backend:{sha,latest}
|
||
transnet/investor-flow-frontend:{sha,latest}
|
||
│
|
||
▼
|
||
Unraid compose (backend :3001 + frontend :3000)
|
||
data volume: IFLOW_DATA_DIR (cache/single-disk, not /mnt/user FUSE)
|
||
```
|
||
|
||
`cd.yml` is **manual** (`workflow_dispatch`) - rebuild and push images without a new commit.
|
||
|
||
A git-pull + `compose up --build` cron (`deploy/unraid/user-scripts/sync-and-up.sh`) remains as a fallback when you want Unraid to build from source instead of pulling registry tags.
|
||
|
||
---
|
||
|
||
## 3. Multi-tenant data tiers (still valid)
|
||
|
||
| Tier | Content | Isolation |
|
||
|------|---------|-----------|
|
||
| A | quotes, candles, options, filings, institution_filings, insider_tx, sector_map, macro, symbols, dealer maps, short interest, COT, FRED | Shared, no owner |
|
||
| B | threads, x_cookie_posts, reddit_posts, adapter_queue*, queue_* | Shared infrastructure |
|
||
| C | watchlists, portfolio_holdings, portfolio_option_legs, trades, strategies, alerts, theses, emotion_logs, reports, saved filters, confluence user racks, corridor watchlist | `owner_id` / `user_id` |
|
||
| D | users, sessions, admin_audit, x_credentials (singleton), smtp_config, llm_*, tracked_funds (operator-curated), fund_position_records | System / shared fund facts |
|
||
|
||
**Demand set:** `symbol_demand` refcount drives which symbols the queue refreshes. System pins (rotation universe, SPY, VIX, confluence universe) use `system_pin` so they are not refcount-inflated.
|
||
|
||
**Stale-while-revalidate:** CacheRepository returns cached rows and schedules refresh when past TTL.
|
||
|
||
### 3.1 Rate-limit-first data plane (ADR-0009)
|
||
|
||
All vendors (Yahoo, X, FRED, SEC, Reddit, NASDAQ, FINRA, CFTC) have short rate limits. The system is designed so stress yields **stale/static UI**, not stampede:
|
||
|
||
| Rule | Mechanism |
|
||
|------|-----------|
|
||
| Request path never stampede | Serve SQLite / `kv_cache` / static fallback first; **no live Yahoo on tRPC** (queue only) |
|
||
| One outbound owner | `AdapterQueue` only (background drain); UI schedules via `CacheRepository.get` |
|
||
| Steady throttle | Per-source min-interval (`sourceRatePolicy.DEFAULT_SOURCE_MIN_INTERVAL_MS`) |
|
||
| 429 cool-down | Source-wide pause 2→5→15→30→60 min; skip all jobs for that source; no schedule flood |
|
||
| Demand-bounded work | Watchlist/portfolio `subscribe` + `ensureInDemand` / `pinSystemSymbol` |
|
||
| TTL-aware tiers | quote-portfolio 1m, quote-priority 5m, quote-watched 10m, EOD candles 6h, meta daily, holdings weekly |
|
||
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
|
||
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
|
||
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
|
||
| Observability | `queue.health()`: vendorGate+queue_state cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||
| Per-source admin control | pause / resume / stop (clear pending) / start |
|
||
|
||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow; Next **rewrites** for `/api` (30s hard timeout).
|
||
|
||
### 3.2 Dealer Flow data plane
|
||
|
||
- Engine: pure `dealerExposureEngine` on `NormalizedOptionSurface` only (no vendor imports).
|
||
- Default provider: Yahoo via `composeYFinanceWithOptions` + `OPTIONS_CHAIN_PROVIDER` (default `yfinance`).
|
||
- Paid switch later: implement SourceFetch for `tradier`/`polygon`, register in `sourceRatePolicy`, set env - engine unchanged.
|
||
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
|
||
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
|
||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip) via `withExposureConvention` / `dealerMap.get({ convention })`. First-paint UI is GEX | VEX only; Method drawer has Customer book | Dealer inventory.
|
||
- Han-style day script (`hanStyleLevels`) is derived from the active convention and returned on `dealerMap.get`.
|
||
|
||
### 3.3 SEC / institutional data plane
|
||
|
||
- SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS.
|
||
- 13F prefers EFTS CUSIP pagination; on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`).
|
||
- `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done).
|
||
- `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||
- Daily `sec-tickers` materializes `company_tickers.json` into `symbols` (issuer CIK, name, exchange). Merge policy: SEC never overwrites yfinance `sector` / `industry` / `peers` / `ticker_kind` and never purges non-SEC rows (ADR-0011).
|
||
|
||
### 3.4 Vendor gate (hard requirement)
|
||
|
||
Process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm, cftc. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||
|
||
### 3.5 Confluence data plane (ADR-0012)
|
||
|
||
- Evaluators resolve candles through `CandleProvider` (cache + freshest quote folded into a partial daily bar).
|
||
- Hourly evaluation of *today* + incremental as-of replay (~3y, budgeted).
|
||
- Weekly walk-forward derivation of entry/exit zone rules on 11 GICS sector ETFs + SPY.
|
||
- Reliability weights (once a slot has ≥2 resolved fires) scale evidence inside `evaluateRack`.
|
||
|
||
Slow-changing composition (ETF top holdings) uses `kv_cache` + `etfHoldingsFallback.ts` + queued `yfinance:topHoldings:*` refresh.
|
||
|
||
---
|
||
|
||
## 4. Backend module map
|
||
|
||
```
|
||
app/server/src/
|
||
index.ts HTTP server + boot: adapters, schedules, pins, loops
|
||
trpc/router.ts all routers
|
||
trpc/context.ts session, cookies, db, cache, queue injection
|
||
db/schema.sql DDL
|
||
db/*Repository.ts watchlist, portfolio, option legs, emotion logs,
|
||
alertSubscription, fund, confluence, corridor
|
||
cache/CacheRepository.ts content-addressed cache API
|
||
queue/AdapterQueue.ts schedule, pause/stop per source, retry, cool-downs
|
||
queue/sourceRatePolicy.ts 429 detect, cool-down ladders, SCHEDULE_INTERVALS
|
||
adapters/ YFinance, Options, Nasdaq, Finra*, Cot, Edgar,
|
||
SecFetch, SecCompanyTickers, SecLint, X, Reddit
|
||
analysis/ indicators, rotation, seasonality, etfHoldingsFallback,
|
||
tickerContext, dealerExposureEngine, dealerMapService,
|
||
dealerMapExplain, dealerFlowExplainNotes,
|
||
dealerStudyEngine, hanStyleLevels, volumeByPrice
|
||
options/ bsm, types, OptionsChainRouter, ConvexityGate (stub)
|
||
llm/ openaiCompatible client, userLlmEndpoint
|
||
admin/ operator functions + CLI
|
||
auth/ totp, oauth, backup codes
|
||
alerts/AlertEngine.ts
|
||
alerts/producers/ insider, 13D/13F, VIX, rotation, unlock, thesis,
|
||
portfolio risk, confluence, mirror
|
||
services/ emailAlertService, vendorGate, secHttp, secDataFetcher,
|
||
reverse13fRefresh, cusipRegistry, captureIngest,
|
||
analystRatingsService, FinraIngestService, stockFloat
|
||
confluence/ slots, library, rack, engine, seed, evaluators,
|
||
candleProvider, zones, corridor*, backtest
|
||
mirror/ captureParser, fund13fFetcher, mirrorEngine
|
||
risk/ RiskEngine, haltCircuitBreaker, optionRiskContribution
|
||
sizing/ SizingEngine, convictionUnlock, twoAxisMatrix
|
||
strategy/ BacktestEngine, PortfolioBacktestEngine
|
||
screener/ UniverseEvaluator, SectorCrosslink
|
||
derisking/ DeriskingEngine
|
||
thesis/ ThesisMonitor
|
||
macro/ FredAdapter, MacroRegime
|
||
reports/ ReportRunner
|
||
onboarding/ starter
|
||
x/ backfill
|
||
```
|
||
|
||
Boot adapter map (`index.ts`): `yfinance` (composed with options), `nasdaq`, `finra-bulk`, `finra-si`, `sec-fetch`, `sec-sc-fetch`, `sec-tickers`, `sec-lint-holders`, `sec-lint-insiders`, `cot`. X and FRED register only when credentials/keys exist.
|
||
|
||
### 4.1 tRPC surface (mounted)
|
||
|
||
| Router | Procedures (summary) |
|
||
|--------|----------------------|
|
||
| `auth` | signup, login, logout, me, changePassword, enable2fa, confirm2fa, oauthStart, oauthCallback |
|
||
| `onboarding` | starter, complete, updateProfile |
|
||
| `market` | snapshot, snapshots, tickerContext, candles, indicators, truckSales, manufacturingPmi, condition, rotation, add/remove/listCustomEtf, rotationCheckForAlert, seasonality, sectorHoldings |
|
||
| `dashboard` | rollup |
|
||
| `admin` | users + modules + enable/disable/delete, sessions, passwords, queue* (incl. per-source pause/stop), lint, data quality, dealerMapIntegrity, alertStatus, X creds/accounts/prune, FRED key, FINRA URL, pending/approve/reject, audit, serverRestart, gdprExport, smtp* |
|
||
| `alerts` | list, acknowledge, acknowledgeAll, clearAll, unackedCount, create/list/update/delete subscription, listTypes, toggleType |
|
||
| `institutional` | flow, insiderStream, ownershipHistory, buyEvents, analystRatings, shortInterest |
|
||
| `edgar` | filings_index, company_facts, filer_cik_meta, full_text_search, form13f_holdings, form4_tx |
|
||
| `watchlists` | list, listByWatchlist, listWatchlists, create, delete, rename, reorder, moveSymbol, addSymbol, removeSymbol |
|
||
| `portfolio` | holdings, addHolding, updateHolding, removeHolding, optionLegs, addOptionLeg, removeOptionLeg |
|
||
| `options` | chain, greeks |
|
||
| `dealerMap` | get, levels, scenario, velocity, explain |
|
||
| `dealerStudy` | propose, log, list, grade, gradeDue, scorecard, promoteToJournal, exportCsv |
|
||
| `mentorLedger` | importFromHarvest, list, confirm, discard, grade, gradeDue, scorecard |
|
||
| `userLlm` | status, upsertEndpoint, clear, test |
|
||
| `reports` | generate |
|
||
| `screener` | filter, strategy |
|
||
| `strategies` | list, create, get, listPresets, getPreset, forkPreset, suggestTickers |
|
||
| `backtest` | run, evaluateLatest, runPortfolio |
|
||
| `sectorCrosslink` | confirm |
|
||
| `derisking` | suggest, dividendHealth |
|
||
| `macro` | series, calendar, regimeClassify, commentary, regimeHistory |
|
||
| `thesisMonitor` | assess (by symbol), timeline (empty events) |
|
||
| `x` | feed, cashtag_search, timeline, accountsForSymbol |
|
||
| `reddit` | subreddit, search (not queued) |
|
||
| `emotionLogger` | add, getByTrade, delete |
|
||
| `sizing` | compute |
|
||
| `risk` | posture (no `haltStatus`; halt fields currently hardcoded false/null) |
|
||
| `trades` | list, get, save, close, stats |
|
||
| `theses` | list, get, create, update, delete |
|
||
| `symbols` | search, holders |
|
||
| `funds` | list, get, liveBook, adminCreate, adminDelete, adminSetEnabled, adminSync13f, adminIngestCaptures |
|
||
| `mirror` | diff |
|
||
| `confluence` | slots, racks, evaluation, backtest, scorecard, signalHistory, recentZones, saveRack, runEvaluationNow, corridorSnapshot, corridorWatchlist, corridorWatchlistAdd/Remove, corridorBacktest, corridorBacktestRun |
|
||
|
||
**Removed since 2026-07-23:** `optionsConvexity.unlock` / `getPayoff`.
|
||
|
||
### 4.2 Default schedule intervals
|
||
|
||
From `sourceRatePolicy.SCHEDULE_INTERVALS`:
|
||
|
||
| Kind | Interval |
|
||
|------|----------|
|
||
| `yfinance-quote-portfolio` | 1 min |
|
||
| `yfinance-quote-priority` | 5 min |
|
||
| `yfinance-quote-watched` | 10 min |
|
||
| `yfinance-eod` | 6 h |
|
||
| `yfinance-meta` | 1 d |
|
||
| `yfinance-holdings` | 7 d |
|
||
| `sec-fetch` | 1 d |
|
||
| `sec-sc-fetch` | 6 h |
|
||
| `sec-tickers` | 1 d |
|
||
| `sec-lint-holders` / `sec-lint-insiders` | 7 d |
|
||
| `x` | 1 h |
|
||
| `finra-si` | 14 d |
|
||
| `fred` | 1 d |
|
||
|
||
`finra-bulk` is deleted on every boot if present (historical 403).
|
||
|
||
---
|
||
|
||
## 5. Frontend architecture
|
||
|
||
```
|
||
app/src/
|
||
app/ Next App Router pages + /health + /api proxy
|
||
components/ Panels + ui kit + layout + dealer-flow + confluence
|
||
stores/ Zustand (theme, goals, active symbol/watchlist, outlook sections)
|
||
lib/trpc.ts Hand-rolled client (covers the product routers)
|
||
lib/workspace-profile.ts Density + nav visibility
|
||
lib/useFeatureAccess.ts Module access
|
||
lib/chart-theme.ts CSS-variable themed Recharts
|
||
lib/strings.ts Primary-rule sensitive copy
|
||
```
|
||
|
||
### 5.1 Routes
|
||
|
||
| Path | Role |
|
||
|------|------|
|
||
| `/` | Symbol workbench (density-gated sections) |
|
||
| `/chart-lab` | Charts |
|
||
| `/institutional` | Institutional + filings |
|
||
| `/filings` | Filings only |
|
||
| `/options-dd` | Options DD |
|
||
| `/dealer-flow` | Dealer map + Study Desk |
|
||
| `/confluence` | Signal Confluence + Price Corridor |
|
||
| `/market-outlook` | Macro / rotation / notes |
|
||
| `/portfolio` | Book (holdings + option legs; guided setup via `?strategyId=`) |
|
||
| `/risk` | Risk posture + sizing math |
|
||
| `/funds`, `/funds/[id]` | Tracked funds + mirror |
|
||
| `/guided-start` | Preset picker + quiz |
|
||
| `/plan` | Decision plan (server) |
|
||
| `/theses` | Thesis CRUD (server) |
|
||
| `/journal` | Close + reflection (server) |
|
||
| `/strategies` | Preset catalog + fork |
|
||
| `/lab` | Backtest |
|
||
| `/screener` | Filter + strategy screen |
|
||
| `/exits` | Derisking considerations |
|
||
| `/reports` | HTML research notes |
|
||
| `/daily-focus` | Goals (localStorage) |
|
||
| `/alerts` | Events + subscriptions |
|
||
| `/welcome` | First-user desk (signup, signin, pending approval panel, TOTP step, interview). Standalone layout — no `LayoutShell`, no SymbolHeader. |
|
||
| `/settings` | Account + density + user LLM (signed-in only; guest redirects to `/welcome?mode=signin`) |
|
||
| `/more` | Mobile secondary destinations |
|
||
| `/mobile`, `/monitor` | Redirect → `/portfolio` |
|
||
| `/admin`, `/admin/users`, `/admin/queue`, `/admin/audit-logs`, `/admin/x-accounts`, `/admin/smtp` | Admin |
|
||
| `/health` | Frontend liveness `{ ok, service: "investor-flow-web" }` |
|
||
|
||
### 5.2 Client coverage
|
||
|
||
`lib/trpc.ts` now exposes the product surface listed in §4.1 (including confluence, funds, mirror, symbols, trades, theses, sizing, risk, strategies, backtest, screener, reports, derisking, dealerMap/Study, mentorLedger, userLlm).
|
||
|
||
**Still server-only / thin on the client:** `admin.gdprExport`, `macro.series` / `calendar` / `regimeHistory` (commentary is aliased as `market.commentary`), `thesisMonitor`, `reddit` (mounted, no product screen).
|
||
|
||
### 5.3 Shell
|
||
|
||
- Desktop (`lg+`): sidebar + watchlist + main.
|
||
- Tablet (`md`–`lg`): compact watchlist bar + main.
|
||
- Phone (`<md`): header + scroll main + bottom tabs. CSS-first so there is no desktop-layout flash.
|
||
- Nested `<main data-app-scroll>` is the only scroller (`100dvh`). Nav clicks force that pane to the top.
|
||
|
||
---
|
||
|
||
## 6. Caching & adapters
|
||
|
||
| Source kind | Role | Typical freshness |
|
||
|-------------|------|-------------------|
|
||
| yfinance (tiered) | quote, candles, sector, options, short-interest Yahoo slice | 1–10 min quotes / EOD candles |
|
||
| nasdaq | days-to-cover + 24mo short-interest history | on demand + queue |
|
||
| finra-si | FINRA short interest | ~14 d |
|
||
| finra-bulk | bulk ingest (optional; not auto-scheduled) | operator |
|
||
| sec / sec-fetch / sec-sc-fetch | filings, 13F, Form 4, SC 13D/G | daily / 6 h SC |
|
||
| sec-tickers | issuer CIK seed | daily |
|
||
| sec-lint-* | gap detection / backfill | weekly / on demand |
|
||
| x | timelines / cashtags | hourly + demand |
|
||
| reddit | subreddit/search | on demand, not queued |
|
||
| fred | series + commentary inputs | daily |
|
||
| cot | CFTC TFF positioning | confluence consumers |
|
||
| llm | summaries / dealer explain | sparse; 180s client timeout |
|
||
|
||
**AdapterQueue features (built):** global and per-source pause/resume/stop/start, per-job errors with stacks, retry job/source, clear done, schedules, startup in_flight→pending recovery, demand hygiene, daily housekeeping.
|
||
|
||
---
|
||
|
||
## 7. AuthZ model
|
||
|
||
| Gate | Behavior |
|
||
|------|----------|
|
||
| `publicProcedure` | No session required (many market/edgar/watchlist reads still public - intentional for local demo; tighten later if multi-tenant hardens) |
|
||
| `protectedProcedure` | Session + `users.status === 'active'` |
|
||
| `adminProcedure` | `is_admin` flag |
|
||
| Module access | `users.modules` JSON. Default `["research","settings"]`. Admin flag injects `admin`. Frontend `FeatureGate` + sidebar filter. |
|
||
| Density | `users.density` (`focused` / `standard` / `full`) filters nav items and overview sections. Independent of modules. |
|
||
|
||
Watchlist list currently falls back to `userId ?? 'anonymous'` - convenient for local dev, weak isolation if exposed beyond localhost.
|
||
|
||
Production: `trpc/context.ts` throws if `IFLOW_SESSION_SECRET` is unset and `NODE_ENV !== 'development'`.
|
||
|
||
---
|
||
|
||
## 8. Testing
|
||
|
||
| Suite | Command | Audit result (2026-08-18) |
|
||
|-------|---------|---------------------------|
|
||
| Backend | `cd app/server && npm test` | **903 pass / 0 fail / 1 skip** |
|
||
| Backend types | `cd app/server && npm run typecheck` | Run in CI |
|
||
| Frontend unit | `cd app && node --test --experimental-strip-types "src/**/*.test.ts"` | **58 pass / 0 fail** |
|
||
| CI | `.github/workflows/ci.yml` | Frontend tests + backend tests + backend typecheck, then image push on `main` |
|
||
|
||
Node may warn that frontend test files are typeless in `package.json` (Next app is not `"type": "module"`). Do not add `"type": "module"` to the Next package; it would change how Next loads config.
|
||
|
||
---
|
||
|
||
## 9. Schema notes
|
||
|
||
Single `strategies` table (the 2026-07 duplicate-definition debt is gone). New / notable tables since that audit:
|
||
|
||
- `symbols` (issuer CIK + search index)
|
||
- `portfolio_option_legs`
|
||
- `dealer_map_snapshots`, `dealer_study_setups`, `mentor_sources`, `mentor_calls`
|
||
- `user_llm_endpoints`
|
||
- `finra_short_interest`, `finra_short_interest_biweekly`, `finra_config`, `stock_float`
|
||
- `tracked_funds`, `fund_position_records`
|
||
- `confluence_racks`, `confluence_evaluations`, `confluence_signal_history`, `confluence_zone_rules`
|
||
- `corridor_snapshots`, `corridor_watchlist`, `corridor_backtest`
|
||
- `producer_run_log`, `notification_outbox`, `smtp_config`
|
||
- `halt_state` (exists; not currently folded into `risk.posture`)
|
||
|
||
Remaining debt:
|
||
|
||
1. Parallel filter tables: `screener_filters` vs `saved_filters` (no product loop).
|
||
2. `options_unlock` table leftover after the unlock ladder was removed.
|
||
3. `thesisMonitor` does not yet populate events from filings/insiders.
|
||
4. `risk.posture` does not read/write `halt_state`.
|
||
|
||
---
|
||
|
||
## 10. LLM / provenance (as built vs design)
|
||
|
||
| Designed | As built |
|
||
|----------|----------|
|
||
| Full LLMGateway with classifyPayload gate | Tables `llm_providers`, `llm_dispatch_audit`, `llm_summaries`; no complete gateway module under `src/` |
|
||
| Ornith default `is_local=true` | Env `LLM_PROVIDER_URL` / `ORNITH_LLM_URL` on the backend container (LAN IP, not `localhost`) |
|
||
| Per-user endpoint | **Live** - `userLlm.*` + Settings UI; encrypted key; used for `dealerMap.explain` |
|
||
| Sensitive thesis data never leaves host | Thesis content is not flowing through LLM product paths |
|
||
|
||
ADR-0006/0008 still govern intent; the full gateway is incomplete. Dealer Flow Layer-0 notes are in-app (`dealerFlowExplainNotes.ts`). Optional offline harvest/distill scripts write that file **and** a personal Obsidian copy - the app never reads Obsidian at runtime.
|
||
|
||
---
|
||
|
||
## 11. Deployment and CI/CD
|
||
|
||
This is the as-built operator path. Day-to-day commands live in [`DEPLOY_UNRAID.md`](./DEPLOY_UNRAID.md).
|
||
|
||
### 11.1 Images
|
||
|
||
| Piece | Path | Runtime |
|
||
|---|---|---|
|
||
| Backend image | `app/server/Dockerfile` | Node 22, `tini`, `node --experimental-strip-types src/index.ts`, data at `/app/data` |
|
||
| Frontend image | `app/Dockerfile` | Next standalone, `IFLOW_BACKEND_URL=http://backend:3001` |
|
||
| Build-from-git compose | `docker-compose.yml` | Used on a laptop or an Unraid checkout |
|
||
| Pull-prebuilt compose | `deploy/unraid/compose.pull.yml` | After CI has published tags |
|
||
| Gitea runner | `deploy/unraid/act-runner-compose.yml` | `gitea/act_runner:0.2.13`, Docker socket, label `ubuntu-latest` |
|
||
| Cron fallback | `deploy/unraid/user-scripts/sync-and-up.sh` | `git fetch` + ff-only merge + `compose up --build` when HEAD moved |
|
||
|
||
Health:
|
||
|
||
- Backend `GET /health` → `{ ok, queue: queue.health() }`
|
||
- Frontend `GET /health` → `{ ok, service: "investor-flow-web" }`
|
||
|
||
Compose `depends_on: backend.service_healthy` so the web container does not start until tRPC is up.
|
||
|
||
### 11.2 Gitea Actions
|
||
|
||
**`ci.yml` (automatic)**
|
||
|
||
| Trigger | Jobs |
|
||
|---------|------|
|
||
| push / PR to `main`, `workflow_dispatch` | `test` |
|
||
| push to `main` after `test` succeeds | `images` |
|
||
|
||
`test`: Node 22, `npm ci` in `app/` and `app/server/`, frontend unit tests, backend `npm test`, backend `npm run typecheck`.
|
||
|
||
`images`: Docker Buildx against the insecure HTTP registry, login with `REGISTRY_TOKEN` (or `github.token`), push:
|
||
|
||
- `${REGISTRY}/transnet/investor-flow-backend:${{ github.sha }}` and `:latest`
|
||
- `${REGISTRY}/transnet/investor-flow-frontend:${{ github.sha }}` and `:latest`
|
||
- registry cache tags `:buildcache`
|
||
|
||
Default `REGISTRY` is `10.37.0.86:3003` (override with a Gitea Actions variable).
|
||
|
||
**`cd.yml` (manual)**
|
||
|
||
Same image build/push without the test gate. Use when you need to republish images from the current `main` SHA.
|
||
|
||
### 11.3 How a change reaches Unraid
|
||
|
||
1. Commit and `git push origin main` to Gitea.
|
||
2. act_runner picks up the workflow (label `ubuntu-latest`).
|
||
3. Tests run. On failure, images are **not** published.
|
||
4. On success, new `:latest` and `:${sha}` tags land in the Gitea registry.
|
||
5. Unraid must **pull and recreate** containers. Publishing is not the same as rolling the running stack.
|
||
|
||
Two supported run modes:
|
||
|
||
**A. Pull prebuilt (preferred after CI is green)**
|
||
|
||
```bash
|
||
# .env image names point at the registry (see deploy/unraid/.env.example)
|
||
docker compose -f deploy/unraid/compose.pull.yml pull
|
||
docker compose -f deploy/unraid/compose.pull.yml up -d
|
||
```
|
||
|
||
**B. Build on Unraid from the git checkout**
|
||
|
||
```bash
|
||
cd /mnt/cache/appdata/investor-flow/repo
|
||
git pull --ff-only
|
||
docker compose --env-file /mnt/cache/appdata/investor-flow/.env up -d --build
|
||
```
|
||
|
||
Mode B is what `sync-and-up.sh` automates. Mode A is what the registry tags are for. Do not mix them on the same project name without knowing which image tags `.env` pins.
|
||
|
||
### 11.4 Secrets and data
|
||
|
||
Required in production `.env`:
|
||
|
||
- `IFLOW_SESSION_SECRET` - `openssl rand -hex 32`
|
||
- `SEC_OPERATOR_EMAIL` - real address (SEC + Yahoo user-agent)
|
||
- `IFLOW_DATA_DIR` - cache or single-disk path, **not** `/mnt/user` (FUSE can corrupt SQLite WAL)
|
||
|
||
Recommended:
|
||
|
||
- `IFLOW_CRYPTO_KEY` - required to decrypt existing X/FRED rows if you copy a Mac DB
|
||
- `LLM_PROVIDER_URL` / `ORNITH_LLM_URL` - Ornith LAN URL (not `localhost` inside the container)
|
||
- `YF_OPERATOR_EMAIL`, `FRED_API_KEY`, OAuth client ids (optional)
|
||
|
||
SQLite lives at `/app/data/investor-flow.db` inside the backend container. Persist that directory. Stop the backend (or use the SQLite backup API) before copying `investor-flow.db*`.
|
||
|
||
### 11.5 Local dev (unchanged)
|
||
|
||
```bash
|
||
cd app/server && npm run dev # :3001
|
||
cd app && npm run dev # :3000
|
||
```
|
||
|
||
Or `docker compose --env-file .env up -d --build` from the repo root.
|
||
|
||
---
|
||
|
||
## 12. Documentation map
|
||
|
||
| Doc | Role |
|
||
|-----|------|
|
||
| `docs/FUNCTIONAL_DESIGN.md` | What users can do / pending |
|
||
| `docs/TECH_DESIGN.md` | This file - how it is built |
|
||
| `docs/DEPLOY_UNRAID.md` | Operator process: Unraid + Gitea CI/CD |
|
||
| `docs/VENDOR_INTEGRATIONS.md` | How to add a vendor without bypassing the gate |
|
||
| `CONTEXT.md` | Ubiquitous language (glossary only) |
|
||
| `docs/adr/*` | Durable decisions (0001–0012) |
|
||
| Obsidian `investor-flow.md` | Session orientation wiki (status here wins) |
|
||
|
||
Deprecated as status sources: root `HANDOFF.md` (2026-06-30 orchestrator snapshot), slice DECOMPOSITION checkbox state.
|