feat: Unraid deploy, dealer-flow heatmap, confluence zones, 13F capture
Ship Node production images, Unraid compose, and Gitea CI/CD (test then push registry images; cron script if no runner). Rebuild dealer flow as a heatmap-first map with integrity gates and chart helpers. Add confluence zone rules, session clock, capture evidence, and tighter 13F/queue/options paths, plus the matching UI and tests.
This commit is contained in:
Investor Flow Build
@@ -0,0 +1,118 @@
|
||||
# Deploy Investor Flow on Unraid
|
||||
|
||||
Two containers: **backend** (`:3001`, SQLite + queue) and **frontend** (`:3000`, Next.js, proxies `/api` to the backend). Git and Actions already live on this Unraid box at `http://unraid.local:3003/transnet/investor-flow`.
|
||||
|
||||
The current Gitea repo has Actions enabled but **no runner registered**, so every CI run is cancelled. Follow "One-time: Gitea runner" below if you want push-to-main to test and publish images.
|
||||
|
||||
## What you get
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | Build-from-git stack (works on Unraid or a laptop) |
|
||||
| `deploy/unraid/compose.pull.yml` | Pull prebuilt images from the Gitea registry |
|
||||
| `deploy/unraid/.env.example` | Secrets + ports + data dir |
|
||||
| `deploy/unraid/act-runner-compose.yml` | Gitea act_runner so CI/CD actually runs |
|
||||
| `deploy/unraid/user-scripts/sync-and-up.sh` | Cron CD: `git pull` + `compose up --build` |
|
||||
| `.github/workflows/ci.yml` | Tests on every push / PR |
|
||||
| `.github/workflows/cd.yml` | Build + push images on `main` |
|
||||
|
||||
## Unraid layout (recommended)
|
||||
|
||||
Use a **cache (or single-disk) path** for SQLite. `/mnt/user/...` is FUSE and can corrupt WAL files.
|
||||
|
||||
```
|
||||
/mnt/cache/appdata/investor-flow/
|
||||
repo/ git checkout (this repo)
|
||||
data/ investor-flow.db + harvest files
|
||||
.env secrets (not in git)
|
||||
```
|
||||
|
||||
## First-time stack
|
||||
|
||||
SSH to Unraid (or Unraid terminal):
|
||||
|
||||
```bash
|
||||
mkdir -p /mnt/cache/appdata/investor-flow/data
|
||||
git clone http://unraid.local:3003/transnet/investor-flow.git \
|
||||
/mnt/cache/appdata/investor-flow/repo
|
||||
|
||||
cp /mnt/cache/appdata/investor-flow/repo/deploy/unraid/.env.example \
|
||||
/mnt/cache/appdata/investor-flow/.env
|
||||
nano /mnt/cache/appdata/investor-flow/.env
|
||||
```
|
||||
|
||||
Required in `.env`:
|
||||
|
||||
- `IFLOW_SESSION_SECRET` — `openssl rand -hex 32`
|
||||
- `SEC_OPERATOR_EMAIL` — real address (SEC + Yahoo user-agent)
|
||||
- `LLM_PROVIDER_URL` — Ornith LAN URL, e.g. `http://10.37.0.165:30081/v1` (not `localhost`)
|
||||
- `IFLOW_DATA_DIR=/mnt/cache/appdata/investor-flow/data`
|
||||
|
||||
If you copy today's Mac DB into `data/`, also set `IFLOW_CRYPTO_KEY` to the same value the Mac used. If the Mac never set one, encrypted X/FRED rows used the built-in dev key; re-enter those keys in Admin after a fresh Unraid DB instead of guessing.
|
||||
|
||||
Optional: copy the existing database (stop the Mac backend first so WAL is clean):
|
||||
|
||||
```bash
|
||||
# from the Mac, after stopping :3001
|
||||
scp app/server/data/investor-flow.db* \
|
||||
root@unraid.local:/mnt/cache/appdata/investor-flow/data/
|
||||
```
|
||||
|
||||
Build and start:
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
docker compose --env-file /mnt/cache/appdata/investor-flow/.env up -d --build
|
||||
```
|
||||
|
||||
Open `http://unraid.local:3000`. Backend health: `http://unraid.local:3001/health`.
|
||||
|
||||
Compose Manager plugin: create a stack whose project directory is the repo checkout and whose env file is `/mnt/cache/appdata/investor-flow/.env`.
|
||||
|
||||
Reverse proxy (SWAG / NPM): point the host at **frontend :3000 only**. The browser talks same-origin `/api`; the Next container reaches the backend on the compose network.
|
||||
|
||||
## One-time: Gitea runner (unlocks CI + image CD)
|
||||
|
||||
1. Gitea → Site Administration → Actions → Runners → **Create new runner**. Copy the token.
|
||||
2. Unraid Docker: add insecure registry `10.37.0.86:3003` (Gitea HTTP) so pulls/pushes work. Restart Docker.
|
||||
3. Start the runner:
|
||||
|
||||
```bash
|
||||
cd /mnt/cache/appdata/investor-flow/repo
|
||||
export GITEA_RUNNER_REGISTRATION_TOKEN=... # from step 1
|
||||
export GITEA_INSTANCE_URL=http://10.37.0.86:3003
|
||||
docker compose -f deploy/unraid/act-runner-compose.yml up -d
|
||||
```
|
||||
|
||||
4. Confirm the runner is **Idle** on the Gitea runners page.
|
||||
5. Repo → Settings → Actions → Secrets: `REGISTRY_TOKEN` = a Gitea token with `write:package` and `write:repository`.
|
||||
6. Push to `main`. CI should run tests; CD should push:
|
||||
|
||||
- `10.37.0.86:3003/transnet/investor-flow-backend:latest`
|
||||
- `10.37.0.86:3003/transnet/investor-flow-frontend:latest`
|
||||
|
||||
Then switch `.env` image names to those registry tags and use `deploy/unraid/compose.pull.yml` so Unraid no longer builds on the array.
|
||||
|
||||
## CD that works before a runner exists
|
||||
|
||||
Unraid → User Scripts → new script, paste `deploy/unraid/user-scripts/sync-and-up.sh`, schedule every 10 minutes (or after you push). It fast-forwards `main` and rebuilds only when HEAD moved.
|
||||
|
||||
A push to Gitea is then: write code → `git push origin main` → cron on Unraid rebuilds containers.
|
||||
|
||||
## Day-2 operations
|
||||
|
||||
| Task | How |
|
||||
|---|---|
|
||||
| Logs | `docker compose logs -f --tail=200 backend frontend` |
|
||||
| Restart | `docker compose up -d` |
|
||||
| Update (pull mode) | `docker compose -f deploy/unraid/compose.pull.yml pull && docker compose -f deploy/unraid/compose.pull.yml up -d` |
|
||||
| Backup DB | copy `/mnt/cache/appdata/investor-flow/data/investor-flow.db*` (stop backend first, or use SQLite backup) |
|
||||
| Ornith | keep `LLM_PROVIDER_URL` on the LAN IP; user LLM endpoint in Settings can stay `http://10.37.0.165:30081/v1` |
|
||||
|
||||
The backend process must stay up for dealer-map snapshots and the queue. `restart: unless-stopped` plus Unraid array autostart covers that.
|
||||
|
||||
## What this is not
|
||||
|
||||
- Not Vercel. Next standalone + the Node backend both run on Unraid.
|
||||
- `/reports` is still the thin HTML stub. A daily GEX briefing job is separate.
|
||||
- Do not publish `IFLOW_SESSION_SECRET`, `IFLOW_CRYPTO_KEY`, or Gitea tokens.
|
||||
@@ -43,8 +43,8 @@
|
||||
| Institutional dashboard | `/institutional` | **Live** | dashboard rollup, flow, insider stream, filings; Q-o-Q / M-o-M |
|
||||
| Filings | `/filings` | **Live** | EDGAR index + 13F + Form 4 detail |
|
||||
| Options DD | `/options-dd` | **Live** | chain + greeks (read-only teaching surface) |
|
||||
| Dealer Flow | `/dealer-flow` | **Live (MVP)** | GEX/VEX/OI metric modes + strike profile; delayed Yahoo OK; **integrity gates** (incomplete vs degraded vs complete); IV hygiene; keep last good map; as-of ET; `dealer-map-replay` CLI + Admin → Queue **Dealer map integrity** buttons; Layer-0 + optional L1 |
|
||||
| Study Desk | (on `/dealer-flow`) | **Live** | Educational setups from map; hist rank; log; auto-grade; scorecard; promote to journal draft; CSV export |
|
||||
| Dealer Flow | `/dealer-flow` | **Live (MVP)** | Heatmap-first IA: GEX/VEX is the only map toggle (convention lives in Method); reading / method / study in drawers; point-in-time book clock; **integrity gates** (incomplete vs degraded vs complete); IV hygiene; keep last good map; as-of ET; `dealer-map-replay` CLI + Admin → Queue **Dealer map integrity** buttons; Layer-0 + optional L1 |
|
||||
| Study Desk | (drawer on `/dealer-flow`) | **Live** | Educational setups from map; hist rank; log; auto-grade; scorecard; promote to journal draft; CSV export |
|
||||
| Mentor ledger | (Study Desk tab) | **Live (MVP)** | Local import from harvest → confirm drafts → path-match grade + mentor scorecard (not buy signals) |
|
||||
| Market outlook | `/market-outlook` | **Live (Phase 1–3 core)** | Beginner language. Condition strip; stronger/weaker-vs-market map; seasonality + calendar; rank snapshots; auto leadership check; truck/factory; macro notes. True ETF flow/COT still later. |
|
||||
| Social / X feed | on overview | **Live** | DB-backed 30d history + admin X credentials/accounts |
|
||||
@@ -123,7 +123,7 @@
|
||||
|------------|--------|
|
||||
| Users, sessions, reset password | **Live** |
|
||||
| Pending approval queue | **Live** |
|
||||
| Adapter queue health, pause/resume, retry, schedules, error stacks | **Live** |
|
||||
| Adapter queue health, pause/resume, retry, schedules, error stacks | **Live** | Cooldown column ticks 429 pauses; blank is not-paused. Unhealthy Yahoo shows backlog/notes. T0/focused quotes drain first. |
|
||||
| SEC queue fetch + lint holders/insiders + data quality | **Live** |
|
||||
| X credentials, accounts, prune | **Live** |
|
||||
| FRED key | **Live** |
|
||||
|
||||
+22
-7
@@ -71,7 +71,7 @@ All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is
|
||||
| Incremental candles | Warm symbols re-fetch ~14d lookback, not full 10y every tick |
|
||||
| Poison quarantine | Delisted / not-found symbols fail permanent; not requeued; demand cleared |
|
||||
| Per-kind drain budgets | Quotes cannot starve symbol meta / candles forever |
|
||||
| Observability | `queue.health()`: cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||||
| Observability | `queue.health()`: vendorGate+queue_state cooldowns, `pendingByKind`, demand size, SPY candle lag, `dataPlaneHealthy` |
|
||||
|
||||
**Anti-patterns (do not reintroduce):** N parallel Yahoo charts on a click; live `quoteSummary` without cache on every panel open; treating 429 as a 2s job retry that keeps hammering the same edge; calling `subscribe` on every page view (inflates refcount - use `ensureInDemand`); `dealerMap.get` calling Yahoo directly; frontend looping expiries to paint Dealer Flow.
|
||||
|
||||
@@ -83,7 +83,7 @@ All vendors (Yahoo, X, FRED, SEC, Reddit) have short rate limits. The system is
|
||||
- Request path: SQLite recompute + schedule-on-miss; max 4–6 nearest expiries; 15m map TTL; daily `dealer_map_snapshots` for velocity.
|
||||
- Integrity: pure `dealerMapIntegrity` hard/soft checks (missing expiries/OI/greeks fail; delay does not); `data_quality` kind `dealer_map`; replay via `dealerMapReplay` + `scripts/dealer-map-replay.ts` (as-of chain ts).
|
||||
- SEC institutional (alert-critical): SC-first fetch seeds `sec:cusip:SYMBOL`; offline curated CUSIP registry (`cusipRegistry`) so resolve does not depend solely on EFTS; 13F prefers EFTS CUSIP pagination, and on EFTS 403/outage falls back to **reverse 13F** (`reverse13fRefresh`: prior holders + tracked funds + major managers via `data.sec.gov`); `SecFetchAdapter` **throws** on hard resolve failure so queue retries (no silent done); `requeueUnhealthySecSymbols` caps heal requeues per tick.
|
||||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. UI toggle: Classic | Dealer (HS).
|
||||
- **Dealer GEX sign convention:** maps are stored as classic OI GEX (`classic_call_pos_put_neg`: call +, put −). Request path can re-express as `dealer_inventory` (full GEX/VEX sign flip - Heatseeker-style dealer short when customers long) via `withExposureConvention` / `dealerMap.get({ convention })`. First-paint UI is GEX | VEX only; Method drawer has Customer book | Dealer inventory.
|
||||
- **Vendor rate-limit enforcement (hard requirement, all sources + future):** process-wide `vendorGate` with **open registration** (`registerVendorIntegration`). Built-in families: yfinance, sec, fred, finra, nasdaq, reddit, x, llm. New vendors must register family + bind `source_kind` before `AdapterQueue` construction (throws otherwise). Prefer `VendorSourceAdapter` / `defineVendorAdapter` so `fetchOne` is auto-gated. HTTP via `vendorFetch` / `secHttp`; SDKs via `withVendorGate`. CI guard bans bare `fetch(` in adapters/services. See `docs/VENDOR_INTEGRATIONS.md`.
|
||||
- LLM: per-user OpenAI-compatible `base_url` + encrypted key + model (`userLlm.*`); not OpenAI-only.
|
||||
- Dealer Flow plain-English notes: in-app `dealerFlowExplainNotes.ts` only (L0/L1). Optional offline scripts harvest X handles and distill into that file **and** write a personal Obsidian vault copy - the app never reads Obsidian at runtime.
|
||||
@@ -268,16 +268,31 @@ ADR-0006/0008 still govern intent; implementation is incomplete.
|
||||
|
||||
## 11. Deployment
|
||||
|
||||
Local dev (typical):
|
||||
|
||||
```bash
|
||||
# Local dev (typical)
|
||||
cd app/server && npm run dev # :3001
|
||||
cd app && npm run dev # :3000
|
||||
|
||||
# Compose
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
Secrets: `.env.example` documents session secret, SEC operator email, X cookie fields, Reddit, FRED, LLM URL.
|
||||
Production is two Node containers (not Bun). Unraid is the intended host; git + Gitea Actions already live at `unraid.local:3003/transnet/investor-flow`.
|
||||
|
||||
```bash
|
||||
# Build-from-git (laptop or Unraid checkout)
|
||||
docker compose --env-file .env up -d --build
|
||||
```
|
||||
|
||||
| Piece | Path |
|
||||
|---|---|
|
||||
| Backend image | `app/server/Dockerfile` (Node 22, `node --experimental-strip-types`) |
|
||||
| Frontend image | `app/Dockerfile` (Next standalone, proxies `/api` via `IFLOW_BACKEND_URL`) |
|
||||
| Unraid operator guide | `docs/DEPLOY_UNRAID.md` |
|
||||
| Gitea CI | `.github/workflows/ci.yml` |
|
||||
| Gitea CD (registry push) | `.github/workflows/cd.yml` |
|
||||
| act_runner | `deploy/unraid/act-runner-compose.yml` |
|
||||
| Cron CD (no runner) | `deploy/unraid/user-scripts/sync-and-up.sh` |
|
||||
|
||||
Secrets: `.env.example` and `deploy/unraid/.env.example`. Production refuses to start without `IFLOW_SESSION_SECRET`. Persist SQLite on a cache/single-disk path (`IFLOW_DATA_DIR`), not `/mnt/user`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -92,6 +92,9 @@ vendor family with 1.5s min-interval pacing.
|
||||
into the demand set on startup.
|
||||
- No advisory output is produced. All picture-quality labels describe
|
||||
evidence; they never recommend action.
|
||||
- The closed loop is not self-executing: the `confluence.eval` procedure
|
||||
must be triggered to produce evaluations. The backtest procedure is
|
||||
read-only and query-driven. Future work can schedule periodic evaluation.
|
||||
- The closed loop now runs on a schedule: hourly evaluation of *today*,
|
||||
incremental as-of replay of cached candles (~3y lookback, budgeted), and
|
||||
weekly walk-forward derivation of entry/exit zone rules learned on the
|
||||
11 GICS sector ETFs + SPY. Reliability weights (once a slot has ≥2
|
||||
resolved fires) scale evidence inside `evaluateRack`. The Confluence page
|
||||
shows the last 3 entry + 3 exit windows that match the active rule.
|
||||
Reference in New Issue
Block a user