Commit Graph
7 Commits
Author SHA1 Message Date
Investor Flow Build 3fa6b32916 feat: Phase 3 FINRA bulk adapter + Phase 4 three-way merge with discrepancy flagging
Phase 3 — FINRA bulk short-interest ingest:
- Add finra_short_interest table to schema (per-symbol, per-settlement-date, per-exchange, with short/exempt/total volume, avg daily vol, days to cover)
- Create FinraIngestService: downloads FINRA consolidated pipe-delimited file from configurable base URL, parses Market|Symbol|ShortVolume|ShortExemptVolume|TotalVolume, stores in finra_short_interest table
- Create FinraBulkAdapter: SourceFetch that calls downloadAndIngestFinra, registers as finra-bulk source kind with finraShortinterest cache handler
- finraShortinterest handler reads latest settlement row per symbol from finra_short_interest table (no per-symbol kv_cache write; data is bulk-ingested)
- Register in index.ts adapter map + HANDLERS + del case

Phase 4 — three-way merge with discrepancy detection:
- shortInterest tRPC procedure now reads all 3 caches (yfinance, nasdaq, finra-bulk) in parallel
- Reconciliation hierarchy: FINRA (shares short) > NASDAQ > Yahoo
- daysToCover: NASDAQ (specific) > FINRA (computed) > Yahoo (short ratio fallback)
- settlementDate: FINRA > NASDAQ > Yahoo
- Discrepancy detection: compares sharesShort across available sources, flags >10% difference with discrepancyPct + discrepancyBetween
- Updated ShortInterestPanel: FINRA source badge, discrepancy warning banner, three-source disclaimer
- Updated trpc.ts client type for new shape
2026-07-25 13:03:49 -04:00
Investor Flow Build 76f60dc766 feat: Phase 2 NASDAQ adapter for days-to-cover + 24mo history
- Add nasdaq SourceKind + nasdaqShortinterest KindHandler (kv_cache with nasdaq: prefix)
- Create NasdaqAdapter: calls api.nasdaq.com/quote/SYM/short-interest, parses settlementDate/sharesShort/daysToCover/avgVolume, returns 24mo history; handles non-NASDAq gracefully
- Register NasdaqAdapter in server index.ts adapters map
- Update shortInterest procedure: reads both yfinance + nasdaq caches in parallel, reconciles (NASDAQ for daysToCover/sharesShort, Yahoo for %float/shortRatio and NYSE fallback)
- Update ShortInterestPanel: display daysToCover, settlement date, source badges, 6-month history table, updated disclaimer
- Update trpc.ts client type for merged response shape
2026-07-25 12:59:43 -04:00
Investor Flow Build 5b9f770aa4 Phase 5: alert subscriptions UI + multiple watchlists + queue fixes
CI / Test & Type-Check (push) Canceled after 0s
UI:
- /alerts page: event history with acknowledge, subscription create/manage with toggle
- /admin/smtp: SMTP config form (host, port, auth, test)
- Watchlist sidebar: dropdown selector for multiple watchlists, create/delete
- Sidebar: alerts count badge, SMTP link under admin
- Mobile tab nav: alerts tab added
- Client trpc.ts: all new API methods + types

Backend:
- watchlists.listByWatchlist procedure + listSymbolsByWatchlist repo fn
- yfinance min-interval 1500->2000ms to reduce Edge 429s
- Fixed e.date.slice error in yfinance-adjustments with typeof guard
- Removed defunct BITF from watchlist+queue
- Cleared 83 failed + 12 backoff queue jobs

Docs:
- FUNCTIONAL_DESIGN.md: alerts + multiple watchlists + SMTP documented
- TECH_DESIGN.md: new modules, tRPC procs, routes updated
2026-07-23 20:51:47 -04:00
Investor Flow Build e262187c3c fix: backfill symbol_demand for sidebar-added symbols + analyst ratings schema fix
- Add await ctx.cache.subscribe() to addSymbol mutation so symbols
  added via the sidebar get registered in symbol_demand and yfinance
  jobs are queued immediately
- Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue
- Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false
  to quoteSummary() to handle Yahoo schema drift
- Add error detail logging for analyst ratings schema failures
- Update .gitignore with common ignores
2026-07-23 18:02:24 -04:00
Investor Flow Build 5ef2b2f060 feat(sec-lint): lint+backfill system for SEC data gaps (B1-B4)
- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop
- Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules)
- No-op cache handlers so drain->cache.set doesn't throw on lint keys
- tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult
- tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once
- tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind)
- InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header
- Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons
- DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter
2026-07-12 20:19:12 -04:00
Investor Flow BuildandClaude ca385c1960 feat(queue): adapter queue overhaul — pause/resume, error capture, scheduling
- Pause/resume persisted in queue_state; drain loop checks isPaused() and skips
- Error capture: every failed/backoff attempt logged to queue_errors with the
  full stack trace; admin queue page expands a failed job to stream it
- Retry controls: retryJob(key), retrySource(kind), clearDone(olderThanMs)
- Per-source scheduling: queue_schedules table + 30s enqueueDueSchedules loop
  (seed defaults sec-fetch 24h, yfinance 5min); admin UI lists/adds/deletes
- Startup recovery: interrupted in_flight jobs reset to pending on boot
- fix(edgar): archive URLs use the filer CIK (accession-number prefix), not the
  company CIK — resolves Cloudflare 429 that left SEC backfills sparse/empty
- Migration: add queue_errors, queue_schedules, queue_state tables plus error/
  scheduled_for columns on adapter_queue (idempotent ALTER on startup)

Verified: full sec-fetch backfill now succeeds for all watched symbols
(NVDA 14,675 institution filings, CIFR 274 insider txns, TSLA 6,011, etc.).
503 backend tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-12 12:55:59 -04:00
Investor Flow Build 82079b3e66 slice 1e-1f: tRPC router (auth + market.snapshot) + node:http server
auth.signup/login/logout/me with signed HMAC session cookies + scrypt hashing;
market.snapshot mega-endpoint (quote+candles+sector). node:http server mounts
tRPC at /api/trpc + /health + background drain loop. Live-verified: real NVDA
194.97/65 candles/Technology served after stale-while-revalidate drain. 36 tests green.
2026-06-29 17:40:42 -04:00