CANDLE_PROVIDER resolves a symbol's candles per-granularity (1d/1wk) from the
shared cache, with a reversible fold-in of the freshest live quote so a
mid-session evaluation sees the current price. Pure fold logic + todayIso are
unit-tested; CacheCandleProvider is a thin cache-backed shim. Evaluators use
this seam instead of cache.get inline, so a future realtime/replay source can
slot in without touching slot logic.
Parse the annual FinFutYY.txt zip (leveraged-funds long/short + OI) into a
CotSeries. Registers the 'cot' source_kind, 'cot_weekly' TTL (7d), the 'cftc'
vendor family (1 req/1.5s pacing), and the adapter in the queue registry.
Uses the file-header column names so parser is robust to layout changes.
New functions in indicators.ts:
- sma: Simple Moving Average
- crossover: golden/death cross detector
- roc: Rate of Change (momentum)
- timeSeriesMomentum: 12-1 month momentum (Moskowitz et al.)
- relativeStrength: symbol vs benchmark momentum differential
- realizedVol: annualized realized volatility from log returns
- volatilityPercentile: percentile rank of current vol vs history
- fibLevels: Fibonacci retracement and extension levels
- fibCluster: proximity check for Fib confluence zones
New file volumeByPrice.ts:
- buildVolumeProfile: ported from frontend volumeProfile.ts
- Identical algorithm: bin clamping, uniform volume split, POC/VAH/VAL
All functions are pure (zero deps, no I/O). Tests added for all new
functions with edge cases, error handling, and numerical correctness.
The stale-while-revalidate pattern queues background fetches via the AdapterQueue,
but with 227+ pending jobs, short interest keys take too long to process.
Users saw empty panels on every symbol because the fetch never completed before
the response returned.
Fix:
- shortInterest procedure: when Yahoo + NASDAQ caches are both empty, fetch
directly via the adapters, store in cache, and return — all within the request.
Subsequent requests hit the cache. Bypasses the clogged queue entirely.
- FinraBulkAdapter: guard against per-symbol keys (non-'latest' and non-date IDs)
by returning a skipped result instead of trying to download a file named
after the ticker.
- finraShortInterest handler: return isStale=false when no data is in the table,
so per-symbol reads never trigger queue fetches (FINRA is schedule-only).
Router stale computation now excludes FINRA from the combined stale flag.
- Add yfinance:shortinterest: and nasdaq:nasdaqShortinterest:
to subscribe() initial queue list so short interest data is fetched proactively
when a symbol is first viewed, rather than only on stale read
- FINRA remains bulk-schedule only via queue_schedules
- Fix default base URL: https://cdn.finra.org/equity/regsho/daily (was www.finra.org/sites/default/files)
- Fix filename pattern: CNMSshvol{YYYYMMDD}.txt (was CAshvol with year/month subdir)
- Fix parser: actual file columns are Date|Symbol|ShortVolume|ShortExemptVolume|TotalVolume|Market
(was incorrectly expecting Market|Symbol|... — first col is Date, last col is Market)
- Dedupe by symbol (volume is total across all listed exchanges, not per-exchange)
- Store combined market codes (e.g. 'B,Q,N') as exchange string
- Use BEGIN/COMMIT instead of db.transaction() (node:sqlite DatabaseSync doesn't have transaction())
- Update finra_config defaults in schema.sql + migration + admin fallback
- Add finra-bulk case to enqueueDueSchedules() — queues a single
finra-bulk:finraShortinterest:latest key per cycle
- Register default twice-monthly schedule (14-day interval) in seedDefaultSchedules()
- Removes need for any external cron; the existing 30s scheduler loop handles it
- Add short_interest TtlClass (24h) to CacheRepository.ts with shortInterestHandler KindHandler (kv_cache JSON pattern), register in HANDLERS and del switch case
- Add case 'shortinterest' to YFinanceAdapter.fetchOne using quoteSummary(defaultKeyStatistics): parses sharesShort, shortPercentOfFloat, shortRatio, dateShortInterest, sharesShortPriorMonth, floatShares, sharesOutstanding
- Add institutional.shortInterest tRPC procedure (cache-read pattern, returns stale flag + provenance)
- Add api.institutional.shortInterest to frontend trpc client
- Create ShortInterestPanel.tsx: mirrors AnalystRatings pattern with loading/error/retry states, formatted shares/ratio/percentage display with source date
- Mount ShortInterestPanel on overview page after AnalystRatings
- Add 6s server-side timeout to Yahoo Finance quoteSummary (Promise.race) so the backend responds with stale data instead of hanging indefinitely. Stale cache is served on error (router.ts:1895-1898), matching ADR-0009.
- Hoist YahooFinance to a module-level singleton (getYf()) so the crumb/cookie is fetched once, not per call. Matches YFinanceAdapter pattern.
- Add suppressNotices for yahooSurvey to reduce log noise.
- Add accession column to institution_filings (schema.sql + migration in client.ts) and thread it through both secDataFetcher.ts INSERT paths (13D/G and 13F-HR). Fixes recurring `no such column: accession` error in new13daProducer alert tick.
- new13daProducer.ts needs no changes - its SELECT accession query now works.
- trpcQuery/trpcMutate now throw 'Request timed out (8s)' instead of
generic 'The operation was aborted.' so users know WHY the request failed
- AnalystRatings adds cancelled guard (race condition fix) so stale
promise results from symbol changes are ignored (matches OverviewPanel
pattern)
- Add Retry button on error state for timeout or other failures
- useEffect cleanup returns cancelled flag setter
- Add await ctx.cache.subscribe() to addSymbol mutation so symbols
added via the sidebar get registered in symbol_demand and yfinance
jobs are queued immediately
- Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue
- Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false
to quoteSummary() to handle Yahoo schema drift
- Add error detail logging for analyst ratings schema failures
- Update .gitignore with common ignores
- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop
- Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules)
- No-op cache handlers so drain->cache.set doesn't throw on lint keys
- tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult
- tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once
- tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind)
- InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header
- Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons
- DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter
- Pause/resume persisted in queue_state; drain loop checks isPaused() and skips
- Error capture: every failed/backoff attempt logged to queue_errors with the
full stack trace; admin queue page expands a failed job to stream it
- Retry controls: retryJob(key), retrySource(kind), clearDone(olderThanMs)
- Per-source scheduling: queue_schedules table + 30s enqueueDueSchedules loop
(seed defaults sec-fetch 24h, yfinance 5min); admin UI lists/adds/deletes
- Startup recovery: interrupted in_flight jobs reset to pending on boot
- fix(edgar): archive URLs use the filer CIK (accession-number prefix), not the
company CIK — resolves Cloudflare 429 that left SEC backfills sparse/empty
- Migration: add queue_errors, queue_schedules, queue_state tables plus error/
scheduled_for columns on adapter_queue (idempotent ALTER on startup)
Verified: full sec-fetch backfill now succeeds for all watched symbols
(NVDA 14,675 institution filings, CIFR 274 insider txns, TSLA 6,011, etc.).
503 backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix OptionsAdapter for yahoo-finance2 v3 API ({ date: expiry } instead of positional arg)
- Fix institutional.flow and insiderStream endpoints to query correct DB tables
- Fix anonymous user INSERT (pw_hash column, not password_hash)
- Create secDataFetcher module for 13F/Form 4 data population
- Add institutional and marketOutlook endpoints to frontend trpc client
- Build out Chart Lab, Institutional, Market Outlook pages with real components
- Add focus-visible rings, aria-labels, hex tokens, shape icons across all panels
- Integrate EmotionLogger, MobileTabNav, useKeyboardShortcuts
- Add localStorage persistence for emotion logs
- Normalize spacing, type scale, hover effects across all components
Chart Lab data path: market.candles (cached OHLCV by timeframe) + market.indicators
(EMA/RSI/relativeVolume via analysis/indicators.ts). By ornith-35 (local, ~75s);
reviewed by orchestrator (128/128 tests, no regression). Conflict-free: impl=ornith-35,
reviewer=orchestrator.
Investor Flow Build