The stale-while-revalidate pattern queues background fetches via the AdapterQueue,
but with 227+ pending jobs, short interest keys take too long to process.
Users saw empty panels on every symbol because the fetch never completed before
the response returned.
Fix:
- shortInterest procedure: when Yahoo + NASDAQ caches are both empty, fetch
directly via the adapters, store in cache, and return — all within the request.
Subsequent requests hit the cache. Bypasses the clogged queue entirely.
- FinraBulkAdapter: guard against per-symbol keys (non-'latest' and non-date IDs)
by returning a skipped result instead of trying to download a file named
after the ticker.
- finraShortInterest handler: return isStale=false when no data is in the table,
so per-symbol reads never trigger queue fetches (FINRA is schedule-only).
Router stale computation now excludes FINRA from the combined stale flag.
- Add short_interest TtlClass (24h) to CacheRepository.ts with shortInterestHandler KindHandler (kv_cache JSON pattern), register in HANDLERS and del switch case
- Add case 'shortinterest' to YFinanceAdapter.fetchOne using quoteSummary(defaultKeyStatistics): parses sharesShort, shortPercentOfFloat, shortRatio, dateShortInterest, sharesShortPriorMonth, floatShares, sharesOutstanding
- Add institutional.shortInterest tRPC procedure (cache-read pattern, returns stale flag + provenance)
- Add api.institutional.shortInterest to frontend trpc client
- Create ShortInterestPanel.tsx: mirrors AnalystRatings pattern with loading/error/retry states, formatted shares/ratio/percentage display with source date
- Mount ShortInterestPanel on overview page after AnalystRatings
- Add await ctx.cache.subscribe() to addSymbol mutation so symbols
added via the sidebar get registered in symbol_demand and yfinance
jobs are queued immediately
- Backfill PEP, WYNN, STZ, CELH into symbol_demand + adapter_queue
- Upgrade yahoo-finance2 3.15.3 -> 3.15.4 and pass validateResult:false
to quoteSummary() to handle Yahoo schema drift
- Add error detail logging for analyst ratings schema failures
- Update .gitignore with common ignores
- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop
- Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules)
- No-op cache handlers so drain->cache.set doesn't throw on lint keys
- tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult
- tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once
- tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind)
- InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header
- Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons
- DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter
- Pause/resume persisted in queue_state; drain loop checks isPaused() and skips
- Error capture: every failed/backoff attempt logged to queue_errors with the
full stack trace; admin queue page expands a failed job to stream it
- Retry controls: retryJob(key), retrySource(kind), clearDone(olderThanMs)
- Per-source scheduling: queue_schedules table + 30s enqueueDueSchedules loop
(seed defaults sec-fetch 24h, yfinance 5min); admin UI lists/adds/deletes
- Startup recovery: interrupted in_flight jobs reset to pending on boot
- fix(edgar): archive URLs use the filer CIK (accession-number prefix), not the
company CIK — resolves Cloudflare 429 that left SEC backfills sparse/empty
- Migration: add queue_errors, queue_schedules, queue_state tables plus error/
scheduled_for columns on adapter_queue (idempotent ALTER on startup)
Verified: full sec-fetch backfill now succeeds for all watched symbols
(NVDA 14,675 institution filings, CIFR 274 insider txns, TSLA 6,011, etc.).
503 backend tests pass.
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix OptionsAdapter for yahoo-finance2 v3 API ({ date: expiry } instead of positional arg)
- Fix institutional.flow and insiderStream endpoints to query correct DB tables
- Fix anonymous user INSERT (pw_hash column, not password_hash)
- Create secDataFetcher module for 13F/Form 4 data population
- Add institutional and marketOutlook endpoints to frontend trpc client
- Build out Chart Lab, Institutional, Market Outlook pages with real components
- Add focus-visible rings, aria-labels, hex tokens, shape icons across all panels
- Integrate EmotionLogger, MobileTabNav, useKeyboardShortcuts
- Add localStorage persistence for emotion logs
- Normalize spacing, type scale, hover effects across all components
Chart Lab data path: market.candles (cached OHLCV by timeframe) + market.indicators
(EMA/RSI/relativeVolume via analysis/indicators.ts). By ornith-35 (local, ~75s);
reviewed by orchestrator (128/128 tests, no regression). Conflict-free: impl=ornith-35,
reviewer=orchestrator.
totp.ts + backup-codes.ts produced by ornith (remote model, reviewed+approved);
router/context/schema integration by orchestrator (delicate edit-existing — ornith
timed out on a dispatch of that size in-session). 58/58 backend tests green.
2FA flow: enable2fa -> confirm2fa -> login now requires a valid TOTP.
auth.signup/login/logout/me with signed HMAC session cookies + scrypt hashing;
market.snapshot mega-endpoint (quote+candles+sector). node:http server mounts
tRPC at /api/trpc + /health + background drain loop. Live-verified: real NVDA
194.97/65 candles/Technology served after stale-while-revalidate drain. 36 tests green.
Investor Flow Build