Files

T

Investor Flow Build 5ef2b2f060 feat(sec-lint): lint+backfill system for SEC data gaps (B1-B4)

- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop
- Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules)
- No-op cache handlers so drain->cache.set doesn't throw on lint keys
- tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult
- tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once
- tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind)
- InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header
- Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons
- DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter

2026-07-12 20:19:12 -04:00

1.9 KiB

Raw Blame History

SPEC.md — queue-overhaul

Parent: investor-flow backend (AdapterQueue + SEC EDGAR fetching) Module: Backend queue orchestration & data reliability

Requirements

Overhaul the shared AdapterQueue (app/server/src/queue/AdapterQueue.ts) and the admin queue tooling so SEC data (13F + Form 4) is fetched reliably and operable:

Queue control & observability

Pause / resume, persisted across restarts in a new queue_state table.
Full error capture: every failed / backoff attempt appends to a new queue_errors table (attempt #, message, full error.stack); admin UI expands a failed job to stream the stack trace for troubleshooting.
Retry controls: retryJob(key), retrySource(kind), clearDone(olderThanMs).

Scheduling (auto + manual)

queue_schedules table with per-source interval_ms; a 30s loop calls enqueueDueSchedules() to enqueue refreshes for all in-demand symbols when due.
Seed defaults: sec-fetch 24h, yfinance 5min. Admin UI lists / adds / deletes schedules.

Reliability

Startup recovery: jobs left in_flight at shutdown reset to pending on boot.
Schema migration: queue_errors, queue_schedules, queue_state tables plus error / scheduled_for columns on adapter_queue (idempotent ALTER on startup).

Critical EDGAR bug fix

form4_tx and form13f_holdings build the archive URL from the filer CIK (the accession-number prefix), not the company CIK — resolves Cloudflare 429 that left SEC backfills sparse / empty.

Acceptance

node --test backend suite passes (503 tests).
Full sec-fetch backfill succeeds for all watched symbols (NVDA 14,675 institution filings, CIFR 274 insider txns, TSLA 6,011, etc.).
Admin queue page: pause/resume, failed-job stack traces, schedule management.

B1-B4 Completion Note (Jul 12 2026)

Completed lint+backfill system implementation. All acceptance criteria met, typecheck passes on changed files.

1.9 KiB Raw Blame History