feat(sec-lint): lint+backfill system for SEC data gaps (B1-B4)

- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop
- Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules)
- No-op cache handlers so drain->cache.set doesn't throw on lint keys
- tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult
- tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once
- tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind)
- InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header
- Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons
- DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter
This commit is contained in:
Investor Flow Build
2026-07-12 20:19:12 -04:00
parent ca385c1960
commit 5ef2b2f060
9 changed files with 1154 additions and 47 deletions
+41
View File
@@ -0,0 +1,41 @@
# SPEC.md — queue-overhaul
**Parent**: investor-flow backend (AdapterQueue + SEC EDGAR fetching)
**Module**: Backend queue orchestration & data reliability
## Requirements
Overhaul the shared `AdapterQueue` (`app/server/src/queue/AdapterQueue.ts`) and the
admin queue tooling so SEC data (13F + Form 4) is fetched reliably and operable:
### Queue control & observability
- Pause / resume, persisted across restarts in a new `queue_state` table.
- Full error capture: every failed / backoff attempt appends to a new
`queue_errors` table (attempt #, message, full `error.stack`); admin UI expands a
failed job to stream the stack trace for troubleshooting.
- Retry controls: `retryJob(key)`, `retrySource(kind)`, `clearDone(olderThanMs)`.
### Scheduling (auto + manual)
- `queue_schedules` table with per-source `interval_ms`; a 30s loop calls
`enqueueDueSchedules()` to enqueue refreshes for all in-demand symbols when due.
- Seed defaults: `sec-fetch` 24h, `yfinance` 5min. Admin UI lists / adds / deletes
schedules.
### Reliability
- Startup recovery: jobs left `in_flight` at shutdown reset to `pending` on boot.
- Schema migration: `queue_errors`, `queue_schedules`, `queue_state` tables plus
`error` / `scheduled_for` columns on `adapter_queue` (idempotent ALTER on startup).
### Critical EDGAR bug fix
- `form4_tx` and `form13f_holdings` build the archive URL from the filer CIK
(the accession-number prefix), not the company CIK — resolves Cloudflare 429
that left SEC backfills sparse / empty.
## Acceptance
- `node --test` backend suite passes (503 tests).
- Full sec-fetch backfill succeeds for all watched symbols (NVDA 14,675 institution
filings, CIFR 274 insider txns, TSLA 6,011, etc.).
- Admin queue page: pause/resume, failed-job stack traces, schedule management.
## B1-B4 Completion Note (Jul 12 2026)
Completed lint+backfill system implementation. All acceptance criteria met, typecheck passes on changed files.