feat(sec-lint): lint+backfill system for SEC data gaps (B1-B4)
- SecLintAdapter implements SourceFetch, runs via shared queue/drain loop - Two new SourceKinds: sec-lint-holders, sec-lint-insiders (weekly schedules) - No-op cache handlers so drain->cache.set doesn't throw on lint keys - tRPC admin.queueLint(symbol, kind) — run lint for one symbol, returns LintResult - tRPC admin.queueLintAll(kind) — backfill ALL watched symbols at once - tRPC admin.dataQualityList() — query data_quality rows (filterable by symbol/kind) - InstitutionalDashboard: 'Lint holders' button + status badge in detail panel header - Admin queue page: Data Quality section with per-row status badges, 'Lint all' buttons - DEFAULT_RATE_MS includes 167ms (~6 req/s) for lint kinds matching EDGAR limiter
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# SPEC.md — queue-overhaul
|
||||
|
||||
**Parent**: investor-flow backend (AdapterQueue + SEC EDGAR fetching)
|
||||
**Module**: Backend queue orchestration & data reliability
|
||||
|
||||
## Requirements
|
||||
|
||||
Overhaul the shared `AdapterQueue` (`app/server/src/queue/AdapterQueue.ts`) and the
|
||||
admin queue tooling so SEC data (13F + Form 4) is fetched reliably and operable:
|
||||
|
||||
### Queue control & observability
|
||||
- Pause / resume, persisted across restarts in a new `queue_state` table.
|
||||
- Full error capture: every failed / backoff attempt appends to a new
|
||||
`queue_errors` table (attempt #, message, full `error.stack`); admin UI expands a
|
||||
failed job to stream the stack trace for troubleshooting.
|
||||
- Retry controls: `retryJob(key)`, `retrySource(kind)`, `clearDone(olderThanMs)`.
|
||||
|
||||
### Scheduling (auto + manual)
|
||||
- `queue_schedules` table with per-source `interval_ms`; a 30s loop calls
|
||||
`enqueueDueSchedules()` to enqueue refreshes for all in-demand symbols when due.
|
||||
- Seed defaults: `sec-fetch` 24h, `yfinance` 5min. Admin UI lists / adds / deletes
|
||||
schedules.
|
||||
|
||||
### Reliability
|
||||
- Startup recovery: jobs left `in_flight` at shutdown reset to `pending` on boot.
|
||||
- Schema migration: `queue_errors`, `queue_schedules`, `queue_state` tables plus
|
||||
`error` / `scheduled_for` columns on `adapter_queue` (idempotent ALTER on startup).
|
||||
|
||||
### Critical EDGAR bug fix
|
||||
- `form4_tx` and `form13f_holdings` build the archive URL from the filer CIK
|
||||
(the accession-number prefix), not the company CIK — resolves Cloudflare 429
|
||||
that left SEC backfills sparse / empty.
|
||||
|
||||
## Acceptance
|
||||
- `node --test` backend suite passes (503 tests).
|
||||
- Full sec-fetch backfill succeeds for all watched symbols (NVDA 14,675 institution
|
||||
filings, CIFR 274 insider txns, TSLA 6,011, etc.).
|
||||
- Admin queue page: pause/resume, failed-job stack traces, schedule management.
|
||||
|
||||
## B1-B4 Completion Note (Jul 12 2026)
|
||||
Completed lint+backfill system implementation. All acceptance criteria met, typecheck passes on changed files.
|
||||
Reference in New Issue
Block a user