42 lines
1.9 KiB
Markdown
42 lines
1.9 KiB
Markdown
|
# SPEC.md — queue-overhaul
|
||
|
|
|
||
|
|
**Parent**: investor-flow backend (AdapterQueue + SEC EDGAR fetching)
|
||
|
|
**Module**: Backend queue orchestration & data reliability
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
Overhaul the shared `AdapterQueue` (`app/server/src/queue/AdapterQueue.ts`) and the
|
||
|
|
admin queue tooling so SEC data (13F + Form 4) is fetched reliably and operable:
|
||
|
|
|
||
|
|
### Queue control & observability
|
||
|
|
- Pause / resume, persisted across restarts in a new `queue_state` table.
|
||
|
|
- Full error capture: every failed / backoff attempt appends to a new
|
||
|
|
`queue_errors` table (attempt #, message, full `error.stack`); admin UI expands a
|
||
|
|
failed job to stream the stack trace for troubleshooting.
|
||
|
|
- Retry controls: `retryJob(key)`, `retrySource(kind)`, `clearDone(olderThanMs)`.
|
||
|
|
|
||
|
|
### Scheduling (auto + manual)
|
||
|
|
- `queue_schedules` table with per-source `interval_ms`; a 30s loop calls
|
||
|
|
`enqueueDueSchedules()` to enqueue refreshes for all in-demand symbols when due.
|
||
|
|
- Seed defaults: `sec-fetch` 24h, `yfinance` 5min. Admin UI lists / adds / deletes
|
||
|
|
schedules.
|
||
|
|
|
||
|
|
### Reliability
|
||
|
|
- Startup recovery: jobs left `in_flight` at shutdown reset to `pending` on boot.
|
||
|
|
- Schema migration: `queue_errors`, `queue_schedules`, `queue_state` tables plus
|
||
|
|
`error` / `scheduled_for` columns on `adapter_queue` (idempotent ALTER on startup).
|
||
|
|
|
||
|
|
### Critical EDGAR bug fix
|
||
|
|
- `form4_tx` and `form13f_holdings` build the archive URL from the filer CIK
|
||
|
|
(the accession-number prefix), not the company CIK — resolves Cloudflare 429
|
||
|
|
that left SEC backfills sparse / empty.
|
||
|
|
|
||
|
|
## Acceptance
|
||
|
|
- `node --test` backend suite passes (503 tests).
|
||
|
|
- Full sec-fetch backfill succeeds for all watched symbols (NVDA 14,675 institution
|
||
|
|
filings, CIFR 274 insider txns, TSLA 6,011, etc.).
|
||
|
|
- Admin queue page: pause/resume, failed-job stack traces, schedule management.
|
||
|
|
|
||
|
|
## B1-B4 Completion Note (Jul 12 2026)
|
||
|
|
Completed lint+backfill system implementation. All acceptance criteria met, typecheck passes on changed files.
|