# SPEC.md — queue-overhaul

**Parent**: investor-flow backend (AdapterQueue + SEC EDGAR fetching)
**Module**: Backend queue orchestration & data reliability

## Requirements

Overhaul the shared `AdapterQueue` (`app/server/src/queue/AdapterQueue.ts`) and the
admin queue tooling so SEC data (13F + Form 4) is fetched reliably and operable:

### Queue control & observability
- Pause / resume, persisted across restarts in a new `queue_state` table.
- Full error capture: every failed / backoff attempt appends to a new
  `queue_errors` table (attempt #, message, full `error.stack`); admin UI expands a
  failed job to stream the stack trace for troubleshooting.
- Retry controls: `retryJob(key)`, `retrySource(kind)`, `clearDone(olderThanMs)`.

### Scheduling (auto + manual)
- `queue_schedules` table with per-source `interval_ms`; a 30s loop calls
  `enqueueDueSchedules()` to enqueue refreshes for all in-demand symbols when due.
- Seed defaults: `sec-fetch` 24h, `yfinance` 5min. Admin UI lists / adds / deletes
  schedules.

### Reliability
- Startup recovery: jobs left `in_flight` at shutdown reset to `pending` on boot.
- Schema migration: `queue_errors`, `queue_schedules`, `queue_state` tables plus
  `error` / `scheduled_for` columns on `adapter_queue` (idempotent ALTER on startup).

### Critical EDGAR bug fix
- `form4_tx` and `form13f_holdings` build the archive URL from the filer CIK
  (the accession-number prefix), not the company CIK — resolves Cloudflare 429
  that left SEC backfills sparse / empty.

## Acceptance
- `node --test` backend suite passes (503 tests).
- Full sec-fetch backfill succeeds for all watched symbols (NVDA 14,675 institution
  filings, CIFR 274 insider txns, TSLA 6,011, etc.).
- Admin queue page: pause/resume, failed-job stack traces, schedule management.

## B1-B4 Completion Note (Jul 12 2026)
Completed lint+backfill system implementation. All acceptance criteria met, typecheck passes on changed files.