transnet/investor-flow
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(queue): adapter queue overhaul — pause/resume, error capture, scheduling

Browse Source 
- Pause/resume persisted in queue_state; drain loop checks isPaused() and skips
- Error capture: every failed/backoff attempt logged to queue_errors with the
  full stack trace; admin queue page expands a failed job to stream it
- Retry controls: retryJob(key), retrySource(kind), clearDone(olderThanMs)
- Per-source scheduling: queue_schedules table + 30s enqueueDueSchedules loop
  (seed defaults sec-fetch 24h, yfinance 5min); admin UI lists/adds/deletes
- Startup recovery: interrupted in_flight jobs reset to pending on boot
- fix(edgar): archive URLs use the filer CIK (accession-number prefix), not the
  company CIK — resolves Cloudflare 429 that left SEC backfills sparse/empty
- Migration: add queue_errors, queue_schedules, queue_state tables plus error/
  scheduled_for columns on adapter_queue (idempotent ALTER on startup)

Verified: full sec-fetch backfill now succeeds for all watched symbols
(NVDA 14,675 institution filings, CIFR 274 insider txns, TSLA 6,011, etc.).
503 backend tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Investor Flow Build
2026-07-12 12:55:59 -04:00
co-authored by Claude
parent 55f07e6b42
commit ca385c1960
10 changed files with 1345 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/server/src/adapters/__tests__/EdgarAdapter.test.ts
+6 -6
View File
@@ -98,10 +98,10 @@ function makeFilingsResponse() {
name: 'TEST COMPANY INC',
filings: {
recent: [
{ form: '10-K', dateReporter: '2026-03-15', accessionNumber: '0001234567-26-000001', accessionNormalization: '2026-03-15', reportDate: '2026-02-28', reportFile: 'http://example.com/10k.pdf', primaryDocument: 'form10k.pdf' },
{ form: '10-Q', dateReporter: '2026-01-15', accessionNumber: '0001234567-26-000002', accessionNormalization: '2026-01-15', reportDate: '2025-12-31', reportFile: 'http://example.com/10q.pdf', primaryDocument: 'form10q.pdf' },
{ form: '8-K', dateReporter: '2025-12-01', accessionNumber: '0001234567-25-000003', accessionNormalization: '2025-12-01', reportDate: '2025-12-01', reportFile: 'http://example.com/8k.pdf', primaryDocument: 'form8k.pdf' },
{ form: 'SC 13G', dateReporter: '2025-06-30', accessionNumber: '0001234567-25-000004', accessionNormalization: '2025-06-30', reportDate: '2025-06-30', reportFile: 'http://example.com/13g.pdf', primaryDocument: 'form13g.pdf' },
{ form: '10-K', filingDate: '2026-03-15', accessionNumber: '0001234567-26-000001', accessionNormalization: '2026-03-15', reportDate: '2026-02-28', reportFile: 'http://example.com/10k.pdf', primaryDocument: 'form10k.pdf' },
{ form: '10-Q', filingDate: '2026-01-15', accessionNumber: '0001234567-26-000002', accessionNormalization: '2026-01-15', reportDate: '2026-01-15', reportFile: 'http://example.com/10q.pdf', primaryDocument: 'form10q.pdf' },
{ form: '8-K', filingDate: '2025-12-01', accessionNumber: '0001234567-25-000003', accessionNormalization: '2025-12-01', reportDate: '2025-12-01', reportFile: 'http://example.com/8k.pdf', primaryDocument: 'form8k.pdf' },
{ form: 'SC 13G', filingDate: '2025-06-30', accessionNumber: '0001234567-25-000004', accessionNormalization: '2025-06-30', reportDate: '2025-06-30', reportFile: 'http://example.com/13g.pdf', primaryDocument: 'form13g.pdf' },
],
},
};
@@ -195,7 +195,7 @@ test('filings_index filters by dateRange (from + to)', async () => {
assert.equal(getCallCount(), 1);
const filings = result.value as Array<Record<string, unknown>>;
// 10-K (2026-03-15), 10-Q (2026-01-15), 8-K (2025-12-01) are in range; SC 13G (2025-06-30) is out
// 10-K (2026-02-28), 10-Q (2026-01-15), 8-K (2025-12-01) are in range; SC 13G (2025-06-30) is out
assert.equal(filings.length, 3, 'should include filings within date range');
const forms = filings.map((f) => f.form);
assert.ok(forms.includes('10-K'));
@@ -222,7 +222,7 @@ test('filings_index filters by formTypes + dateRange combined', async () => {
assert.equal(getCallCount(), 1);
const filings = result.value as Array<Record<string, unknown>>;
// 10-K (2026-03-15) and 10-Q (2026-01-15) both have dateReporter >= 2026-01-01
// 10-K (2026-02-28) and 10-Q (2026-01-15) both have reportDate >= 2026-01-01
assert.equal(filings.length, 2, '10-K and 10-Q are in date range with form filter');
const forms = filings.map((f) => f.form);
assert.ok(forms.includes('10-K'));