feat: multiple updates - alerts, auth, sectors, rotation service, financials ingestion, task specs, and agent framework

2026-06-06 22:01:40 -04:00
parent 7cbc5c120d
commit a16050c80a
47 changed files with 2434 additions and 373 deletions
@@ -0,0 +1,44 @@
+# Task: Wire News Ingestion as Celery Task
+
+## Current State
+
+`src/data-pipeline/ingest_news.py` exists as a standalone script but is NOT wired as a Celery task. Only `ingest_prices` and `ingest_sec` exist in `src/backend/tasks/`.
+
+## Goal
+
+Make news/sentiment data ingestion runnable as a scheduled Celery task, integrated into the pipeline orchestrator.
+
+## Requirements
+
+1. Create `src/backend/tasks/ingest_news.py` Celery task
+2. Reuse logic from `src/data-pipeline/ingest_news.py` (don't duplicate)
+3. Task must be idempotent — deduplicate by (ticker, source, published_date)
+4. Task must store news articles and sentiment signals into PostgreSQL
+5. Add to Celery Beat schedule in `celery_app.py`
+6. Add to docker-compose worker service
+
+## Acceptance Criteria
+
+- `celery -A celery_app worker` processes news ingestion tasks
+- Running the task twice with same input produces no duplicate records
+- News articles and sentiment data visible in database after task completes
+- Celery Beat runs it on a configurable schedule (default: daily)
+- Frontend can query news via existing `/api/v1/stocks/{ticker}/sentiment` endpoint
+
+## Constraints
+
+- Use TimescaleDB hypertables for time-series news data
+- Follow existing task patterns from `ingest_prices.py` and `ingest_sec.py`
+- Keep under 200 lines per framework rule
+- Read existing migrations before writing schema
+
+## Files to Create/Modify
+
+- `src/backend/tasks/ingest_news.py` (new)
+- `src/backend/celery_app.py` (add schedule entry)
+- `docker-compose.dev.yml` (verify worker includes task)
+- `src/data-pipeline/ingest_news.py` (verify it's importable, not just executable)
+
+## Next Steps After This Task
+
+Wire pipeline orchestrator (phase2-pipeline-orchestrator)