Task: Wire News Ingestion as Celery Task

Current State

src/data-pipeline/ingest_news.py exists as a standalone script but is NOT wired as a Celery task. Only ingest_prices and ingest_sec exist in src/backend/tasks/.

Goal

Make news/sentiment data ingestion runnable as a scheduled Celery task, integrated into the pipeline orchestrator.

Requirements

Create src/backend/tasks/ingest_news.py Celery task
Reuse logic from src/data-pipeline/ingest_news.py (don't duplicate)
Task must be idempotent — deduplicate by (ticker, source, published_date)
Task must store news articles and sentiment signals into PostgreSQL
Add to Celery Beat schedule in celery_app.py
Add to docker-compose worker service

Acceptance Criteria

celery -A celery_app worker processes news ingestion tasks
Running the task twice with same input produces no duplicate records
News articles and sentiment data visible in database after task completes
Celery Beat runs it on a configurable schedule (default: daily)
Frontend can query news via existing /api/v1/stocks/{ticker}/sentiment endpoint

Constraints

Use TimescaleDB hypertables for time-series news data
Follow existing task patterns from ingest_prices.py and ingest_sec.py
Keep under 200 lines per framework rule
Read existing migrations before writing schema

Files to Create/Modify

src/backend/tasks/ingest_news.py (new)
src/backend/celery_app.py (add schedule entry)
docker-compose.dev.yml (verify worker includes task)
src/data-pipeline/ingest_news.py (verify it's importable, not just executable)

Next Steps After This Task

Wire pipeline orchestrator (phase2-pipeline-orchestrator)

1.7 KiB Raw Permalink Blame History