45 lines
1.7 KiB
Markdown
45 lines
1.7 KiB
Markdown
|
# Task: Wire News Ingestion as Celery Task
|
||
|
|
|
||
|
|
## Current State
|
||
|
|
|
||
|
|
`src/data-pipeline/ingest_news.py` exists as a standalone script but is NOT wired as a Celery task. Only `ingest_prices` and `ingest_sec` exist in `src/backend/tasks/`.
|
||
|
|
|
||
|
|
## Goal
|
||
|
|
|
||
|
|
Make news/sentiment data ingestion runnable as a scheduled Celery task, integrated into the pipeline orchestrator.
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
1. Create `src/backend/tasks/ingest_news.py` Celery task
|
||
|
|
2. Reuse logic from `src/data-pipeline/ingest_news.py` (don't duplicate)
|
||
|
|
3. Task must be idempotent — deduplicate by (ticker, source, published_date)
|
||
|
|
4. Task must store news articles and sentiment signals into PostgreSQL
|
||
|
|
5. Add to Celery Beat schedule in `celery_app.py`
|
||
|
|
6. Add to docker-compose worker service
|
||
|
|
|
||
|
|
## Acceptance Criteria
|
||
|
|
|
||
|
|
- `celery -A celery_app worker` processes news ingestion tasks
|
||
|
|
- Running the task twice with same input produces no duplicate records
|
||
|
|
- News articles and sentiment data visible in database after task completes
|
||
|
|
- Celery Beat runs it on a configurable schedule (default: daily)
|
||
|
|
- Frontend can query news via existing `/api/v1/stocks/{ticker}/sentiment` endpoint
|
||
|
|
|
||
|
|
## Constraints
|
||
|
|
|
||
|
|
- Use TimescaleDB hypertables for time-series news data
|
||
|
|
- Follow existing task patterns from `ingest_prices.py` and `ingest_sec.py`
|
||
|
|
- Keep under 200 lines per framework rule
|
||
|
|
- Read existing migrations before writing schema
|
||
|
|
|
||
|
|
## Files to Create/Modify
|
||
|
|
|
||
|
|
- `src/backend/tasks/ingest_news.py` (new)
|
||
|
|
- `src/backend/celery_app.py` (add schedule entry)
|
||
|
|
- `docker-compose.dev.yml` (verify worker includes task)
|
||
|
|
- `src/data-pipeline/ingest_news.py` (verify it's importable, not just executable)
|
||
|
|
|
||
|
|
## Next Steps After This Task
|
||
|
|
|
||
|
|
Wire pipeline orchestrator (phase2-pipeline-orchestrator)
|