invest-copilot/tasks/phase2-pipeline-orchestrator/SPEC.md

Task: Wire Pipeline Orchestrator

Current State

src/data-pipeline/pipeline.py exists but is not connected to Celery Beat or the backend. The four ingestion scripts (prices, sec, financials, news) run standalone — no dependency ordering, no error handling between stages, no visibility into pipeline health.

Goal

Create a proper pipeline orchestrator that:

1. Chains ingestion tasks in correct dependency order
2. Provides health/status endpoints
3. Integrates with existing Celery infrastructure

Requirements

1. Define dependency graph: prices → sec → financials → news (prices must run first)
2. Create src/backend/tasks/pipeline.py with orchestration logic
3. Add /api/v1/sync/status endpoint (already exists per README — verify it works)
4. Create Celery chain/group for full pipeline run
5. Add manual trigger endpoint: POST /api/v1/sync/pipeline/run
6. Add to docker-compose worker service

Acceptance Criteria

• Full pipeline runs end-to-end via single API call
• Dependency ordering enforced (prices before sec, etc.)
• Pipeline status visible via /api/v1/sync/status
• Failed stage does not block other independent stages
• Pipeline health check returns last run time, status, errors

Constraints

• Use Celery chains for ordered stages, Celery groups for parallel stages
• Follow existing task patterns — don't reinvent error handling
• Keep under 200 lines per framework rule
• Read existing migrations before writing schema

Files to Create/Modify

• src/backend/tasks/pipeline.py (new)
• src/backend/routers/data_sync.py (add pipeline trigger endpoint)
• src/backend/celery_app.py (add pipeline schedule)
• docker-compose.dev.yml (verify worker includes task)

Next Steps After This Task

Wire backtest integration (phase2-backtest-integration)