feat: multiple updates - alerts, auth, sectors, rotation service, financials ingestion, task specs, and agent framework

tasks/phase2-pipeline-orchestrator/SPEC.md

@@ -0,0 +1,47 @@
+# Task: Wire Pipeline Orchestrator
+
+## Current State
+
+`src/data-pipeline/pipeline.py` exists but is not connected to Celery Beat or the backend. The four ingestion scripts (prices, sec, financials, news) run standalone — no dependency ordering, no error handling between stages, no visibility into pipeline health.
+
+## Goal
+
+Create a proper pipeline orchestrator that:
+1. Chains ingestion tasks in correct dependency order
+2. Provides health/status endpoints
+3. Integrates with existing Celery infrastructure
+
+## Requirements
+
+1. Define dependency graph: prices → sec → financials → news (prices must run first)
+2. Create `src/backend/tasks/pipeline.py` with orchestration logic
+3. Add `/api/v1/sync/status` endpoint (already exists per README — verify it works)
+4. Create Celery chain/group for full pipeline run
+5. Add manual trigger endpoint: `POST /api/v1/sync/pipeline/run`
+6. Add to docker-compose worker service
+
+## Acceptance Criteria
+
+- Full pipeline runs end-to-end via single API call
+- Dependency ordering enforced (prices before sec, etc.)
+- Pipeline status visible via `/api/v1/sync/status`
+- Failed stage does not block other independent stages
+- Pipeline health check returns last run time, status, errors
+
+## Constraints
+
+- Use Celery chains for ordered stages, Celery groups for parallel stages
+- Follow existing task patterns — don't reinvent error handling
+- Keep under 200 lines per framework rule
+- Read existing migrations before writing schema
+
+## Files to Create/Modify
+
+- `src/backend/tasks/pipeline.py` (new)
+- `src/backend/routers/data_sync.py` (add pipeline trigger endpoint)
+- `src/backend/celery_app.py` (add pipeline schedule)
+- `docker-compose.dev.yml` (verify worker includes task)
+
+## Next Steps After This Task
+
+Wire backtest integration (phase2-backtest-integration)