feat: multiple updates - alerts, auth, sectors, rotation service, financials ingestion, task specs, and agent framework

2026-06-06 22:01:40 -04:00
parent 7cbc5c120d
commit a16050c80a
47 changed files with 2434 additions and 373 deletions
@@ -1,101 +1,18 @@
-# BRAIN.md — Semantic Memory Layer
+# BRAIN.md

-This file captures observed patterns, architectural decisions, invariants, and lessons learned from working within the invest-copilot codebase. It is updated iteratively as new structural insights emerge.
+## Architecture (invest-copilot)
+- **Backend:** FastAPI (`src/backend/main.py`) — routers, services, models, schemas, Celery tasks
+- **Frontend:** Next.js 15 App Router — pages, components (shadcn/ui), Zustand stores, TanStack Query hooks
+- **Data:** TimescaleDB (PostgreSQL), Redis cache, MinIO storage
+- **Pipeline:** `src/data-pipeline/` — price/SEC/news ingestion, `pipeline.py` orchestrator
+- **Real-time:** Celery → Redis Pub/Sub → SSE/WebSocket (`stream.py`) → frontend

---
+## Key Patterns
+- Frontend → FastAPI Router → Service → SQLAlchemy → TimescaleDB
+- Auth: JWT + session via middleware
+- Strategy → DB → Celery evaluation → Alert → SSE push

-## Architecture Invariants
-
-### Backend (FastAPI)
- **Entry:** `src/backend/main.py` — FastAPI app with middleware, CORS, lifespan events
- **Models:** `src/backend/models/` — SQLAlchemy ORM models (stock, price, watchlist, strategy, alert, sec_filing, insider_trade, screener, sector_rotation)
- **Schemas:** `src/backend/schemas/` — Pydantic v2 schemas for request/response validation
- **Routers:** `src/backend/routers/` — REST endpoints + `stream.py` for SSE/WebSocket
- **Services:** `src/backend/services/` — Business logic (auth, market_data, rotation, screener, sec, sentiment)
- **Tasks:** `src/backend/tasks/` — Celery workers (ingest_prices, ingest_sec, sector_scan)
- **Config:** `config.py` — Environment-based configuration
- **Database:** `database.py` — SQLAlchemy engine/session setup (PostgreSQL + TimescaleDB)
- **Cache:** `cache.py` — Redis cache layer
- **Storage:** `storage.py` — MinIO/S3-compatible object storage
-
-### Frontend (Next.js 15)
- **App Router:** `src/frontend/src/app/` — Pages: dashboard, stock/[ticker], watchlists, strategies, screeners, sectors, alerts, sync, login, register
- **Components:** `src/frontend/src/components/` — UI primitives (shadcn/ui), layout (Sidebar, TopNav, Header), domain-specific (StockCard, PriceChart, StockTable, StockProfile, FilingCard)
- **State:** `src/frontend/src/store/` — Zustand stores (UI, Watchlist, Strategies)
- **Data Fetching:** `src/frontend/src/hooks/` — useStockData, useWatchlistData, useSSE
- **Lib:** `src/frontend/src/lib/` — API client, constants, utilities
- **Types:** `src/frontend/src/types/` + `src/shared/types.ts` — Shared TypeScript types
-
-### Data Pipeline
- `src/data-pipeline/` — Python scripts for data ingestion (prices, SEC filings, financials, news)
- `pipeline.py` — Orchestrator
- `migration.sql` — Database schema migrations
- `tickers.json` — Ticker universe
-
-### Infrastructure
- `docker-compose.dev.yml` / `docker-compose.prod.yml` — Docker Compose configurations
- `nginx/` — Reverse proxy configuration
- `.env.example` — Environment variable template
-
---
-
-## Observed Patterns
-
-### Data Flow (typical API request)
-```
-Frontend (TanStack Query) → FastAPI Router → Service Layer → SQLAlchemy Model → PostgreSQL
-Frontend (Zustand) ← TanStack Query ← Router ← Service ← Model ← DB
-```
-
-### Real-time Data Flow
-```
-Celery Task (ingest) → Redis Pub/Sub → SSE/WebSocket (stream.py) → Frontend (useSSE hook)
-```
-
-### Auth Flow
-```
-Login → JWT token + session → Middleware validation → Protected routes
-```
-
-### Strategy → Alert Pipeline
-```
-User creates strategy (schemas/strategy) → Stored in DB → Celery worker evaluates → Alert created (models/alert) → Push/SSE notification
-```
-
---
-
-## Key Dependencies & Contracts
-
-| Component | Depends On | Contract |
-|-----------|-----------|----------|
-| Routers | Services, Schemas | HTTP request/response via Pydantic |
-| Services | Models, Cache, Storage | Business logic, data transformation |
-| Models | Database | SQLAlchemy ORM, table relationships |
-| Tasks | Services, Database | Async Celery jobs, idempotent ingestion |
-| Frontend | Backend API | REST endpoints + SSE/WebSocket streams |
-| Frontend Store | API hooks | Zustand state management |
-
---
-
-## Security Considerations (Observed)
- JWT-based auth with session support
- CORS middleware configured
- Pydantic validation on all inputs
- Need to verify: rate limiting on API endpoints, input sanitization in screener queries, WebSocket connection limits
-
---
-
-## Lessons Learned / Notes
-
-<!-- Add observations here as they emerge from work sessions. -->
-<!-- Example: "Price ingestion tasks must be idempotent — deduplicate by (ticker, timestamp) pair." -->
-<!-- Example: "Zustand stores should mirror TanStack Query cache to avoid double-fetching." -->
-
---
-
-## Pattern Attractors (Future Exploration)
-
- Sector rotation detection logic in `rotation_service.py` — how does it correlate with ETF flows?
- Screener query builder in `screener_service.py` — potential SQL injection surface if not parameterized
- SSE stream in `stream.py` — connection lifecycle and reconnection strategy
- Celery task deduplication — are ingestion tasks idempotent?
+## Invariants
+- TimescaleDB hypertables for all time-series data
+- Never assume DB schema — read migrations first
+- Research → design → implement flow