641 lines
22 KiB
Markdown
641 lines
22 KiB
Markdown
# Invest Copilot — Data Edge Research
|
||
|
||
> Created: 2025-05-26
|
||
> Purpose: Identify all data sources, signals, and alternative data that create an investment "edge" — information that gives our users an informational advantage over retail competitors.
|
||
>
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [Core Market Data (Table Stakes)](#1-core-market-data-table-stakes)
|
||
2. [Fundamental Data (The Foundation)](#2-fundamental-data-the-foundation)
|
||
3. [Institutional & Insider Signals](#3-institutional--insider-signals)
|
||
4. [Options Flow & Sentiment](#4-options-flow--sentiment)
|
||
5. [Short Interest & Squeeze Potential](#5-short-interest--squeeze-potential)
|
||
6. [SEC Filings & Regulatory Intelligence](#6-sec-filings--regulatory-intelligence)
|
||
7. [Alternative Data (True Alpha)](#7-alternative-data-true-alpha)
|
||
8. [Macro & Economic Indicators](#8-macro--economic-indicators)
|
||
9. [Sector & Rotation Signals](#9-sector--rotation-signals)
|
||
10. [AI/ML Feature Engineering](#10-aiml-feature-engineering)
|
||
11. [API Providers & Cost Analysis](#11-api-providers--cost-analysis)
|
||
|
||
---
|
||
|
||
## 1. Core Market Data (Table Stakes)
|
||
|
||
*Everyone has this. You need it, but it doesn't create an edge by itself.*
|
||
|
||
| Data Point | Why It Matters | Frequency |
|
||
|-----------|---------------|-----------|
|
||
| Real-time price (bid/ask/last) | Entry/exit timing | Tick-by-tick |
|
||
| Volume (absolute + relative) | Conviction behind moves | Tick-by-tick |
|
||
| VWAP (Volume Weighted Avg Price) | Institutional benchmark | Real-time |
|
||
| 52-week high/low | Psychological levels | Daily |
|
||
| Market cap / Float | Liquidity assessment | Daily |
|
||
| Average volume (10d/30d/90d) | Normalization baseline | Daily |
|
||
| Intraday OHLCV (1m/5m/15m/1h) | Chart patterns, entry timing | Intraday |
|
||
| Pre-market / After-hours price | Gap risk, overnight sentiment | Extended hours |
|
||
| Split/dividend-adjusted prices | Historical accuracy | Event-driven |
|
||
| Relative strength vs sector/index | Outperformance/underperformance | Daily |
|
||
|
||
---
|
||
|
||
## 2. Fundamental Data (The Foundation)
|
||
|
||
*Where the real story lives. This is where you separate investors from gamblers.*
|
||
|
||
### Income Statement
|
||
- Revenue (quarterly + YoY + QoQ growth rates)
|
||
- Gross margin, operating margin, net margin (trend analysis)
|
||
- EBITDA / EBIT
|
||
- EPS (GAAP + non-GAAP + diluted)
|
||
- R&D spend (critical for tech — shows future investment)
|
||
- SG&A as % of revenue (efficiency metric)
|
||
- Free cash flow conversion
|
||
|
||
### Balance Sheet
|
||
- Total assets/liabilities/debt
|
||
- Net debt / EBITDA ratio (solvency)
|
||
- Current ratio, quick ratio (liquidity)
|
||
- Share count changes (dilution detection)
|
||
- Cash & equivalents vs short-term debt
|
||
- Goodwill & intangible assets (quality of earnings)
|
||
|
||
### Cash Flow
|
||
- Operating cash flow (quality of earnings)
|
||
- Capex (growth vs maintenance)
|
||
- Free cash flow (FCF = OCF − Capex)
|
||
- Share buybacks (management confidence signal)
|
||
- Dividend payments & changes
|
||
|
||
### Key Ratios (Computed)
|
||
- P/E, PEG, P/S, P/B, P/FCF
|
||
- ROE, ROA, ROIC
|
||
- Debt/Equity, Interest Coverage
|
||
- Altman Z-Score (bankruptcy risk)
|
||
- Piotroski F-Score (9-factor quality score)
|
||
|
||
---
|
||
|
||
## 3. Institutional & Insider Signals
|
||
|
||
*One of the strongest predictive signals available to retail.*
|
||
|
||
### Institutional Ownership
|
||
- Top 10 holders (Vanguard, BlackRock, Fidelity, etc.)
|
||
- % of float held by institutions
|
||
- **Quarterly change in institutional ownership** (rising = bullish signal)
|
||
- 13F filings (lagged 45 days, but comprehensive)
|
||
- Hedge fund holdings (13F — track specific funds like Renaissance, Citadel, Bridgewater)
|
||
- Mutual fund net inflows/outflows
|
||
- **New positions** vs **increased positions** vs **sold positions**
|
||
|
||
### Insider Activity (FORM 4)
|
||
- **Insider buys** — strongest signal (they spend their own money)
|
||
- CEO/CFO buys are highest conviction
|
||
- Open market buys > exercise of options
|
||
- Cluster buying (multiple insiders buying) = very strong signal
|
||
- **Insider sells** — need context
|
||
- 10b5-1 plans = routine, not signal
|
||
- Unplanned sells = potential red flag
|
||
- Cluster sells = very bearish
|
||
- **Form 4 filing date vs transaction date** — speed matters
|
||
- **SEC Form 144** (proposed sales — early warning)
|
||
|
||
---
|
||
|
||
## 4. Options Flow & Sentiment
|
||
|
||
*Options market often moves before the stock. This is real-time institutional positioning.*
|
||
|
||
### Options Data
|
||
- **Unusual options activity** — volume >> open interest
|
||
- Large block trades (100+ contracts)
|
||
- Out-of-the-money calls (bullish speculation)
|
||
- Put/call ratio spikes (fear/greed)
|
||
- **Put/Call ratio** by ticker and overall market
|
||
- **Implied volatility** (IV) vs historical volatility (HV)
|
||
- IV > HV = options expensive (potential move)
|
||
- IV rank / IV percentile
|
||
- **Options chain** — max pain, gamma exposure
|
||
- **Block trades** — dark pool prints
|
||
- **Dark pool volume %**
|
||
|
||
### Sentiment Indicators
|
||
- **Put/Call ratio** breakdown (equity, index, single stock)
|
||
- **CBOE Volatility Index (VIX)** and components
|
||
- **CNN Fear & Greed Index**
|
||
- **AAII sentiment survey**
|
||
- **Google Trends** for stock/sector searches
|
||
- **Reddit/Twitter sentiment** (r/wallstreetbets, r/investing)
|
||
|
||
---
|
||
|
||
## 5. Short Interest & Squeeze Potential
|
||
|
||
*Short squeeze setups can create 100%+ moves in days.*
|
||
|
||
### Short Data
|
||
- **Short interest** (% of float)
|
||
- **Days to cover** (short ratio)
|
||
- **Short interest trend** (rising = bearish, but also squeeze fuel)
|
||
- **Squeeze probability score**:
|
||
- High short interest (>20%)
|
||
- Low float (<50M shares)
|
||
- High days-to-cover (>5)
|
||
- Rising price + volume
|
||
- Recent catalyst (earnings, FDA, product)
|
||
|
||
### Borrowing Data
|
||
- **Stock borrow fees** (high fees = hard to borrow = squeeze potential)
|
||
- **Locate availability**
|
||
- **Cost to borrow** (% annual)
|
||
|
||
---
|
||
|
||
## 6. SEC Filings & Regulatory Intelligence
|
||
|
||
*Raw regulatory filings are the most authoritative source — before analysts catch up.*
|
||
|
||
### Key Filings
|
||
- **10-K** (annual) — comprehensive financial picture
|
||
- **10-Q** (quarterly) — quarterly updates
|
||
- **8-K** (current) — material events (earnings, M&A, leadership changes)
|
||
- **DEF 14A** (proxy) — executive comp, board changes
|
||
- **S-1** / **S-3** — new offerings (dilution risk)
|
||
- **SC 13D/G** — activist positions (>5% ownership)
|
||
- **Form 4** — insider transactions (daily)
|
||
- **Form 144** — proposed insider sales
|
||
|
||
### NLP Extraction Targets
|
||
- **MD&A changes** — management commentary shifts
|
||
- **Risk factor additions** — new risks = new concerns
|
||
- **Auditor changes** — red flag if auditor resigns
|
||
- **Going concern** mentions = existential threat
|
||
- **Related party transactions** — potential tunneling
|
||
- **Segment revenue breakdown** — growth drivers
|
||
|
||
---
|
||
|
||
## 7. Alternative Data (True Alpha)
|
||
|
||
*This is where you create real edge. These are hedge fund-grade signals.*
|
||
|
||
### Consumer Behavior
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| App download counts | Sensor Tower, App Annie | Early revenue signal for consumer apps |
|
||
| App usage/engagement | SimilarWeb, data.ai | Retention, engagement trends |
|
||
| Web traffic | SimilarWeb, SEMrush | Interest, funnel performance |
|
||
| Credit card spend | YipitData, Earnest Research | Real-time revenue proxy |
|
||
| Grocery/retail receipts | Earnest Research | Consumer discretionary health |
|
||
| Shipping/tracking data | Project44, Descartes | Supply chain visibility, inventory |
|
||
|
||
### Corporate Activity
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| Job postings | Employment data, LinkedIn | Growth signaling, expansion plans |
|
||
| Job posting changes | Indeed, LinkedIn | Hiring freeze = cost cutting signal |
|
||
| Patent filings | USPTO, Google Patents | Innovation pipeline |
|
||
| Building permits | Municipal records | Physical expansion plans |
|
||
| Executive hires/leaves | LinkedIn, SEC filings | Leadership quality, stability |
|
||
| Earnings call transcripts | Seeking Alpha, Motley Fool | NLP on management tone, guidance |
|
||
|
||
### Supply Chain
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| Supplier revenue changes | Supplier financials | Proxy for customer demand |
|
||
| Supplier capex increases | Supplier filings | Capacity expansion = demand confidence |
|
||
| Supplier stock performance | Supplier tickers | Leading indicator for customers |
|
||
| Container shipping rates | Drewry, Clarksons | Global trade volume proxy |
|
||
| Oil/commodity prices | Bloomberg, CME | Input cost pressure |
|
||
|
||
### Sentiment & Social
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| Reddit sentiment | Pushshift, Reddit API | Retail sentiment extremes = contrarian |
|
||
| Twitter/X sentiment | X API | Real-time reaction, influencer moves |
|
||
| StockTwits sentiment | StockTwits API | Retail trader positioning |
|
||
| Google Trends | Google Trends API | Interest spike detection |
|
||
| YouTube/video content | YouTube Data API | Media coverage analysis |
|
||
| News sentiment | NewsAPI, GDELT | Sentiment scoring, event detection |
|
||
|
||
### Physical/Economic Proxies
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| Satellite imagery | Planet, Sentinel | Retail parking lots, construction |
|
||
| Credit card transaction data | YipitData, Flexport | Consumer spending in real-time |
|
||
| Mobile location data | SafeGraph, Foursquare | Foot traffic to stores |
|
||
| Energy consumption | Utility data | Industrial activity proxy |
|
||
| Water usage data | Various providers | Agricultural/industrial activity |
|
||
|
||
### Macro Indicators (Beyond the headline)
|
||
| Signal | Source | Edge |
|
||
|--------|--------|------|
|
||
| Yield curve (2s10, 3m10) | FRED, Treasury.gov | Recession predictor |
|
||
| Inverted yield curve depth/duration | FRED | Recession probability |
|
||
| Leading Economic Index (LEI) | Conference Board | 6-12 month outlook |
|
||
| PMI (ISM Manufacturing/Services) | ISM | Economic activity pulse |
|
||
| Consumer confidence | Conference Board | Consumer spending predictor |
|
||
| Jobless claims (weekly) | DOL | Labor market health |
|
||
| Initial vs continuing claims ratio | DOL | Trend vs noise |
|
||
| Building permits/housing starts | Census Bureau | Housing market leading indicator |
|
||
| Consumer credit changes | NY Fed | Consumer financial stress |
|
||
|
||
---
|
||
|
||
## 8. Macro & Economic Indicators
|
||
|
||
*For sector rotation and macro regime detection.*
|
||
|
||
### Interest Rate Environment
|
||
- Fed funds rate / Fed expectations (CME FedWatch)
|
||
- Treasury yields (2Y, 5Y, 10Y, 30Y)
|
||
- **Yield curve spread** (10Y-2Y, 10Y-3M) — recession signal
|
||
- **TED spread** (credit risk)
|
||
- **TIPS breakeven** (inflation expectations)
|
||
- SOFR, LIBOR successor rates
|
||
- **Commercial paper spreads**
|
||
- **High yield spreads** (ICE BofA HY OAS)
|
||
|
||
### Inflation
|
||
- CPI (headline + core)
|
||
- PCE (Fed's preferred measure)
|
||
- PPI (producer prices — leading indicator)
|
||
- Wage growth (average hourly earnings)
|
||
- Shelter/rent component (largest CPI component)
|
||
|
||
### Growth
|
||
- GDP growth (advance, second, final)
|
||
- Non-farm payrolls
|
||
- Unemployment rate
|
||
- ISM Manufacturing PMI (>50 = expansion)
|
||
- ISM Services PMI
|
||
- Retail sales
|
||
- Industrial production
|
||
|
||
### Currency & Commodities
|
||
- DXY (US Dollar Index)
|
||
- USD/EUR, USD/JPY, USD/CNY
|
||
- Gold (fear/deflation hedge)
|
||
- Oil (WTI/Brent — inflation, growth)
|
||
- Copper (economic activity — "Dr. Copper")
|
||
- Bitcoin (risk-on proxy)
|
||
|
||
---
|
||
|
||
## 9. Sector & Rotation Signals
|
||
|
||
*For the ETF/Index tracking and rotation detection feature.*
|
||
|
||
### ETF-Level Data
|
||
| ETF | Sector | Signal |
|
||
|-----|--------|--------|
|
||
| XLK | Technology | Tech leadership |
|
||
| XLF | Financials | Risk appetite, rate sensitivity |
|
||
| XLI | Industrial | Economic activity |
|
||
| XLY | Consumer Discretionary | Consumer confidence |
|
||
| XLP | Consumer Staples | Defensive positioning |
|
||
| XLE | Energy | Commodity cycle |
|
||
| XLV | Healthcare | Defensive, innovation |
|
||
| XLU | Utilities | Defensive, rate sensitivity |
|
||
| XLB | Materials | Cyclical, commodities |
|
||
| XLR | Real Estate | Rate sensitivity, housing |
|
||
| XLRE | Real Estate | Same as above (alternate) |
|
||
| XLG | Large Cap Growth | Growth tilt |
|
||
| XSC | Small Cap | Economic outlook (IWR alternative) |
|
||
|
||
### Rotation Indicators (The Edge)
|
||
1. **Relative Strength Score** — sector vs SPY over 20d/50d/200d
|
||
2. **RSI Divergence** — sector making new high while SPY doesn't = leadership
|
||
3. **Money Flow** — sector inflow vs outflow tracking
|
||
4. **Sector ETF spread** — XLK vs XLE ratio changing
|
||
5. **Breadth** — stocks above 50MA and 200MA within sector
|
||
6. **Volume concentration** — volume shifting to specific sectors
|
||
|
||
### Rotation Detection Algorithm
|
||
```
|
||
Rotation = when 3+ of these conditions align:
|
||
1. Sector ETF breaks above 50-day MA
|
||
2. Sector ETF RSI crosses above 50
|
||
3. Sector ETF volume > 20-day average
|
||
4. Sector's top 3 stocks outperform SPY
|
||
5. Sector relative strength vs SPY trending up (10d)
|
||
6. Institutional money flow data shows inflows
|
||
7. Analyst upgrades concentrated in sector
|
||
```
|
||
|
||
---
|
||
|
||
## 10. AI/ML Feature Engineering
|
||
|
||
*Transforming raw data into predictive features.*
|
||
|
||
### Technical Indicators (Engineered)
|
||
- Moving averages (20, 50, 100, 200 day) + crossovers
|
||
- RSI (14-day) + overbought/oversold
|
||
- MACD (12, 26, 9) + signal line crossovers
|
||
- Bollinger Bands (20, 2) + position relative to bands
|
||
- ATR (14-day) — volatility measure
|
||
- Volume MA + volume spike detection (3x avg)
|
||
- Gap analysis (pre-market gap % + fill probability)
|
||
- Support/resistance levels (pivot points, swing highs/lows)
|
||
- Fibonacci retracement levels
|
||
- Ichimoku Cloud components
|
||
|
||
### Sentiment Features
|
||
- Put/Call ratio (10-day rolling avg + spike)
|
||
- Short interest change (weekly)
|
||
- Insider buy/sell ratio (quarterly)
|
||
- Analyst rating changes (upgrades - downgrades)
|
||
- Analyst price target revisions (upgrades - downgrades)
|
||
- Social sentiment score (normalized -3 to +3)
|
||
- News sentiment (Vader/BERT-based score)
|
||
|
||
### Fundamental Features
|
||
- Revenue growth acceleration/deceleration
|
||
- Margin expansion/contraction rate
|
||
- Cash flow vs net income divergence
|
||
- Working capital changes
|
||
- Inventory turnover changes
|
||
- Days sales outstanding (DSO) changes
|
||
- Altman Z-Score trend
|
||
- Piotroski F-Score
|
||
|
||
### Composite Scores (The Real Edge)
|
||
1. **Momentum Score** (0-100) — price + volume + relative strength
|
||
2. **Value Score** (0-100) — P/E vs sector, P/B, PEG, FCF yield
|
||
3. **Quality Score** (0-100) — ROIC, margin stability, debt, FCF conversion
|
||
4. **Sentiment Score** (0-100) — insider activity + institutional flows + analyst ratings
|
||
5. **Catalyst Score** (0-100) — upcoming events, earnings proximity, news flow
|
||
6. **Risk Score** (0-100) — volatility, beta, short interest, debt
|
||
|
||
---
|
||
|
||
## 11. API Providers & Cost Analysis
|
||
|
||
### Free / Low-Cost Tier
|
||
| Provider | Data | Cost | Rate Limit |
|
||
|----------|------|------|-----------|
|
||
| **Yahoo Finance** (yfinance) | Prices, fundamentals, options | Free | ~1,000/hr |
|
||
| **Finnhub** | Real-time + fundamentals + alternatives | Free tier | 60 calls/min |
|
||
| **Alpha Vantage** | Prices, fundamentals, alternatives | Free tier | 5 calls/min |
|
||
| **FRED** | Macro/economic data | Free | None |
|
||
| **SEC EDGAR** | All filings | Free | None |
|
||
| **Quandl/Nasdaq Data** | Economic data | Free tier | Limited |
|
||
| **Polygon.io** | Real-time + options | Free tier | Limited |
|
||
|
||
### Paid Tier
|
||
| Provider | Data | Cost | Edge Level |
|
||
|----------|------|------|-----------|
|
||
| **Finnhub** Pro | All + alternatives + sentiment | $100/mo | Medium |
|
||
| **Polygon.io** | Real-time + options + fundamentals | $29-$199/mo | Medium |
|
||
| **Twelve Data** | Prices, fundamentals, crypto | $49-$299/mo | Medium |
|
||
| **Dataroma** | Institutional holdings | $50/mo | Medium |
|
||
| **InsiderMonkey** | Insider + institutional | $50-$200/mo | Medium |
|
||
| **YipitData** | Consumer spending, shipping | $500+/mo | High |
|
||
| **Seeking Alpha Pro** | Earnings transcripts, articles | $240/yr | Medium |
|
||
| **Koyfin** | Bloomberg-lite terminal | $50-$200/mo | Medium |
|
||
| **Bloomberg Terminal** | Everything | $25k/yr | Highest |
|
||
| **Refinitiv (LSEG)** | Everything | $25k+/yr | Highest |
|
||
|
||
### Recommended Stack (MVP → Scale)
|
||
|
||
**Phase 1 (MVP — Free/Low Cost):**
|
||
- `yfinance` or `Finnhub Free` — prices + fundamentals
|
||
- `FRED` — macro data (all free, official government source)
|
||
- `SEC EDGAR API` — all filings (free, official)
|
||
- `Alpha Vantage Free` — alternatives (sentiment, tech indicators)
|
||
|
||
**Phase 2 (Growth — ~$150/mo):**
|
||
- `Finnhub Pro` — real-time data + alternatives
|
||
- `Polygon.io` — options data + real-time
|
||
- `Dataroma` — clean institutional ownership data
|
||
- `Seeking Alpha Pro` — earnings transcripts
|
||
|
||
**Phase 3 (Scale — ~$500/mo):**
|
||
- `YipitData` or `Earnest Research` — consumer spending
|
||
- `Project44` or `Descartes` — supply chain
|
||
- `SimilarWeb` — web traffic
|
||
- `SafeGraph` — foot traffic
|
||
|
||
---
|
||
|
||
## Recommended Data Model (Database Schema)
|
||
|
||
### Core Tables
|
||
|
||
```sql
|
||
-- Time series price data (granular)
|
||
stock_prices (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
open DECIMAL, high DECIMAL, low DECIMAL, close DECIMAL,
|
||
volume BIGINT,
|
||
vwap DECIMAL,
|
||
source VARCHAR,
|
||
PRIMARY KEY (ticker, date, source)
|
||
);
|
||
|
||
-- Fundamental data (quarterly)
|
||
fundamentals (
|
||
ticker VARCHAR,
|
||
quarter DATE,
|
||
revenue DECIMAL,
|
||
gross_margin DECIMAL,
|
||
operating_margin DECIMAL,
|
||
net_margin DECIMAL,
|
||
eps DECIMAL,
|
||
pe_ratio DECIMAL,
|
||
market_cap DECIMAL,
|
||
pb_ratio DECIMAL,
|
||
ps_ratio DECIMAL,
|
||
roe DECIMAL,
|
||
roic DECIMAL,
|
||
debt_equity DECIMAL,
|
||
current_ratio DECIMAL,
|
||
fcf DECIMAL,
|
||
shares_outstanding BIGINT,
|
||
PRIMARY KEY (ticker, quarter)
|
||
);
|
||
|
||
-- Institutional ownership (quarterly)
|
||
institutional_holdings (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
holder_name VARCHAR,
|
||
shares BIGINT,
|
||
pct_float DECIMAL,
|
||
holding_type VARCHAR,
|
||
filing_form VARCHAR,
|
||
PRIMARY KEY (ticker, date, holder_name)
|
||
);
|
||
|
||
-- Insider transactions (daily)
|
||
insider_transactions (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
insider_name VARCHAR,
|
||
title VARCHAR,
|
||
transaction_type VARCHAR, -- BUY, SELL, EXERCISE
|
||
shares BIGINT,
|
||
price DECIMAL,
|
||
value DECIMAL,
|
||
form_4_date DATE,
|
||
PRIMARY KEY (ticker, date, insider_name, shares)
|
||
);
|
||
|
||
-- SEC filings
|
||
sec_filings (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
form_type VARCHAR, -- 10-K, 10-Q, 8-K, DEF 14A, etc.
|
||
url VARCHAR,
|
||
filing_date DATE,
|
||
period_end DATE,
|
||
nlp_summary TEXT,
|
||
nlp_sentiment DECIMAL,
|
||
risk_factors_added INT,
|
||
risk_factors_removed INT,
|
||
PRIMARY KEY (ticker, date, form_type)
|
||
);
|
||
|
||
-- Options data
|
||
options_chain (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
expiry DATE,
|
||
strike DECIMAL,
|
||
option_type VARCHAR, -- CALL, PUT
|
||
volume BIGINT,
|
||
open_interest BIGINT,
|
||
implied_vol DECIMAL,
|
||
last_price DECIMAL,
|
||
PRIMARY KEY (ticker, date, expiry, strike, option_type)
|
||
);
|
||
|
||
-- Short interest (bi-monthly)
|
||
short_interest (
|
||
ticker VARCHAR,
|
||
date DATE,
|
||
short_shares BIGINT,
|
||
float BIGINT,
|
||
short_pct FLOAT,
|
||
days_to_cover FLOAT,
|
||
borrow_fee FLOAT,
|
||
PRIMARY KEY (ticker, date)
|
||
);
|
||
|
||
-- Sector rotation (daily)
|
||
sector_performance (
|
||
ticker VARCHAR, -- ETF ticker (XLK, XLF, etc.)
|
||
date DATE,
|
||
close DECIMAL,
|
||
change_pct DECIMAL,
|
||
volume BIGINT,
|
||
rs_vs_spy DECIMAL, -- relative strength vs SPY
|
||
above_ma50 BOOLEAN,
|
||
above_ma200 BOOLEAN,
|
||
rsi_14 DECIMAL,
|
||
PRIMARY KEY (ticker, date)
|
||
);
|
||
|
||
-- Watchlists
|
||
watchlists (
|
||
id UUID PRIMARY KEY,
|
||
user_id UUID,
|
||
name VARCHAR,
|
||
created_at TIMESTAMP,
|
||
updated_at TIMESTAMP
|
||
);
|
||
|
||
watchlist_items (
|
||
watchlist_id UUID REFERENCES watchlists(id),
|
||
ticker VARCHAR,
|
||
added_at TIMESTAMP,
|
||
notes TEXT,
|
||
PRIMARY KEY (watchlist_id, ticker)
|
||
);
|
||
|
||
-- Strategies
|
||
strategies (
|
||
id UUID PRIMARY KEY,
|
||
user_id UUID,
|
||
name VARCHAR,
|
||
description TEXT,
|
||
created_at TIMESTAMP,
|
||
updated_at TIMESTAMP,
|
||
is_active BOOLEAN
|
||
);
|
||
|
||
strategy_rules (
|
||
strategy_id UUID REFERENCES strategies(id),
|
||
rule_type VARCHAR, -- TECHNICAL, FUNDAMENTAL, SENTIMENT
|
||
condition VARCHAR,
|
||
threshold DECIMAL,
|
||
direction VARCHAR, -- ABOVE, BELOW, CROSS_ABOVE, CROSS_BELOW
|
||
PRIMARY KEY (strategy_id, condition)
|
||
);
|
||
|
||
-- Alerts
|
||
alerts (
|
||
id UUID PRIMARY KEY,
|
||
user_id UUID,
|
||
watchlist_id UUID,
|
||
strategy_id UUID,
|
||
triggered_at TIMESTAMP,
|
||
ticker VARCHAR,
|
||
alert_type VARCHAR,
|
||
message TEXT,
|
||
is_read BOOLEAN,
|
||
data JSONB -- raw data that triggered the alert
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
## Data Pipeline Architecture
|
||
|
||
```
|
||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||
│ API/Scraper │───▶│ Raw Data │───▶│ Normalization │
|
||
│ (yfinance, │ │ Lake (S3/MinIO)│ │ & Enrichment │
|
||
│ FRED, EDGAR) │ │ │ │ (PostgreSQL) │
|
||
└─────────────────┘ └─────────────────┘ └─────────┬───────┘
|
||
│
|
||
▼
|
||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||
│ Real-time │◀───│ Feature │◀───│ API Layer │
|
||
│ WebSocket │ │ Engineering │ │ (FastAPI) │
|
||
│ (prices, │ │ (technical, │ │ │
|
||
│ options, │ │ sentiment) │ │ │
|
||
│ alerts) │ └─────────────────┘ └─────────────────┘
|
||
└─────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Summary: The Edge Pyramid
|
||
|
||
```
|
||
┌─────────────────┐
|
||
│ AI/ML Signals │ ← Composite scores, anomaly detection
|
||
├─────────────────┤
|
||
│ Alternative Data │ ← Consumer spend, web traffic, jobs
|
||
├─────────────────┤
|
||
│ Institutional │ ← 13F flows, insider buys, hedge funds
|
||
├─────────────────┤
|
||
│ Options/Short │ ← Put/call, squeeze potential
|
||
├─────────────────┤
|
||
│ SEC Filings │ ← NLP on 8-Ks, MD&A changes
|
||
├─────────────────┤
|
||
│ Fundamentals │ ← Financials, ratios, quality scores
|
||
├─────────────────┤
|
||
│ Technical │ ← Charts, indicators, volume
|
||
└─────────────────┘
|
||
│ Market Data │ ← Prices, volume (table stakes)
|
||
```
|
||
|
||
**The real edge lives in layers 4-7.** Layers 1-3 everyone has. Layer 8 (AI/ML) is where you combine all signals into predictive composites.
|