# Invest Copilot — Data Edge Research

> Created: 2025-05-26
> Purpose: Identify all data sources, signals, and alternative data that create an investment "edge" — information that gives our users an informational advantage over retail competitors.
>

---

## Table of Contents

1. [Core Market Data (Table Stakes)](#1-core-market-data-table-stakes)
2. [Fundamental Data (The Foundation)](#2-fundamental-data-the-foundation)
3. [Institutional & Insider Signals](#3-institutional--insider-signals)
4. [Options Flow & Sentiment](#4-options-flow--sentiment)
5. [Short Interest & Squeeze Potential](#5-short-interest--squeeze-potential)
6. [SEC Filings & Regulatory Intelligence](#6-sec-filings--regulatory-intelligence)
7. [Alternative Data (True Alpha)](#7-alternative-data-true-alpha)
8. [Macro & Economic Indicators](#8-macro--economic-indicators)
9. [Sector & Rotation Signals](#9-sector--rotation-signals)
10. [AI/ML Feature Engineering](#10-aiml-feature-engineering)
11. [API Providers & Cost Analysis](#11-api-providers--cost-analysis)

---

## 1. Core Market Data (Table Stakes)

*Everyone has this. You need it, but it doesn't create an edge by itself.*

| Data Point | Why It Matters | Frequency |
|-----------|---------------|-----------|
| Real-time price (bid/ask/last) | Entry/exit timing | Tick-by-tick |
| Volume (absolute + relative) | Conviction behind moves | Tick-by-tick |
| VWAP (Volume Weighted Avg Price) | Institutional benchmark | Real-time |
| 52-week high/low | Psychological levels | Daily |
| Market cap / Float | Liquidity assessment | Daily |
| Average volume (10d/30d/90d) | Normalization baseline | Daily |
| Intraday OHLCV (1m/5m/15m/1h) | Chart patterns, entry timing | Intraday |
| Pre-market / After-hours price | Gap risk, overnight sentiment | Extended hours |
| Split/dividend-adjusted prices | Historical accuracy | Event-driven |
| Relative strength vs sector/index | Outperformance/underperformance | Daily |

---

## 2. Fundamental Data (The Foundation)

*Where the real story lives. This is where you separate investors from gamblers.*

### Income Statement
- Revenue (quarterly + YoY + QoQ growth rates)
- Gross margin, operating margin, net margin (trend analysis)
- EBITDA / EBIT
- EPS (GAAP + non-GAAP + diluted)
- R&D spend (critical for tech — shows future investment)
- SG&A as % of revenue (efficiency metric)
- Free cash flow conversion

### Balance Sheet
- Total assets/liabilities/debt
- Net debt / EBITDA ratio (solvency)
- Current ratio, quick ratio (liquidity)
- Share count changes (dilution detection)
- Cash & equivalents vs short-term debt
- Goodwill & intangible assets (quality of earnings)

### Cash Flow
- Operating cash flow (quality of earnings)
- Capex (growth vs maintenance)
- Free cash flow (FCF = OCF − Capex)
- Share buybacks (management confidence signal)
- Dividend payments & changes

### Key Ratios (Computed)
- P/E, PEG, P/S, P/B, P/FCF
- ROE, ROA, ROIC
- Debt/Equity, Interest Coverage
- Altman Z-Score (bankruptcy risk)
- Piotroski F-Score (9-factor quality score)

---

## 3. Institutional & Insider Signals

*One of the strongest predictive signals available to retail.*

### Institutional Ownership
- Top 10 holders (Vanguard, BlackRock, Fidelity, etc.)
- % of float held by institutions
- **Quarterly change in institutional ownership** (rising = bullish signal)
- 13F filings (lagged 45 days, but comprehensive)
- Hedge fund holdings (13F — track specific funds like Renaissance, Citadel, Bridgewater)
- Mutual fund net inflows/outflows
- **New positions** vs **increased positions** vs **sold positions**

### Insider Activity (FORM 4)
- **Insider buys** — strongest signal (they spend their own money)
  - CEO/CFO buys are highest conviction
  - Open market buys > exercise of options
  - Cluster buying (multiple insiders buying) = very strong signal
- **Insider sells** — need context
  - 10b5-1 plans = routine, not signal
  - Unplanned sells = potential red flag
  - Cluster sells = very bearish
- **Form 4 filing date vs transaction date** — speed matters
- **SEC Form 144** (proposed sales — early warning)

---

## 4. Options Flow & Sentiment

*Options market often moves before the stock. This is real-time institutional positioning.*

### Options Data
- **Unusual options activity** — volume >> open interest
  - Large block trades (100+ contracts)
  - Out-of-the-money calls (bullish speculation)
  - Put/call ratio spikes (fear/greed)
- **Put/Call ratio** by ticker and overall market
- **Implied volatility** (IV) vs historical volatility (HV)
  - IV > HV = options expensive (potential move)
  - IV rank / IV percentile
- **Options chain** — max pain, gamma exposure
- **Block trades** — dark pool prints
- **Dark pool volume %**

### Sentiment Indicators
- **Put/Call ratio** breakdown (equity, index, single stock)
- **CBOE Volatility Index (VIX)** and components
- **CNN Fear & Greed Index**
- **AAII sentiment survey**
- **Google Trends** for stock/sector searches
- **Reddit/Twitter sentiment** (r/wallstreetbets, r/investing)

---

## 5. Short Interest & Squeeze Potential

*Short squeeze setups can create 100%+ moves in days.*

### Short Data
- **Short interest** (% of float)
- **Days to cover** (short ratio)
- **Short interest trend** (rising = bearish, but also squeeze fuel)
- **Squeeze probability score**:
  - High short interest (>20%)
  - Low float (<50M shares)
  - High days-to-cover (>5)
  - Rising price + volume
  - Recent catalyst (earnings, FDA, product)

### Borrowing Data
- **Stock borrow fees** (high fees = hard to borrow = squeeze potential)
- **Locate availability**
- **Cost to borrow** (% annual)

---

## 6. SEC Filings & Regulatory Intelligence

*Raw regulatory filings are the most authoritative source — before analysts catch up.*

### Key Filings
- **10-K** (annual) — comprehensive financial picture
- **10-Q** (quarterly) — quarterly updates
- **8-K** (current) — material events (earnings, M&A, leadership changes)
- **DEF 14A** (proxy) — executive comp, board changes
- **S-1** / **S-3** — new offerings (dilution risk)
- **SC 13D/G** — activist positions (>5% ownership)
- **Form 4** — insider transactions (daily)
- **Form 144** — proposed insider sales

### NLP Extraction Targets
- **MD&A changes** — management commentary shifts
- **Risk factor additions** — new risks = new concerns
- **Auditor changes** — red flag if auditor resigns
- **Going concern** mentions = existential threat
- **Related party transactions** — potential tunneling
- **Segment revenue breakdown** — growth drivers

---

## 7. Alternative Data (True Alpha)

*This is where you create real edge. These are hedge fund-grade signals.*

### Consumer Behavior
| Signal | Source | Edge |
|--------|--------|------|
| App download counts | Sensor Tower, App Annie | Early revenue signal for consumer apps |
| App usage/engagement | SimilarWeb, data.ai | Retention, engagement trends |
| Web traffic | SimilarWeb, SEMrush | Interest, funnel performance |
| Credit card spend | YipitData, Earnest Research | Real-time revenue proxy |
| Grocery/retail receipts | Earnest Research | Consumer discretionary health |
| Shipping/tracking data | Project44, Descartes | Supply chain visibility, inventory |

### Corporate Activity
| Signal | Source | Edge |
|--------|--------|------|
| Job postings | Employment data, LinkedIn | Growth signaling, expansion plans |
| Job posting changes | Indeed, LinkedIn | Hiring freeze = cost cutting signal |
| Patent filings | USPTO, Google Patents | Innovation pipeline |
| Building permits | Municipal records | Physical expansion plans |
| Executive hires/leaves | LinkedIn, SEC filings | Leadership quality, stability |
| Earnings call transcripts | Seeking Alpha, Motley Fool | NLP on management tone, guidance |

### Supply Chain
| Signal | Source | Edge |
|--------|--------|------|
| Supplier revenue changes | Supplier financials | Proxy for customer demand |
| Supplier capex increases | Supplier filings | Capacity expansion = demand confidence |
| Supplier stock performance | Supplier tickers | Leading indicator for customers |
| Container shipping rates | Drewry, Clarksons | Global trade volume proxy |
| Oil/commodity prices | Bloomberg, CME | Input cost pressure |

### Sentiment & Social
| Signal | Source | Edge |
|--------|--------|------|
| Reddit sentiment | Pushshift, Reddit API | Retail sentiment extremes = contrarian |
| Twitter/X sentiment | X API | Real-time reaction, influencer moves |
| StockTwits sentiment | StockTwits API | Retail trader positioning |
| Google Trends | Google Trends API | Interest spike detection |
| YouTube/video content | YouTube Data API | Media coverage analysis |
| News sentiment | NewsAPI, GDELT | Sentiment scoring, event detection |

### Physical/Economic Proxies
| Signal | Source | Edge |
|--------|--------|------|
| Satellite imagery | Planet, Sentinel | Retail parking lots, construction |
| Credit card transaction data | YipitData, Flexport | Consumer spending in real-time |
| Mobile location data | SafeGraph, Foursquare | Foot traffic to stores |
| Energy consumption | Utility data | Industrial activity proxy |
| Water usage data | Various providers | Agricultural/industrial activity |

### Macro Indicators (Beyond the headline)
| Signal | Source | Edge |
|--------|--------|------|
| Yield curve (2s10, 3m10) | FRED, Treasury.gov | Recession predictor |
| Inverted yield curve depth/duration | FRED | Recession probability |
| Leading Economic Index (LEI) | Conference Board | 6-12 month outlook |
| PMI (ISM Manufacturing/Services) | ISM | Economic activity pulse |
| Consumer confidence | Conference Board | Consumer spending predictor |
| Jobless claims (weekly) | DOL | Labor market health |
| Initial vs continuing claims ratio | DOL | Trend vs noise |
| Building permits/housing starts | Census Bureau | Housing market leading indicator |
| Consumer credit changes | NY Fed | Consumer financial stress |

---

## 8. Macro & Economic Indicators

*For sector rotation and macro regime detection.*

### Interest Rate Environment
- Fed funds rate / Fed expectations (CME FedWatch)
- Treasury yields (2Y, 5Y, 10Y, 30Y)
- **Yield curve spread** (10Y-2Y, 10Y-3M) — recession signal
- **TED spread** (credit risk)
- **TIPS breakeven** (inflation expectations)
- SOFR, LIBOR successor rates
- **Commercial paper spreads**
- **High yield spreads** (ICE BofA HY OAS)

### Inflation
- CPI (headline + core)
- PCE (Fed's preferred measure)
- PPI (producer prices — leading indicator)
- Wage growth (average hourly earnings)
- Shelter/rent component (largest CPI component)

### Growth
- GDP growth (advance, second, final)
- Non-farm payrolls
- Unemployment rate
- ISM Manufacturing PMI (>50 = expansion)
- ISM Services PMI
- Retail sales
- Industrial production

### Currency & Commodities
- DXY (US Dollar Index)
- USD/EUR, USD/JPY, USD/CNY
- Gold (fear/deflation hedge)
- Oil (WTI/Brent — inflation, growth)
- Copper (economic activity — "Dr. Copper")
- Bitcoin (risk-on proxy)

---

## 9. Sector & Rotation Signals

*For the ETF/Index tracking and rotation detection feature.*

### ETF-Level Data
| ETF | Sector | Signal |
|-----|--------|--------|
| XLK | Technology | Tech leadership |
| XLF | Financials | Risk appetite, rate sensitivity |
| XLI | Industrial | Economic activity |
| XLY | Consumer Discretionary | Consumer confidence |
| XLP | Consumer Staples | Defensive positioning |
| XLE | Energy | Commodity cycle |
| XLV | Healthcare | Defensive, innovation |
| XLU | Utilities | Defensive, rate sensitivity |
| XLB | Materials | Cyclical, commodities |
| XLR | Real Estate | Rate sensitivity, housing |
| XLRE | Real Estate | Same as above (alternate) |
| XLG | Large Cap Growth | Growth tilt |
| XSC | Small Cap | Economic outlook (IWR alternative) |

### Rotation Indicators (The Edge)
1. **Relative Strength Score** — sector vs SPY over 20d/50d/200d
2. **RSI Divergence** — sector making new high while SPY doesn't = leadership
3. **Money Flow** — sector inflow vs outflow tracking
4. **Sector ETF spread** — XLK vs XLE ratio changing
5. **Breadth** — stocks above 50MA and 200MA within sector
6. **Volume concentration** — volume shifting to specific sectors

### Rotation Detection Algorithm
```
Rotation = when 3+ of these conditions align:
  1. Sector ETF breaks above 50-day MA
  2. Sector ETF RSI crosses above 50
  3. Sector ETF volume > 20-day average
  4. Sector's top 3 stocks outperform SPY
  5. Sector relative strength vs SPY trending up (10d)
  6. Institutional money flow data shows inflows
  7. Analyst upgrades concentrated in sector
```

---

## 10. AI/ML Feature Engineering

*Transforming raw data into predictive features.*

### Technical Indicators (Engineered)
- Moving averages (20, 50, 100, 200 day) + crossovers
- RSI (14-day) + overbought/oversold
- MACD (12, 26, 9) + signal line crossovers
- Bollinger Bands (20, 2) + position relative to bands
- ATR (14-day) — volatility measure
- Volume MA + volume spike detection (3x avg)
- Gap analysis (pre-market gap % + fill probability)
- Support/resistance levels (pivot points, swing highs/lows)
- Fibonacci retracement levels
- Ichimoku Cloud components

### Sentiment Features
- Put/Call ratio (10-day rolling avg + spike)
- Short interest change (weekly)
- Insider buy/sell ratio (quarterly)
- Analyst rating changes (upgrades - downgrades)
- Analyst price target revisions (upgrades - downgrades)
- Social sentiment score (normalized -3 to +3)
- News sentiment (Vader/BERT-based score)

### Fundamental Features
- Revenue growth acceleration/deceleration
- Margin expansion/contraction rate
- Cash flow vs net income divergence
- Working capital changes
- Inventory turnover changes
- Days sales outstanding (DSO) changes
- Altman Z-Score trend
- Piotroski F-Score

### Composite Scores (The Real Edge)
1. **Momentum Score** (0-100) — price + volume + relative strength
2. **Value Score** (0-100) — P/E vs sector, P/B, PEG, FCF yield
3. **Quality Score** (0-100) — ROIC, margin stability, debt, FCF conversion
4. **Sentiment Score** (0-100) — insider activity + institutional flows + analyst ratings
5. **Catalyst Score** (0-100) — upcoming events, earnings proximity, news flow
6. **Risk Score** (0-100) — volatility, beta, short interest, debt

---

## 11. API Providers & Cost Analysis

### Free / Low-Cost Tier
| Provider | Data | Cost | Rate Limit |
|----------|------|------|-----------|
| **Yahoo Finance** (yfinance) | Prices, fundamentals, options | Free | ~1,000/hr |
| **Finnhub** | Real-time + fundamentals + alternatives | Free tier | 60 calls/min |
| **Alpha Vantage** | Prices, fundamentals, alternatives | Free tier | 5 calls/min |
| **FRED** | Macro/economic data | Free | None |
| **SEC EDGAR** | All filings | Free | None |
| **Quandl/Nasdaq Data** | Economic data | Free tier | Limited |
| **Polygon.io** | Real-time + options | Free tier | Limited |

### Paid Tier
| Provider | Data | Cost | Edge Level |
|----------|------|------|-----------|
| **Finnhub** Pro | All + alternatives + sentiment | $100/mo | Medium |
| **Polygon.io** | Real-time + options + fundamentals | $29-$199/mo | Medium |
| **Twelve Data** | Prices, fundamentals, crypto | $49-$299/mo | Medium |
| **Dataroma** | Institutional holdings | $50/mo | Medium |
| **InsiderMonkey** | Insider + institutional | $50-$200/mo | Medium |
| **YipitData** | Consumer spending, shipping | $500+/mo | High |
| **Seeking Alpha Pro** | Earnings transcripts, articles | $240/yr | Medium |
| **Koyfin** | Bloomberg-lite terminal | $50-$200/mo | Medium |
| **Bloomberg Terminal** | Everything | $25k/yr | Highest |
| **Refinitiv (LSEG)** | Everything | $25k+/yr | Highest |

### Recommended Stack (MVP → Scale)

**Phase 1 (MVP — Free/Low Cost):**
- `yfinance` or `Finnhub Free` — prices + fundamentals
- `FRED` — macro data (all free, official government source)
- `SEC EDGAR API` — all filings (free, official)
- `Alpha Vantage Free` — alternatives (sentiment, tech indicators)

**Phase 2 (Growth — ~$150/mo):**
- `Finnhub Pro` — real-time data + alternatives
- `Polygon.io` — options data + real-time
- `Dataroma` — clean institutional ownership data
- `Seeking Alpha Pro` — earnings transcripts

**Phase 3 (Scale — ~$500/mo):**
- `YipitData` or `Earnest Research` — consumer spending
- `Project44` or `Descartes` — supply chain
- `SimilarWeb` — web traffic
- `SafeGraph` — foot traffic

---

## Recommended Data Model (Database Schema)

### Core Tables

```sql
-- Time series price data (granular)
stock_prices (
  ticker VARCHAR,
  date DATE,
  open DECIMAL, high DECIMAL, low DECIMAL, close DECIMAL,
  volume BIGINT,
  vwap DECIMAL,
  source VARCHAR,
  PRIMARY KEY (ticker, date, source)
);

-- Fundamental data (quarterly)
fundamentals (
  ticker VARCHAR,
  quarter DATE,
  revenue DECIMAL,
  gross_margin DECIMAL,
  operating_margin DECIMAL,
  net_margin DECIMAL,
  eps DECIMAL,
  pe_ratio DECIMAL,
  market_cap DECIMAL,
  pb_ratio DECIMAL,
  ps_ratio DECIMAL,
  roe DECIMAL,
  roic DECIMAL,
  debt_equity DECIMAL,
  current_ratio DECIMAL,
  fcf DECIMAL,
  shares_outstanding BIGINT,
  PRIMARY KEY (ticker, quarter)
);

-- Institutional ownership (quarterly)
institutional_holdings (
  ticker VARCHAR,
  date DATE,
  holder_name VARCHAR,
  shares BIGINT,
  pct_float DECIMAL,
  holding_type VARCHAR,
  filing_form VARCHAR,
  PRIMARY KEY (ticker, date, holder_name)
);

-- Insider transactions (daily)
insider_transactions (
  ticker VARCHAR,
  date DATE,
  insider_name VARCHAR,
  title VARCHAR,
  transaction_type VARCHAR,  -- BUY, SELL, EXERCISE
  shares BIGINT,
  price DECIMAL,
  value DECIMAL,
  form_4_date DATE,
  PRIMARY KEY (ticker, date, insider_name, shares)
);

-- SEC filings
sec_filings (
  ticker VARCHAR,
  date DATE,
  form_type VARCHAR,  -- 10-K, 10-Q, 8-K, DEF 14A, etc.
  url VARCHAR,
  filing_date DATE,
  period_end DATE,
  nlp_summary TEXT,
  nlp_sentiment DECIMAL,
  risk_factors_added INT,
  risk_factors_removed INT,
  PRIMARY KEY (ticker, date, form_type)
);

-- Options data
options_chain (
  ticker VARCHAR,
  date DATE,
  expiry DATE,
  strike DECIMAL,
  option_type VARCHAR,  -- CALL, PUT
  volume BIGINT,
  open_interest BIGINT,
  implied_vol DECIMAL,
  last_price DECIMAL,
  PRIMARY KEY (ticker, date, expiry, strike, option_type)
);

-- Short interest (bi-monthly)
short_interest (
  ticker VARCHAR,
  date DATE,
  short_shares BIGINT,
  float BIGINT,
  short_pct FLOAT,
  days_to_cover FLOAT,
  borrow_fee FLOAT,
  PRIMARY KEY (ticker, date)
);

-- Sector rotation (daily)
sector_performance (
  ticker VARCHAR,  -- ETF ticker (XLK, XLF, etc.)
  date DATE,
  close DECIMAL,
  change_pct DECIMAL,
  volume BIGINT,
  rs_vs_spy DECIMAL,  -- relative strength vs SPY
  above_ma50 BOOLEAN,
  above_ma200 BOOLEAN,
  rsi_14 DECIMAL,
  PRIMARY KEY (ticker, date)
);

-- Watchlists
watchlists (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

watchlist_items (
  watchlist_id UUID REFERENCES watchlists(id),
  ticker VARCHAR,
  added_at TIMESTAMP,
  notes TEXT,
  PRIMARY KEY (watchlist_id, ticker)
);

-- Strategies
strategies (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  description TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  is_active BOOLEAN
);

strategy_rules (
  strategy_id UUID REFERENCES strategies(id),
  rule_type VARCHAR,  -- TECHNICAL, FUNDAMENTAL, SENTIMENT
  condition VARCHAR,
  threshold DECIMAL,
  direction VARCHAR,  -- ABOVE, BELOW, CROSS_ABOVE, CROSS_BELOW
  PRIMARY KEY (strategy_id, condition)
);

-- Alerts
alerts (
  id UUID PRIMARY KEY,
  user_id UUID,
  watchlist_id UUID,
  strategy_id UUID,
  triggered_at TIMESTAMP,
  ticker VARCHAR,
  alert_type VARCHAR,
  message TEXT,
  is_read BOOLEAN,
  data JSONB  -- raw data that triggered the alert
);
```

---

## Data Pipeline Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  API/Scraper    │───▶│  Raw Data       │───▶│  Normalization  │
│  (yfinance,     │    │  Lake (S3/MinIO)│    │  & Enrichment   │
│   FRED, EDGAR)  │    │                 │    │  (PostgreSQL)   │
└─────────────────┘    └─────────────────┘    └─────────┬───────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Real-time      │◀───│  Feature        │◀───│  API Layer      │
│  WebSocket      │    │  Engineering    │    │  (FastAPI)      │
│  (prices,      │    │  (technical,    │    │                 │
│   options,     │    │   sentiment)    │    │                 │
│   alerts)      │    └─────────────────┘    └─────────────────┘
└─────────────────┘
```

---

## Summary: The Edge Pyramid

```
           ┌─────────────────┐
           │  AI/ML Signals  │    ← Composite scores, anomaly detection
           ├─────────────────┤
           │ Alternative Data │    ← Consumer spend, web traffic, jobs
           ├─────────────────┤
           │ Institutional    │    ← 13F flows, insider buys, hedge funds
           ├─────────────────┤
           │ Options/Short    │    ← Put/call, squeeze potential
           ├─────────────────┤
           │ SEC Filings      │    ← NLP on 8-Ks, MD&A changes
           ├─────────────────┤
           │ Fundamentals     │    ← Financials, ratios, quality scores
           ├─────────────────┤
           │ Technical        │    ← Charts, indicators, volume
           └─────────────────┘
           │ Market Data     │    ← Prices, volume (table stakes)
```

**The real edge lives in layers 4-7.** Layers 1-3 everyone has. Layer 8 (AI/ML) is where you combine all signals into predictive composites.