pi.dev/invest-copilot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
invest-copilot/docs/data-edge-research.md
T

641 lines
22 KiB
Markdown
Raw Permalink Normal View History

Initial commit: invest-copilot app
2026-05-30 11:28:59 -04:00
# Invest Copilot — Data Edge Research
> Created: 2025-05-26
> Purpose: Identify all data sources, signals, and alternative data that create an investment "edge" — information that gives our users an informational advantage over retail competitors.
>
---
## Table of Contents
1. [Core Market Data (Table Stakes)](#1-core-market-data-table-stakes)
2. [Fundamental Data (The Foundation)](#2-fundamental-data-the-foundation)
3. [Institutional & Insider Signals](#3-institutional--insider-signals)
4. [Options Flow & Sentiment](#4-options-flow--sentiment)
5. [Short Interest & Squeeze Potential](#5-short-interest--squeeze-potential)
6. [SEC Filings & Regulatory Intelligence](#6-sec-filings--regulatory-intelligence)
7. [Alternative Data (True Alpha)](#7-alternative-data-true-alpha)
8. [Macro & Economic Indicators](#8-macro--economic-indicators)
9. [Sector & Rotation Signals](#9-sector--rotation-signals)
10. [AI/ML Feature Engineering](#10-aiml-feature-engineering)
11. [API Providers & Cost Analysis](#11-api-providers--cost-analysis)
---
## 1. Core Market Data (Table Stakes)
*Everyone has this. You need it, but it doesn't create an edge by itself.*
| Data Point | Why It Matters | Frequency |
|-----------|---------------|-----------|
| Real-time price (bid/ask/last) | Entry/exit timing | Tick-by-tick |
| Volume (absolute + relative) | Conviction behind moves | Tick-by-tick |
| VWAP (Volume Weighted Avg Price) | Institutional benchmark | Real-time |
| 52-week high/low | Psychological levels | Daily |
| Market cap / Float | Liquidity assessment | Daily |
| Average volume (10d/30d/90d) | Normalization baseline | Daily |
| Intraday OHLCV (1m/5m/15m/1h) | Chart patterns, entry timing | Intraday |
| Pre-market / After-hours price | Gap risk, overnight sentiment | Extended hours |
| Split/dividend-adjusted prices | Historical accuracy | Event-driven |
| Relative strength vs sector/index | Outperformance/underperformance | Daily |
---
## 2. Fundamental Data (The Foundation)
*Where the real story lives. This is where you separate investors from gamblers.*
### Income Statement
- Revenue (quarterly + YoY + QoQ growth rates)
- Gross margin, operating margin, net margin (trend analysis)
- EBITDA / EBIT
- EPS (GAAP + non-GAAP + diluted)
- R&D spend (critical for tech — shows future investment)
- SG&A as % of revenue (efficiency metric)
- Free cash flow conversion
### Balance Sheet
- Total assets/liabilities/debt
- Net debt / EBITDA ratio (solvency)
- Current ratio, quick ratio (liquidity)
- Share count changes (dilution detection)
- Cash & equivalents vs short-term debt
- Goodwill & intangible assets (quality of earnings)
### Cash Flow
- Operating cash flow (quality of earnings)
- Capex (growth vs maintenance)
- Free cash flow (FCF = OCF − Capex)
- Share buybacks (management confidence signal)
- Dividend payments & changes
### Key Ratios (Computed)
- P/E, PEG, P/S, P/B, P/FCF
- ROE, ROA, ROIC
- Debt/Equity, Interest Coverage
- Altman Z-Score (bankruptcy risk)
- Piotroski F-Score (9-factor quality score)
---
## 3. Institutional & Insider Signals
*One of the strongest predictive signals available to retail.*
### Institutional Ownership
- Top 10 holders (Vanguard, BlackRock, Fidelity, etc.)
- % of float held by institutions
- **Quarterly change in institutional ownership** (rising = bullish signal)
- 13F filings (lagged 45 days, but comprehensive)
- Hedge fund holdings (13F — track specific funds like Renaissance, Citadel, Bridgewater)
- Mutual fund net inflows/outflows
- **New positions** vs **increased positions** vs **sold positions**
### Insider Activity (FORM 4)
- **Insider buys** — strongest signal (they spend their own money)
- CEO/CFO buys are highest conviction
- Open market buys > exercise of options
- Cluster buying (multiple insiders buying) = very strong signal
- **Insider sells** — need context
- 10b5-1 plans = routine, not signal
- Unplanned sells = potential red flag
- Cluster sells = very bearish
- **Form 4 filing date vs transaction date** — speed matters
- **SEC Form 144** (proposed sales — early warning)
---
## 4. Options Flow & Sentiment
*Options market often moves before the stock. This is real-time institutional positioning.*
### Options Data
- **Unusual options activity** — volume >> open interest
- Large block trades (100+ contracts)
- Out-of-the-money calls (bullish speculation)
- Put/call ratio spikes (fear/greed)
- **Put/Call ratio** by ticker and overall market
- **Implied volatility** (IV) vs historical volatility (HV)
- IV > HV = options expensive (potential move)
- IV rank / IV percentile
- **Options chain** — max pain, gamma exposure
- **Block trades** — dark pool prints
- **Dark pool volume %**
### Sentiment Indicators
- **Put/Call ratio** breakdown (equity, index, single stock)
- **CBOE Volatility Index (VIX)** and components
- **CNN Fear & Greed Index**
- **AAII sentiment survey**
- **Google Trends** for stock/sector searches
- **Reddit/Twitter sentiment** (r/wallstreetbets, r/investing)
---
## 5. Short Interest & Squeeze Potential
*Short squeeze setups can create 100%+ moves in days.*
### Short Data
- **Short interest** (% of float)
- **Days to cover** (short ratio)
- **Short interest trend** (rising = bearish, but also squeeze fuel)
- **Squeeze probability score**:
- High short interest (>20%)
- Low float (<50M shares)
- High days-to-cover (>5)
- Rising price + volume
- Recent catalyst (earnings, FDA, product)
### Borrowing Data
- **Stock borrow fees** (high fees = hard to borrow = squeeze potential)
- **Locate availability**
- **Cost to borrow** (% annual)
---
## 6. SEC Filings & Regulatory Intelligence
*Raw regulatory filings are the most authoritative source — before analysts catch up.*
### Key Filings
- **10-K** (annual) — comprehensive financial picture
- **10-Q** (quarterly) — quarterly updates
- **8-K** (current) — material events (earnings, M&A, leadership changes)
- **DEF 14A** (proxy) — executive comp, board changes
- **S-1** / **S-3** — new offerings (dilution risk)
- **SC 13D/G** — activist positions (>5% ownership)
- **Form 4** — insider transactions (daily)
- **Form 144** — proposed insider sales
### NLP Extraction Targets
- **MD&A changes** — management commentary shifts
- **Risk factor additions** — new risks = new concerns
- **Auditor changes** — red flag if auditor resigns
- **Going concern** mentions = existential threat
- **Related party transactions** — potential tunneling
- **Segment revenue breakdown** — growth drivers
---
## 7. Alternative Data (True Alpha)
*This is where you create real edge. These are hedge fund-grade signals.*
### Consumer Behavior
| Signal | Source | Edge |
|--------|--------|------|
| App download counts | Sensor Tower, App Annie | Early revenue signal for consumer apps |
| App usage/engagement | SimilarWeb, data.ai | Retention, engagement trends |
| Web traffic | SimilarWeb, SEMrush | Interest, funnel performance |
| Credit card spend | YipitData, Earnest Research | Real-time revenue proxy |
| Grocery/retail receipts | Earnest Research | Consumer discretionary health |
| Shipping/tracking data | Project44, Descartes | Supply chain visibility, inventory |
### Corporate Activity
| Signal | Source | Edge |
|--------|--------|------|
| Job postings | Employment data, LinkedIn | Growth signaling, expansion plans |
| Job posting changes | Indeed, LinkedIn | Hiring freeze = cost cutting signal |
| Patent filings | USPTO, Google Patents | Innovation pipeline |
| Building permits | Municipal records | Physical expansion plans |
| Executive hires/leaves | LinkedIn, SEC filings | Leadership quality, stability |
| Earnings call transcripts | Seeking Alpha, Motley Fool | NLP on management tone, guidance |
### Supply Chain
| Signal | Source | Edge |
|--------|--------|------|
| Supplier revenue changes | Supplier financials | Proxy for customer demand |
| Supplier capex increases | Supplier filings | Capacity expansion = demand confidence |
| Supplier stock performance | Supplier tickers | Leading indicator for customers |
| Container shipping rates | Drewry, Clarksons | Global trade volume proxy |
| Oil/commodity prices | Bloomberg, CME | Input cost pressure |
### Sentiment & Social
| Signal | Source | Edge |
|--------|--------|------|
| Reddit sentiment | Pushshift, Reddit API | Retail sentiment extremes = contrarian |
| Twitter/X sentiment | X API | Real-time reaction, influencer moves |
| StockTwits sentiment | StockTwits API | Retail trader positioning |
| Google Trends | Google Trends API | Interest spike detection |
| YouTube/video content | YouTube Data API | Media coverage analysis |
| News sentiment | NewsAPI, GDELT | Sentiment scoring, event detection |
### Physical/Economic Proxies
| Signal | Source | Edge |
|--------|--------|------|
| Satellite imagery | Planet, Sentinel | Retail parking lots, construction |
| Credit card transaction data | YipitData, Flexport | Consumer spending in real-time |
| Mobile location data | SafeGraph, Foursquare | Foot traffic to stores |
| Energy consumption | Utility data | Industrial activity proxy |
| Water usage data | Various providers | Agricultural/industrial activity |
### Macro Indicators (Beyond the headline)
| Signal | Source | Edge |
|--------|--------|------|
| Yield curve (2s10, 3m10) | FRED, Treasury.gov | Recession predictor |
| Inverted yield curve depth/duration | FRED | Recession probability |
| Leading Economic Index (LEI) | Conference Board | 6-12 month outlook |
| PMI (ISM Manufacturing/Services) | ISM | Economic activity pulse |
| Consumer confidence | Conference Board | Consumer spending predictor |
| Jobless claims (weekly) | DOL | Labor market health |
| Initial vs continuing claims ratio | DOL | Trend vs noise |
| Building permits/housing starts | Census Bureau | Housing market leading indicator |
| Consumer credit changes | NY Fed | Consumer financial stress |
---
## 8. Macro & Economic Indicators
*For sector rotation and macro regime detection.*
### Interest Rate Environment
- Fed funds rate / Fed expectations (CME FedWatch)
- Treasury yields (2Y, 5Y, 10Y, 30Y)
- **Yield curve spread** (10Y-2Y, 10Y-3M) — recession signal
- **TED spread** (credit risk)
- **TIPS breakeven** (inflation expectations)
- SOFR, LIBOR successor rates
- **Commercial paper spreads**
- **High yield spreads** (ICE BofA HY OAS)
### Inflation
- CPI (headline + core)
- PCE (Fed's preferred measure)
- PPI (producer prices — leading indicator)
- Wage growth (average hourly earnings)
- Shelter/rent component (largest CPI component)
### Growth
- GDP growth (advance, second, final)
- Non-farm payrolls
- Unemployment rate
- ISM Manufacturing PMI (>50 = expansion)
- ISM Services PMI
- Retail sales
- Industrial production
### Currency & Commodities
- DXY (US Dollar Index)
- USD/EUR, USD/JPY, USD/CNY
- Gold (fear/deflation hedge)
- Oil (WTI/Brent — inflation, growth)
- Copper (economic activity — "Dr. Copper")
- Bitcoin (risk-on proxy)
---
## 9. Sector & Rotation Signals
*For the ETF/Index tracking and rotation detection feature.*
### ETF-Level Data
| ETF | Sector | Signal |
|-----|--------|--------|
| XLK | Technology | Tech leadership |
| XLF | Financials | Risk appetite, rate sensitivity |
| XLI | Industrial | Economic activity |
| XLY | Consumer Discretionary | Consumer confidence |
| XLP | Consumer Staples | Defensive positioning |
| XLE | Energy | Commodity cycle |
| XLV | Healthcare | Defensive, innovation |
| XLU | Utilities | Defensive, rate sensitivity |
| XLB | Materials | Cyclical, commodities |
| XLR | Real Estate | Rate sensitivity, housing |
| XLRE | Real Estate | Same as above (alternate) |
| XLG | Large Cap Growth | Growth tilt |
| XSC | Small Cap | Economic outlook (IWR alternative) |
### Rotation Indicators (The Edge)
1. **Relative Strength Score** — sector vs SPY over 20d/50d/200d
2. **RSI Divergence** — sector making new high while SPY doesn't = leadership
3. **Money Flow** — sector inflow vs outflow tracking
4. **Sector ETF spread** — XLK vs XLE ratio changing
5. **Breadth** — stocks above 50MA and 200MA within sector
6. **Volume concentration** — volume shifting to specific sectors
### Rotation Detection Algorithm
```
Rotation = when 3+ of these conditions align:
1. Sector ETF breaks above 50-day MA
2. Sector ETF RSI crosses above 50
3. Sector ETF volume > 20-day average
4. Sector's top 3 stocks outperform SPY
5. Sector relative strength vs SPY trending up (10d)
6. Institutional money flow data shows inflows
7. Analyst upgrades concentrated in sector
```
---
## 10. AI/ML Feature Engineering
*Transforming raw data into predictive features.*
### Technical Indicators (Engineered)
- Moving averages (20, 50, 100, 200 day) + crossovers
- RSI (14-day) + overbought/oversold
- MACD (12, 26, 9) + signal line crossovers
- Bollinger Bands (20, 2) + position relative to bands
- ATR (14-day) — volatility measure
- Volume MA + volume spike detection (3x avg)
- Gap analysis (pre-market gap % + fill probability)
- Support/resistance levels (pivot points, swing highs/lows)
- Fibonacci retracement levels
- Ichimoku Cloud components
### Sentiment Features
- Put/Call ratio (10-day rolling avg + spike)
- Short interest change (weekly)
- Insider buy/sell ratio (quarterly)
- Analyst rating changes (upgrades - downgrades)
- Analyst price target revisions (upgrades - downgrades)
- Social sentiment score (normalized -3 to +3)
- News sentiment (Vader/BERT-based score)
### Fundamental Features
- Revenue growth acceleration/deceleration
- Margin expansion/contraction rate
- Cash flow vs net income divergence
- Working capital changes
- Inventory turnover changes
- Days sales outstanding (DSO) changes
- Altman Z-Score trend
- Piotroski F-Score
### Composite Scores (The Real Edge)
1. **Momentum Score** (0-100) — price + volume + relative strength
2. **Value Score** (0-100) — P/E vs sector, P/B, PEG, FCF yield
3. **Quality Score** (0-100) — ROIC, margin stability, debt, FCF conversion
4. **Sentiment Score** (0-100) — insider activity + institutional flows + analyst ratings
5. **Catalyst Score** (0-100) — upcoming events, earnings proximity, news flow
6. **Risk Score** (0-100) — volatility, beta, short interest, debt
---
## 11. API Providers & Cost Analysis
### Free / Low-Cost Tier
| Provider | Data | Cost | Rate Limit |
|----------|------|------|-----------|
| **Yahoo Finance** (yfinance) | Prices, fundamentals, options | Free | ~1,000/hr |
| **Finnhub** | Real-time + fundamentals + alternatives | Free tier | 60 calls/min |
| **Alpha Vantage** | Prices, fundamentals, alternatives | Free tier | 5 calls/min |
| **FRED** | Macro/economic data | Free | None |
| **SEC EDGAR** | All filings | Free | None |
| **Quandl/Nasdaq Data** | Economic data | Free tier | Limited |
| **Polygon.io** | Real-time + options | Free tier | Limited |
### Paid Tier
| Provider | Data | Cost | Edge Level |
|----------|------|------|-----------|
| **Finnhub** Pro | All + alternatives + sentiment | $100/mo | Medium |
| **Polygon.io** | Real-time + options + fundamentals | $29-$199/mo | Medium |
| **Twelve Data** | Prices, fundamentals, crypto | $49-$299/mo | Medium |
| **Dataroma** | Institutional holdings | $50/mo | Medium |
| **InsiderMonkey** | Insider + institutional | $50-$200/mo | Medium |
| **YipitData** | Consumer spending, shipping | $500+/mo | High |
| **Seeking Alpha Pro** | Earnings transcripts, articles | $240/yr | Medium |
| **Koyfin** | Bloomberg-lite terminal | $50-$200/mo | Medium |
| **Bloomberg Terminal** | Everything | $25k/yr | Highest |
| **Refinitiv (LSEG)** | Everything | $25k+/yr | Highest |
### Recommended Stack (MVP → Scale)
**Phase 1 (MVP — Free/Low Cost):**
- `yfinance` or `Finnhub Free` — prices + fundamentals
- `FRED` — macro data (all free, official government source)
- `SEC EDGAR API` — all filings (free, official)
- `Alpha Vantage Free` — alternatives (sentiment, tech indicators)
**Phase 2 (Growth — ~$150/mo):**
- `Finnhub Pro` — real-time data + alternatives
- `Polygon.io` — options data + real-time
- `Dataroma` — clean institutional ownership data
- `Seeking Alpha Pro` — earnings transcripts
**Phase 3 (Scale — ~$500/mo):**
- `YipitData` or `Earnest Research` — consumer spending
- `Project44` or `Descartes` — supply chain
- `SimilarWeb` — web traffic
- `SafeGraph` — foot traffic
---
## Recommended Data Model (Database Schema)
### Core Tables
```sql
-- Time series price data (granular)
stock_prices (
ticker VARCHAR,
date DATE,
open DECIMAL, high DECIMAL, low DECIMAL, close DECIMAL,
volume BIGINT,
vwap DECIMAL,
source VARCHAR,
PRIMARY KEY (ticker, date, source)
);
-- Fundamental data (quarterly)
fundamentals (
ticker VARCHAR,
quarter DATE,
revenue DECIMAL,
gross_margin DECIMAL,
operating_margin DECIMAL,
net_margin DECIMAL,
eps DECIMAL,
pe_ratio DECIMAL,
market_cap DECIMAL,
pb_ratio DECIMAL,
ps_ratio DECIMAL,
roe DECIMAL,
roic DECIMAL,
debt_equity DECIMAL,
current_ratio DECIMAL,
fcf DECIMAL,
shares_outstanding BIGINT,
PRIMARY KEY (ticker, quarter)
);
-- Institutional ownership (quarterly)
institutional_holdings (
ticker VARCHAR,
date DATE,
holder_name VARCHAR,
shares BIGINT,
pct_float DECIMAL,
holding_type VARCHAR,
filing_form VARCHAR,
PRIMARY KEY (ticker, date, holder_name)
);
-- Insider transactions (daily)
insider_transactions (
ticker VARCHAR,
date DATE,
insider_name VARCHAR,
title VARCHAR,
transaction_type VARCHAR, -- BUY, SELL, EXERCISE
shares BIGINT,
price DECIMAL,
value DECIMAL,
form_4_date DATE,
PRIMARY KEY (ticker, date, insider_name, shares)
);
-- SEC filings
sec_filings (
ticker VARCHAR,
date DATE,
form_type VARCHAR, -- 10-K, 10-Q, 8-K, DEF 14A, etc.
url VARCHAR,
filing_date DATE,
period_end DATE,
nlp_summary TEXT,
nlp_sentiment DECIMAL,
risk_factors_added INT,
risk_factors_removed INT,
PRIMARY KEY (ticker, date, form_type)
);
-- Options data
options_chain (
ticker VARCHAR,
date DATE,
expiry DATE,
strike DECIMAL,
option_type VARCHAR, -- CALL, PUT
volume BIGINT,
open_interest BIGINT,
implied_vol DECIMAL,
last_price DECIMAL,
PRIMARY KEY (ticker, date, expiry, strike, option_type)
);
-- Short interest (bi-monthly)
short_interest (
ticker VARCHAR,
date DATE,
short_shares BIGINT,
float BIGINT,
short_pct FLOAT,
days_to_cover FLOAT,
borrow_fee FLOAT,
PRIMARY KEY (ticker, date)
);
-- Sector rotation (daily)
sector_performance (
ticker VARCHAR, -- ETF ticker (XLK, XLF, etc.)
date DATE,
close DECIMAL,
change_pct DECIMAL,
volume BIGINT,
rs_vs_spy DECIMAL, -- relative strength vs SPY
above_ma50 BOOLEAN,
above_ma200 BOOLEAN,
rsi_14 DECIMAL,
PRIMARY KEY (ticker, date)
);
-- Watchlists
watchlists (
id UUID PRIMARY KEY,
user_id UUID,
name VARCHAR,
created_at TIMESTAMP,
updated_at TIMESTAMP
);
watchlist_items (
watchlist_id UUID REFERENCES watchlists(id),
ticker VARCHAR,
added_at TIMESTAMP,
notes TEXT,
PRIMARY KEY (watchlist_id, ticker)
);
-- Strategies
strategies (
id UUID PRIMARY KEY,
user_id UUID,
name VARCHAR,
description TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP,
is_active BOOLEAN
);
strategy_rules (
strategy_id UUID REFERENCES strategies(id),
rule_type VARCHAR, -- TECHNICAL, FUNDAMENTAL, SENTIMENT
condition VARCHAR,
threshold DECIMAL,
direction VARCHAR, -- ABOVE, BELOW, CROSS_ABOVE, CROSS_BELOW
PRIMARY KEY (strategy_id, condition)
);
-- Alerts
alerts (
id UUID PRIMARY KEY,
user_id UUID,
watchlist_id UUID,
strategy_id UUID,
triggered_at TIMESTAMP,
ticker VARCHAR,
alert_type VARCHAR,
message TEXT,
is_read BOOLEAN,
data JSONB -- raw data that triggered the alert
);
```
---
## Data Pipeline Architecture
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ API/Scraper │───▶│ Raw Data │───▶│ Normalization │
│ (yfinance, │ │ Lake (S3/MinIO)│ │ & Enrichment │
│ FRED, EDGAR) │ │ │ │ (PostgreSQL) │
└─────────────────┘ └─────────────────┘ └─────────┬───────┘
│
▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Real-time │◀───│ Feature │◀───│ API Layer │
│ WebSocket │ │ Engineering │ │ (FastAPI) │
│ (prices, │ │ (technical, │ │ │
│ options, │ │ sentiment) │ │ │
│ alerts) │ └─────────────────┘ └─────────────────┘
└─────────────────┘
```
---
## Summary: The Edge Pyramid
```
┌─────────────────┐
│ AI/ML Signals │ ← Composite scores, anomaly detection
├─────────────────┤
│ Alternative Data │ ← Consumer spend, web traffic, jobs
├─────────────────┤
│ Institutional │ ← 13F flows, insider buys, hedge funds
├─────────────────┤
│ Options/Short │ ← Put/call, squeeze potential
├─────────────────┤
│ SEC Filings │ ← NLP on 8-Ks, MD&A changes
├─────────────────┤
│ Fundamentals │ ← Financials, ratios, quality scores
├─────────────────┤
│ Technical │ ← Charts, indicators, volume
└─────────────────┘
│ Market Data │ ← Prices, volume (table stakes)
```
**The real edge lives in layers 4-7.** Layers 1-3 everyone has. Layer 8 (AI/ML) is where you combine all signals into predictive composites.
Reference in New Issue Copy Permalink