pi.dev/invest-copilot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a16050c80a06e6d89814582a4e2e8a5fdc4bcbbf
invest-copilot/docs/data-edge-research.md
T

22 KiB
Raw Blame History

Invest Copilot — Data Edge Research

Created: 2025-05-26 Purpose: Identify all data sources, signals, and alternative data that create an investment "edge" — information that gives our users an informational advantage over retail competitors.

Table of Contents

  1. Core Market Data (Table Stakes)
  2. Fundamental Data (The Foundation)
  3. Institutional & Insider Signals
  4. Options Flow & Sentiment
  5. Short Interest & Squeeze Potential
  6. SEC Filings & Regulatory Intelligence
  7. Alternative Data (True Alpha)
  8. Macro & Economic Indicators
  9. Sector & Rotation Signals
  10. AI/ML Feature Engineering
  11. API Providers & Cost Analysis

1. Core Market Data (Table Stakes)

Everyone has this. You need it, but it doesn't create an edge by itself.

Data Point Why It Matters Frequency
Real-time price (bid/ask/last) Entry/exit timing Tick-by-tick
Volume (absolute + relative) Conviction behind moves Tick-by-tick
VWAP (Volume Weighted Avg Price) Institutional benchmark Real-time
52-week high/low Psychological levels Daily
Market cap / Float Liquidity assessment Daily
Average volume (10d/30d/90d) Normalization baseline Daily
Intraday OHLCV (1m/5m/15m/1h) Chart patterns, entry timing Intraday
Pre-market / After-hours price Gap risk, overnight sentiment Extended hours
Split/dividend-adjusted prices Historical accuracy Event-driven
Relative strength vs sector/index Outperformance/underperformance Daily

2. Fundamental Data (The Foundation)

Where the real story lives. This is where you separate investors from gamblers.

Income Statement

  • Revenue (quarterly + YoY + QoQ growth rates)
  • Gross margin, operating margin, net margin (trend analysis)
  • EBITDA / EBIT
  • EPS (GAAP + non-GAAP + diluted)
  • R&D spend (critical for tech — shows future investment)
  • SG&A as % of revenue (efficiency metric)
  • Free cash flow conversion

Balance Sheet

  • Total assets/liabilities/debt
  • Net debt / EBITDA ratio (solvency)
  • Current ratio, quick ratio (liquidity)
  • Share count changes (dilution detection)
  • Cash & equivalents vs short-term debt
  • Goodwill & intangible assets (quality of earnings)

Cash Flow

  • Operating cash flow (quality of earnings)
  • Capex (growth vs maintenance)
  • Free cash flow (FCF = OCF − Capex)
  • Share buybacks (management confidence signal)
  • Dividend payments & changes

Key Ratios (Computed)

  • P/E, PEG, P/S, P/B, P/FCF
  • ROE, ROA, ROIC
  • Debt/Equity, Interest Coverage
  • Altman Z-Score (bankruptcy risk)
  • Piotroski F-Score (9-factor quality score)

3. Institutional & Insider Signals

One of the strongest predictive signals available to retail.

Institutional Ownership

  • Top 10 holders (Vanguard, BlackRock, Fidelity, etc.)
  • % of float held by institutions
  • Quarterly change in institutional ownership (rising = bullish signal)
  • 13F filings (lagged 45 days, but comprehensive)
  • Hedge fund holdings (13F — track specific funds like Renaissance, Citadel, Bridgewater)
  • Mutual fund net inflows/outflows
  • New positions vs increased positions vs sold positions

Insider Activity (FORM 4)

  • Insider buys — strongest signal (they spend their own money)
    • CEO/CFO buys are highest conviction
    • Open market buys > exercise of options
    • Cluster buying (multiple insiders buying) = very strong signal
  • Insider sells — need context
    • 10b5-1 plans = routine, not signal
    • Unplanned sells = potential red flag
    • Cluster sells = very bearish
  • Form 4 filing date vs transaction date — speed matters
  • SEC Form 144 (proposed sales — early warning)

4. Options Flow & Sentiment

Options market often moves before the stock. This is real-time institutional positioning.

Options Data

  • Unusual options activity — volume >> open interest
    • Large block trades (100+ contracts)
    • Out-of-the-money calls (bullish speculation)
    • Put/call ratio spikes (fear/greed)
  • Put/Call ratio by ticker and overall market
  • Implied volatility (IV) vs historical volatility (HV)
    • IV > HV = options expensive (potential move)
    • IV rank / IV percentile
  • Options chain — max pain, gamma exposure
  • Block trades — dark pool prints
  • Dark pool volume %

Sentiment Indicators

  • Put/Call ratio breakdown (equity, index, single stock)
  • CBOE Volatility Index (VIX) and components
  • CNN Fear & Greed Index
  • AAII sentiment survey
  • Google Trends for stock/sector searches
  • Reddit/Twitter sentiment (r/wallstreetbets, r/investing)

5. Short Interest & Squeeze Potential

Short squeeze setups can create 100%+ moves in days.

Short Data

  • Short interest (% of float)
  • Days to cover (short ratio)
  • Short interest trend (rising = bearish, but also squeeze fuel)
  • Squeeze probability score:
    • High short interest (>20%)
    • Low float (<50M shares)
    • High days-to-cover (>5)
    • Rising price + volume
    • Recent catalyst (earnings, FDA, product)

Borrowing Data

  • Stock borrow fees (high fees = hard to borrow = squeeze potential)
  • Locate availability
  • Cost to borrow (% annual)

6. SEC Filings & Regulatory Intelligence

Raw regulatory filings are the most authoritative source — before analysts catch up.

Key Filings

  • 10-K (annual) — comprehensive financial picture
  • 10-Q (quarterly) — quarterly updates
  • 8-K (current) — material events (earnings, M&A, leadership changes)
  • DEF 14A (proxy) — executive comp, board changes
  • S-1 / S-3 — new offerings (dilution risk)
  • SC 13D/G — activist positions (>5% ownership)
  • Form 4 — insider transactions (daily)
  • Form 144 — proposed insider sales

NLP Extraction Targets

  • MD&A changes — management commentary shifts
  • Risk factor additions — new risks = new concerns
  • Auditor changes — red flag if auditor resigns
  • Going concern mentions = existential threat
  • Related party transactions — potential tunneling
  • Segment revenue breakdown — growth drivers

7. Alternative Data (True Alpha)

This is where you create real edge. These are hedge fund-grade signals.

Consumer Behavior

Signal Source Edge
App download counts Sensor Tower, App Annie Early revenue signal for consumer apps
App usage/engagement SimilarWeb, data.ai Retention, engagement trends
Web traffic SimilarWeb, SEMrush Interest, funnel performance
Credit card spend YipitData, Earnest Research Real-time revenue proxy
Grocery/retail receipts Earnest Research Consumer discretionary health
Shipping/tracking data Project44, Descartes Supply chain visibility, inventory

Corporate Activity

Signal Source Edge
Job postings Employment data, LinkedIn Growth signaling, expansion plans
Job posting changes Indeed, LinkedIn Hiring freeze = cost cutting signal
Patent filings USPTO, Google Patents Innovation pipeline
Building permits Municipal records Physical expansion plans
Executive hires/leaves LinkedIn, SEC filings Leadership quality, stability
Earnings call transcripts Seeking Alpha, Motley Fool NLP on management tone, guidance

Supply Chain

Signal Source Edge
Supplier revenue changes Supplier financials Proxy for customer demand
Supplier capex increases Supplier filings Capacity expansion = demand confidence
Supplier stock performance Supplier tickers Leading indicator for customers
Container shipping rates Drewry, Clarksons Global trade volume proxy
Oil/commodity prices Bloomberg, CME Input cost pressure

Sentiment & Social

Signal Source Edge
Reddit sentiment Pushshift, Reddit API Retail sentiment extremes = contrarian
Twitter/X sentiment X API Real-time reaction, influencer moves
StockTwits sentiment StockTwits API Retail trader positioning
Google Trends Google Trends API Interest spike detection
YouTube/video content YouTube Data API Media coverage analysis
News sentiment NewsAPI, GDELT Sentiment scoring, event detection

Physical/Economic Proxies

Signal Source Edge
Satellite imagery Planet, Sentinel Retail parking lots, construction
Credit card transaction data YipitData, Flexport Consumer spending in real-time
Mobile location data SafeGraph, Foursquare Foot traffic to stores
Energy consumption Utility data Industrial activity proxy
Water usage data Various providers Agricultural/industrial activity

Macro Indicators (Beyond the headline)

Signal Source Edge
Yield curve (2s10, 3m10) FRED, Treasury.gov Recession predictor
Inverted yield curve depth/duration FRED Recession probability
Leading Economic Index (LEI) Conference Board 6-12 month outlook
PMI (ISM Manufacturing/Services) ISM Economic activity pulse
Consumer confidence Conference Board Consumer spending predictor
Jobless claims (weekly) DOL Labor market health
Initial vs continuing claims ratio DOL Trend vs noise
Building permits/housing starts Census Bureau Housing market leading indicator
Consumer credit changes NY Fed Consumer financial stress

8. Macro & Economic Indicators

For sector rotation and macro regime detection.

Interest Rate Environment

  • Fed funds rate / Fed expectations (CME FedWatch)
  • Treasury yields (2Y, 5Y, 10Y, 30Y)
  • Yield curve spread (10Y-2Y, 10Y-3M) — recession signal
  • TED spread (credit risk)
  • TIPS breakeven (inflation expectations)
  • SOFR, LIBOR successor rates
  • Commercial paper spreads
  • High yield spreads (ICE BofA HY OAS)

Inflation

  • CPI (headline + core)
  • PCE (Fed's preferred measure)
  • PPI (producer prices — leading indicator)
  • Wage growth (average hourly earnings)
  • Shelter/rent component (largest CPI component)

Growth

  • GDP growth (advance, second, final)
  • Non-farm payrolls
  • Unemployment rate
  • ISM Manufacturing PMI (>50 = expansion)
  • ISM Services PMI
  • Retail sales
  • Industrial production

Currency & Commodities

  • DXY (US Dollar Index)
  • USD/EUR, USD/JPY, USD/CNY
  • Gold (fear/deflation hedge)
  • Oil (WTI/Brent — inflation, growth)
  • Copper (economic activity — "Dr. Copper")
  • Bitcoin (risk-on proxy)

9. Sector & Rotation Signals

For the ETF/Index tracking and rotation detection feature.

ETF-Level Data

ETF Sector Signal
XLK Technology Tech leadership
XLF Financials Risk appetite, rate sensitivity
XLI Industrial Economic activity
XLY Consumer Discretionary Consumer confidence
XLP Consumer Staples Defensive positioning
XLE Energy Commodity cycle
XLV Healthcare Defensive, innovation
XLU Utilities Defensive, rate sensitivity
XLB Materials Cyclical, commodities
XLR Real Estate Rate sensitivity, housing
XLRE Real Estate Same as above (alternate)
XLG Large Cap Growth Growth tilt
XSC Small Cap Economic outlook (IWR alternative)

Rotation Indicators (The Edge)

  1. Relative Strength Score — sector vs SPY over 20d/50d/200d
  2. RSI Divergence — sector making new high while SPY doesn't = leadership
  3. Money Flow — sector inflow vs outflow tracking
  4. Sector ETF spread — XLK vs XLE ratio changing
  5. Breadth — stocks above 50MA and 200MA within sector
  6. Volume concentration — volume shifting to specific sectors

Rotation Detection Algorithm

Rotation = when 3+ of these conditions align:
  1. Sector ETF breaks above 50-day MA
  2. Sector ETF RSI crosses above 50
  3. Sector ETF volume > 20-day average
  4. Sector's top 3 stocks outperform SPY
  5. Sector relative strength vs SPY trending up (10d)
  6. Institutional money flow data shows inflows
  7. Analyst upgrades concentrated in sector

10. AI/ML Feature Engineering

Transforming raw data into predictive features.

Technical Indicators (Engineered)

  • Moving averages (20, 50, 100, 200 day) + crossovers
  • RSI (14-day) + overbought/oversold
  • MACD (12, 26, 9) + signal line crossovers
  • Bollinger Bands (20, 2) + position relative to bands
  • ATR (14-day) — volatility measure
  • Volume MA + volume spike detection (3x avg)
  • Gap analysis (pre-market gap % + fill probability)
  • Support/resistance levels (pivot points, swing highs/lows)
  • Fibonacci retracement levels
  • Ichimoku Cloud components

Sentiment Features

  • Put/Call ratio (10-day rolling avg + spike)
  • Short interest change (weekly)
  • Insider buy/sell ratio (quarterly)
  • Analyst rating changes (upgrades - downgrades)
  • Analyst price target revisions (upgrades - downgrades)
  • Social sentiment score (normalized -3 to +3)
  • News sentiment (Vader/BERT-based score)

Fundamental Features

  • Revenue growth acceleration/deceleration
  • Margin expansion/contraction rate
  • Cash flow vs net income divergence
  • Working capital changes
  • Inventory turnover changes
  • Days sales outstanding (DSO) changes
  • Altman Z-Score trend
  • Piotroski F-Score

Composite Scores (The Real Edge)

  1. Momentum Score (0-100) — price + volume + relative strength
  2. Value Score (0-100) — P/E vs sector, P/B, PEG, FCF yield
  3. Quality Score (0-100) — ROIC, margin stability, debt, FCF conversion
  4. Sentiment Score (0-100) — insider activity + institutional flows + analyst ratings
  5. Catalyst Score (0-100) — upcoming events, earnings proximity, news flow
  6. Risk Score (0-100) — volatility, beta, short interest, debt

11. API Providers & Cost Analysis

Free / Low-Cost Tier

Provider Data Cost Rate Limit
Yahoo Finance (yfinance) Prices, fundamentals, options Free ~1,000/hr
Finnhub Real-time + fundamentals + alternatives Free tier 60 calls/min
Alpha Vantage Prices, fundamentals, alternatives Free tier 5 calls/min
FRED Macro/economic data Free None
SEC EDGAR All filings Free None
Quandl/Nasdaq Data Economic data Free tier Limited
Polygon.io Real-time + options Free tier Limited

Paid Tier

Provider Data Cost Edge Level
Finnhub Pro All + alternatives + sentiment $100/mo Medium
Polygon.io Real-time + options + fundamentals $29-$199/mo Medium
Twelve Data Prices, fundamentals, crypto $49-$299/mo Medium
Dataroma Institutional holdings $50/mo Medium
InsiderMonkey Insider + institutional $50-$200/mo Medium
YipitData Consumer spending, shipping $500+/mo High
Seeking Alpha Pro Earnings transcripts, articles $240/yr Medium
Koyfin Bloomberg-lite terminal $50-$200/mo Medium
Bloomberg Terminal Everything $25k/yr Highest
Refinitiv (LSEG) Everything $25k+/yr Highest

Recommended Stack (MVP → Scale)

Phase 1 (MVP — Free/Low Cost):

  • yfinance or Finnhub Free — prices + fundamentals
  • FRED — macro data (all free, official government source)
  • SEC EDGAR API — all filings (free, official)
  • Alpha Vantage Free — alternatives (sentiment, tech indicators)

Phase 2 (Growth — ~$150/mo):

  • Finnhub Pro — real-time data + alternatives
  • Polygon.io — options data + real-time
  • Dataroma — clean institutional ownership data
  • Seeking Alpha Pro — earnings transcripts

Phase 3 (Scale — ~$500/mo):

  • YipitData or Earnest Research — consumer spending
  • Project44 or Descartes — supply chain
  • SimilarWeb — web traffic
  • SafeGraph — foot traffic

Recommended Data Model (Database Schema)

Core Tables

-- Time series price data (granular)
stock_prices (
  ticker VARCHAR,
  date DATE,
  open DECIMAL, high DECIMAL, low DECIMAL, close DECIMAL,
  volume BIGINT,
  vwap DECIMAL,
  source VARCHAR,
  PRIMARY KEY (ticker, date, source)
);

-- Fundamental data (quarterly)
fundamentals (
  ticker VARCHAR,
  quarter DATE,
  revenue DECIMAL,
  gross_margin DECIMAL,
  operating_margin DECIMAL,
  net_margin DECIMAL,
  eps DECIMAL,
  pe_ratio DECIMAL,
  market_cap DECIMAL,
  pb_ratio DECIMAL,
  ps_ratio DECIMAL,
  roe DECIMAL,
  roic DECIMAL,
  debt_equity DECIMAL,
  current_ratio DECIMAL,
  fcf DECIMAL,
  shares_outstanding BIGINT,
  PRIMARY KEY (ticker, quarter)
);

-- Institutional ownership (quarterly)
institutional_holdings (
  ticker VARCHAR,
  date DATE,
  holder_name VARCHAR,
  shares BIGINT,
  pct_float DECIMAL,
  holding_type VARCHAR,
  filing_form VARCHAR,
  PRIMARY KEY (ticker, date, holder_name)
);

-- Insider transactions (daily)
insider_transactions (
  ticker VARCHAR,
  date DATE,
  insider_name VARCHAR,
  title VARCHAR,
  transaction_type VARCHAR,  -- BUY, SELL, EXERCISE
  shares BIGINT,
  price DECIMAL,
  value DECIMAL,
  form_4_date DATE,
  PRIMARY KEY (ticker, date, insider_name, shares)
);

-- SEC filings
sec_filings (
  ticker VARCHAR,
  date DATE,
  form_type VARCHAR,  -- 10-K, 10-Q, 8-K, DEF 14A, etc.
  url VARCHAR,
  filing_date DATE,
  period_end DATE,
  nlp_summary TEXT,
  nlp_sentiment DECIMAL,
  risk_factors_added INT,
  risk_factors_removed INT,
  PRIMARY KEY (ticker, date, form_type)
);

-- Options data
options_chain (
  ticker VARCHAR,
  date DATE,
  expiry DATE,
  strike DECIMAL,
  option_type VARCHAR,  -- CALL, PUT
  volume BIGINT,
  open_interest BIGINT,
  implied_vol DECIMAL,
  last_price DECIMAL,
  PRIMARY KEY (ticker, date, expiry, strike, option_type)
);

-- Short interest (bi-monthly)
short_interest (
  ticker VARCHAR,
  date DATE,
  short_shares BIGINT,
  float BIGINT,
  short_pct FLOAT,
  days_to_cover FLOAT,
  borrow_fee FLOAT,
  PRIMARY KEY (ticker, date)
);

-- Sector rotation (daily)
sector_performance (
  ticker VARCHAR,  -- ETF ticker (XLK, XLF, etc.)
  date DATE,
  close DECIMAL,
  change_pct DECIMAL,
  volume BIGINT,
  rs_vs_spy DECIMAL,  -- relative strength vs SPY
  above_ma50 BOOLEAN,
  above_ma200 BOOLEAN,
  rsi_14 DECIMAL,
  PRIMARY KEY (ticker, date)
);

-- Watchlists
watchlists (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

watchlist_items (
  watchlist_id UUID REFERENCES watchlists(id),
  ticker VARCHAR,
  added_at TIMESTAMP,
  notes TEXT,
  PRIMARY KEY (watchlist_id, ticker)
);

-- Strategies
strategies (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  description TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  is_active BOOLEAN
);

strategy_rules (
  strategy_id UUID REFERENCES strategies(id),
  rule_type VARCHAR,  -- TECHNICAL, FUNDAMENTAL, SENTIMENT
  condition VARCHAR,
  threshold DECIMAL,
  direction VARCHAR,  -- ABOVE, BELOW, CROSS_ABOVE, CROSS_BELOW
  PRIMARY KEY (strategy_id, condition)
);

-- Alerts
alerts (
  id UUID PRIMARY KEY,
  user_id UUID,
  watchlist_id UUID,
  strategy_id UUID,
  triggered_at TIMESTAMP,
  ticker VARCHAR,
  alert_type VARCHAR,
  message TEXT,
  is_read BOOLEAN,
  data JSONB  -- raw data that triggered the alert
);

Data Pipeline Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  API/Scraper    │───▶│  Raw Data       │───▶│  Normalization  │
│  (yfinance,     │    │  Lake (S3/MinIO)│    │  & Enrichment   │
│   FRED, EDGAR)  │    │                 │    │  (PostgreSQL)   │
└─────────────────┘    └─────────────────┘    └─────────┬───────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Real-time      │◀───│  Feature        │◀───│  API Layer      │
│  WebSocket      │    │  Engineering    │    │  (FastAPI)      │
│  (prices,      │    │  (technical,    │    │                 │
│   options,     │    │   sentiment)    │    │                 │
│   alerts)      │    └─────────────────┘    └─────────────────┘
└─────────────────┘

Summary: The Edge Pyramid

           ┌─────────────────┐
           │  AI/ML Signals  │    ← Composite scores, anomaly detection
           ├─────────────────┤
           │ Alternative Data │    ← Consumer spend, web traffic, jobs
           ├─────────────────┤
           │ Institutional    │    ← 13F flows, insider buys, hedge funds
           ├─────────────────┤
           │ Options/Short    │    ← Put/call, squeeze potential
           ├─────────────────┤
           │ SEC Filings      │    ← NLP on 8-Ks, MD&A changes
           ├─────────────────┤
           │ Fundamentals     │    ← Financials, ratios, quality scores
           ├─────────────────┤
           │ Technical        │    ← Charts, indicators, volume
           └─────────────────┘
           │ Market Data     │    ← Prices, volume (table stakes)

The real edge lives in layers 4-7. Layers 1-3 everyone has. Layer 8 (AI/ML) is where you combine all signals into predictive composites.

Reference in New Issue View Git Blame Copy Permalink