Files

T

gitea c70d6035cc Initial commit: invest-copilot app

2026-05-30 11:29:20 -04:00

22 KiB

Raw Permalink Blame History

Invest Copilot — Data Edge Research

Created: 2025-05-26 Purpose: Identify all data sources, signals, and alternative data that create an investment "edge" — information that gives our users an informational advantage over retail competitors.

Core Market Data (Table Stakes)
Fundamental Data (The Foundation)
Institutional & Insider Signals
Options Flow & Sentiment
Short Interest & Squeeze Potential
SEC Filings & Regulatory Intelligence
Alternative Data (True Alpha)
Macro & Economic Indicators
Sector & Rotation Signals
AI/ML Feature Engineering
API Providers & Cost Analysis

1. Core Market Data (Table Stakes)

Everyone has this. You need it, but it doesn't create an edge by itself.

Data Point	Why It Matters	Frequency
Real-time price (bid/ask/last)	Entry/exit timing	Tick-by-tick
Volume (absolute + relative)	Conviction behind moves	Tick-by-tick
VWAP (Volume Weighted Avg Price)	Institutional benchmark	Real-time
52-week high/low	Psychological levels	Daily
Market cap / Float	Liquidity assessment	Daily
Average volume (10d/30d/90d)	Normalization baseline	Daily
Intraday OHLCV (1m/5m/15m/1h)	Chart patterns, entry timing	Intraday
Pre-market / After-hours price	Gap risk, overnight sentiment	Extended hours
Split/dividend-adjusted prices	Historical accuracy	Event-driven
Relative strength vs sector/index	Outperformance/underperformance	Daily

2. Fundamental Data (The Foundation)

Where the real story lives. This is where you separate investors from gamblers.

Income Statement

Revenue (quarterly + YoY + QoQ growth rates)
Gross margin, operating margin, net margin (trend analysis)
EBITDA / EBIT
EPS (GAAP + non-GAAP + diluted)
R&D spend (critical for tech — shows future investment)
SG&A as % of revenue (efficiency metric)
Free cash flow conversion

Balance Sheet

Total assets/liabilities/debt
Net debt / EBITDA ratio (solvency)
Current ratio, quick ratio (liquidity)
Share count changes (dilution detection)
Cash & equivalents vs short-term debt
Goodwill & intangible assets (quality of earnings)

Cash Flow

Operating cash flow (quality of earnings)
Capex (growth vs maintenance)
Free cash flow (FCF = OCF − Capex)
Share buybacks (management confidence signal)
Dividend payments & changes

Key Ratios (Computed)

P/E, PEG, P/S, P/B, P/FCF
ROE, ROA, ROIC
Debt/Equity, Interest Coverage
Altman Z-Score (bankruptcy risk)
Piotroski F-Score (9-factor quality score)

3. Institutional & Insider Signals

One of the strongest predictive signals available to retail.

Institutional Ownership

Top 10 holders (Vanguard, BlackRock, Fidelity, etc.)
% of float held by institutions
Quarterly change in institutional ownership (rising = bullish signal)
13F filings (lagged 45 days, but comprehensive)
Hedge fund holdings (13F — track specific funds like Renaissance, Citadel, Bridgewater)
Mutual fund net inflows/outflows
New positions vs increased positions vs sold positions

Insider Activity (FORM 4)

Insider buys — strongest signal (they spend their own money)
- CEO/CFO buys are highest conviction
- Open market buys > exercise of options
- Cluster buying (multiple insiders buying) = very strong signal
Insider sells — need context
- 10b5-1 plans = routine, not signal
- Unplanned sells = potential red flag
- Cluster sells = very bearish
Form 4 filing date vs transaction date — speed matters
SEC Form 144 (proposed sales — early warning)

4. Options Flow & Sentiment

Options market often moves before the stock. This is real-time institutional positioning.

Options Data

Unusual options activity — volume >> open interest
- Large block trades (100+ contracts)
- Out-of-the-money calls (bullish speculation)
- Put/call ratio spikes (fear/greed)
Put/Call ratio by ticker and overall market
Implied volatility (IV) vs historical volatility (HV)
- IV > HV = options expensive (potential move)
- IV rank / IV percentile
Options chain — max pain, gamma exposure
Block trades — dark pool prints
Dark pool volume %

Sentiment Indicators

Put/Call ratio breakdown (equity, index, single stock)
CBOE Volatility Index (VIX) and components
CNN Fear & Greed Index
AAII sentiment survey
Google Trends for stock/sector searches
Reddit/Twitter sentiment (r/wallstreetbets, r/investing)

5. Short Interest & Squeeze Potential

Short squeeze setups can create 100%+ moves in days.

Short Data

Short interest (% of float)
Days to cover (short ratio)
Short interest trend (rising = bearish, but also squeeze fuel)
Squeeze probability score:
- High short interest (>20%)
- Low float (<50M shares)
- High days-to-cover (>5)
- Rising price + volume
- Recent catalyst (earnings, FDA, product)

Borrowing Data

Stock borrow fees (high fees = hard to borrow = squeeze potential)
Locate availability
Cost to borrow (% annual)

6. SEC Filings & Regulatory Intelligence

Raw regulatory filings are the most authoritative source — before analysts catch up.

Key Filings

10-K (annual) — comprehensive financial picture
10-Q (quarterly) — quarterly updates
8-K (current) — material events (earnings, M&A, leadership changes)
DEF 14A (proxy) — executive comp, board changes
S-1 / S-3 — new offerings (dilution risk)
SC 13D/G — activist positions (>5% ownership)
Form 4 — insider transactions (daily)
Form 144 — proposed insider sales

NLP Extraction Targets

MD&A changes — management commentary shifts
Risk factor additions — new risks = new concerns
Auditor changes — red flag if auditor resigns
Going concern mentions = existential threat
Related party transactions — potential tunneling
Segment revenue breakdown — growth drivers

7. Alternative Data (True Alpha)

This is where you create real edge. These are hedge fund-grade signals.

Consumer Behavior

Signal	Source	Edge
App download counts	Sensor Tower, App Annie	Early revenue signal for consumer apps
App usage/engagement	SimilarWeb, data.ai	Retention, engagement trends
Web traffic	SimilarWeb, SEMrush	Interest, funnel performance
Credit card spend	YipitData, Earnest Research	Real-time revenue proxy
Grocery/retail receipts	Earnest Research	Consumer discretionary health
Shipping/tracking data	Project44, Descartes	Supply chain visibility, inventory

Corporate Activity

Signal	Source	Edge
Job postings	Employment data, LinkedIn	Growth signaling, expansion plans
Job posting changes	Indeed, LinkedIn	Hiring freeze = cost cutting signal
Patent filings	USPTO, Google Patents	Innovation pipeline
Building permits	Municipal records	Physical expansion plans
Executive hires/leaves	LinkedIn, SEC filings	Leadership quality, stability
Earnings call transcripts	Seeking Alpha, Motley Fool	NLP on management tone, guidance

Supply Chain

Signal	Source	Edge
Supplier revenue changes	Supplier financials	Proxy for customer demand
Supplier capex increases	Supplier filings	Capacity expansion = demand confidence
Supplier stock performance	Supplier tickers	Leading indicator for customers
Container shipping rates	Drewry, Clarksons	Global trade volume proxy
Oil/commodity prices	Bloomberg, CME	Input cost pressure

Signal	Source	Edge
Reddit sentiment	Pushshift, Reddit API	Retail sentiment extremes = contrarian
Twitter/X sentiment	X API	Real-time reaction, influencer moves
StockTwits sentiment	StockTwits API	Retail trader positioning
Google Trends	Google Trends API	Interest spike detection
YouTube/video content	YouTube Data API	Media coverage analysis
News sentiment	NewsAPI, GDELT	Sentiment scoring, event detection

Physical/Economic Proxies

Signal	Source	Edge
Satellite imagery	Planet, Sentinel	Retail parking lots, construction
Credit card transaction data	YipitData, Flexport	Consumer spending in real-time
Mobile location data	SafeGraph, Foursquare	Foot traffic to stores
Energy consumption	Utility data	Industrial activity proxy
Water usage data	Various providers	Agricultural/industrial activity

Macro Indicators (Beyond the headline)

Signal	Source	Edge
Yield curve (2s10, 3m10)	FRED, Treasury.gov	Recession predictor
Inverted yield curve depth/duration	FRED	Recession probability
Leading Economic Index (LEI)	Conference Board	6-12 month outlook
PMI (ISM Manufacturing/Services)	ISM	Economic activity pulse
Consumer confidence	Conference Board	Consumer spending predictor
Jobless claims (weekly)	DOL	Labor market health
Initial vs continuing claims ratio	DOL	Trend vs noise
Building permits/housing starts	Census Bureau	Housing market leading indicator
Consumer credit changes	NY Fed	Consumer financial stress

8. Macro & Economic Indicators

For sector rotation and macro regime detection.

Interest Rate Environment

Fed funds rate / Fed expectations (CME FedWatch)
Treasury yields (2Y, 5Y, 10Y, 30Y)
Yield curve spread (10Y-2Y, 10Y-3M) — recession signal
TED spread (credit risk)
TIPS breakeven (inflation expectations)
SOFR, LIBOR successor rates
Commercial paper spreads
High yield spreads (ICE BofA HY OAS)

Inflation

CPI (headline + core)
PCE (Fed's preferred measure)
PPI (producer prices — leading indicator)
Wage growth (average hourly earnings)
Shelter/rent component (largest CPI component)

Growth

GDP growth (advance, second, final)
Non-farm payrolls
Unemployment rate
ISM Manufacturing PMI (>50 = expansion)
ISM Services PMI
Retail sales
Industrial production

Currency & Commodities

DXY (US Dollar Index)
USD/EUR, USD/JPY, USD/CNY
Gold (fear/deflation hedge)
Oil (WTI/Brent — inflation, growth)
Copper (economic activity — "Dr. Copper")
Bitcoin (risk-on proxy)

9. Sector & Rotation Signals

For the ETF/Index tracking and rotation detection feature.

ETF-Level Data

ETF	Sector	Signal
XLK	Technology	Tech leadership
XLF	Financials	Risk appetite, rate sensitivity
XLI	Industrial	Economic activity
XLY	Consumer Discretionary	Consumer confidence
XLP	Consumer Staples	Defensive positioning
XLE	Energy	Commodity cycle
XLV	Healthcare	Defensive, innovation
XLU	Utilities	Defensive, rate sensitivity
XLB	Materials	Cyclical, commodities
XLR	Real Estate	Rate sensitivity, housing
XLRE	Real Estate	Same as above (alternate)
XLG	Large Cap Growth	Growth tilt
XSC	Small Cap	Economic outlook (IWR alternative)

Rotation Indicators (The Edge)

Relative Strength Score — sector vs SPY over 20d/50d/200d
RSI Divergence — sector making new high while SPY doesn't = leadership
Money Flow — sector inflow vs outflow tracking
Sector ETF spread — XLK vs XLE ratio changing
Breadth — stocks above 50MA and 200MA within sector
Volume concentration — volume shifting to specific sectors

Rotation Detection Algorithm

Rotation = when 3+ of these conditions align:
  1. Sector ETF breaks above 50-day MA
  2. Sector ETF RSI crosses above 50
  3. Sector ETF volume > 20-day average
  4. Sector's top 3 stocks outperform SPY
  5. Sector relative strength vs SPY trending up (10d)
  6. Institutional money flow data shows inflows
  7. Analyst upgrades concentrated in sector

10. AI/ML Feature Engineering

Transforming raw data into predictive features.

Technical Indicators (Engineered)

Moving averages (20, 50, 100, 200 day) + crossovers
RSI (14-day) + overbought/oversold
MACD (12, 26, 9) + signal line crossovers
Bollinger Bands (20, 2) + position relative to bands
ATR (14-day) — volatility measure
Volume MA + volume spike detection (3x avg)
Gap analysis (pre-market gap % + fill probability)
Support/resistance levels (pivot points, swing highs/lows)
Fibonacci retracement levels
Ichimoku Cloud components

Sentiment Features

Put/Call ratio (10-day rolling avg + spike)
Short interest change (weekly)
Insider buy/sell ratio (quarterly)
Analyst rating changes (upgrades - downgrades)
Analyst price target revisions (upgrades - downgrades)
Social sentiment score (normalized -3 to +3)
News sentiment (Vader/BERT-based score)

Fundamental Features

Revenue growth acceleration/deceleration
Margin expansion/contraction rate
Cash flow vs net income divergence
Working capital changes
Inventory turnover changes
Days sales outstanding (DSO) changes
Altman Z-Score trend
Piotroski F-Score

Composite Scores (The Real Edge)

Momentum Score (0-100) — price + volume + relative strength
Value Score (0-100) — P/E vs sector, P/B, PEG, FCF yield
Quality Score (0-100) — ROIC, margin stability, debt, FCF conversion
Sentiment Score (0-100) — insider activity + institutional flows + analyst ratings
Catalyst Score (0-100) — upcoming events, earnings proximity, news flow
Risk Score (0-100) — volatility, beta, short interest, debt

11. API Providers & Cost Analysis

Free / Low-Cost Tier

Provider	Data	Cost	Rate Limit
Yahoo Finance (yfinance)	Prices, fundamentals, options	Free	~1,000/hr
Finnhub	Real-time + fundamentals + alternatives	Free tier	60 calls/min
Alpha Vantage	Prices, fundamentals, alternatives	Free tier	5 calls/min
FRED	Macro/economic data	Free	None
SEC EDGAR	All filings	Free	None
Quandl/Nasdaq Data	Economic data	Free tier	Limited
Polygon.io	Real-time + options	Free tier	Limited

Paid Tier

Provider	Data	Cost	Edge Level
Finnhub Pro	All + alternatives + sentiment	$100/mo	Medium
Polygon.io	Real-time + options + fundamentals	$29-$199/mo	Medium
Twelve Data	Prices, fundamentals, crypto	$49-$299/mo	Medium
Dataroma	Institutional holdings	$50/mo	Medium
InsiderMonkey	Insider + institutional	$50-$200/mo	Medium
YipitData	Consumer spending, shipping	$500+/mo	High
Seeking Alpha Pro	Earnings transcripts, articles	$240/yr	Medium
Koyfin	Bloomberg-lite terminal	$50-$200/mo	Medium
Bloomberg Terminal	Everything	$25k/yr	Highest
Refinitiv (LSEG)	Everything	$25k+/yr	Highest

Recommended Stack (MVP → Scale)

Phase 1 (MVP — Free/Low Cost):

yfinance or Finnhub Free — prices + fundamentals
FRED — macro data (all free, official government source)
SEC EDGAR API — all filings (free, official)
Alpha Vantage Free — alternatives (sentiment, tech indicators)

Phase 2 (Growth — ~$150/mo):

Finnhub Pro — real-time data + alternatives
Polygon.io — options data + real-time
Dataroma — clean institutional ownership data
Seeking Alpha Pro — earnings transcripts

Phase 3 (Scale — ~$500/mo):

YipitData or Earnest Research — consumer spending
Project44 or Descartes — supply chain
SimilarWeb — web traffic
SafeGraph — foot traffic

Recommended Data Model (Database Schema)

Core Tables

-- Time series price data (granular)
stock_prices (
  ticker VARCHAR,
  date DATE,
  open DECIMAL, high DECIMAL, low DECIMAL, close DECIMAL,
  volume BIGINT,
  vwap DECIMAL,
  source VARCHAR,
  PRIMARY KEY (ticker, date, source)
);

-- Fundamental data (quarterly)
fundamentals (
  ticker VARCHAR,
  quarter DATE,
  revenue DECIMAL,
  gross_margin DECIMAL,
  operating_margin DECIMAL,
  net_margin DECIMAL,
  eps DECIMAL,
  pe_ratio DECIMAL,
  market_cap DECIMAL,
  pb_ratio DECIMAL,
  ps_ratio DECIMAL,
  roe DECIMAL,
  roic DECIMAL,
  debt_equity DECIMAL,
  current_ratio DECIMAL,
  fcf DECIMAL,
  shares_outstanding BIGINT,
  PRIMARY KEY (ticker, quarter)
);

-- Institutional ownership (quarterly)
institutional_holdings (
  ticker VARCHAR,
  date DATE,
  holder_name VARCHAR,
  shares BIGINT,
  pct_float DECIMAL,
  holding_type VARCHAR,
  filing_form VARCHAR,
  PRIMARY KEY (ticker, date, holder_name)
);

-- Insider transactions (daily)
insider_transactions (
  ticker VARCHAR,
  date DATE,
  insider_name VARCHAR,
  title VARCHAR,
  transaction_type VARCHAR,  -- BUY, SELL, EXERCISE
  shares BIGINT,
  price DECIMAL,
  value DECIMAL,
  form_4_date DATE,
  PRIMARY KEY (ticker, date, insider_name, shares)
);

-- SEC filings
sec_filings (
  ticker VARCHAR,
  date DATE,
  form_type VARCHAR,  -- 10-K, 10-Q, 8-K, DEF 14A, etc.
  url VARCHAR,
  filing_date DATE,
  period_end DATE,
  nlp_summary TEXT,
  nlp_sentiment DECIMAL,
  risk_factors_added INT,
  risk_factors_removed INT,
  PRIMARY KEY (ticker, date, form_type)
);

-- Options data
options_chain (
  ticker VARCHAR,
  date DATE,
  expiry DATE,
  strike DECIMAL,
  option_type VARCHAR,  -- CALL, PUT
  volume BIGINT,
  open_interest BIGINT,
  implied_vol DECIMAL,
  last_price DECIMAL,
  PRIMARY KEY (ticker, date, expiry, strike, option_type)
);

-- Short interest (bi-monthly)
short_interest (
  ticker VARCHAR,
  date DATE,
  short_shares BIGINT,
  float BIGINT,
  short_pct FLOAT,
  days_to_cover FLOAT,
  borrow_fee FLOAT,
  PRIMARY KEY (ticker, date)
);

-- Sector rotation (daily)
sector_performance (
  ticker VARCHAR,  -- ETF ticker (XLK, XLF, etc.)
  date DATE,
  close DECIMAL,
  change_pct DECIMAL,
  volume BIGINT,
  rs_vs_spy DECIMAL,  -- relative strength vs SPY
  above_ma50 BOOLEAN,
  above_ma200 BOOLEAN,
  rsi_14 DECIMAL,
  PRIMARY KEY (ticker, date)
);

-- Watchlists
watchlists (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

watchlist_items (
  watchlist_id UUID REFERENCES watchlists(id),
  ticker VARCHAR,
  added_at TIMESTAMP,
  notes TEXT,
  PRIMARY KEY (watchlist_id, ticker)
);

-- Strategies
strategies (
  id UUID PRIMARY KEY,
  user_id UUID,
  name VARCHAR,
  description TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  is_active BOOLEAN
);

strategy_rules (
  strategy_id UUID REFERENCES strategies(id),
  rule_type VARCHAR,  -- TECHNICAL, FUNDAMENTAL, SENTIMENT
  condition VARCHAR,
  threshold DECIMAL,
  direction VARCHAR,  -- ABOVE, BELOW, CROSS_ABOVE, CROSS_BELOW
  PRIMARY KEY (strategy_id, condition)
);

-- Alerts
alerts (
  id UUID PRIMARY KEY,
  user_id UUID,
  watchlist_id UUID,
  strategy_id UUID,
  triggered_at TIMESTAMP,
  ticker VARCHAR,
  alert_type VARCHAR,
  message TEXT,
  is_read BOOLEAN,
  data JSONB  -- raw data that triggered the alert
);

Data Pipeline Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  API/Scraper    │───▶│  Raw Data       │───▶│  Normalization  │
│  (yfinance,     │    │  Lake (S3/MinIO)│    │  & Enrichment   │
│   FRED, EDGAR)  │    │                 │    │  (PostgreSQL)   │
└─────────────────┘    └─────────────────┘    └─────────┬───────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Real-time      │◀───│  Feature        │◀───│  API Layer      │
│  WebSocket      │    │  Engineering    │    │  (FastAPI)      │
│  (prices,      │    │  (technical,    │    │                 │
│   options,     │    │   sentiment)    │    │                 │
│   alerts)      │    └─────────────────┘    └─────────────────┘
└─────────────────┘

Summary: The Edge Pyramid

           ┌─────────────────┐
           │  AI/ML Signals  │    ← Composite scores, anomaly detection
           ├─────────────────┤
           │ Alternative Data │    ← Consumer spend, web traffic, jobs
           ├─────────────────┤
           │ Institutional    │    ← 13F flows, insider buys, hedge funds
           ├─────────────────┤
           │ Options/Short    │    ← Put/call, squeeze potential
           ├─────────────────┤
           │ SEC Filings      │    ← NLP on 8-Ks, MD&A changes
           ├─────────────────┤
           │ Fundamentals     │    ← Financials, ratios, quality scores
           ├─────────────────┤
           │ Technical        │    ← Charts, indicators, volume
           └─────────────────┘
           │ Market Data     │    ← Prices, volume (table stakes)

The real edge lives in layers 4-7. Layers 1-3 everyone has. Layer 8 (AI/ML) is where you combine all signals into predictive composites.

22 KiB Raw Permalink Blame History Unescape Escape