fix: correct FINRA URL, filename format, parser column order, and default base URL

- Fix default base URL: https://cdn.finra.org/equity/regsho/daily (was www.finra.org/sites/default/files)
- Fix filename pattern: CNMSshvol{YYYYMMDD}.txt (was CAshvol with year/month subdir)
- Fix parser: actual file columns are Date|Symbol|ShortVolume|ShortExemptVolume|TotalVolume|Market
  (was incorrectly expecting Market|Symbol|... — first col is Date, last col is Market)
- Dedupe by symbol (volume is total across all listed exchanges, not per-exchange)
- Store combined market codes (e.g. 'B,Q,N') as exchange string
- Use BEGIN/COMMIT instead of db.transaction() (node:sqlite DatabaseSync doesn't have transaction())
- Update finra_config defaults in schema.sql + migration + admin fallback
This commit is contained in:
Investor Flow Build
2026-07-25 13:14:05 -04:00
parent 8a00363dd0
commit 48872b7805
4 changed files with 50 additions and 32 deletions
+1 -1
View File
@@ -238,7 +238,7 @@ function runMigrations(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS finra_config (
id TEXT PRIMARY KEY DEFAULT 'singleton',
base_url TEXT NOT NULL DEFAULT 'https://www.finra.org/sites/default/files',
base_url TEXT NOT NULL DEFAULT 'https://cdn.finra.org/equity/regsho/daily',
updated_at TEXT NOT NULL
)
`);