111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
|
# Adding a vendor integration
|
||
|
|
|
||
|
|
Rate limits are a **first-class product constraint** (ADR-0009). Every external
|
||
|
|
vendor - existing and future - must share the same process-wide gate.
|
||
|
|
|
||
|
|
If a new integration can call the network without `vendorGate`, that is a bug.
|
||
|
|
|
||
|
|
## Required steps
|
||
|
|
|
||
|
|
### 1. Register the family (before any traffic)
|
||
|
|
|
||
|
|
```ts
|
||
|
|
import { registerVendorIntegration } from '../services/vendorGate.ts';
|
||
|
|
|
||
|
|
registerVendorIntegration({
|
||
|
|
family: 'polygon', // unique budget name
|
||
|
|
sourceKinds: ['polygon'], // AdapterQueue source_kind(s)
|
||
|
|
policy: {
|
||
|
|
minIntervalMs: 200, // gap between completed calls
|
||
|
|
maxInflight: 1, // 1 = single-flight
|
||
|
|
drainJobBudget: 2, // max jobs per drain cycle
|
||
|
|
hostPattern: 'polygon\\.io', // docs / optional default allowlist
|
||
|
|
},
|
||
|
|
});
|
||
|
|
```
|
||
|
|
|
||
|
|
Call this at process startup (e.g. next to adapter registration in `index.ts`)
|
||
|
|
or inside the adapter module top-level so importing the adapter registers it.
|
||
|
|
|
||
|
|
### 2. Extend SourceKind if needed
|
||
|
|
|
||
|
|
Add the string to `SourceKind` in `app/server/src/cache/CacheRepository.ts`.
|
||
|
|
|
||
|
|
### 3. Implement the adapter (prefer base class)
|
||
|
|
|
||
|
|
```ts
|
||
|
|
import { VendorSourceAdapter, type FetchResult } from './SourceAdapter.ts';
|
||
|
|
import { vendorFetch } from '../services/vendorGate.ts';
|
||
|
|
|
||
|
|
export class PolygonAdapter extends VendorSourceAdapter {
|
||
|
|
readonly sourceKind = 'polygon' as const;
|
||
|
|
|
||
|
|
protected async fetchOneUngated(key: CacheKey): Promise<FetchResult> {
|
||
|
|
// All HTTP:
|
||
|
|
const resp = await vendorFetch('polygon', url, {
|
||
|
|
hostAllowlist: /polygon\.io/i,
|
||
|
|
});
|
||
|
|
// Library SDKs:
|
||
|
|
// return withVendorGate('polygon', () => client.get(...));
|
||
|
|
// but VendorSourceAdapter already gates the whole job — do not double-gate
|
||
|
|
// unless you make additional calls from a service outside fetchOne.
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Helpers:
|
||
|
|
|
||
|
|
| API | Use when |
|
||
|
|
|-----|----------|
|
||
|
|
| `VendorSourceAdapter` | New queue adapter (auto-gates `fetchOne`) |
|
||
|
|
| `defineVendorAdapter({...})` | Tiny one-off adapter without a class |
|
||
|
|
| `withVendorGate(family, fn)` | SDK / subprocess calls |
|
||
|
|
| `vendorFetch(family, url, opts)` | Raw HTTP |
|
||
|
|
| `secHttp` / `secFetch` | SEC only (UA + host rules) |
|
||
|
|
|
||
|
|
### 4. Register with AdapterQueue
|
||
|
|
|
||
|
|
```ts
|
||
|
|
adapters.set('polygon', new PolygonAdapter());
|
||
|
|
// Constructor throws if 'polygon' was not bindSourceKind'd.
|
||
|
|
```
|
||
|
|
|
||
|
|
### 5. Schedule (optional)
|
||
|
|
|
||
|
|
Add default interval in `SCHEDULE_INTERVALS` / `sourceRatePolicy` if the source
|
||
|
|
should refresh on a timer.
|
||
|
|
|
||
|
|
## What AdapterQueue enforces
|
||
|
|
|
||
|
|
- **Construction:** every adapter `source_kind` must have a family binding.
|
||
|
|
- **Drain:** at most `drainJobBudget` jobs per family per cycle.
|
||
|
|
- **429/403:** cools **all** source_kinds in that family + process-wide gate.
|
||
|
|
- **Heal / schedules:** skip family while cooling.
|
||
|
|
|
||
|
|
## What CI enforces
|
||
|
|
|
||
|
|
`src/services/__tests__/vendorHttpGuard.test.ts`:
|
||
|
|
|
||
|
|
- Fails if bare `fetch(` appears under `adapters/`, `services/`, `macro/`, `mirror/`
|
||
|
|
outside allowlisted gate modules.
|
||
|
|
- Covers runtime registration of a fictional future vendor.
|
||
|
|
|
||
|
|
## Checklist for PR review
|
||
|
|
|
||
|
|
- [ ] `registerVendorIntegration` (or family + bind) present
|
||
|
|
- [ ] `SourceKind` updated
|
||
|
|
- [ ] No bare `fetch` / ungated SDK in the adapter
|
||
|
|
- [ ] `hostAllowlist` set for HTTP
|
||
|
|
- [ ] Rate-limit errors surface as thrown messages matching `isRateLimitError`
|
||
|
|
- [ ] Stale/cached path preferred on UI (never stampede from clicks)
|
||
|
|
|
||
|
|
## Anti-patterns
|
||
|
|
|
||
|
|
| Don't | Do |
|
||
|
|
|-------|-----|
|
||
|
|
| `await fetch(vendorUrl)` in an adapter | `vendorFetch(family, url, { hostAllowlist })` |
|
||
|
|
| New `TokenBucket` per adapter | Shared `vendorGate` policy |
|
||
|
|
| Cool only one `source_kind` on 429 | Family cool-down (automatic if gated) |
|
||
|
|
| Call vendor from tRPC handler live | Cache / queue / static fallback |
|
||
|
|
| Skip registration "just for a prototype" | Register with strict policy even for prototypes |
|