gitflarePublicPartly agent-ready50
cf-d1-queues-pulse
Event analytics on Cloudflare D1 and Queues: ingestion API, batched consumer, rollups, live dashboard, load generator, dead-letter view.
- LicenseApache-2.0
- LanguageTypeScript
- Last landed
8a65841Oct 20, 2026
8a65841Architecture
Flow
POST /api/events validates events, stamps ts, and calls sendBatch on the EVENTS queue; it returns 202 with accepted/rejected counts.
Batching
The consumer gets up to 100 messages or waits 2 s. Good messages are aggregated in memory per (minute, type, path, country), so N events become far fewer rows. All upserts go in a single D1 batch(), which is one round trip and atomic.
Idempotent rollup upserts
Rows use INSERT ... ON CONFLICT(minute,type,path,country) DO UPDATE SET count = count + excluded.count, sum = sum + excluded.sum. Concurrent consumers merge correctly. Acks happen only after the batch succeeds. If the batch fails, messages are retried. Delivery is at-least-once, so a crash between commit and ack can double-count a batch. Rollups are approximate counters, not a ledger.
Retries and the DLQ
Poison messages (failing validation) call msg.retry(). After max_retries: 3 Queues moves them to pulse-events-dlq. A DLQ consumer stores body, error and attempts in D1 dead_letters. /dlq lists them. Replay re-enqueues to EVENTS and deletes the row; purge deletes all.
Percentiles and lag
value_samples keeps a capped reservoir of values per minute; p50/p95 are computed from it. Consumer lag is now minus the newest event ts.
Retention
An hourly cron deletes rollups older than 7 days.