…
⚡
caller
on_pass_expiring()
→calls directly
one process · one stack
1 fetch_pass(id)
↓
2 notify_user(p)✗ throws
↓
3 delete_pass(p)
⏱
Latency — caller blocks for 1+2+3 summed
✗
Error blast radius — step 2 throws → step 3 never runs
📈
Spikes — 1M expiries at 00:00 → 1M calls at once
🔗
Coupling — caller must know & call every reaction
producer
expiry watcher
→enqueue()
← head (oldest)tail →
e0
e1
e2
e3
e4
→dequeue()
worker
1 consumer
FIFO queue. Producer just appends and returns. Worker drains at its own pace.
read msg
→
ack here = at-most-once
→
⚙ process
→
ack here = at-least-once
✗ ack on read → at-most-once
message deleted before the work. Crash mid-process → it's gone, lost forever. Fast and lossy — fine only when dropping is acceptable (sampled metrics, best-effort pings).
✓ ack after success → at-least-once
process first, ack only when done. Crash before ack → broker redelivers → may run twice → needs idempotency. The usual default.
Same queue, two guarantees — the only variable is when you ack relative to the work. No free lunch: exactly-once = at-least-once + idempotent processing.
producer
expiry watcher
→
one FIFO queue
e0
e1
e2
e3
e4
→
worker · A
same job
→
worker · B
same job
✓ ~2× throughput — each message goes to exactly one worker.
✗ no ordering across workers · ✗ same entity may run twice / concurrently
✗ no ordering across workers · ✗ same entity may run twice / concurrently
EVT #91
EVT #91 ↻ redelivered
def handle(evt): # may arrive more than once # 1 · delete is naturally idempotent (2nd time = no-op) db.execute("DELETE FROM passes WHERE id=%s", evt.pass_id) # 2 · dedupe the unsafe-to-repeat side effect first = db.execute( "INSERT INTO seen_events(id) VALUES(%s)" " ON CONFLICT DO NOTHING RETURNING 1", evt.id) if first: # True only the first time send_push(evt.user_id) # never double-notify
Delivered ×2 → one delete, one push. Redelivery is now harmless.
producer
expiry watcher
→
shared queue
e
→got it ✓
notify
→already removed ✗
delete-pass
Destructive read → consumers compete. notify acks & deletes it; delete-pass never sees the event.
producer
append-only
→append
immutable log — nothing is deleted on read
▲notify @6
▲delete @4
Two cursors, one shared copy. Reading advances a cursor — it never removes the event.
producer
expiry watcher
→
log split into 4 partitions
P0
·
·
P1
·
P2
·
·
·
P3
·
Partition = unit of parallelism. N independent, individually-ordered sub-logs. Order holds within a partition only.
topic: 4 partitions
P0
P1
P2
P3
P0,P1 ⟶
P2,P3 ⟶
consumer group: notify
consumer c1
owns P0, P1
consumer c2
owns P2, P3
consumer c3
idle — no partition left
One partition → exactly one consumer in the group. That ownership is the once-per-group guarantee. Ceiling = partition count. Offsets committed per
(group, partition).1 read offset N
→
⚙ process
→
3 commit N
✗ crash before commit
committed offset still N-1. On rebalance the partition moves to another consumer, which resumes at N → event re-delivered. at-least-once → handlers must be idempotent.
✓ commit then crash
offset already advanced past N. The new owner starts at N+1 → not re-delivered. Work is safe.
✗ commit before process
if you commit first then crash mid-work, the event is gone. at-most-once → can lose data. Rarely what you want.
Rebalance reassigns a dead consumer's partitions to survivors, each resuming from the last committed offset. Process-then-commit + idempotency = self-healing.