exactly once semantics in kafka (atomic broadcast)

Exactly once semantics in Kafka are defined in terms of atomic broadcast:

Messages are published and they will be delivered one time exactly by one or more receiving applications.

Design docs:

Notes

Original impossibility of consensus with one faulty process describes failures within extremely limited scenarios
few main problems:
- If the producer writes to the log but doesn’t get an ack
  - solved by the idempotence of kafka, producer can write as much as it wants (and kafka will just drop duplicates)
- consumer reads from the log but doesn’t update its offset
  - Consumer ensures the dervied states it creates and the offsets in the upstream need to stay in sync
  - so we can:
    - store offsets in the same db as derive state, and only update them as a transaction, restarting reads current offset from the db
    - write both state updates and offsets in idempotent fashion
  - not technically “exactly once”, more like “effectively once”
- leverages kafka streams
  - 3 components:
    - log compaction in kafka allows it to be used for a journal and snapshot state changes
    - reading data from kafka = advancing offsets
    - writing to kafka
  - wrapping all 3 in a kafka stream as a whole transaction allows for the atomic broadcasts