Apache Kafka is widely acclaimed for its robust real-time data streaming capabilities. A key factor contributing to its widespread adoption is its flexibility regarding message delivery semantics. Kafka allows developers to tailor message delivery guarantees between producers and consumers according to specific application needs. These guarantees can be categorized into three core types:

Messages may be lost at most once

• , but they are never redelivered.
• At least once: Messages are never lost, but duplicates might be delivered.
• Exactly once: Every message is delivered precisely once—ensuring no duplicates or data loss, though achieving this is more complex.

Understanding Kafka’s Message Delivery Guarantees

To grasp the essence of Kafka’s delivery guarantees, it’s crucial to consider the interaction between two key components: the producer (which publishes messages) and the consumer (which reads messages). Both sides play a vital role in ensuring the desired delivery semantics.

Producer Message Guarantees

On the producer side, Kafka guarantees message durability once a message is “committed” to the log. A message is deemed committed if at least one broker replicating the partition remains functional. This provides a strong assurance that committed messages won’t be lost, even in the event of broker failures, provided the proper replication settings are in place.

Before version 0.11.0.0, Kafka only supported “at least once” delivery semantics. Due to potential network issues, a producer might resend a message if it didn’t receive an acknowledgement from the broker, leading to duplicate entries in the log.

However, starting from version 0.11.0.0, Kafka introduced idempotent producers to address this issue. Idempotent producers ensure that even if a message is resent, it won’t result in duplicates. This is achieved by assigning each producer a unique ID and each message a sequence number, enabling Kafka brokers to detect and discard duplicate messages.

Kafka also provides transactional guarantees, enabling producers to atomically send messages across multiple partitions. This means that all messages within the transaction are successfully written, or none are—ensuring consistency, especially in use cases involving Kafka Streams where messages may be distributed across multiple topics.

Consumer Message Guarantees

On the consumer side, Kafka offers varying levels of message delivery semantics depending on how message offsets (which track the consumer’s position in the log) are managed.

• At most once: This occurs when a consumer commits its offset (i.e., its position in the log) before processing the message. If a failure occurs after the offset is committed but before the message is processed, the message will be skipped, resulting in potential data loss.

• At least once: Here, the consumer processes the message first and then commits the offset. If the consumer crashes after processing but before saving the offset, it will re-read and reprocess the same message upon restart. This results in potential duplicate processing but ensures no data is lost.

• Exactly once: Kafka’s exactly-once semantics (EOS) come into play when using Kafka Streams or its transactional producer-consumer features. In this case, offsets are stored as part of the same transaction as the consumer’s output. If the transaction fails, the offset and any side effects are rolled back, ensuring a consistent state and avoiding duplicate processing.

Choosing the Right Delivery Guarantee for Your Application

The flexibility Kafka offers allows developers to choose the appropriate delivery semantics based on their application’s tolerance for message loss, duplication, and latency.

• At most once is useful when occasional message loss is acceptable, and minimizing latency is more important than ensuring every message is processed.
• At least once is the default choice for many real-time applications where no data should be lost, though occasional duplicates can be tolerated and handled by the application.
• Exactly once is critical in scenarios requiring strict guarantees, such as financial transactions, e-commerce order processing, or any system that demands perfect data consistency.

Conclusion

Kafka’s message delivery semantics offer a versatile toolkit for building resilient, scalable, and efficient data streaming pipelines. Whether you prioritize performance, data reliability, or strict consistency, Kafka provides the necessary configurations—such as idempotent producers and transactional support—to meet a variety of use cases. By allowing developers to choose between at most once, at least once, and exactly once guarantees, Kafka ensures it can be customized to fit the specific needs of different applications, from low-latency systems to mission-critical services.