Choosing the right cleanup policy in Kafka configuration

Question

I am using kafka_2.10-0.10.0.1. I created the topics with 1 partitions. I know that the default cleanup policy is "delete". I want to keep all the records in the topic all the time (without deleting any record). What's the right way : make "log.cleaner.enable=false" or "log.cleanup.policy=compact"?

Answer 1

Topics have broker-wide configs that apply by default to any topic that doesn't have a config, but topics can also have topic-specific configs that override or complement broker-wide topic configs.

Broker-wide topic configs are set in your service.properties file. Topic specific configs are set using the bin/kafka-topics.sh script or AdminClient if you are using Java.

The relevant broker-wide config for you is log.retention.ms and the equivalent topic-specific config is retention.ms. If you set log.retention.ms to -1, all topics without this the retention.ms config will have an infinite retention period. Likewise, if you set -1 for retention.ms on a specific topic, it will have an infinite retention period.

To set retention.ms on a new topic:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
  --partitions 20 --replication-factor 3 --config retention.ms=-1

You can also modify an existing topic to set retention.ms to -1:

> bin/kafka-configs.sh --zookeeper zk_host:port/chroot --entity-type topics --entity-name my_topic_name --alter --add-config retention.ms=-1

See the full list of topic-specific configs here: https://kafka.apache.org/documentation/#topicconfigs and more about topic operations: https://kafka.apache.org/documentation/#basic_ops