11


Whats is considered best-practise when creating topics for Apache Kafka?
Does everyone allow automatic creation of topics or how do you do it? Do you bundle the topic-creation-step with the starting of the kafka-instance?

I have a docker-based Kafka-installation which is gone be used by multiple applications. How can I keep the topic-creation for every application separate from the startup of the Kafka-container?. Looking at Confluents music-demo they create the topics by spinning up a new kafka-image, calling the "create-topic-script" and then leave the container to die. This feels abit "hacky" but maby its the only way?

Regards

Jiinxy
  • 399
  • 2
  • 4
  • 16

3 Answers3

14

Whats is considered best-practise when creating topics for Apache Kafka? Does everyone allow automatic creation of topics or how do you do it?

It depends on what you're doing. You can definitely use topic auto creation, but then the automatically created topics will have the default broker-wide configuration in terms of partitions and replication factor.

For Kafka Streams, a Confluent engineer writes that manually creating topics before starting the application is recommended:

I also want to point out, that it is highly recommended to not use auto topic create for Streams, but to manually create all input/output topics before you start your Streams application.

For more details, see http://docs.confluent.io/current/streams/developer-guide.html#managing-topics-of-a-kafka-streams-application

With regard to:

Do you bundle the topic-creation-step with the starting of the kafka-instance?

Yes. If you have a Java application, you can use AdminClient in the main method of your application before you start your application. If you have some other kind of application, you can run an init script that calls bin/kafka-topics.sh before your application. If you're using Kubernetes, you can use a Kubernetes Init Container. But there are obviously any number of ways you can do this.

This feels abit "hacky" but maby its the only way?

I don't think this is hacky. Pretty normal to have init steps, I think.

Finally, also note that you may want to configure the retention policy on your topics. This can be done with broker-wide defaults or on a per-topic basis: https://stackoverflow.com/a/48504305/741970.

Dmitry Minkovsky
  • 33,253
  • 26
  • 106
  • 152
  • Thank you for that excellent answer. Currently it works using the `bin/kafka-topics.sh` and maby I just have to get used to the idea. I will definitly look into the scripting using `AdminClient`. – Jiinxy Jan 30 '18 at 09:24
2

You have two ways to create a Kafka topic, each one depends on your needs :

  • Set the property auto.create.topics.enable to true (it should be by default), and then the topic will be created when a value is published to the broker. Then be sure to check the following properties : default.replication.factor for the default number of replicas of the created topic, and num.partitions for the default number of partitions
  • Create manually a topic with the provided with Kafka utility.

This is the official way to manually create a topic :

bin/kafka-topics.sh --create --zookeeper <your_zookeeper_host>:2181 --replication-factor <number_of_replicas> --partitions <number_of_partitions> --topic <name_of_your_topic>
bachrc
  • 977
  • 1
  • 10
  • 19
  • Yes these were the options that I identified aswell. The questions was mostly which one to use. But as you say it probably comes down to needs. @Dimitry also pointed out the possibility to use `AdminClient`which I will look into. – Jiinxy Jan 30 '18 at 09:28
1

Another option to manage topics is to take a declarative, git-ops approach. This separates topic management from the runtime of applications completely. Whether this constitutes "best practice," or not, I think depends on the situation. For some use-cases/teams this can work very well.

See the following question/answer for tooling supporting this approach.

How to declaratively manage Kafka topics?

peterevans
  • 24,033
  • 6
  • 67
  • 72