Kafka in docker - cant consume messages from another host (python)

Question

So I am running a bitnami kafka image and cant figure out how to consume messages from another hosts. Here is the setup:

version: "3"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    container_name: zookeeper
    restart: always
    ports:
      - "2181:2181"
    volumes:
      - "./zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
     - kafka_network
  kafka:
    image: docker.io/bitnami/kafka:3.1
    container_name: kafka
    restart: always
    ports:
      - "9092:9092"
      - "9093:9093"
    volumes:
      - "./kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LOG_SEGMENT_DELETE_DELAY_MS=10000
      - KAFKA_CFG_LOG_SEGMENT_BYTES=1073741824
      - KAFKA_CFG_LOG_RETENTION_HOURS=168
      - KAFKA_CFG_LOG_RETENTION_BYTES=1073741824
      - KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS=10000
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092
    depends_on:
      - zookeeper
    networks:
      - kafka_network
  
networks:
  kafka_network:
    driver: bridge
    name: kafka_network

I have few python scripts consuming messages running in other containers on same host - works good. But now I am trying to consume messages from another host (from dockerized python script):

from kafka import KafkaConsumer

consumer = KafkaConsumer('stream', bootstrap_servers="MY_IP_ADDRESS:9092")
for msg in consumer:
    print(msg)

The above connects successfully (returns an error on port/ip change), but then it hangs infinitely without returning anything. (yes I am sure there are messages to consume)

My best guess is it is something with docker-networks and/or

- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092

parameter as this is the most common solution I've found, yet I've tried many variations of this with IP/localhost/outside&inside listeners and still can't consume any message

What am I missing?

You are correct that the advertised listener is the problem. You need to put the host's external IP there. Also, port 9093 isn't doing anything, so you should remove it from Compose file — OneCricketeer, May 03 '22 at 18:58

Kafka in docker - cant consume messages from another host (python)

0 Answers0