0

I'm fairly new to kafka and Graylog. I setup Log4j2(Spring Boot) with Kafka, so it sends logger messages to the given topic. After I confirmed that messages are there I went to Graylog. I configured Raw / Plaintext Kafka in Graylog to consume from the topic. Graylog connects to Kafka successfully and then tries to consume these messages, but fails.

I'm using docker-compose to start everything up.

docker-compose.yml:

version: '3.7'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  mongodb:
    image: mongo:latest
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  graylog:
    image: graylog/graylog:4.2
    environment:
      - GRAYLOG_PASSWORD_SECRET=mfyz11sanane22banane
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000 # Graylog web interface and REST API
      - 5555:5555 # Raw/Plaintext TCP
      - 1514:1514 # Syslog TCP
      - 1514:1514/udp # Syslog UDP
      - 12201:12201 # GELF TCP
      - 12201:12201/udp # GELF UDP

Graylog error:

18 16:44:10,267 INFO : org.graylog.shaded.kafka09.client.ClientUtils$ - Fetching metadata from broker BrokerEndPoint(1001,localhost,9092) with correlation id 163 for 1 topic(s) Set(audit)

2021-11-18 16:44:10,267 INFO : org.graylog.shaded.kafka09.producer.SyncProducer - Connected to localhost:9092 for producing

2021-11-18 16:44:10,267 INFO : org.graylog.shaded.kafka09.producer.SyncProducer - Disconnecting from localhost:9092

2021-11-18 16:44:10,267 WARN : org.graylog.shaded.kafka09.client.ClientUtils$ - Fetching topic metadata with correlation id 163 for topics [Set(audit)] from broker [BrokerEndPoint(1001,localhost,9092)] failed

java.nio.channels.ClosedChannelException: null

at org.graylog.shaded.kafka09.network.BlockingChannel.send(BlockingChannel.scala:110) ~[graylog.jar:?]

at org.graylog.shaded.kafka09.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75) ~[graylog.jar:?]

at org.graylog.shaded.kafka09.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74) ~[graylog.jar:?]

at org.graylog.shaded.kafka09.producer.SyncProducer.send(SyncProducer.scala:119) ~[graylog.jar:?]

at org.graylog.shaded.kafka09.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) [graylog.jar:?]

at org.graylog.shaded.kafka09.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) [graylog.jar:?]

at org.graylog.shaded.kafka09.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) [graylog.jar:?]

at org.graylog.shaded.kafka09.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [graylog.jar:?]

2021-11-18 16:44:10,268 INFO : org.graylog.shaded.kafka09.producer.SyncProducer - Disconnecting from localhost:9092

2021-11-18 16:44:10,268 WARN : org.graylog.shaded.kafka09.consumer.ConsumerFetcherManager$LeaderFinderThread - [graylog2_c37a55e7193a-1637253801870-a8b076a7-leader-finder-thread], Failed to find leader for Set([audit,0])
Scasty
  • 1
  • 1
  • 2
  • According to the error, it doesn't connect successfully... 1) Set `KAFKA_ADVERTISED_HOST_NAME: kafka` to communicate between containers 2) You have not configured the Kafka address for Graylog container, and it appears to default to localhost 3) `links` is a deprecated Docker compose property, so remove it – OneCricketeer Nov 19 '21 at 17:31

0 Answers0