0

i investigate too much topic in stackoverflow but i couldn't find solution for me. I have a docker-compose file and i am trying to set true parameters for configuration. Other than that i have a django api that manages web requests.

My goal is when a user visits someone elses "post" i want to send a message to kafka "x user visited y users 'z' titled post". Finished all my configurations. In endpoint i configured a producer that sends message to kafka. After request to endpoint it gives me that error in title. How can i solve that ?

django view.py;

 def get(self,request,post_id,format=None):
    """
    Gets a post
    """
    post = self.get_post(post_id)
    serializer = PostSerializer(post)

    send_data(post.owner, request.user, post, datetime.now().strftime("%d/%m/%Y %H:%M:%S") )

    return Response(serializer.data)

send data func;

def send_data(visitor_user:User, owner_user:User, visitedPost:Post, timestamp:date):
  from kafka.errors import KafkaError
  from kafka import KafkaProducer
  producer = KafkaProducer( bootstrap_servers=['kafka:9092'] )


  visitor = visitor_user.first_name + " " + visitor_user.last_name + "(username:" + visitor_user.username + ") "
  owner = owner_user.first_name + owner_user.last_name + "(username:" + owner_user.username + ") "
  post = "'" + visitedPost.title + "' titled post."
  message = timestamp + "->" + visitor + "read" + owner + " 's " + post

  result = producer.send( 'visitation-log', message.encode() )

docker-compose.yml;

version : "3"

services:
    db:
       image: postgres
       environment:
        - "POSTGRES_HOST_AUTH_METHOD=trust"
    dj: 
       container_name: dja 
       build: django # django's Dockerfile path
       command: python manage.py runserver 0.0.0.0:80
       volumes:
       - ./django:/code

       ports:
        - "80:80"
       depends_on:
        - db

    ng:
        container_name: ngtrip
        build: angular
        ports:
            - "8080:80"

    zookeeper:
        image: wurstmeister/zookeeper
        container_name: zookeeper
        restart: always
        ports:
            - 2181:2181

    kafka:
        image: wurstmeister/kafka
        container_name: kafka
        hostname: localhost
        restart: always
        ports:
            - 9092:9092
        depends_on:
            - zookeeper
        links:
            - zookeeper:zookeeper
        environment:
            KAFKA_CREATE_TOPICS: "visitation-log:1:3,Topic2:1:1:compact"
            KAFKA_LISTENERS: PLAINTEXT://localhost:9092
            KAFKA_ADVERTISED_HOST: localhost
            KAFKA_ADVERTISED_POST: 9092
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

UPDATE

I organized my kafka environment configs like belove thanks to help of cricket and it is now producing messages from django with "kafka:9092" .

kafka:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    container_name: kafka
    restart: always
    ports:
        - "29092:9092"
    environment:
        KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:29092
        KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
        KAFKA_ADVERTISED_LISTENERS: INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
        KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

${} fields are empty string for now because i didn't attach anything to them. But i must ask another thing. My consumer is located outside of docker. Its in my local. When i set ConsumerConfig bootstrap.server config to "kafka:29092" or anything else it doesn't matter when i run the app it stills checking localhost:9092. How can i solve that?

FINAL UPDATE

It seems it is a total rookie mistake. I changed image to confluent kafka image and about that packaging problem i used spotify's docker maven plugin and created a image. Final docker-compose file;

zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
        ZOOKEEPER_CLIENT_PORT: 2181
        ZOOKEEPER_TICK_TIME: 2000

kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
        - zookeeper
    ports:
        - 9092:9092
    environment:
        KAFKA_BROKER_ID: 1
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
        KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
        KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
        KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
consumer:
    image: kafkaconsumer:latest
    ports:
        - "8070:8070"
    depends_on:
      - kafka
    environment:
      SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:29092
    restart: always

i can access to kafka in docker with kafka:29092 and outside the kafka i use localhost:9092 with these configuration. Consumer image is builted by docker-maven plugin.

Kadir Akın
  • 33
  • 1
  • 7
  • `KAFKA_ADVERTISED_HOST` should **not** be `localhost`. Where did you copy that docker section from? The `KAFKA_LISTENERS` and the `hostname` should not either. – OneCricketeer Mar 28 '20 at 02:28
  • I did a lot of changes and i dont remember anymore. It seems i need to read documentation more but right now it has to finish. What i am trying to do use python as a producer and spring boot as a consumer. My django api and kafka cluster is inside a docker container and i couldn't transfer my spring app to docker container because when i use mvn install it was trying to connect and fails so it is not produced jar file to use in Dockerfile or i failed somehow. So that's the story. How can i solve that deal? There was a spring part in docker-compose btw. But i assumed that wasn't the real prblm – Kadir Akın Mar 28 '20 at 03:16
  • I can't really help with app specfic things. Read the blog post linked in the duplicate post. If you want to build a Docker image from Maven, use Jib https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin – OneCricketeer Mar 28 '20 at 05:56
  • I think i figured it out about this spring problem. Even i declare in application.properties that bootstrap server is kafka:9092 and its in configconsumer, it still search for brokers in localhost:9092 . I saw that in runtime log in spring. Sorry for inconvenience. – Kadir Akın Mar 28 '20 at 12:27
  • Yeah - https://rmoff.net/2018/08/02/kafka-listeners-explained – OneCricketeer Mar 28 '20 at 16:49
  • I edited the question at the same time with your comment :) . Thanks for help. – Kadir Akın Mar 28 '20 at 16:50
  • You need `- "29092:29092"` in your ports list, by the way – OneCricketeer Mar 28 '20 at 16:51
  • Should i add or change to "29092:29092" ? Because when i add to the port lists docker-compose up returns 500 (ports are not available) and when i change my producer fails – Kadir Akın Mar 28 '20 at 17:15
  • Port 9092 only belongs within the docker network. Django would use `kafka:9092` and it will work fine without even having `ports:` at all in the compose file for kafka. Then if you want to connect outside the container from the host, then you need a port forward, and your settings have defined the outside host IP/name to be used on port 29092, so that is the one you should be forwarding. – OneCricketeer Mar 28 '20 at 17:16
  • I did what you said. and just changed ports to - "29092:29092" and my django weirdly fails. In docker ps kafka forwarding is now like this 0.0.0.0:29092->29092/tcp . – Kadir Akın Mar 28 '20 at 17:37
  • Again, django is in a container, it would not connect to the host's ports. It would connect to the docker network ports, which are 9092. Is suggest you expose the kafka address in an environment variable, like `os.environ['BOOTSTRAP_SERVERS']` – OneCricketeer Mar 29 '20 at 06:03

0 Answers0