i investigate too much topic in stackoverflow but i couldn't find solution for me. I have a docker-compose file and i am trying to set true parameters for configuration. Other than that i have a django api that manages web requests.
My goal is when a user visits someone elses "post" i want to send a message to kafka "x user visited y users 'z' titled post". Finished all my configurations. In endpoint i configured a producer that sends message to kafka. After request to endpoint it gives me that error in title. How can i solve that ?
django view.py;
def get(self,request,post_id,format=None):
"""
Gets a post
"""
post = self.get_post(post_id)
serializer = PostSerializer(post)
send_data(post.owner, request.user, post, datetime.now().strftime("%d/%m/%Y %H:%M:%S") )
return Response(serializer.data)
send data func;
def send_data(visitor_user:User, owner_user:User, visitedPost:Post, timestamp:date):
from kafka.errors import KafkaError
from kafka import KafkaProducer
producer = KafkaProducer( bootstrap_servers=['kafka:9092'] )
visitor = visitor_user.first_name + " " + visitor_user.last_name + "(username:" + visitor_user.username + ") "
owner = owner_user.first_name + owner_user.last_name + "(username:" + owner_user.username + ") "
post = "'" + visitedPost.title + "' titled post."
message = timestamp + "->" + visitor + "read" + owner + " 's " + post
result = producer.send( 'visitation-log', message.encode() )
docker-compose.yml;
version : "3"
services:
db:
image: postgres
environment:
- "POSTGRES_HOST_AUTH_METHOD=trust"
dj:
container_name: dja
build: django # django's Dockerfile path
command: python manage.py runserver 0.0.0.0:80
volumes:
- ./django:/code
ports:
- "80:80"
depends_on:
- db
ng:
container_name: ngtrip
build: angular
ports:
- "8080:80"
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
restart: always
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka
container_name: kafka
hostname: localhost
restart: always
ports:
- 9092:9092
depends_on:
- zookeeper
links:
- zookeeper:zookeeper
environment:
KAFKA_CREATE_TOPICS: "visitation-log:1:3,Topic2:1:1:compact"
KAFKA_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_ADVERTISED_HOST: localhost
KAFKA_ADVERTISED_POST: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
UPDATE
I organized my kafka environment configs like belove thanks to help of cricket and it is now producing messages from django with "kafka:9092" .
kafka:
image: wurstmeister/kafka
command: [start-kafka.sh]
container_name: kafka
restart: always
ports:
- "29092:9092"
environment:
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
${} fields are empty string for now because i didn't attach anything to them. But i must ask another thing. My consumer is located outside of docker. Its in my local. When i set ConsumerConfig bootstrap.server config to "kafka:29092" or anything else it doesn't matter when i run the app it stills checking localhost:9092. How can i solve that?
FINAL UPDATE
It seems it is a total rookie mistake. I changed image to confluent kafka image and about that packaging problem i used spotify's docker maven plugin and created a image. Final docker-compose file;
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
consumer:
image: kafkaconsumer:latest
ports:
- "8070:8070"
depends_on:
- kafka
environment:
SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:29092
restart: always
i can access to kafka in docker with kafka:29092 and outside the kafka i use localhost:9092 with these configuration. Consumer image is builted by docker-maven plugin.