Connect two instances of docker-compose

Question

I have a dockerized application with a few services running using docker-compose. I'd like to connect this application with ElasticSearch/Logstash/Kibana (ELK) using another docker-compose application, docker-elk. Both of them are running in the same docker machine in development. In production, that will probably not be the case.

How can I configure my application's docker-compose.yml to link to the ELK stack?

You can't. you either need to use your host machine network to connect to apps or use overlay networking with you docker compose which I believe still not mature enough for production use cases. — Boynux, Dec 21 '15 at 17:37

score 9 · Accepted Answer · edited May 23 '17 at 11:46

Update Jun 2016

The answer below is outdated starting with docker 1.10. See this other similar answer for the new solution. https://stackoverflow.com/a/34476794/1556338

Old answer

Create a network:

 $ docker network create --driver bridge my-net

Reference that network as an environment variable (${NETWORK})in the docker-compose.yml files. Eg:

pg:
  image: postgres:9.4.4
  container_name: pg
  net: ${NETWORK}
  ports:
    - "5432"
myapp:
  image: quay.io/myco/myapp
  container_name: myapp
  environment:
    DATABASE_URL: "http://pg:5432"
  net: ${NETWORK}
  ports:
    - "3000:3000"

Note that pg in http://pg:5432 will resolve to the ip address of the pg service (container). No need to hardcode ip addresses; An entry for pg is automatically added to the /etc/host of the myapp container.

Call docker-compose, passing it the network you created:

$ NETWORK=my-net docker-compose up -d -f docker-compose.yml -f other-compose.yml

I've created a bridge network above which only works within one node (host). Good for dev. If you need to get two nodes to talk to each other, you need to create an overlay network. Same principle though. You pass the network name to the docker-compose up command.

score 8 · Answer 2 · answered Apr 13 '17 at 07:32

You could also create a network with docker outside your docker-compose :

docker network create my-shared-network

And in your docker-compose.yml :

version: '2'
services:
  pg:
    image: postgres:9.4.4
    container_name: pg
    expose:
      - "5432"
networks:
  default:
    external:
      name: my-shared-network

And in your second docker-compose.yml :

version: '2'
  myapp:
    image: quay.io/myco/myapp
    container_name: myapp
    environment:
      DATABASE_URL: "http://pg:5432"
    net: ${NETWORK}
    expose:
      - "3000"
networks:
  default:
    external:
      name: my-shared-network

And both instances will be able to see each other, without open ports on host, you just need to expose ports, and there will see each other through the network : "my-shared-network".

Is `net: ${NETWORK}` required in the second docker-compose.yml? — Marcus Ahlberg, Feb 16 '20 at 10:26
If one of the service has to start before another, how can we implement link or depends_on. — Tara Prasad Gurung, Aug 25 '20 at 15:01

Rob van Laarhoven · Answer 3 · 2015-12-21T13:37:20.300

2

Take a look at multi-host docker networking

Networking is a feature of Docker Engine that allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. The networked containers can even span multiple hosts, so you don’t have to worry about what host your container lands on. They seamlessly communicate with each other wherever they are – thus enabling true distributed applications.

edited Dec 21 '15 at 13:37

answered Dec 21 '15 at 11:37

Rob van Laarhoven

8,461
2
29
48

1

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/low-quality-posts/10638334) – Mel Dec 21 '15 at 12:58
@tmoreau I agree. But this question is too broad, the topic too complex, to give a short answer. I removed the link so OP can google. – Rob van Laarhoven Dec 21 '15 at 13:10
If the question is too broad, flag as such. Your answer could be a comment. – Mel Dec 21 '15 at 13:11
3

That's a matter of opinion. The answer is too broad, not the question. Answers are for adding solutions to the question. Comments to add essential information, ask for clarification, ask for extra information, etc. Although short the answer may serve as a future reference. – Rob van Laarhoven Dec 21 '15 at 13:36

score 2 · Answer 4 · answered Dec 22 '15 at 00:03

If you set a predictable project name for the first composition you can use external_links to reference external containers by name from a different compose file.

In the next docker-compose release (1.6) you will be able to use user defined networks, and have both compositions join the same network.

Connect two instances of docker-compose

4 Answers4

Linked