20

I am using docker compose to run couple of service which depends on each other. Here is part of the docker-compose:

  backend:
    build: .
    command: bash -c "npm run build && npm start"
    ports: 
      - "3015:3015"
    depends_on:
      - couchdb
      - redis
      - uds-mock-server
    volumes:
      - /app/node_modules
      - .:/app
    user: root
  api-test:
    restart: always 
    build: .
    depends_on:
      - backend
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
      interval: 200s
      timeout: 200s
      retries: 5
    user: root

As you see I have two service over there and backend should first run and the server needs to be ready then api-test can start. backend has an endpoint: localhost:2015/readiness and whenever it returns 200 then api test can start. When I run while building the order is respected so backend first followed by api-mock but when docker compose starts running them api-test run quicker and since it relies on the backend to be ready it fails.

Base on the following:

Docker Compose wait for container X before starting Y

and

Docker healthcheck in composer file

It is suggested that I should use healthcheck which I do in api test:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
  interval: 200s
  timeout: 200s
  retries: 5

If I get this correctly api-test should run and call the readiness endpoint and wait until it hears from backend readiness (up to 200s) if it fails it waits for 200s and then try again for 5 times. But what I see is api-test keep failing and restarting and does not even give a chance to backend to run and it keeps doing this like a loop. Am I missing anything? Any help is really appreciated

Learner
  • 1,337
  • 2
  • 14
  • 31

2 Answers2

17

The two examples are based on the condition form of depends_on which is no longer supported in compose version 3. So, unless your docker-compose version is <3 the healthcheck will not help you much. The healthcheck sets the status of the container (starting, healthy or unhealthy) but docker-compose does not wait until backend container is healthy before starting the app-test. There is a detailed explanation about how depends_on works in Control startup and shutdown order in Compose

As a side note, the healthcheck in your compose file sets the status of the app-test container and not backend.

Therefore to control when the api-test can start, you have to wrap the service command of the container. For your particular case the following will do the job:

bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; <service_command>'

It tries to connect to backend every 5 seconds (the connection timeout is 2s). When the received HTTP status code is 200 OK the loop ends and it executes the <service_command>

The relevant docker-compose part:

  api-test:
    restart: always
    command: bash -c 'while [[ "$$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up;npm start'
    depends_on:
      - backend
      ...

Hope this helps.

b0gusb
  • 3,785
  • 2
  • 11
  • 31
  • 1
    when I run your command I get this: ERROR: Invalid interpolation format for "command" option in service "de-backend": "bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; npm start'" – Learner Dec 03 '19 at 13:42
  • 2
    @Learner In the docker-compose.yml file you should escape the `$` with another `$`. See [Variable Substitution](https://docs.docker.com/compose/compose-file/#variable-substitution). I have updated my answer. – b0gusb Dec 03 '19 at 14:11
  • Thanks. That worked but when I run the compose I see the api-test service keeps sending .. though readiness is readiness is ready with 200 it seems to be an infinite loop so this never happens echo backend is up; npm start – Learner Dec 03 '19 at 14:15
  • What do you get if you run `curl -v uds-mock-server:4000/readiness` inside the api-test container? Is the uds-mock-server the service you wish to check is ready? – b0gusb Dec 03 '19 at 14:22
  • Yes it is mock-server. I changed the code a bit because of requirements changes – Learner Dec 03 '19 at 14:28
  • when I echo the condition: "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" I get this 000 – Learner Dec 03 '19 at 14:28
  • Try to run curl in verbose mode to see what's going on. – b0gusb Dec 03 '19 at 14:45
  • getting this backend_1 | * Trying xxxx... backend_1 | * TCP_NODELAY set backend_1 | * connect to xxx port 4000 failed: Connection refused backend_1 | * Failed to connect to mock-server port 4000: Connection refused backend_1 | * Closing connection 0 backend_1 | .. [000] – Learner Dec 03 '19 at 14:58
  • ooops my bad http not https Thanks – Learner Dec 03 '19 at 15:26
  • docker supports clean health checks. No need to hack something together. – The Fool Nov 24 '20 at 00:26
13

it's not localhost connection string.

It should be service name of backend container : 

test: ["CMD", "curl", "-f", "http://backend:3015/readiness"]
Thanh Nguyen Van
  • 8,237
  • 5
  • 30
  • 46
  • 1
    This one fixed it for me! – legel Jun 30 '21 at 19:37
  • 1
    is there any way to use semicolon in statement? for example: test: ["CMD", "netstat", "-an | grep 45800 > /dev/null; if [ 0 != $? ]; then exit 1; fi;"] It gives an error to me: ERROR: Invalid interpolation format for "healthcheck" option in service "frontend": "-an | grep 45800 > /dev/null; if [ 0 != $? ]; then exit 1; fi;" – Jaswinder Aug 21 '21 at 04:58
  • This solved it for the, but what does the "-f" flag do? – Beyar Jan 29 '22 at 21:53