4

My pipeline consists of very long jobs that build docker images by compiling a set of packages; all of them require ~ 3 hours on my runner.

I set up my CI so that only one job can run at a time, in order to reduce the disk I/O overhead. But in this way many jobs fail, supposedly because they hit a job start timeout which seems to be bout 1h, with this message:

There has been a timeout failure or the job got stuck. Check your timeout limits or try again

I have not been able to find how to increase the job start timeout, since all the documentation I found is relative to the job duration timeout.

So I’d need help with this, thanks. I'm using GitLab CE 12.8.1 and Gitlab Runner 11.5.1.

Yasen
  • 682
  • 4
  • 9
Nicola Mori
  • 141
  • 1
  • 3
  • I know what you are talking about. We can paraphrase it using gitlab specific terminology. You want to specify timeout that controls conversion between pending and started job. For now it seems impossible in gitlab. – puchu Mar 09 '24 at 12:08

1 Answers1

5

There are three types of timeouts in Gitlab CI:

  • Project timeout
  • Runner timeout
  • Jobs timeout

Project timeout

According to GitLab docs:

Timeout defines the maximum amount of time in minutes that a job is able run. This is configurable under your project’s Settings > CI/CD > General pipelines settings. The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs’ running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed

Runner timeout

According to GitLab docs:

For each Runner you can specify a maximum job timeout.

Runner timeout settings are defined in runner's edit page

Jobs timeout

GitLab CI/CD Pipeline Configuration Reference | GitLab

Job's timeout allows you to configure a timeout for a specific job. For example:

build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m

Precedence of different types of timeout

The job-level timeout can exceed the project-level timeout but can not exceed the Runner-specific timeout.

If runner timeout smaller than project defined timeout, will take the precedence.

Examples of precedence of timeout directive

See Configuring GitLab Runners | GitLab

name: **Example 1 - Runner timeout bigger than project timeout**
  project_timeout: 2h
  runner_timeout: 24h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 2 - Runner timeout not configured**
  project_timeout: 2h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 3 - Runner AND job timeout are not configured**
  project_timeout: 24h
  resulting_timeout: 24h

name: **Example 4 - Runner timeout smaller than project timeout**
  project_timeout: 2h
  runner_timeout: 30m
  resulting_timeout: 30m

name: **Example 5 - Runner timeout smaller than Project timeout, Job timeout is bigger than Runner timeout**
  project_timeout: 2h
  runner_timeout: 30m
  job_timeout: 1h
  resulting_timeout: 30m
Yasen
  • 682
  • 4
  • 9
  • thanks for your reply, but as far as I understand these are all runtime timeout, limiting the time a job can effectively run after it is started. But my problem, as I tried to explain in my original post, is that my jobs die because they are not started due to unavailable runners, since my only runner runs one job at each time and the others are killed while waiting to be run. Maybe your reply effectively provides a solution for this case but I misunderstood it? – Nicola Mori Mar 24 '20 at 18:07
  • Hi, @nicola-mori. Just set Project timeout to value more than 3h. And leave Runner timeout value empty. See Example 3 – Yasen Mar 24 '20 at 18:28
  • Hi, my Project timeout (i.e. that configured in Settings -> CI / CD Settings -> General pipelines -> Timeout) is currently 2d, and my runner does not have a timeout. Does this correspond to what you are suggesting? Or do have to put the settings you are suggesting somewhere (I guess in .gitlab-ci.yml)? Sorry for the noob questions but I'm a beginner... – Nicola Mori Mar 24 '20 at 18:57
  • Hi. Please show your .gitlab-ci.yml – Yasen Mar 25 '20 at 05:52
  • You can find it here: https://basket.fi.infn.it:443/f/0de7f11b48/?dl=1 – Nicola Mori Mar 25 '20 at 09:11
  • Sorry, nothing to say – Yasen Mar 25 '20 at 13:43