How do I push to a repo from within a gitlab CI pipeline?

Question

In my CI pipeline I am generating an artifact public/graph.png that visualises some aspect of my code. In a later step I want to commit that to the repo from within the CI pipeline. Here's the pertinent part of .gitlab-ci.yml:

commit-graph:
  stage: pages
  script:
    - git config user.email "cipipeline@example.com"
    - git config user.name "CI Pipeline"
    - cd /group/project
    - mv public/graph.png .
    - git add graph.png
    - git commit -m "committing graph.png [ci skip]"
    - echo $CI_COMMIT_REF_NAME
    - git push origin HEAD:$CI_COMMIT_REF_NAME

When the pipeline runs within gitlab it fails with:

$ git config user.email "cipipeline@dhgitlab.dunnhumby.co.uk"
$ git config user.name "CI Pipeline"
$ cd /group/project
$ mv public/graph.png .
$ git add graph.png
$ git commit -m "committing graph.png [ci skip]"
[detached HEAD 22a50d1] committing graph.png [ci skip]
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 graph.png
$ echo $CI_COMMIT_REF_NAME
jamiet/my-branch
$ git push origin HEAD:$CI_COMMIT_REF_NAME
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/group/project/project.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Not sure what I'm doing wrong and don't know enough about SSL to understand that error. Can anyone advise?

We are hosting gitlab ourselves by the way.

score 9 · Answer 1 · edited Jul 14 '21 at 09:38

I found this GitLab forum link helpful As suggested by the user you need to generate SSH key, associate it with new GitLab user dedicated for this job and add key to the runner. Small drawback is you need to use swap origin in gitlab for original ssh source (instead of sandboxed one used inside the job) which leads to committer being changed to mentioned new account instead of person who triggered pipeline. Source from link:

# for your information
whoami
printenv

# we need to extract the ssh/git URL as the runner uses a tokenized URL
export CI_PUSH_REPO=`echo $CI_REPOSITORY_URL | perl -pe 's#.*@(.+?(\:\d+)?)/#git@\1:#'`

# runner runs on a detached HEAD, create a temporary local branch for editing
git checkout -b ci_processing
git config --global user.name "My Runner"
git config --global user.email "runner@gitlab.example.org"
git remote set-url --push origin "${CI_PUSH_REPO}"

# make your changes
touch test.txt

# push changes
# always return true so that the build does not fail if there are no changes
git push origin ci_processing:${CI_COMMIT_REF_NAME} || true

Just with current version of GitLab you need to change source variable name as follows:

export CI_PUSH_REPO=`echo $CI_REPOSITORY_URL | perl -pe 's#.*@(.+?(\:\d+)?)/#git@\1:#'`

Błażej Michalik · Answer 2 · 2021-09-27T13:02:59.523

Some finger push-ups still required, but here's a less brittle way of pushing to the repository from its own CI, that I use in my daily work. It pushes to master directly from a detached head:

Generate an RSA key and add it as a Project Deploy Key with write access (the public part).
Put the private part into your CI/CD variables from inside of your project settings as SSH_PUSH_KEY. Make sure to set it as protected.
Add a CI_KNOWN_HOSTS variable, with the SSH fingerprint of your GitLab instance (remember that thing ssh asks you about the first time you try to connect to a host? That.).

Use ssh-keyscan <gitlab-host> to get it. It will look similar to this:

my.gitlab.instance.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEArlUMUmNj59PpoLyy4EsKbwhPUfXxuAzFN7dMKDXVvKMmN8344HqQV1tRx6fcmH+0BXK1JAP4f10V0VnYti3e1c5f9dhpl8pIqKLMJgdGDq3MLqjihL3bp5xm8nDsPTm5FoEPPYK1I3M2wr18pBB19evz64NHrK5R/HO5LyTrybVasFumt8cZoH6crnCFgfQWV1mHAG3j41Q0z4yxu6g8zBWESZcVVn90HxQH7+LDHx11122233344491MQGl5fZcKqVWsWQVEssaK87iBsWUxvsuoeVUrj4YRcmbi6F4+ZZZZZZZwwww3ZboWsSWxTk5ESR6WWHccBm8GQflXyY3ZQ==

Set up your job inside .gitlab-ci.yml as follows. Set stage and resource_group options appropriately - without the latter you might run into race conditions. Also, make sure to set only properly, as otherwise your CI might trigger itself:

"This CI job pushes to its own repo":
    stage: my_push_stage
    resource_group: this_option_comes_handy_when_pushing
    only:
        - triggers
    before_script:
        - mkdir ~/.ssh/
        - echo "${CI_KNOWN_HOSTS}" > ~/.ssh/known_hosts
        - echo "${SSH_PUSH_KEY}" > ~/.ssh/id_rsa
        - chmod 600 ~/.ssh/id_rsa
        - git config user.email "ci@example.com"
        - git config user.name "CI"
        - git remote remove ssh_origin || true  # Local repo state may be cached
        - git remote add ssh_origin "git@$CI_SERVER_HOST:$CI_PROJECT_PATH.git"
    script:
        - touch "xyz"  # Make an edit
        - git add "xyz"
        - git commit -m "My CI commit"
        - git push ssh_origin HEAD:master  # ❗ this pushes to master, 
                                           # use $CI_BUILD_REF_NAME if you want to push to current branch
        - git tag MyCiTag  # If you need to add a tag you can do that too
        - git push --tags ssh_origin

you can use ssh-keyscan $CI_SERVER_HOST > ~/.ssh/known_hosts rather than a variable — Tom Chiverton, Jan 07 '22 at 17:45
@TomChiverton While this works, it's basically the same as setting `GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no"`. — Błażej Michalik, Jun 01 '22 at 10:01

conspicillatus · Answer 3 · 2022-02-01T06:03:20.847

8

You can add the CI_SERVER_CLS_CA_FILE to sslCAInfo git config.

checkout alchemy:
    stage: prepare
    script:
        - git config --global "http.${CI_SERVER_URL}.sslCAInfo" "$CI_SERVER_TLS_CA_FILE"
        - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/sparklemuffin/alchemy.git

@Sjoerd's approach, in the comments, to export GIT_SSL_CAINFO instead, is a bit shorter.

export GIT_SSL_CAINFO=$CI_SERVER_TLS_CA_FILE

Gitlab creates CI_SERVER_TLS_CA_FILE and configures git to use it for initially cloning the repository. For some reason this configuration is no longer available later on.

edited Feb 01 '22 at 06:03

answered Oct 22 '20 at 18:45

conspicillatus

133
3
9

2

I used a similar solution: I put `export GIT_SSL_CAINFO="$CI_SERVER_TLS_CA_FILE"` at the top of my script. It [doesn't work if you put it under `variables`](https://gitlab.com/gitlab-org/gitlab/-/issues/292220). – Sjoerd Feb 18 '21 at 11:43

score 1 · Accepted Answer · edited Apr 22 '19 at 14:11

1

Solved it. Issuing git config --global http.sslverify "false" prior to the push solved that particular problem (it exposed another problem but that's for another thread :) )

edited Apr 22 '19 at 14:11

Caleb

4,731
1
44
61

answered Aug 06 '18 at 22:27

jamiet

8,334
9
57
120

28

Good question, but -1 to this answer because it encourages avoiding the problem rather than fixing it. There are good ways to get a free SSL certificate these days, there should be no excuse not to fix the SSL problem rather than disabling it entirely. – Caleb Apr 22 '19 at 14:13

Matruskan · Answer 5 · 2019-11-20T19:59:43.423

I can commit from Gitlab-CI with a selected user with a minor change based on tsr's answer https://stackoverflow.com/a/57800614/5269825 :

# set remote URL to https://oauth2:<AccessToken>@server.com/project.git
CI_PUSH_REPO=`echo "$CI_REPOSITORY_URL $ACCESS_TOKEN_PARAM" | sed 's/^.*\(@.*\)\s\(.*\)/https:\/\/oauth2:\2\1/g'`
git config http.sslverify false
git remote set-url --push origin "${CI_PUSH_REPO}"
git config user.name "Token Owner"
git config user.email "tokenowner@email.com"

# runner runs on a detached HEAD, create a temporary local branch for editing
git checkout -b ci_processing
# make your changes

# push changes
# always return true so that the build does not fail if there are no changes
git push origin ci_processing:${CI_BUILD_REF_NAME} || true

The ACCESS_TOKEN_PARAM must be configured at the project's CI/CD Variables configuration.

The idea of using Oauth2 and Access Token was taken from https://stackoverflow.com/a/52074198/5269825 and https://stackoverflow.com/a/52154378/5269825.

Also, pushing changes can trigger a new pipeline!

score 1 · Answer 6 · answered Oct 22 '21 at 11:31

1

Instead of declaring CI_KNOWN_HOST, you can try get it realtime:

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- ssh-keyscan -t rsa $CI_SERVER_HOST >> ~/.ssh/known_hosts

answered Oct 22 '21 at 11:31

ChesTeRcs

51
5

score 0 · Answer 7 · answered Sep 07 '20 at 15:17

In my case deploy keys option was optimal (compared to personal tokens or CI token - only supports basic auth) with Gitlab Shell Runner. In case someone is struggling with pushing to from Gitlab CI, this can be done sharing the public key of the runner with the Gitlab server

score 0 · Answer 8 · answered Nov 05 '21 at 11:47

This is a working example as of today environment: k8s gitlab: 13.x gitlab runner: 13.x goal of this job named convert:- Converts the excel files to json files and commits and updates the branch of the repo.

convert:
  variables:
    REPO: "gitlab.com/group/myproject.git" # example
    BRANCH: "BRANCHNAME" # example
    # recommended to store the following as project variables to hide secrets away from the gitlab ci file.
    GITLAB_USER_ID: "gitlab_user" # example
    CI_USERNAME: "gitlab_user" # example
    CI_PUSH_TOKEN: "<api token from gitlab" # example
    GITLAB_USER_EMAIL: "gitlab_user@company.com" # example
  stage: convert
  image:
    name: python:3.7-buster
    entrypoint: ["/bin/ash"]
  before_script:
    - pip3 install openpyxl
    - ls -altr
  script:
    - echo 'converting excel to json'
    - python excel2json.py
    - git remote set-url origin https://${CI_USERNAME}:${CI_PUSH_TOKEN}@$REPO
    - git config --global user.email '${GITLAB_USER_EMAIL}'
    - git config --global user.name '${GITLAB_USER_ID}'
    - git add -A && git commit -m 'added/updated json files'
    - git push origin HEAD:$BRANCH

Note: CI_USERNAME == GITLAB_USER_ID. Both are same in my case.

score 0 · Answer 9 · answered Nov 28 '21 at 13:49

Another way:

Create dedicated Gitlab account
Add CI/CD variable GIT_CICDUSER_PASSWORD containing it's password to the project inside the main account running the pipeline

Then a job can look like:

task_requiring_a_push:
  stage: some-stage
  variables:
    GIT_STRATEGY: none
  script:
    - git config --global user.email "user@mail.com"
    - git config --global user.name "CI/CD User"
    - git clone https://<dedicated_gitlab_user_username>:$GIT_CICDUSER_PASSWORD@gitlab.com/$CI_PROJECT_PATH .
    - git checkout $CI_COMMIT_REF_NAME
    - # do something
    - git add *
    - git commit -m "message"
    - git push --push-option=ci.skip origin $CI_COMMIT_REF_NAME
only:
    - release-*

How do I push to a repo from within a gitlab CI pipeline?

9 Answers9

Linked