How to configure registries.yaml to point to several Harbor proxy cache registry?

Question

I have configured a Harbor (with SSL and FQDN: harbor.example.com) as a proxy cache for several docker registries (docker.io, quay.io and k8s.gcr.io). I can pull any images without any problem from any PC/VM which can access the Harbor netowrk, i.e.:

# From hub.docker.com
docker pull harbor.example.com/dockerhub/library/ubuntu:20.04
# From quay.com
docker pull harbor.example.com/q/metallb/speaker:v0.12.1
# From k8s.gcr.com
docker pull harbor.example.com/g/ingress-nginx/controller:v1.1.2@sha256:28b11ce69e57843de44e3db6413e98d09de0f6688e33d4bd384002a44f78405c

I want to use this Harbor to, from a local k3s cluster (which was an airgap installation) and its only way to pull images is using the previously mentioned Harbor registry (because it has no way to reach the internet), pull images from every docker proxy cache created with, for example, and following the lasts examples:

k3s crictl pull ubuntu:20.04
k3s crictl pull quay.io/metallb/controller:v0.12.1
k3s crictl pull k8s.gcr.io/ingress-nginx/controller:v1.1.2@sha256:28b11ce69e57843de44e3db6413e98d09de0f6688e33d4bd384002a44f78405c

In order to get it done, and following the instructions from the rke2 official docx, I'm trying to modify the /etc/rancher/k3s/registries.yaml to rewrite the link , i.e., from docker.io to harbor.example.com/dockerhub, and/or from quay.io to harbor.example.com/q and or k8s.grc.io to harbor.example.com/g but I can't find a way to achieve correctly... I guess it should be something like the folowing (but for every Harbor's cache proxy):

mirrors:
  quay.io:
    endpoint:
      - "https://harbor.virtalus.com"
    rewrite:
      "\/(.*)": "q/$1"

So the question is: how to properly configure containerd's registries.yaml to point it to several Harbor's proxy cache?

Answer 1

I just got this working with Harbor Version v2.10.0-6abb4eab using mirror and a tweaked URL instead of trying to rewrite image names on the fly:

1. Follow the docs at https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/ to create a project called proxy_cache, setup security or make the mirror public
2. If using your own CA copy the pem file somewhere on each k3s node
3. registries.yaml should look something like this. I have robot user and my own CA.

mirrors:
  docker.io:
    endpoint:
      - "https://harbor.example.com/v2/proxy_cache"
configs:
  "harbor.example.com":
    auth:
      username: "robot$k3s" # this is the registry username
      password: "thisisaverysecurepasswordtoaccesspublicfiles" # this is the registry password
    tls:
      # cert_file: # path to the cert file used in the registry (mTLS)
      # key_file:  # path to the key file used in the registry (mTLS)
      ca_file: "/etc/rancher/k3s/ca.pem"  # path to the ca file used in the registry (if using TLS)

4. Restart k3s
5. Reference images as normal, eg

      containers:
      - image: docker.io/kong/httpbin:0.1.0

or

      containers:
      - image: kong/httpbin:0.1.0

I automated the copying the CA and the registry.yaml to my k3s servers with some ansibles like this:

- name: registries
  hosts: server
  become: true
  become_user: root
  tasks:
  - name: registries.yaml
    ansible.builtin.copy:
      src: registries.yaml 
      dest: /etc/rancher/k3s/registries.yaml
      owner: root
      group: root
      mode: '0600'

name: ca.pem
ansible.builtin.copy:
  src: ~/ca/ca.pem
  dest: /etc/rancher/k3s/ca.pem
  owner: root
  group: root
  mode: '0600'

name: k3s
ansible.builtin.systemd_service:
  state: restarted
  name: k3s