Private Docker Registry

Motivation

I have been using docker containers for a while and quite amazed by its simplicity and power. Someday in the previous week when I was migrating/re-deploying my pleroma instance from my SurfaceGo ubuntu machine to my home lab, I found I need more private repos on docker hub. Docker hub by default only provides 1 private repo for normal users, which is clearly not what I want. So I decided to build my own docker registry and deploy it as a service in my home lab machine.

My set up is:

Run a front-facing machine with a public IPv4 address, and add a DNS A record (registry.mxcao.me) to point to this address. The front-facing machine will route all valid traffic to the registry container running in my Rancher cluster.

Since my home lab do not have a public IPv4 address dedicated for such purpose, I use a DigitalOcean instance to serve the purpose, along with my pleroma service routing (feel free to play around pleroma.mxcao.me).

To be able to route my traffic to the registry container, I use WireGuard to connect these two services. Definitely check out my previous post about setting up WireGuard if you are interested.

Prerequisite

Check you have the following items ready:

• Your own domain so that it can be reached from the public Internet
• A front-facing machine with
  • a public IPv4 address (e.g. 1.2.3.4)
  • nginx
  • WireGuard
• Docker (or Rancher alike) installed on the home lab to host our registry service

Steps

Set up DNS record

I want my registry service accessible from anywhere with a memorizable address and therefore I need to set up a A record to point pleroma.mxcao.me to 1.2.3.4. You can omit this step if you can accept the inconvenience of accessing your registry via IP addresses.

Prepare WireGuard for the service

As mentioned, I decided to use WireGuard to achieve point-to-point communication between my DO machine and the container. This set up involves:

• Generate private/public key-pairs for the registry service container to use
• Add a peer in the DO WireGuard configuration
• Set up WireGuard in the registry host (note here we use port forwarding to map registry service’s listening port to the node running it, therefore I can simply set up WireGuard in the node)
  • For my particular case, I built and deployed the WireGuard module to RancherOS following this great guide
    • I did not follow all steps in this guide, and you should decide if having the WireGuard on the host is enough.

Now let’s assume you configured the WireGuard client with IPv4 address 10.9.60.5.

Prepare NGINX for traffic routing

I decided to implement basic authentication for my private registry in a reverse proxy that sits in front of the registry. This is simpler than configure the native basic auth registry feature I think, please correct me if I’m wrong.

I use simple htpasswd file as an example, but as mentioned in their documentation, any other nginx authentication backend should be fairly easy to implement.

Let’s obtain a certificate for the subdomain registry.mxcao.me via Let’s Encrypt:

• Make sure your nginx service is not running

• Setup your SSL cert, using your method of choice or certbot. If using certbot, first install it:

  sudo apt install certbot

  Then set it up:

  sudo mkdir -p /var/lib/letsencrypt/
  sudo certbot certonly --email <your@emailaddress> -d registry.mxcao.me --standalone

If successful, you should have your certificate and chain files in /etc/letsencrypt/live/registry.mxcao.me

In your nginx configuration folder (it can be conf.d or sites-available), add below routing rules.

Remember to change the docker-registry IP address and the certificate file path to suit your needs.

upstream docker-registry {
  server 10.9.60.5:5000;
}

## Set a variable to help us decide if we need to add the
## 'Docker-Distribution-Api-Version' header.
## The registry always sets this header.
## In the case of nginx performing auth, the header is unset
## since nginx is auth-ing before proxying.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
  '' 'registry/2.0';
}

server {
  listen 80;
  listen [::]:80;
  server_name registry.mxcao.me;
  location / {
    return 301 https://$server_name$request_uri;
  }
}

server {
  server_name registry.mxcao.me;

  listen 443 ssl;
  listen [::]:443 ssl;

  # SSL
  ssl_certificate /etc/letsencrypt/live/registry.mxcao.me/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/registry.mxcao.me/privkey.pem;

  # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
  ssl_protocols TLSv1.1 TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/moby/moby/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication to v2 use auth_basic setting.
    auth_basic "Registry realm";
    auth_basic_user_file /etc/nginx/registry_auth/nginx.htpasswd;

    ## If $docker_distribution_api_version is empty, the header is not added.
    ## See the map directive above where this variable is defined.
    add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

    proxy_pass http://docker-registry;
    # required for docker client's sake
    proxy_set_header Host $http_host;
    # pass on real client's IP
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 900;
  }
}

As you can see in the above configuration, we need to put the password file nginx.htpasswd in /etc/nginx/registry_auth, create this folder if not already exist.

For some reason, I can’t make it work following the documentation to generate the password file. Running docker run --rm --entrypoint htpasswd registry:2 -Bbn testuser testpassword > nginx.htpasswd simply not gonna work for nginx to authenticate properly.

A work-around is posted here, and to summarize you can run the following commands to populate the nginx.htpasswd password file.

sudo sh -c "echo -n 'testuser:' >> nginx.htpasswd"
# This command will prompt you to enter the pwd
sudo sh -c "openssl passwd -apr1 >> nginx.htpasswd"

So far so good! Start your nginx service and now we are ready to spawn the actual registry service.

Set up registry service

This is my particular set up case and you can easily port to the docker commands to serve the registry.

In my rancher cluster, I added another workload with the following tweaks:

• Port mapping from container’s 5000/tcp to host’s 5000/tcp
• Env variable REGISTRY_STORAGE_DELETE_ENABLED set to true to enable image deletion
• Mount a volume from the host to the container’s path /var/lib/registry

Done! Now you can test your private registry set up by running docker login -u=testuser -p=testpwd registry.mxcao.me in your own machine. The expected output should be:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

That’s it! And you can now push unlimited number of repos/images to your own, private registry. Enjoy.