Skip to content
Hands-on IT IT Drills

Chapter 3: Mirrored sources as the only supply path (Harbor, Nexus, Artifactory)

In this chapter you will learn:

  • the difference between a proxy cache, a curated mirror and an offline mirror
  • how to enforce the “only supply path” technically: client configuration, egress blocking, policy
  • how to use the Nexus REST API to create mirrors for container images, PyPI, Debian packages and npm
  • how to build the demo-app so that it obtains everything exclusively through the mirror, and prove it with a blocked network
  • how proxy cache projects and replication work in Harbor, and remote/virtual repositories in Artifactory

What is it about?

Think of a hospital pharmacy. The wards don’t order on their own from some wholesaler on the internet. Everything goes through the pharmacy. The pharmacy knows its suppliers, checks the goods, stores them and hands them out to the wards. When a recall comes, it knows immediately which ward received which batch. And because the wards have no other supply path at all, nobody can “just quickly” order something unchecked.

A mirror for software packages is this pharmacy. Instead of every build ordering directly from Docker Hub, PyPI, npm or Debian, all builds fetch their ingredients from a central repository manager such as Nexus, Harbor or Artifactory. The first time, the mirror downloads from the internet, stores a copy and from then on serves it itself.

That alone is convenience. It only becomes security through the second part of the heading: only supply path. The build environment must not be able to reach the internet at all, only the mirror. Then the mirror is the only place where you have to control, log and block.

   BEFORE: everyone fetches           AFTER: only supply path

  +-------+   +----------------+     +-------+               +------------+
  | Build |-->| Docker Hub     |     | Build |--+            | Docker Hub |
  |       |-->| PyPI           |     +-------+  |  +-------+ | PyPI       |
  |       |-->| deb.debian.org |                +->|Mirror |>| Debian     |
  |       |-->| npm, curl ...  |     +-------+     | Nexus | | npm        |
  +-------+   +----------------+     | Build |---->|       | +------------+
                                     +-------+     +-------+
   Control:   none                       |      only the mirror may
   Logging:   scattered                  X      reach the internet
   Blocking:  impossible                 |  Firewall: direct path
                                         v  to the internet blocked
                                      Internet

The concepts made simple

Three kinds of mirrors

Not every mirror is the same. The three basic forms differ in when and how checking happens.

Kind How it works Strength Weakness
Proxy/cache downloads on demand, stores convenient, usable immediately lets everything through
Curated mirror only approved packages full control maintenance effort
Offline mirror copy without an internet connection air-gapped possible freshness, transport

Proxy/cache (pull-through cache). The mirror is a middleman. If a build asks for flask==3.0.3, the mirror looks in its warehouse. If the package is there, it delivers it. If not, it fetches it from the original, stores it and then delivers. Advantages: you have a central log, faster builds and independence from outages or rate limits at the original. Deleted or manipulated upstream versions no longer reach you once you have them in the cache. Disadvantage: without further rules, a proxy lets every package through, including a brand-new malicious package.

Curated mirror. Only packages that someone has approved are available. This can happen manually (request and review) or rule-based: “only packages that are older than 14 days, have no critical vulnerability and carry an allowed license”. Commercial products such as Sonatype Repository Firewall or JFrog Curation automate this. With built-in means, you build it from a proxy plus an internal (hosted) repository and an approval process.

Offline mirror. In an environment without internet (air-gapped), there is no upstream. Packages are collected outside, checked and imported via a controlled transfer (storage media, data diode). Tools for this include skopeo sync or Harbor replication for images, pip download for Python packages and aptly or apt-mirror for Debian. Chapter 4 shows how the transfer works securely.

Remember: In practice, the forms are combined. Typical: a proxy with rules in the build zone, behind it a curated repository for production, and an offline mirror for highly sensitive networks.

Repository types: proxy, hosted, group

Nexus and Artifactory know the same three building blocks, only their names differ slightly.

Nexus Artifactory Purpose
proxy remote mirrors an external source
hosted local your own, internally built packages
group virtual bundles several repos under one URL

A group repository is practical: clients know only one URL, and behind it are internal and mirrored packages. But exactly here lurks a trap: dependency confusion. If there is an internal package firma-utils in version 1.2 and an attacker uploads a package with the same name in version 99.0 to PyPI, the client may take the higher version from the proxy.

Security: Countermeasures against dependency confusion: internal package names with a unique prefix or scope (@firma/... with npm), reserve internal names publicly, set routing or exclusion rules for internal names on the proxy in the mirror. And never use pip install --extra-index-url, because pip treats all indexes as equal.

Enforcing the “only supply path”

A mirror that can be bypassed is a recommendation, not a control. Enforcement is a matter of several layers:

Layer Measure Enforces?
1. Client configuration pip.conf, .npmrc, apt sources, registry mirror no, just convenient
2. Network egress blocking, firewall, NetworkPolicy yes
3. Admission only images from an allowed registry (Chapter 9) yes, in the cluster
4. Checking linter searches for foreign URLs in the build detects deviations
5. Monitoring alert on blocked connections makes it visible

Layer 1 ensures that the normal path is the right one. Developers should not have to fight. Layer 2 is the actual enforcement. Build hosts and runner pods may only reach the mirror and a few internal services (DNS, Git server). Everything else is dropped at the firewall or by a Kubernetes NetworkPolicy. Important: the mirror itself of course needs outbound access, but only to the upstream addresses you have configured.

For Kubernetes runners, an egress block looks like this (addresses are examples):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: build-egress-nur-spiegel
  namespace: ci-runner
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - ipBlock:
            cidr: 10.20.30.40/32        # Nexus
      ports:
        - { protocol: TCP, port: 8081 }
        - { protocol: TCP, port: 8082 }
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

As soon as a NetworkPolicy with Egress selects a pod, all outbound traffic for that pod that is not explicitly allowed is forbidden. Prerequisite: the cluster’s network plugin (CNI) actually enforces NetworkPolicies, for example Calico or Cilium.

Layer 3 takes effect at deployment. A Kyverno or OPA rule rejects every pod whose image does not start with nexus.intern:8082/ or harbor.intern/. You build that in Chapter 9.

What all needs to be mirrored

A typical container build draws from three worlds:

  • Container images: base images such as python:3.12-slim. Clients: Docker, containerd, Podman, BuildKit.
  • Operating system packages: apt-get install on Debian/Ubuntu, dnf install on RHEL/Fedora. The package lists are signed with GPG; the mirror passes the signatures through unchanged.
  • Language packages: PyPI (pip), npm, Maven Central, Go modules, NuGet, crates.io.

Client configuration at a glance:

Ecosystem Configuration Example value
pip /etc/pip.conf or PIP_INDEX_URL index-url = https://nexus.intern/repository/pypi-proxy/simple
npm .npmrc registry=https://nexus.intern/repository/npm-proxy/
apt /etc/apt/sources.list.d/*.sources URIs: https://nexus.intern/repository/debian-bookworm/
dnf /etc/yum.repos.d/*.repo baseurl=https://nexus.intern/repository/rocky-baseos/

For Maven, you add a mirror for all repositories in settings.xml:

<settings>
  <mirrors>
    <mirror>
      <id>nexus</id>
      <mirrorOf>*</mirrorOf>
      <url>https://nexus.intern/repository/maven-public/</url>
    </mirror>
  </mirrors>
</settings>

For container images, there are two ways. Either you write the mirror directly into the image reference (nexus.intern:8082/library/python:3.12-slim). Or you configure the container runtime so that it redirects requests for docker.io to the mirror. With containerd, this is done via hosts.toml:

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

[host."https://nexus.intern:8082"]
  capabilities = ["pull", "resolve"]

Warning: containerd falls back to the server entry if the mirror does not respond. Without egress blocking, the redirect is therefore not enforcement. With Docker, the registry-mirrors option in daemon.json also applies only to images from Docker Hub, not to ghcr.io or quay.io.

Remember: A mirror does not change the digest of an image. python:3.12-slim@sha256:abc… is the same content via Docker Hub and via Nexus. Digest pinning from Chapter 2 and mirrors therefore complement each other perfectly.

Harbor: proxy cache projects and replication

Harbor is a CNCF registry specifically for OCI artifacts (images, Helm charts, signatures, SBOMs). Harbor is not intended for language packages. For images, it brings two fitting features.

Proxy cache project. You create a registry endpoint (for example Docker Hub) and a project that serves as a proxy cache for this endpoint. Clients then pull via harbor.intern/<project>/<image>. Via the Harbor API (v2.0), it looks like this in condensed form:

# 1. Create a registry endpoint for Docker Hub
curl -s -u "admin:${HARBOR_PW}" -H 'Content-Type: application/json' \
  -X POST https://harbor.intern/api/v2.0/registries -d '{
    "name": "dockerhub",
    "type": "docker-hub",
    "url": "https://hub.docker.com",
    "credential": {"type": "basic", "access_key": "hubuser",
                   "access_secret": "<Token aus Vault>"}
  }'

# 2. Create a proxy cache project (registry_id from GET /api/v2.0/registries)
curl -s -u "admin:${HARBOR_PW}" -H 'Content-Type: application/json' \
  -X POST https://harbor.intern/api/v2.0/projects -d '{
    "project_name": "dockerhub-proxy",
    "registry_id": 1,
    "metadata": {"public": "false"}
  }'

# 3. Use it
docker pull harbor.intern/dockerhub-proxy/library/python:3.12-slim

For official Docker Hub images, library/ belongs in the path (the placeholder <Token aus Vault> means “token from Vault”). Nobody can push into a proxy cache project; it is purely read access to the upstream.

Replication. Harbor can copy artifacts between registries based on rules, via push (away from Harbor) or pull (toward Harbor), scheduled, event-driven or manually. That way you distribute approved base images from a central Harbor instance to the sites or into a target zone. A rule via the API, condensed:

{
  "name": "basisimages-in-zielzone",
  "dest_registry": { "id": 3 },
  "dest_namespace": "basis",
  "filters": [
    { "type": "name", "value": "basis/**" },
    { "type": "tag", "value": "v*" }
  ],
  "trigger": {
    "type": "scheduled",
    "trigger_settings": { "cron": "0 0 2 * * *" }
  },
  "override": true,
  "enabled": true
}

The rule is created with POST /api/v2.0/replication/policies. Without src_registry, the local Harbor instance is the source (push replication). Harbor uses a cron format with six fields; the first field is seconds. So this rule runs daily at 02:00. For the transfer across zone boundaries, replication alone is not enough, because checking and approval are missing. Chapter 4 adds that.

Artifactory: remote and virtual repositories

JFrog Artifactory is a commercial repository manager with very broad format support. The model corresponds to Nexus:

  • Remote repository: proxy for an external source, for example pypi-remote on https://files.pythonhosted.org.
  • Local repository: your own artifacts, for example docker-local for built images.
  • Virtual repository: bundles local and remote under one name, for example pypi from pypi-local and pypi-remote. The resolution order is configurable.

The client URLs follow a fixed scheme:

pip:    https://art.intern/artifactory/api/pypi/pypi/simple
npm:    https://art.intern/artifactory/api/npm/npm/
Docker: art.intern/docker/python:3.12-slim   (depending on the Docker access method)

For automation, there is the JFrog CLI jf and the REST API. The principles “only supply path” and “egress blocking” apply in the same way.

Hands-on Lab: Nexus as the only supply path

You start Nexus Repository, create four proxy repositories via the REST API and build the demo-app so that it obtains everything through Nexus. Finally, you block the internet for the build container and prove that it still works.

Prerequisites from Chapter 0: Docker runs with at least 8 GB RAM, ~/seclab/demo-app exists, curl, jq and crane are installed.

  Host (localhost)
  +-------------------------------------------------------------+
  |  Docker network "bridge" (with internet)                    |
  |      +---------------------------+                          |
  |      | nexus  :8081 (UI, PyPI,   |----> Docker Hub, PyPI,   |
  |      |        apt, npm)          |      deb.debian.org, npm |
  |      |        :8082 (Docker)     |                          |
  |      +---------------------------+                          |
  |                  |                                          |
  |  Docker network "seclab-intern" (--internal, NO internet)   |
  |      +------------------+                                   |
  |      | build container  |---- only http://nexus:8081 -----> |
  |      +------------------+                                   |
  +-------------------------------------------------------------+

Step 1: Create the working directory and the internal network

mkdir -p ~/seclab/kap03
cd ~/seclab/kap03
cp -r ~/seclab/demo-app demo-app
docker network create --internal seclab-intern
docker network inspect seclab-intern -f '{{.Internal}}'

Expected output (the first line is the ID of the new network):

3f9c1a...
true

--internal means: containers in this network can reach each other, but not the outside. This is our simulation of the firewall in front of the build hosts.

Step 2: Start Nexus

docker run -d --name nexus \
  -p 8081:8081 -p 8082:8082 \
  -v nexus-data:/nexus-data \
  -e INSTALL4J_ADD_VM_PARAMS="-Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g" \
  sonatype/nexus3
docker network connect seclab-intern nexus

Nexus is now attached to two networks: the default network bridge with internet (for the upstream) and seclab-intern (for the builds). Port 8081 is the UI and REST API as well as the access point for PyPI, apt and npm. We reserve port 8082 for the Docker proxy, because Docker clients expect a registry at the root of a host and port.

Startup takes one to three minutes. Wait until the status API responds:

NEXUS=http://localhost:8081
until [ "$(curl -s -o /dev/null -w '%{http_code}' \
  "$NEXUS/service/rest/v1/status")" = "200" ]; do
  echo "warte auf Nexus ..."; sleep 10
done
echo "Nexus ist bereit"

Expected output (“waiting for Nexus …” until “Nexus is ready”):

warte auf Nexus ...
warte auf Nexus ...
Nexus ist bereit

Tip: The heap setting via INSTALL4J_ADD_VM_PARAMS keeps Nexus at around 2 GB of memory in the lab. For production, Sonatype specifies considerably more. Since version 3.77, the free variant is called “Community Edition” and has usage limits; they don’t matter for the lab.

Step 3: Get and change the admin password

On first start, Nexus generates a random admin password in a file:

INITIAL_PW=$(docker exec nexus cat /nexus-data/admin.password)
echo "$INITIAL_PW"

Expected output: a UUID, for example 5c1d8a3e-....

Change it via the API to a lab password:

NEXUS_PW='lab-nexus-123'   # lab value, never in production
curl -s -o /dev/null -w '%{http_code}\n' -u "admin:${INITIAL_PW}" \
  -X PUT -H 'Content-Type: text/plain' --data "${NEXUS_PW}" \
  "$NEXUS/service/rest/v1/security/users/admin/change-password"

Expected output:

204

204 No Content means success. From now on we use a small helper function to keep the commands short:

nx() { curl -s -u "admin:${NEXUS_PW}" -H 'Content-Type: application/json' "$@"; }
nx "$NEXUS/service/rest/v1/security/users?userId=admin" | jq -r '.[0].userId'

Expected output:

admin

Warning: Newer Nexus versions (Community Edition) require you to accept the license (EULA) at the first login. To do so, log in once in the UI at http://localhost:8081 as admin and follow the wizard. Choose “Enable anonymous access” there; we need it in the next step. Older versions don’t ask for a EULA.

Step 4: Enable anonymous read access and the Docker realm

So that build clients can read without logging in, we enable anonymous access. For anonymous Docker pulls, Nexus additionally needs the “Docker Bearer Token Realm”:

nx -X PUT "$NEXUS/service/rest/v1/security/anonymous" \
  -d '{"enabled": true, "userId": "anonymous",
       "realmName": "NexusAuthorizingRealm"}' | jq '.enabled'

REALMS=$(nx "$NEXUS/service/rest/v1/security/realms/active" \
  | jq -c '. + ["DockerToken"] | unique')
nx -o /dev/null -w '%{http_code}\n' -X PUT \
  "$NEXUS/service/rest/v1/security/realms/active" -d "$REALMS"
nx "$NEXUS/service/rest/v1/security/realms/active"

Expected output:

true
204
["DockerToken","NexusAuthenticatingRealm"]

The exact list of realms may differ depending on the version. What matters is that DockerToken is included.

Security: Anonymous reading is convenient in the lab. In production, you give each pipeline its own technical user with read permissions or use tokens. Then you see in the log which pipeline fetched which package.

Step 5: Create a Docker proxy for Docker Hub

Create the configuration as a JSON file:

cat > docker-hub.json <<'EOF'
{
  "name": "docker-hub",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "https://registry-1.docker.io",
    "contentMaxAge": 1440,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true },
  "docker": { "v1Enabled": false, "forceBasicAuth": false, "httpPort": 8082 },
  "dockerProxy": { "indexType": "HUB" }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @docker-hub.json \
  "$NEXUS/service/rest/v1/repositories/docker/proxy"

Expected output:

201

The most important fields:

  • remoteUrl: the upstream, here the Docker Hub registry API.
  • contentMaxAge / metadataMaxAge: minutes for which content and metadata (such as the tag-to-digest mapping) are considered fresh.
  • docker.httpPort: 8082: Nexus opens its own HTTP connector on port 8082 for this repository.
  • forceBasicAuth: false: allows anonymous pulls via the token realm from step 4.
  • indexType: HUB: search and index run against Docker Hub.

Now the test. Docker allows HTTP registries on localhost without extra configuration:

docker pull localhost:8082/library/python:3.12-slim
crane digest localhost:8082/library/python:3.12-slim
crane digest python:3.12-slim

Expected output (shortened):

3.12-slim: Pulling from library/python
...
Status: Downloaded newer image for localhost:8082/library/python:3.12-slim
localhost:8082/library/python:3.12-slim
sha256:4a3f...
sha256:4a3f...

Both digests are identical. The mirror delivers bit-for-bit the same image as Docker Hub.

Tip: Docker Hub limits anonymous pulls per IP address. Because only Nexus now queries Docker Hub and caches everything, the number of pulls drops drastically. In production, you also store a Docker Hub account in the proxy under httpClient.authentication.

Step 6: Create a PyPI proxy

cat > pypi-proxy.json <<'EOF'
{
  "name": "pypi-proxy",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "https://pypi.org",
    "contentMaxAge": 1440,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @pypi-proxy.json \
  "$NEXUS/service/rest/v1/repositories/pypi/proxy"
curl -s "$NEXUS/repository/pypi-proxy/simple/flask/" \
  | grep -o 'flask-3.0.3[^<"#]*' | sort -u

Expected output:

201
flask-3.0.3-py3-none-any.whl
flask-3.0.3.tar.gz

The page /simple/flask/ is the “Simple Repository Index” according to PEP 503, exactly what pip queries. Nexus rewrites the download links in it so that they point to Nexus itself and not to files.pythonhosted.org.

Step 7: Create an APT proxy for Debian bookworm

cat > apt-proxy.json <<'EOF'
{
  "name": "debian-bookworm",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "http://deb.debian.org/debian/",
    "contentMaxAge": 1440,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true },
  "apt": { "distribution": "bookworm", "flat": false }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @apt-proxy.json \
  "$NEXUS/service/rest/v1/repositories/apt/proxy"

Expected output:

201

Now we test from inside the internal network. Debian images from bookworm on use the deb822 format under /etc/apt/sources.list.d/debian.sources. We replace this file with one that points to Nexus:

cat > nexus.sources <<'EOF'
Types: deb
URIs: http://nexus:8081/repository/debian-bookworm/
Suites: bookworm
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
EOF
docker pull localhost:8082/library/debian:bookworm-slim
docker run --rm --network seclab-intern \
  -v "$PWD/nexus.sources:/etc/apt/sources.list.d/debian.sources:ro" \
  localhost:8082/library/debian:bookworm-slim \
  bash -c 'apt-get update && apt-get install -y --no-install-recommends jq \
    && jq --version'

Expected output (shortened):

Get:1 http://nexus:8081/repository/debian-bookworm bookworm InRelease [151 kB]
Get:2 http://nexus:8081/repository/debian-bookworm bookworm/main amd64 Packages ...
...
Setting up jq (1.6-2.1) ...
jq-1.6

Note two things. First: the container sits in the internal network without internet, yet the installation works. Second: Signed-By points to the Debian keyring in the image. apt still verifies the signature of the package lists against the Debian key. A manipulated mirror could not slip in forged packages without apt aborting.

Remember: With Debian, security updates come from a separate archive (security.debian.org, suite bookworm-security). In practice, you create a second APT proxy repository for it. That is part of the exercise.

Step 8: Create an npm proxy

cat > npm-proxy.json <<'EOF'
{
  "name": "npm-proxy",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "https://registry.npmjs.org",
    "contentMaxAge": 1440,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @npm-proxy.json \
  "$NEXUS/service/rest/v1/repositories/npm/proxy"
curl -s "$NEXUS/repository/npm-proxy/lodash" | jq -r '."dist-tags".latest'

Expected output:

201
4.17.21

The version number may be higher on your machine. An npm client uses the proxy with one line in the project’s .npmrc:

registry=http://localhost:8081/repository/npm-proxy/

Finally, check which repositories now exist:

nx "$NEXUS/service/rest/v1/repositories" \
  | jq -r '.[] | select(.type=="proxy") | "\(.format)\t\(.name)"'

Expected output (excerpt; Nexus also ships its own default repositories such as maven-central):

docker  docker-hub
pypi    pypi-proxy
apt     debian-bookworm
npm     npm-proxy
maven2  maven-central

Step 9: Counter-test – nothing works without the mirror

Now we prove that the internal network is really blocked. A build container tries to download directly from PyPI:

docker run --rm --network seclab-intern \
  localhost:8082/library/python:3.12-slim \
  pip download --no-deps --retries 0 --timeout 5 -d /tmp flask==3.0.3

Expected output (shortened; the error message depends on the Docker version):

WARNING: ... Failed to establish a new connection: [Errno -3] Temporary
  failure in name resolution ...
ERROR: Could not find a version that satisfies the requirement flask==3.0.3
  (from versions: none)
ERROR: No matching distribution found for flask==3.0.3

Instead of the name resolution error, [Errno 101] Network is unreachable may also appear. Both mean: no path to the internet.

Now the same container with the mirror. To do this, create a pip.conf:

cat > pip.conf <<'EOF'
[global]
index-url = http://nexus:8081/repository/pypi-proxy/simple
trusted-host = nexus
EOF
docker run --rm --network seclab-intern \
  -v "$PWD/pip.conf:/etc/pip.conf:ro" \
  localhost:8082/library/python:3.12-slim \
  pip download --no-deps -d /tmp flask==3.0.3

Expected output:

Looking in indexes: http://nexus:8081/repository/pypi-proxy/simple
Collecting flask==3.0.3
  Downloading http://nexus:8081/repository/pypi-proxy/packages/flask/3.0.3/...
Saved /tmp/flask-3.0.3-py3-none-any.whl
Successfully downloaded flask

pip reads /etc/pip.conf as the system-wide configuration. trusted-host is needed because we use HTTP instead of HTTPS in the lab. In production, Nexus sits behind TLS with a certificate from your own PKI, and then trusted-host is no longer needed.

Step 10: Fetch the demo-app’s dependencies through the mirror

Now the real build step. The build container in the internal network downloads all of the demo-app’s dependencies as wheels into a directory. It runs with your UID so that the files belong to you:

cd ~/seclab/kap03/demo-app
cp ../pip.conf .
docker run --rm --network seclab-intern \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD/pip.conf:/etc/pip.conf:ro" -v "$PWD":/w -w /w \
  localhost:8082/library/python:3.12-slim \
  pip download --only-binary=:all: -r requirements.txt -d wheels
ls wheels

Expected output (versions of the transitive packages may differ):

blinker-1.8.2-py3-none-any.whl
certifi-2024.8.30-py3-none-any.whl
chardet-4.0.0-py2.py3-none-any.whl
click-8.1.7-py3-none-any.whl
flask-3.0.3-py3-none-any.whl
gunicorn-22.0.0-py3-none-any.whl
idna-2.10-py2.py3-none-any.whl
itsdangerous-2.2.0-py3-none-any.whl
jinja2-3.1.4-py3-none-any.whl
markupsafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64....whl
packaging-24.1-py3-none-any.whl
requests-2.25.0-py2.py3-none-any.whl
urllib3-1.26.20-py2.py3-none-any.whl
werkzeug-3.0.4-py3-none-any.whl

--only-binary=:all: prevents pip from downloading source packages that could execute code during installation. If you generated a requirements.lock with hashes in Chapter 2, you can use it here instead of requirements.txt. The hashes match because Nexus serves the files unchanged.

Step 11: Build the image hermetically

Now we build the image without any network access. The base image comes from the mirror, the packages from the wheels directory:

cat > Dockerfile.mirror <<'EOF'
# syntax=docker/dockerfile:1
ARG BASE_IMAGE=localhost:8082/library/python:3.12-slim
FROM ${BASE_IMAGE} AS build
WORKDIR /build
COPY requirements.txt .
COPY wheels/ /wheels/
RUN pip install --no-cache-dir --no-index --find-links=/wheels \
      --prefix=/install -r requirements.txt

FROM ${BASE_IMAGE}
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY --from=build /install /usr/local
WORKDIR /app
COPY app.py .
USER 10001:10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]
EOF
docker build --network=none -f Dockerfile.mirror \
  -t localhost:5001/demo-app:1.0.0-mirror .

Expected output (last lines):

 => [build 5/5] RUN pip install --no-cache-dir --no-index --find-links=/wheels ...
 ...
 => => naming to localhost:5001/demo-app:1.0.0-mirror

What happens here:

  • --network=none takes the network away from all RUN steps. The build is thus hermetic: it cannot download anything that was not in the build context beforehand.
  • --no-index --find-links=/wheels forces pip to install only from the local directory.
  • The base image is resolved by the Docker daemon, not by the RUN step. It comes from Nexus via localhost:8082.
  • The global ARG BASE_IMAGE before the first FROM applies to both FROM lines. In production, you pass your organization’s mirror here via --build-arg, ideally with a digest.

Warning: If the .dockerignore from Chapter 0 excludes the wheels/ directory, COPY wheels/ fails. Check with cat .dockerignore.

Start the image as a check and push it to the build registry:

docker run -d --rm --name demo-mirror -p 18001:8000 \
  localhost:5001/demo-app:1.0.0-mirror
sleep 2 && curl -s http://localhost:18001/ && docker stop demo-mirror
docker push localhost:5001/demo-app:1.0.0-mirror
crane digest localhost:5001/demo-app:1.0.0-mirror

Expected output:

{"status":"ok"}
demo-mirror
...
sha256:b81f...

Step 12: Check the evidence in the mirror

The mirror now knows exactly what your builds obtained. That is the foundation for audit and vulnerability response:

nx "$NEXUS/service/rest/v1/components?repository=pypi-proxy" \
  | jq -r '.items[] | "\(.name) \(.version)"' | sort
nx "$NEXUS/service/rest/v1/components?repository=docker-hub" \
  | jq -r '.items[] | "\(.name):\(.version)"'

Expected output (excerpt):

flask 3.0.3
gunicorn 22.0.0
requests 2.25.0
urllib3 1.26.20
...
library/debian:bookworm-slim
library/python:3.12-slim

The API returns one page per call. With many components, the response contains a field continuationToken, which you append as a parameter for the next page.

Imagine that tomorrow a critical vulnerability in urllib3 1.26.20 becomes known. With this list, you know immediately whether your builds ever obtained that version. Together with SBOMs (Chapter 7), you also see which images contain it.

Step 13: Clean up

docker stop nexus
docker network disconnect seclab-intern nexus
docker network rm seclab-intern
docker image rm localhost:8082/library/debian:bookworm-slim

The volume nexus-data is preserved. With docker start nexus you have the mirror back, cache included. If you want to start completely from scratch: docker rm nexus && docker volume rm nexus-data.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Solution
curl on the status endpoint returns 000 Nexus is still starting wait 1–3 minutes, check docker logs nexus
401 Unauthorized on docker pull Docker realm missing or forceBasicAuth enable DockerToken, forceBasicAuth: false
http: server gave HTTP response to HTTPS client registry not on localhost set insecure-registries or use TLS
400 when creating a repo required field missing in the JSON or name taken read the response text, omit -w
Port 8082 does not respond httpPort not set or port not mapped check -p 8082:8082 and docker.httpPort
pip: not a trusted or secure host HTTP without trusted-host set trusted-host or use HTTPS
apt: NO_PUBKEY or not signed wrong Signed-By path check the path to the Debian keyring
Proxy suddenly delivers nothing autoBlock after upstream errors check the status in the UI, test upstream reachability

Exercise: Extend the mirror

Your organization needs two more supply sources:

  1. Debian security updates for bookworm as a separate APT proxy repository debian-security.
  2. Maven Central as a separate proxy repository maven-proxy.

Create both via the REST API. Check the security updates with an apt-get update in the internal network and Maven with a curl on a POM file, for example org/apache/commons/commons-lang3/3.14.0/commons-lang3-3.14.0.pom.

Hint 1: The Debian security updates live under http://security.debian.org/debian-security/; the suite is called bookworm-security. In the deb822 file, you can write several sections one below the other, separated by a blank line.

Hint 2: The endpoint for Maven is /service/rest/v1/repositories/maven/proxy. Maven repositories additionally need a "maven" block with versionPolicy (for example RELEASE) and layoutPolicy (for example STRICT).

Solution

APT proxy for the security updates:

cat > apt-security.json <<'EOF'
{
  "name": "debian-security",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "http://security.debian.org/debian-security/",
    "contentMaxAge": 1440,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true },
  "apt": { "distribution": "bookworm-security", "flat": false }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @apt-security.json \
  "$NEXUS/service/rest/v1/repositories/apt/proxy"

The sources file with two sections:

Types: deb
URIs: http://nexus:8081/repository/debian-bookworm/
Suites: bookworm
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://nexus:8081/repository/debian-security/
Suites: bookworm-security
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

After apt-get update in the internal network, lines with debian-security bookworm-security InRelease appear.

Maven proxy:

cat > maven-proxy.json <<'EOF'
{
  "name": "maven-proxy",
  "online": true,
  "storage": { "blobStoreName": "default", "strictContentTypeValidation": true },
  "proxy": {
    "remoteUrl": "https://repo1.maven.org/maven2/",
    "contentMaxAge": -1,
    "metadataMaxAge": 1440
  },
  "negativeCache": { "enabled": true, "timeToLive": 1440 },
  "httpClient": { "blocked": false, "autoBlock": true },
  "maven": { "versionPolicy": "RELEASE", "layoutPolicy": "STRICT" }
}
EOF
nx -o /dev/null -w '%{http_code}\n' -X POST -d @maven-proxy.json \
  "$NEXUS/service/rest/v1/repositories/maven/proxy"
POM=org/apache/commons/commons-lang3/3.14.0/commons-lang3-3.14.0.pom
curl -s -o /dev/null -w '%{http_code}\n' "$NEXUS/repository/maven-proxy/$POM"

Expected: 201 twice, then 200. contentMaxAge: -1 means “never recheck”. That is right for Maven releases, because a published release must never change. Why a separate name? Nexus already ships a repository maven-central. In production, you remove unused default repositories so that nobody accidentally obtains packages by bypassing your setup.

Save your progress? Sign up for free

From the field

Architecture in regulated environments. In public authorities and critical infrastructure (KRITIS) companies, the mirror typically sits in a DMZ or a dedicated “supply zone”. Only the mirror may go to the internet, via a forward proxy with an allowlist, and only to the configured upstreams. The build zone reaches only the mirror. The production zone reaches only a second, curated registry that is filled through a controlled transfer (Chapter 4). In air-gapped networks there is no upstream at all; there, the mirror is updated via storage media or a data diode.

Curation with built-in means. Without commercial firewall products, many organizations work with two stages. A proxy in the build zone is open to developers and is continuously scanned. A hosted repository for production contains only packages and images that have gone through an approval process: scan, license check, possibly a minimum age. Nexus OSS additionally offers routing rules with which you can allow or block paths by regular expression, for example to block internal package names on the PyPI proxy.

Operations. A mirror becomes a critical dependency. If it fails, nobody builds anymore. So plan for high availability or at least fast recovery, backups of the blob store and the database, storage monitoring and cleanup policies for old content. Nexus and Harbor provide metrics in Prometheus format that you feed into your monitoring.

Monitoring enforcement. The proof that the mirror really is the only supply path comes from the firewall. Evaluate blocked egress connections from build hosts and alert on them. An increase shows you a team trying to build around the mirror, often unintentionally because a tool has its own built-in download URL. You handle these cases with the teams, not against them (Chapter 10).

Credentials. The mirror needs its own credentials for the upstreams, such as a Docker Hub account, and the pipelines need read access to the mirror. This data belongs in a secrets management system and is rotated regularly (Chapter 6). A separate technical user per pipeline makes the access log meaningful.

Audit. Auditors typically want to see three things: a list of the allowed upstreams with justification, technical proof of the egress block (firewall rules, NetworkPolicies) and logs of which components were obtained when. Keep the repository configuration as code, for example as JSON files in Git as in this lab, or with the Terraform provider for Nexus or Harbor. Then every change to the supply path is a traceable merge request.

Interview questions

Q: What is the difference between a proxy cache and a curated mirror?

A: A proxy cache downloads every requested package from the upstream on demand and stores it. It brings logging, speed and independence, but lets everything through. A curated mirror contains only approved packages, checked manually or rule-based. It gives more control but costs maintenance effort.

Q: How do you make sure the mirror really is the only supply path?

A: With several layers. Client configuration makes the right path convenient. Enforcement happens via the network: egress blocking at the firewall or by NetworkPolicy, only the mirror may reach the internet. In the cluster, additionally an admission policy that admits only images from your own registry. And monitoring of the blocked connections.

Q: What is dependency confusion and how do you protect yourself?

A: An attacker publishes a public package with the name of an internal package and a higher version. Clients that use several indexes as equals take the higher version. Protection: no --extra-index-url, internal names with a scope or prefix, reserve names publicly, block internal names on the proxy in the mirror.

Q: Does a mirror change the digest of an image?

A: No. The digest is the hash of the manifest. A mirror delivers the manifest and layers unchanged, which is why digest pinning and signature verification also work through the mirror.

Q: Why can’t a compromised APT mirror simply swap packages?

A: The package lists are signed by the Debian archive, and the lists contain the hashes of all packages. apt verifies the signature against the local keyring. At most, a mirror can serve old, correctly signed states or hold back updates. The Valid-Until fields in the Release files and monitoring help against that.

Q: How does a proxy cache project in Harbor work?

A: You create a registry endpoint, such as Docker Hub, and a project that references this endpoint. Clients pull via harbor/<project>/<image>. Harbor fetches missing images from the upstream and caches them. Pushing into the project is not possible.

Q: What is a hermetic build and how does it relate to the mirror?

A: A hermetic build has no network access during execution. All inputs are declared and provided beforehand. The mirror is the controlled source from which these inputs are fetched in a preceding step, for example with pip download. The actual build then runs with --network=none.

Summary

What you learned:

  • A mirror is the pharmacy of your supply chain: obtain centrally, store, log, be able to block.
  • Proxy cache, curated mirror and offline mirror complement each other. The choice depends on the zone and the protection level.
  • The “only supply path” only comes about through an egress block. Client configuration alone is merely a recommendation.
  • With the Nexus REST API, you created proxies for Docker Hub, PyPI, Debian and npm, as traceable code in JSON files.
  • A container in the --internal network could not reach PyPI, but installed without problems through Nexus. The actual image build ran hermetically with --network=none.
  • Digests and signatures remain valid through the mirror. Digest pinning, apt signatures and pip hashes keep working.
  • Harbor offers proxy cache projects and rule-based replication for OCI artifacts; Artifactory implements the same model with remote, local and virtual repositories.