Skip to content
Hands-on IT IT Drills

Day 15: Competition – disaster drills

Goal: Under time pressure, you find and fix eight typical Renovate failures, from a broken config to a lockfile that fails on a peer dependency. · Duration: approx. 120 min · Prerequisite: Day 14

Briefing

  • Each round has its own repo in the org shop. aufstellung.sh builds all starting states at once, including a “nightly run” of Renovate.
  • Per round: take in the situation, start the stopwatch, repair until the round’s check passes. Measure first (log, dashboard, API), then change.
  • The hint costs half the points. Read the solution only afterwards.
  • No shortcuts via admin rights: Renovate has to produce the state itself.
 Failure ──► situation (issue, dashboard, PR, log) ──► diagnosis ──► smallest repair
                                                       └──► Renovate run ──► check   

Drill 0: Setup (untimed)

Under repos/, the companion code brings the files of the seven competition repos and the lab tool aufstellung.sh. It builds the failures you are supposed to find – what it does in detail is shown by grep '^# [0-9]' aufstellung.sh; read the rest only after the competition. It creates seven repos (copy, git init, push as on Day 0), publishes @shop/money as on Day 8 and lets Renovate run over everything once (nacht.log).

source ~/renovate-lab/setup/env.sh
export RENOVATE_GITHUB_COM_TOKEN=$(gh auth token)   # optional, see Day 0
mkdir -p ~/renovate-lab/tag15 && cd ~/renovate-lab/tag15
cp ~/renovate-lab/code/tag15/aufstellung.sh .
./aufstellung.sh

Expected (abridged):

R15_AUF

You can run it again at any time; the repos are then created fresh.

The numbers and target versions in this chapter come from the test run; your versions will be newer. Without RENOVATE_GITHUB_COM_TOKEN, the GitHub Actions updates in kasse are missing; nothing else changes.

Warning: Play the rounds in order, 1 to 8. Round 2 uses shop/kasse for its test, and every round ends with a real Renovate run.

Save your progress? Sign up for free

Round 1: A missing comma

Situation: Since yesterday, not a single PR has come from shop/kasse. Instead, there is an issue in the repo.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/issues?type=issues" \
  | jq -r '.[] | "#\(.number) \(.state) \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
grep -A4 'invalid config' nacht.log | cut -c1-88

Expected:

#1 open Action Required: Fix Renovate Configuration
 WARN: Repository has invalid config (repository=shop/kasse)
       "err": {
         "validationSource": "renovate.json",
         "validationError": "Invalid JSON (parsing failed)",
         "validationMessage": "Syntax error: expecting number near  0\n  \"pack",

Task: Renovate opens PRs in kasse again, and the issue is closed.

Time budget: 5 min · Points: 10

Hint: The message in the issue is cryptic; jq and the validator name the line and column.

Solution:

cd ~/renovate-lab/tag15
rm -rf kasse
git clone -q forgejo:shop/kasse.git
cd kasse
renovate-config-validator 2>&1 | grep -E 'WARN|"message"'
jq . renovate.json > /dev/null
sed -n 4,5p renovate.json

Expected:

 WARN: File could not be parsed
         "message": "JSON5: invalid character '\\\"' at 5:3"
jq: parse error: Expected separator between values at line 5, column 16
  "prHourlyLimit": 0
  "packageRules": [

The comma after "prHourlyLimit": 0 is missing:

sed -i.bak 's/"prHourlyLimit": 0$/&,/' renovate.json && rm renovate.json.bak
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "renovate.json: fehlendes Komma"
git push -q
cd .. && renovate shop/kasse | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/issues?type=issues&state=all" \
  | jq -r '.[] | "#\(.number) \(.state) \(.title)"'

Expected:

 INFO: PR created (repository=shop/kasse, branch=renovate/express-4.x)
 INFO: PR created (repository=shop/kasse, branch=renovate/prettier-3.x)
 INFO: PR created (repository=shop/kasse, branch=renovate/node-20.x)
       "result": "done",
#5 open Dependency Dashboard
#1 closed Action Required: Fix Renovate Configuration

Renovate closes the error issue itself. The majors are missing on purpose: the config requires dependencyDashboardApproval for them.

Lesson:

  • A broken repo config means: exit code 0, "result": "config-validation" and an issue, but no PR (Day 9). Without looking at issues or logs, nobody notices.
  • Renovate reads renovate.json leniently as JSON5: a trailing comma at the end of a list passed in testing. Still make the validator a mandatory check in every repo’s CI (Day 2).

Save your progress? Sign up for free

Round 2: The rotated token

Situation: A colleague has “already rotated” the bot token. The new token is in wk-token and is supposed to go into the operations configuration tomorrow. You do the test.

RENOVATE_TOKEN=$(cat wk-token) renovate shop/kasse > token.log 2>&1; echo "exit=$?"
grep -E '^FATAL|errorMessage' token.log

Expected:

exit=1
FATAL: Initialization error
       "errorMessage": "Authentication failure"
           "errorMessage": "Authentication failure",

Task: Find the cause and create a replacement in wk-token with exactly the scopes Renovate needs. The broken token is deleted; no admin scope.

Time budget: 10 min · Points: 15

Hint: The first thing Renovate queries is GET /api/v1/user; in its response, Forgejo names the missing scope.

Solution: Forgejo’s response is clearer than Renovate’s.

curl -s -H "Authorization: token $(cat wk-token)" "$FORGEJO/api/v1/user" \
  | jq -r .message
curl -s -H "Authorization: token $(cat wk-token)" "$FORGEJO/api/v1/repos/shop/kasse" \
  | jq -c .permissions
BOT=renovate-bot:lab-bot-pw-123                     # lab value!
curl -s -u $BOT "$FORGEJO/api/v1/users/renovate-bot/tokens" \
  | jq -c '.[] | select(.name == "renovate-rotation") | .scopes'

Expected:

token does not have at least one of required scope(s): [read:user]
{"admin":false,"push":true,"pull":true}
["read:repository"]

The token only has read:repository. Careful with permissions: push: true is the permission of the user in the repo, not that of the token. If you only add read:user, in testing you run into the next error (read:organization when reading the org), then into remote: Forbidden on push. So go straight for the complete list from Day 0:

curl -s -o /dev/null -w "%{http_code}\n" -u $BOT -X DELETE \
  "$FORGEJO/api/v1/users/renovate-bot/tokens/renovate-rotation"
curl -s -u $BOT -H 'Content-Type: application/json' \
  -X POST "$FORGEJO/api/v1/users/renovate-bot/tokens" -d '{"name": "renovate-rotation",
  "scopes": ["write:repository", "write:issue", "read:user", "read:organization",
  "read:package"]}' | jq -j .sha1 > wk-token
RENOVATE_TOKEN=$(cat wk-token) renovate shop/kasse | grep -E '^FATAL|"result"'

Expected:

204
       "result": "done",

jq -j writes the token without a trailing newline (Day 10). The bot creates it itself via basic auth, so rotation works without admin access to the container.

Lesson:

  • With Forgejo, Authentication failure often means “wrong scope”, not “wrong token”. The actual message only shows up in the debug log or the API response.
  • Rotation is a procedure with a fixed scope list and a test run. read:package only if repos use the Forgejo registry (Round 7).

Save your progress? Sign up for free

Round 3: The regex sees nothing

Situation: The team maintains lint tools in the Makefile of shop/werkzeug and built a regex manager for them (Day 7). For weeks no update has come; the dashboard only lists the base image.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/werkzeug/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -E '^<details><summary>[a-z]|^ - `' dash.md
head -5 ~/renovate-lab/code/tag15/repos/werkzeug/Makefile

Expected:

<details><summary>dockerfile (1)</summary>
 - `node 20.11.0-alpine` → [Updates: `20.20.2-alpine`, `24.21.0-alpine`]
# Werkzeuge für Doku- und API-Checks (Version hier pflegen, nicht im CI-Job)
# renovate: datasource=npm depName=markdownlint-cli2
MARKDOWNLINT_VERSION ?= 0.10.0
# renovate: datasource=npm depName=@redocly/cli
REDOCLY_VERSION ?= 1.0.0

Task: markdownlint-cli2 and @redocly/cli appear in the dashboard and get PRs. The Makefile stays as it is.

Time budget: 10 min · Points: 15

Hint: --platform=local --dry-run=extract with LOG_LEVEL=debug tells you whether the file pattern or the regex fails.

Solution: First separate which part of the manager is failing.

cd ~/renovate-lab/tag15
rm -rf werkzeug
git clone -q forgejo:shop/werkzeug.git
cd werkzeug
LOG_LEVEL=debug renovate --platform=local --dry-run=extract > ../extract.log 2>&1
grep -E 'Matched .* manager regex|No dependencies found' ../extract.log | cut -c1-88
jq -r '.customManagers[0].matchStrings[0]' renovate.json | grep -o '\\n.*'

Expected:

DEBUG: Matched 1 file(s) for manager regex (repository=local)
DEBUG: No dependencies found in file for custom regex manager (repository=local, package
\n\w+_VERSION := (?<currentValue>\S+)

The file is found, the regex does not match. It expects :=, but the Makefile has ?=. From now on the assignment should match all three Make variants:

jq '.customManagers[0].matchStrings[0] |= sub(" := "; " [:?]?= ")' renovate.json \
  > neu.json && mv neu.json renovate.json
renovate --platform=local --dry-run=extract --report-type=file \
  --report-path=../extract.json > /dev/null 2>&1
jq -r '.repositories[].packageFiles.regex[].deps[]
  | "\(.depName) \(.currentValue) (\(.datasource))"' ../extract.json
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "Makefile-Regex: ?= statt :="
git push -q
cd .. && renovate shop/werkzeug | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/werkzeug/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -E '^ - `' dash.md

Expected:

markdownlint-cli2 0.10.0 (npm)
@redocly/cli 1.0.0 (npm)
 INFO: PR created (repository=shop/werkzeug, branch=renovate/redocly-cli-1.x)
 INFO: PR created (repository=shop/werkzeug, branch=renovate/markdownlint-cli2-0.x)
 INFO: PR created (repository=shop/werkzeug, branch=renovate/redocly-cli-2.x)
       "result": "done",
 - `node 20.11.0-alpine` → [Updates: `20.20.2-alpine`, `24.21.0-alpine`]
 - `markdownlint-cli2 0.10.0` → [Updates: `0.23.3`]
 - `@redocly/cli 1.0.0` → [Updates: `1.34.20`, `2.62.1`]

Lesson:

  • A regex manager that finds nothing produces no error and no warning. Only “Detected Dependencies” in the dashboard or --dry-run=extract reveal the gap.
  • Two debug lines separate the error sources: Matched N file(s) (file pattern) and No dependencies found in file (regex). The local extract needs no push.

Save your progress? Sign up for free

Round 4: The automerge that never comes

Situation: In shop/automat, patch and minor updates of the devDependencies are supposed to land automatically (Day 5). The prettier PR has been open since yesterday.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .body | grep 'Automerge\*\*'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" | jq -r .state
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" \
  | jq -r '.statuses[] | "\(.context) = \(.status)"'

Expected:

#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#3 open renovate/express-5.x | Update dependency express to v5
🚦 **Automerge**: Enabled.
gesamt: pending
  ci / test (pull_request) = pending (Waiting to run)
  ci / test (push) = pending (Waiting to run)

Task: The team has decided: from now on, the tests run in the central lab CI, which reports as ci/test (helper ci, as on Day 5). Get PR #2 to merge automatically, but only after a green ci/test. No ignoreTests, no manual merge.

Time budget: 15 min · Points: 15

Hint: Who is actually supposed to run ci / test? Ask the Actions API for runs and runners.

Solution: First the cause, then Renovate’s view.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/actions/runs" \
  | jq -r '.workflow_runs[] | "\(.status) \(.event) \(.title)"' | head -3
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/actions/runners"; echo
LOG_LEVEL=debug renovate shop/automat > automat.log 2>&1
grep 'renovate/prettier' automat.log \
  | grep -oE 'Branch status (yellow|green|red)|PR is not ready[^(]*' | sort | uniq -c

Expected:

waiting push Update dependency express to v5
waiting pull_request Update dependency express to v5
waiting push Update dependency prettier to v3.9.10
[]

   2 Branch status yellow
   1 PR is not ready for merge

Forgejo runs .github/workflows/ci.yml as Actions, but there is no runner. The runs wait forever, the checks stay pending, Renovate sees “yellow” and waits along with them. The old statuses are attached to the old commit, so after switching Actions off a rebase is needed. Branch protection comes first, otherwise there is a nasty surprise:

curl -s -u "$ADMIN" -X POST "$FORGEJO/api/v1/repos/shop/automat/branch_protections" --json '{"rule_name": "main",
  "enable_status_check": true, "status_check_contexts": ["ci/test"]}' \
  | jq -c '{rule_name, enable_status_check, status_check_contexts}'
curl -s -u "$ADMIN" -X PATCH "$FORGEJO/api/v1/repos/shop/automat" --json '{"has_actions":false}' | jq -c '{has_actions}'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- rebase-check|- [x] <!-- rebase-check|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .number
renovate shop/automat | grep -E '"result"'
sleep 5
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls?state=all" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep prettier
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" | jq -r .state
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" \
  | jq -r '.statuses[] | "\(.context) = \(.status)"'
SHA=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/branches/renovate%2Fprettier-3.x" | jq -r .commit.id)
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/statuses/$SHA" \
  --json '{"state":"success","context":"ci/test"}' | jq -r .status
sleep 5
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls?state=all" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep prettier

Expected:

{"rule_name":"main","enable_status_check":true,"status_check_contexts":["ci/test"]}
{"has_actions":false}
       "result": "done",
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
gesamt: 
success
#2 merged refs/pull/2/head | Update dependency prettier to v3.9.10

After the rebase, the new commit has not a single status. Forgejo now waits for the required check and only merges after ci/test = success. Forgejo triggers the merge itself: Renovate had scheduled it when creating the PR (platformAutomerge, Day 5).

Warning: Without branch protection, Forgejo merged in testing one second after the rebase push, without any test at all. A scheduled merge fires as soon as no check is pending anymore, even with zero checks. Renovate itself would have waited.

Lesson:

  • A single check that nobody ever processes blocks every automerge. Renovate reports this only in the debug log as Branch status yellow.
  • In production, the repair means: register a runner or remove the workflow. Switching off Actions is only right if another CI takes over.
  • platformAutomerge on Forgejo only with branch protection and required checks (Day 5). Only they make “test first, then merge” the rule.

Save your progress? Sign up for free

Round 5: The PR flood

Situation: Someone wanted to “just see everything” in shop/flut and released all the brakes. The nightly run flooded the repo.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | wc -l
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep -E 'dotenv' | head -3
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/raw/renovate.json" | jq 'del(."$schema")'

Expected:

      32
#11 open renovate/dotenv-11.x | Update dependency dotenv to v11
#12 open renovate/dotenv-12.x | Update dependency dotenv to v12
#13 open renovate/dotenv-13.x | Update dependency dotenv to v13
{
  "extends": [
    "config:recommended"
  ],
  "prHourlyLimit": 0,
  "prConcurrentLimit": 0,
  "branchConcurrentLimit": 0,
  "separateMinorPatch": true,
  "separateMultipleMajor": true
}

Task: At most five open PRs plus one bulk PR for all minor and patch updates, which still arrives today. New majors only after approval in the dashboard. Renovate cleans up the leftovers itself; no PR is closed by hand.

Time budget: 15 min · Points: 15

Hint: A PR closed by hand is an ignore (Round 6); whatever Renovate no longer needs, it closes itself as “autoclosed”.

Solution: A new config from the building blocks of Day 4:

cd ~/renovate-lab/tag15
rm -rf flut
git clone -q forgejo:shop/flut.git
cd flut
cat > renovate.json <<'EOF'
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended", "group:allNonMajor"],
  "prConcurrentLimit": 5,
  "packageRules": [
    {
      "description": "Majors erst nach Freigabe im Dashboard",
      "matchUpdateTypes": ["major"],
      "dependencyDashboardApproval": true
    }
  ]
}
EOF
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "Flut eindaemmen: Sammel-PR, Limit, Major-Freigabe"
git push -q
cd .. && LOG_LEVEL=debug renovate shop/flut > flut.log 2>&1
grep -c 'Autoclosing PR' flut.log
grep -oE 'Hourly PRs limit reached|Open PR Count.*Hourly PR Count: [0-9]+' flut.log \
  | sort -u
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
sed -n '/^## Pending Approval/,/^## Open/p' | grep -E '^##|^ - ' \
  | cut -c1-88

Expected:

27
Hourly PRs limit reached
Open PR Count: 5, Existing Branch Count: 5, Hourly PR Count: 32
#10 open renovate/axios-1.x | Update dependency axios to v1
#18 open renovate/dotenv-18.x | Update dependency dotenv to v18
#21 open renovate/express-5.x | Update dependency express to v5
#26 open renovate/pino-10.x | Update dependency pino to v10
#32 open renovate/uuid-14.x | Update dependency uuid to v14
## Pending Approval
 - [ ] <!-- approve-branch=renovate/major-eslint-monorepo -->Update dependency eslint to
 - [ ] <!-- approve-branch=renovate/major-jest-monorepo -->Update dependency jest to v30
 - [ ] <!-- approve-all-pending-prs -->🔐 **Create all pending approval PRs at once** 🔐
## Rate-Limited
 - [ ] <!-- unlimit-branch=renovate/all-minor-patch -->Update all non-major dependencies
## Open

27 PRs are gone, with “- autoclosed” in the title. What remains are the five majors that already had a PR: dependencyDashboardApproval only applies to new branches. eslint and jest had different branch names because of separateMultipleMajor; they now wait for approval. The bulk PR is missing: without prHourlyLimit, the default of 2 applies again, and 32 PRs have already been created this hour. You pull it forward with a checkbox (Day 14):

D=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .number')
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues/$D" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- unlimit-branch=renovate/all-minor-patch|- [x] <!-- unlimit-branch=renovate/all-minor-patch|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json \
  "$FORGEJO/api/v1/repos/shop/flut/issues/$D" | jq -r .number
renovate shop/flut | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls?state=open" | jq -r '.[]
  | select(.head.ref == "renovate/all-minor-patch") | .body' | grep -cE '^\| \['
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls?state=closed" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | tail -2

Expected:

 INFO: PR created (repository=shop/flut, branch=renovate/all-minor-patch)
       "result": "done",
6
#30 closed refs/pull/30/head | Update dependency uuid to v12 - autoclosed
#31 closed refs/pull/31/head | Update dependency uuid to v13 - autoclosed

One PR with six packages instead of 25 individual ones. The checkbox also bypasses prConcurrentLimit: six PRs are open now.

Lesson:

  • Clean up via the config, never by mass-closing. Renovate closes what it no longer needs and leaves no ignores behind.
  • After a flood, the default prHourlyLimit slows down the repair (“Rate-Limited” in the dashboard, Hourly PR Count in the debug log).

Save your progress? Sign up for free

Round 6: The ignored major

Situation: In shop/major, someone closed the express 5 PR in the spring. Now the team wants to migrate after all, but Renovate delivers nothing.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/pulls?state=all" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues/3/comments" | jq -r '.[].body' | grep -E '^###|rename' \
  | cut -c1-88

Expected:

#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#3 closed refs/pull/3/head | Update dependency express to v5
### Renovate Ignore Notification
If you accidentally closed this PR, or if you changed your mind: rename this PR to get a

Task: An open PR for express 5, created by Renovate. No config change.

Time budget: 5 min · Points: 10

Hint: The dashboard has a dedicated section for closed PRs.

Solution:

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
sed -n '/^## PR Closed/,/^## /p' | grep -E '^ - ' | cut -c1-88
D=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .number')
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues/$D" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- recreate-branch=renovate/express-5.x|- [x] <!-- recreate-branch=renovate/express-5.x|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json \
  "$FORGEJO/api/v1/repos/shop/major/issues/$D" | jq -r .number
renovate shop/major | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'

Expected:

 - [ ] <!-- recreate-branch=renovate/express-5.x -->[Update dependency express to v5](pu
 INFO: PR created (repository=shop/major, branch=renovate/express-5.x)
       "result": "done",
#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#5 open renovate/express-5.x | Update dependency express to v5

A new PR (#5); the old one stays closed. The way described in the comment also works: change the title of #3 via the API, and the block no longer applies.

Lesson:

  • The block is tied to the title of the closed PR. It applies to all future 5.x versions, not just the one at the time.
  • If you deliberately postpone a major, write that into the config (allowedVersions, Day 6) instead of closing PRs. Then the decision is visible and reviewable.

Save your progress? Sign up for free

Round 7: The private registry

Situation: shop/bezahlung uses @shop/money from the Forgejo registry. Version 1.1.0 has been published for days, but no PR comes.

grep -A2 'lookup failures' nacht.log | cut -c1-88
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -A2 dash.md 'Warning' | cut -c1-88

Expected:

 WARN: Package lookup failures (repository=shop/bezahlung)
       "warnings": ["Failed to look up npm package @shop/money: no-result"],
       "files": ["package.json"]
> ⚠️ **Warning**
> 
> Renovate failed to look up the following dependencies: `Failed to look up npm package 

Task: A PR for @shop/money 1.1.0, a dashboard without a lookup warning. The token does not belong in the repo and does not stay set in your shell.

Time budget: 10 min · Points: 10

Hint: no-result says little; the status code of the registry request in the debug log says everything.

Solution:

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/raw/.npmrc"
LOG_LEVEL=debug renovate --dry-run=lookup shop/bezahlung > lookup.log 2>&1
grep -E 'statusCode=' lookup.log \
  | sed -E 's/.*GET ([^ ]+) .*statusCode=(-?[0-9]+).*/\2 \1/' | sort -u
curl -s -H "Authorization: token $RENOVATE_TOKEN" \
  "$FORGEJO/api/packages/shop/npm/@shop%2fmoney" | jq -c '{name, v: (.versions | keys)}'

Expected:

@shop:registry=http://127.0.0.1:3000/api/packages/shop/npm/
401 http://127.0.0.1:3000/api/packages/shop/npm/@shop%2Fmoney
{"name":"@shop/money","v":["1.0.0","1.1.0"]}

Renovate queries the registry anonymously, but the org is limited. With the bot token it works. As on Day 8, Renovate gets a hostRules rule via the environment, here in a file that you load only for the run in a subshell:

cat > wk-creds.sh <<'EOF'
# wk-creds.sh – hostRules for Forgejo's npm registry (not in the repo!)
export RENOVATE_HOST_RULES=$(jq -nc --arg t "$RENOVATE_TOKEN" \
  --arg h "$FORGEJO/api/packages/" '[{matchHost: $h, hostType: "npm", token: $t}]')
EOF
chmod 600 wk-creds.sh
(source wk-creds.sh && renovate shop/bezahlung) | grep -E 'WARN|PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
echo "RENOVATE_HOST_RULES in dieser Shell: ${RENOVATE_HOST_RULES:-leer}"

Expected:

 INFO: PR created (repository=shop/bezahlung, branch=renovate/shop-money-1.x)
       "result": "done",
#2 open renovate/shop-money-1.x | Update dependency @shop/money to v1.1.0
RENOVATE_HOST_RULES in dieser Shell: leer

Lesson:

  • Renovate does not automatically send the platform token to the package registry, not even on the same host. Every registry needs its own hostRules rule.
  • matchHost with an http:// URL, because the lab runs without TLS (Day 8). In production, https:// and a read token that can only do read:package. Lookup errors only show up in the dashboard and the log, never as an issue of their own.

Save your progress? Sign up for free

Round 8 (bonus): The lockfile goes on strike

Situation: shop/lager wanted “one PR per package” and switched off a group to get it. Since then, three of four PRs are red.

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
for b in react-18.x react-dom-18.x react-19.x react-dom-19.x; do
  printf '%-16s %s\n' $b "$(checks lager renovate/$b | sed -n 2p)"; done
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/raw/renovate.json" | jq -c '{extends, ignorePresets}'

Expected:

#1 open renovate/react-18.x | Update dependency react to v18.3.1
#2 open renovate/react-dom-18.x | Update dependency react-dom to v18.3.1
#3 open renovate/react-19.x | Update dependency react to v19
#4 open renovate/react-dom-19.x | Update dependency react-dom to v19
react-18.x       
react-dom-18.x     renovate/artifacts = failure (Artifact file update failure)
react-19.x         renovate/artifacts = failure (Artifact file update failure)
react-dom-19.x     renovate/artifacts = failure (Artifact file update failure)
{"extends":["config:recommended"],"ignorePresets":["group:monorepos"]}

Task: All open PRs in lager without artifact errors. ignorePresets stays; the team still does not want the big monorepo groups.

Time budget: 10 min · Points: 10 (bonus)

Hint: Read the “Artifact update problem” comment on one of the red PRs.

Solution:

curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/issues/3/comments" | jq -r '.[].body' \
  | grep -E '^###|npm error (While|Found|peer|Conflicting)' | cut -c1-88

Expected:

### ⚠️ Artifact update problem
##### File name: package-lock.json
npm error While resolving: [email protected]
npm error Found: [email protected]
npm error peer react@"^18.2.0" from [email protected]
npm error Conflicting peer dependency: [email protected]

Via a peer dependency, react-dom requires the matching react version. Individually, npm cannot resolve the lockfile (ERESOLVE). Only react 18.3.1 happens to still fit react-dom 18.2.0. The two belong in a targeted group:

cd ~/renovate-lab/tag15
rm -rf lager
git clone -q forgejo:shop/lager.git
cd lager
cat > renovate.json <<'EOF'
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "ignorePresets": ["group:monorepos"],
  "prHourlyLimit": 0,
  "packageRules": [
    {
      "description": "react und react-dom nur gemeinsam (Peer-Dependency)",
      "matchPackageNames": ["react", "react-dom"],
      "groupName": "react"
    }
  ]
}
EOF
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "react und react-dom gruppieren"
git push -q
cd .. && renovate shop/lager | grep -E 'PR created|Autoclosing|WARN|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/pulls" \
  | jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
for b in react major-react; do printf '%-12s %s\n' $b \
  "$(checks lager renovate/$b | head -1)"; done

Expected:

 INFO: PR created (repository=shop/lager, branch=renovate/react)
 INFO: PR created (repository=shop/lager, branch=renovate/major-react)
 INFO: Autoclosing PR (repository=shop/lager)
 INFO: Autoclosing PR (repository=shop/lager)
 INFO: Autoclosing PR (repository=shop/lager)
 INFO: Autoclosing PR (repository=shop/lager)
       "result": "done",
#6 open renovate/react | Update react to v18.3.1
#7 open renovate/major-react | Update react to v19
react        gesamt: 
major-react  gesamt: 

No more renovate/artifacts status; the four old PRs are autoclosed.

Lesson:

  • group:monorepos (in config:recommended) is not a convenience; it prevents exactly this error. Whoever switches it off has to group peer pairs themselves.
  • An artifact error does not break the PR, it makes it incomplete: package.json is changed, the lockfile is not. Merging would be wrong, and Renovate says so in the comment.

Save your progress? Sign up for free

Scoring

Full points only within the time budget and without a hint. With a hint or up to double the time you get half, after that nothing.

Round Time budget Points
1 A missing comma 5 min 10
2 The rotated token 10 min 15
3 The regex sees nothing 10 min 15
4 The automerge that never comes 15 min 15
5 The PR flood 15 min 15
6 The ignored major 5 min 10
7 The private registry 10 min 10
8 The lockfile goes on strike (bonus) 10 min 10
Total 80 min 100

With 70 points or more, you can run Renovate for an organization. For every missed round, write one line into the runbook: symptom, first command, solution.

Checkpoint

After all rounds, what counts is the clean final state, not just the vanished symptom. O saves typing for the address of the org repos.

R1 – kasse has a valid config, no “Action Required” and open PRs:

cd ~/renovate-lab/tag15
O="$FORGEJO/api/v1/repos/shop"
curl -s -u "$ADMIN" "$O/kasse/raw/renovate.json" | jq -c .extends
curl -s -u "$ADMIN" "$O/kasse/issues?type=issues" | jq -r '.[].title'
curl -s -u "$ADMIN" "$O/kasse/pulls" | jq length

Expected:

R15_R1

R2 – the stripped-down token is gone, the bot token is sufficient:

curl -s -u renovate-bot:lab-bot-pw-123 "$FORGEJO/api/v1/users/renovate-bot/tokens" \
  | jq -c '.[] | {name, scopes}'

Expected:

R15_R2

R3 – the Makefile tools are listed in the dashboard of werkzeug:

curl -s -u "$ADMIN" "$O/werkzeug/issues?type=issues" \
  | jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -oE 'markdownlint-cli2|redocly' dash.md | sort -u

Expected:

markdownlint-cli2
redocly

R4 – in automat the prettier PR is merged, Actions are off, ci/test is required:

curl -s -u "$ADMIN" "$O/automat/pulls?state=closed" | jq -r '.[] | "\(.merged) \(.title)"'
curl -s -u "$ADMIN" "$O/automat" | jq .has_actions
curl -s -u "$ADMIN" "$O/automat/branch_protections" | jq -c '.[].status_check_contexts'

Expected:

R15_R4

R5 – flut has at most six open PRs, a limit and autoclosed leftovers:

curl -s -u "$ADMIN" "$O/flut/pulls" | jq length
curl -s -u "$ADMIN" "$O/flut/raw/renovate.json" | jq .prConcurrentLimit
curl -s -u "$ADMIN" "$O/flut/pulls?state=closed" | jq -r '.[].title' | grep -c autoclosed

Expected:

R15_R5

R6 and R7 – the express major PR is back, bezahlung finds @shop/money:

curl -s -u "$ADMIN" "$O/major/pulls" | jq -r '.[].head.ref' | grep express-5
curl -s -u "$ADMIN" "$O/bezahlung/pulls" | jq -r '.[].title'
echo "hostRules in der Shell: ${RENOVATE_HOST_RULES:-keine}"

Expected:

R15_R67

R8 (bonus) – the open PRs in lager have no artifact error:

for b in $(curl -s -u "$ADMIN" "$O/lager/pulls" | jq -r '.[].head.ref'); do
  curl -s -u "$ADMIN" "$O/lager/commits/${b//\//%2F}/status" | jq -r .state
done

Expected:

R15_R8

Challenge

An early-warning system would have reported rounds 1, 4, 5, 7 and 8 before anyone complained. Write fruehwarnung.sh, which reports five signals for all repos of the org shop: an open “Action Required” issue, lookup errors in the dashboard, more than ten open PRs, bot PRs with renovate/artifacts = failure and automerge PRs with status pending. Test it against a fresh setup.

Hint: GET /orgs/shop/repos returns the org repos, and GET /repos/{o}/{r}/commits/{branch}/status the combined status of a branch.

Solution

The file is also available at code/tag15/challenge/fruehwarnung.sh.

cat > fruehwarnung.sh <<'EOF'
#!/usr/bin/env bash
# fruehwarnung.sh – warning signals from the competition for all repos of the org shop.
# Usage: ./fruehwarnung.sh [max-open-PRs]   (default 10, like prConcurrentLimit)
LAB="${LAB:-$HOME/renovate-lab}"
source "$LAB/setup/env.sh"
MAX="${1:-10}"; found=0
api() { curl -sf -u "$ADMIN" "$@"; }
warn() { printf '%-10s %s\n' "$1" "$2"; found=$((found + 1)); }
for r in $(api "$FORGEJO/api/v1/orgs/shop/repos?limit=50" | jq -r '.[].name' | sort); do
  A="$FORGEJO/api/v1/repos/shop/$r"
  iss=$(curl -s -u "$ADMIN" "$A/issues?type=issues&state=open&limit=50")
  jq -e '.[] | select(.title | startswith("Action Required"))' <<<"$iss" >/dev/null \
    && warn "$r" "Config-Fehler-Issue offen"
  jq -r '.[] | select(.title == "Dependency Dashboard") | .body' <<<"$iss" \
    | grep -q 'Failed to look up' && warn "$r" "Lookup-Fehler im Dashboard"
  prs=$(curl -s -u "$ADMIN" "$A/pulls?state=open&limit=50")
  n=$(jq length <<<"$prs"); [ "$n" -gt "$MAX" ] && warn "$r" "PR-Flut: $n offen"
  for b in $(jq -r '.[] | select(.user.login == "renovate-bot") | .head.ref' <<<"$prs")
  do
    st=$(curl -s -u "$ADMIN" "$A/commits/${b//\//%2F}/status")
    jq -e '.statuses[]? | select(.context == "renovate/artifacts"
      and .status == "failure")' <<<"$st" >/dev/null \
      && warn "$r" "Artefakt-Fehler: $b"
    jq -e --arg b "$b" '.[] | select(.head.ref == $b)
      | .body | contains("**Automerge**: Enabled")' <<<"$prs" >/dev/null \
      && [ "$(jq -r .state <<<"$st")" = pending ] \
      && warn "$r" "Automerge wartet auf pending: $b"
  done
done
echo "Warnsignale: $found"
EOF
chmod +x fruehwarnung.sh
./fruehwarnung.sh
./aufstellung.sh > /dev/null && ./fruehwarnung.sh

Expected:

Warnsignale: 0
automat    Automerge wartet auf pending: renovate/prettier-3.x
bezahlung  Lookup-Fehler im Dashboard
flut       PR-Flut: 32 offen
kasse      Config-Fehler-Issue offen
lager      Artefakt-Fehler: renovate/react-dom-19.x
lager      Artefakt-Fehler: renovate/react-19.x
lager      Artefakt-Fehler: renovate/react-dom-18.x
Warnsignale: 7

After your repairs everything is quiet; after the fresh setup the script reports five of the eight rounds. Rounds 3 and 6 remain invisible: a regex that finds nothing and a deliberately closed PR look normal to any machine. Round 2 shows up in the exit code of the Renovate run, not in Forgejo.

Finally, you clean up the competition repos and the rotation token:

for r in kasse werkzeug automat flut major bezahlung lager; do
  curl -s -u "$ADMIN" -X DELETE "$FORGEJO/api/v1/repos/shop/$r)"; done
curl -s -o /dev/null -w "%{http_code}\n" -u renovate-bot:lab-bot-pw-123 -X DELETE \
  "$FORGEJO/api/v1/users/renovate-bot/tokens/renovate-rotation"
rm -f wk-token wk-creds.sh

Expected:

204

Save your progress? Sign up for free

Enterprise Note

  • These drills belong in every quarter as a game day, on a staging instance with real repos. What gets measured is the time to detection, not just the time to repair.
  • An early-warning system like fruehwarnung.sh runs after every Renovate run, plus alerts on exit code ≠ 0 and on "result" other than done in the JSON logs (Day 9).
  • Token rotation as a runbook with a fixed scope list, a test run and deletion of the old token. On GitHub and GitLab, the same applies to app permissions and token scopes.
  • Limits, groups and dependencyDashboardApproval come from the org preset (Day 11). A repo can override them, but a review by the platform team will catch it.
  • Every round ends with a new test: the validator in CI (Round 1), an extract report with a minimum number of dependencies (Round 3), required checks via branch protection (Round 4).