Day 15: Competition – disaster drills
Goal: Under time pressure, you find and fix eight typical Renovate failures, from a broken config to a lockfile that fails on a peer dependency. · Duration: approx. 120 min · Prerequisite: Day 14
Briefing
- Each round has its own repo in the org
shop.aufstellung.shbuilds all starting states at once, including a “nightly run” of Renovate. - Per round: take in the situation, start the stopwatch, repair until the round’s check passes. Measure first (log, dashboard, API), then change.
- The hint costs half the points. Read the solution only afterwards.
- No shortcuts via admin rights: Renovate has to produce the state itself.
Failure ──► situation (issue, dashboard, PR, log) ──► diagnosis ──► smallest repair
└──► Renovate run ──► check
Drill 0: Setup (untimed)
Under repos/, the companion code brings the files of the
seven competition repos and the lab tool aufstellung.sh. It
builds the failures you are supposed to find – what it does in detail is
shown by grep '^# [0-9]' aufstellung.sh; read the rest only
after the competition. It creates seven repos (copy,
git init, push as on Day 0), publishes
@shop/money as on Day 8 and lets Renovate run over
everything once (nacht.log).
source ~/renovate-lab/setup/env.sh
export RENOVATE_GITHUB_COM_TOKEN=$(gh auth token) # optional, see Day 0
mkdir -p ~/renovate-lab/tag15 && cd ~/renovate-lab/tag15
cp ~/renovate-lab/code/tag15/aufstellung.sh .
./aufstellung.shExpected (abridged):
R15_AUF
You can run it again at any time; the repos are then created fresh.
The numbers and target versions in this chapter come from the test
run; your versions will be newer. Without
RENOVATE_GITHUB_COM_TOKEN, the GitHub Actions updates in
kasse are missing; nothing else changes.
Warning: Play the rounds in order, 1 to 8. Round 2 uses
shop/kassefor its test, and every round ends with a real Renovate run.
Save your progress? Sign up for free
Round 1: A missing comma
Situation: Since yesterday, not a single PR has come
from shop/kasse. Instead, there is an issue in the
repo.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/issues?type=issues" \
| jq -r '.[] | "#\(.number) \(.state) \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
grep -A4 'invalid config' nacht.log | cut -c1-88Expected:
#1 open Action Required: Fix Renovate Configuration
WARN: Repository has invalid config (repository=shop/kasse)
"err": {
"validationSource": "renovate.json",
"validationError": "Invalid JSON (parsing failed)",
"validationMessage": "Syntax error: expecting number near 0\n \"pack",
Task: Renovate opens PRs in kasse
again, and the issue is closed.
Time budget: 5 min · Points: 10
Hint: The message in the issue is cryptic;
jq and the validator name the line and column.
Solution:
cd ~/renovate-lab/tag15
rm -rf kasse
git clone -q forgejo:shop/kasse.git
cd kasse
renovate-config-validator 2>&1 | grep -E 'WARN|"message"'
jq . renovate.json > /dev/null
sed -n 4,5p renovate.jsonExpected:
WARN: File could not be parsed
"message": "JSON5: invalid character '\\\"' at 5:3"
jq: parse error: Expected separator between values at line 5, column 16
"prHourlyLimit": 0
"packageRules": [
The comma after "prHourlyLimit": 0 is missing:
sed -i.bak 's/"prHourlyLimit": 0$/&,/' renovate.json && rm renovate.json.bak
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "renovate.json: fehlendes Komma"
git push -q
cd .. && renovate shop/kasse | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/kasse/issues?type=issues&state=all" \
| jq -r '.[] | "#\(.number) \(.state) \(.title)"'Expected:
INFO: PR created (repository=shop/kasse, branch=renovate/express-4.x)
INFO: PR created (repository=shop/kasse, branch=renovate/prettier-3.x)
INFO: PR created (repository=shop/kasse, branch=renovate/node-20.x)
"result": "done",
#5 open Dependency Dashboard
#1 closed Action Required: Fix Renovate Configuration
Renovate closes the error issue itself. The majors are missing on
purpose: the config requires dependencyDashboardApproval
for them.
Lesson:
- A broken repo config means: exit code 0,
"result": "config-validation"and an issue, but no PR (Day 9). Without looking at issues or logs, nobody notices. - Renovate reads
renovate.jsonleniently as JSON5: a trailing comma at the end of a list passed in testing. Still make the validator a mandatory check in every repo’s CI (Day 2).
Save your progress? Sign up for free
Round 2: The rotated token
Situation: A colleague has “already rotated” the bot
token. The new token is in wk-token and is supposed to go
into the operations configuration tomorrow. You do the test.
RENOVATE_TOKEN=$(cat wk-token) renovate shop/kasse > token.log 2>&1; echo "exit=$?"
grep -E '^FATAL|errorMessage' token.logExpected:
exit=1
FATAL: Initialization error
"errorMessage": "Authentication failure"
"errorMessage": "Authentication failure",
Task: Find the cause and create a replacement in
wk-token with exactly the scopes Renovate needs. The broken
token is deleted; no admin scope.
Time budget: 10 min · Points: 15
Hint: The first thing Renovate queries is
GET /api/v1/user; in its response, Forgejo names the
missing scope.
Solution: Forgejo’s response is clearer than Renovate’s.
curl -s -H "Authorization: token $(cat wk-token)" "$FORGEJO/api/v1/user" \
| jq -r .message
curl -s -H "Authorization: token $(cat wk-token)" "$FORGEJO/api/v1/repos/shop/kasse" \
| jq -c .permissions
BOT=renovate-bot:lab-bot-pw-123 # lab value!
curl -s -u $BOT "$FORGEJO/api/v1/users/renovate-bot/tokens" \
| jq -c '.[] | select(.name == "renovate-rotation") | .scopes'Expected:
token does not have at least one of required scope(s): [read:user]
{"admin":false,"push":true,"pull":true}
["read:repository"]
The token only has read:repository. Careful with
permissions: push: true is the permission of
the user in the repo, not that of the token. If you
only add read:user, in testing you run into the next error
(read:organization when reading the org), then into
remote: Forbidden on push. So go straight for the complete
list from Day 0:
curl -s -o /dev/null -w "%{http_code}\n" -u $BOT -X DELETE \
"$FORGEJO/api/v1/users/renovate-bot/tokens/renovate-rotation"
curl -s -u $BOT -H 'Content-Type: application/json' \
-X POST "$FORGEJO/api/v1/users/renovate-bot/tokens" -d '{"name": "renovate-rotation",
"scopes": ["write:repository", "write:issue", "read:user", "read:organization",
"read:package"]}' | jq -j .sha1 > wk-token
RENOVATE_TOKEN=$(cat wk-token) renovate shop/kasse | grep -E '^FATAL|"result"'Expected:
204
"result": "done",
jq -j writes the token without a trailing newline (Day
10). The bot creates it itself via basic auth, so rotation works without
admin access to the container.
Lesson:
- With Forgejo,
Authentication failureoften means “wrong scope”, not “wrong token”. The actual message only shows up in the debug log or the API response. - Rotation is a procedure with a fixed scope list and a test run.
read:packageonly if repos use the Forgejo registry (Round 7).
Save your progress? Sign up for free
Round 3: The regex sees nothing
Situation: The team maintains lint tools in the
Makefile of shop/werkzeug and built a regex
manager for them (Day 7). For weeks no update has come; the dashboard
only lists the base image.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/werkzeug/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -E '^<details><summary>[a-z]|^ - `' dash.md
head -5 ~/renovate-lab/code/tag15/repos/werkzeug/MakefileExpected:
<details><summary>dockerfile (1)</summary>
- `node 20.11.0-alpine` → [Updates: `20.20.2-alpine`, `24.21.0-alpine`]
# Werkzeuge für Doku- und API-Checks (Version hier pflegen, nicht im CI-Job)
# renovate: datasource=npm depName=markdownlint-cli2
MARKDOWNLINT_VERSION ?= 0.10.0
# renovate: datasource=npm depName=@redocly/cli
REDOCLY_VERSION ?= 1.0.0
Task: markdownlint-cli2 and
@redocly/cli appear in the dashboard and get PRs. The
Makefile stays as it is.
Time budget: 10 min · Points: 15
Hint:
--platform=local --dry-run=extract with
LOG_LEVEL=debug tells you whether the file pattern or the
regex fails.
Solution: First separate which part of the manager is failing.
cd ~/renovate-lab/tag15
rm -rf werkzeug
git clone -q forgejo:shop/werkzeug.git
cd werkzeug
LOG_LEVEL=debug renovate --platform=local --dry-run=extract > ../extract.log 2>&1
grep -E 'Matched .* manager regex|No dependencies found' ../extract.log | cut -c1-88
jq -r '.customManagers[0].matchStrings[0]' renovate.json | grep -o '\\n.*'Expected:
DEBUG: Matched 1 file(s) for manager regex (repository=local)
DEBUG: No dependencies found in file for custom regex manager (repository=local, package
\n\w+_VERSION := (?<currentValue>\S+)
The file is found, the regex does not match. It expects
:=, but the Makefile has ?=. From
now on the assignment should match all three Make variants:
jq '.customManagers[0].matchStrings[0] |= sub(" := "; " [:?]?= ")' renovate.json \
> neu.json && mv neu.json renovate.json
renovate --platform=local --dry-run=extract --report-type=file \
--report-path=../extract.json > /dev/null 2>&1
jq -r '.repositories[].packageFiles.regex[].deps[]
| "\(.depName) \(.currentValue) (\(.datasource))"' ../extract.json
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "Makefile-Regex: ?= statt :="
git push -q
cd .. && renovate shop/werkzeug | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/werkzeug/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -E '^ - `' dash.mdExpected:
markdownlint-cli2 0.10.0 (npm)
@redocly/cli 1.0.0 (npm)
INFO: PR created (repository=shop/werkzeug, branch=renovate/redocly-cli-1.x)
INFO: PR created (repository=shop/werkzeug, branch=renovate/markdownlint-cli2-0.x)
INFO: PR created (repository=shop/werkzeug, branch=renovate/redocly-cli-2.x)
"result": "done",
- `node 20.11.0-alpine` → [Updates: `20.20.2-alpine`, `24.21.0-alpine`]
- `markdownlint-cli2 0.10.0` → [Updates: `0.23.3`]
- `@redocly/cli 1.0.0` → [Updates: `1.34.20`, `2.62.1`]
Lesson:
- A regex manager that finds nothing produces no error and no warning.
Only “Detected Dependencies” in the dashboard or
--dry-run=extractreveal the gap. - Two debug lines separate the error sources:
Matched N file(s)(file pattern) andNo dependencies found in file(regex). The local extract needs no push.
Save your progress? Sign up for free
Round 4: The automerge that never comes
Situation: In shop/automat, patch and
minor updates of the devDependencies are supposed to land automatically
(Day 5). The prettier PR has been open since yesterday.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .body | grep 'Automerge\*\*'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" | jq -r .state
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" \
| jq -r '.statuses[] | "\(.context) = \(.status)"'Expected:
#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#3 open renovate/express-5.x | Update dependency express to v5
🚦 **Automerge**: Enabled.
gesamt: pending
ci / test (pull_request) = pending (Waiting to run)
ci / test (push) = pending (Waiting to run)
Task: The team has decided: from now on, the tests
run in the central lab CI, which reports as ci/test (helper
ci, as on Day 5). Get PR #2 to merge automatically, but
only after a green ci/test. No ignoreTests, no
manual merge.
Time budget: 15 min · Points: 15
Hint: Who is actually supposed to run
ci / test? Ask the Actions API for runs and runners.
Solution: First the cause, then Renovate’s view.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/actions/runs" \
| jq -r '.workflow_runs[] | "\(.status) \(.event) \(.title)"' | head -3
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/actions/runners"; echo
LOG_LEVEL=debug renovate shop/automat > automat.log 2>&1
grep 'renovate/prettier' automat.log \
| grep -oE 'Branch status (yellow|green|red)|PR is not ready[^(]*' | sort | uniq -cExpected:
waiting push Update dependency express to v5
waiting pull_request Update dependency express to v5
waiting push Update dependency prettier to v3.9.10
[]
2 Branch status yellow
1 PR is not ready for merge
Forgejo runs .github/workflows/ci.yml as Actions, but
there is no runner. The runs wait forever, the checks stay
pending, Renovate sees “yellow” and waits along with them.
The old statuses are attached to the old commit, so after switching
Actions off a rebase is needed. Branch protection comes first, otherwise
there is a nasty surprise:
curl -s -u "$ADMIN" -X POST "$FORGEJO/api/v1/repos/shop/automat/branch_protections" --json '{"rule_name": "main",
"enable_status_check": true, "status_check_contexts": ["ci/test"]}' \
| jq -c '{rule_name, enable_status_check, status_check_contexts}'
curl -s -u "$ADMIN" -X PATCH "$FORGEJO/api/v1/repos/shop/automat" --json '{"has_actions":false}' | jq -c '{has_actions}'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- rebase-check|- [x] <!-- rebase-check|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json "$FORGEJO/api/v1/repos/shop/automat/issues/2" | jq -r .number
renovate shop/automat | grep -E '"result"'
sleep 5
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls?state=all" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep prettier
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" | jq -r .state
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/commits/renovate%2Fprettier-3.x/status" \
| jq -r '.statuses[] | "\(.context) = \(.status)"'
SHA=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/branches/renovate%2Fprettier-3.x" | jq -r .commit.id)
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/statuses/$SHA" \
--json '{"state":"success","context":"ci/test"}' | jq -r .status
sleep 5
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/automat/pulls?state=all" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep prettierExpected:
{"rule_name":"main","enable_status_check":true,"status_check_contexts":["ci/test"]}
{"has_actions":false}
"result": "done",
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
gesamt:
success
#2 merged refs/pull/2/head | Update dependency prettier to v3.9.10
After the rebase, the new commit has not a single status. Forgejo now
waits for the required check and only merges after
ci/test = success. Forgejo triggers the merge itself:
Renovate had scheduled it when creating the PR
(platformAutomerge, Day 5).
Warning: Without branch protection, Forgejo merged in testing one second after the rebase push, without any test at all. A scheduled merge fires as soon as no check is
pendinganymore, even with zero checks. Renovate itself would have waited.
Lesson:
- A single check that nobody ever processes blocks every automerge.
Renovate reports this only in the debug log as
Branch status yellow. - In production, the repair means: register a runner or remove the workflow. Switching off Actions is only right if another CI takes over.
platformAutomergeon Forgejo only with branch protection and required checks (Day 5). Only they make “test first, then merge” the rule.
Save your progress? Sign up for free
Round 5: The PR flood
Situation: Someone wanted to “just see everything”
in shop/flut and released all the brakes. The nightly run
flooded the repo.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | wc -l
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | grep -E 'dotenv' | head -3
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/raw/renovate.json" | jq 'del(."$schema")'Expected:
32
#11 open renovate/dotenv-11.x | Update dependency dotenv to v11
#12 open renovate/dotenv-12.x | Update dependency dotenv to v12
#13 open renovate/dotenv-13.x | Update dependency dotenv to v13
{
"extends": [
"config:recommended"
],
"prHourlyLimit": 0,
"prConcurrentLimit": 0,
"branchConcurrentLimit": 0,
"separateMinorPatch": true,
"separateMultipleMajor": true
}Task: At most five open PRs plus one bulk PR for all minor and patch updates, which still arrives today. New majors only after approval in the dashboard. Renovate cleans up the leftovers itself; no PR is closed by hand.
Time budget: 15 min · Points: 15
Hint: A PR closed by hand is an ignore (Round 6); whatever Renovate no longer needs, it closes itself as “autoclosed”.
Solution: A new config from the building blocks of Day 4:
cd ~/renovate-lab/tag15
rm -rf flut
git clone -q forgejo:shop/flut.git
cd flut
cat > renovate.json <<'EOF'
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", "group:allNonMajor"],
"prConcurrentLimit": 5,
"packageRules": [
{
"description": "Majors erst nach Freigabe im Dashboard",
"matchUpdateTypes": ["major"],
"dependencyDashboardApproval": true
}
]
}
EOF
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "Flut eindaemmen: Sammel-PR, Limit, Major-Freigabe"
git push -q
cd .. && LOG_LEVEL=debug renovate shop/flut > flut.log 2>&1
grep -c 'Autoclosing PR' flut.log
grep -oE 'Hourly PRs limit reached|Open PR Count.*Hourly PR Count: [0-9]+' flut.log \
| sort -u
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
sed -n '/^## Pending Approval/,/^## Open/p' | grep -E '^##|^ - ' \
| cut -c1-88Expected:
27
Hourly PRs limit reached
Open PR Count: 5, Existing Branch Count: 5, Hourly PR Count: 32
#10 open renovate/axios-1.x | Update dependency axios to v1
#18 open renovate/dotenv-18.x | Update dependency dotenv to v18
#21 open renovate/express-5.x | Update dependency express to v5
#26 open renovate/pino-10.x | Update dependency pino to v10
#32 open renovate/uuid-14.x | Update dependency uuid to v14
## Pending Approval
- [ ] <!-- approve-branch=renovate/major-eslint-monorepo -->Update dependency eslint to
- [ ] <!-- approve-branch=renovate/major-jest-monorepo -->Update dependency jest to v30
- [ ] <!-- approve-all-pending-prs -->🔐 **Create all pending approval PRs at once** 🔐
## Rate-Limited
- [ ] <!-- unlimit-branch=renovate/all-minor-patch -->Update all non-major dependencies
## Open
27 PRs are gone, with “- autoclosed” in the title. What remains are
the five majors that already had a PR:
dependencyDashboardApproval only applies to new branches.
eslint and jest had different branch names because of
separateMultipleMajor; they now wait for approval. The bulk
PR is missing: without prHourlyLimit, the default of 2
applies again, and 32 PRs have already been created this hour. You pull
it forward with a checkbox (Day 14):
D=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .number')
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/issues/$D" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- unlimit-branch=renovate/all-minor-patch|- [x] <!-- unlimit-branch=renovate/all-minor-patch|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json \
"$FORGEJO/api/v1/repos/shop/flut/issues/$D" | jq -r .number
renovate shop/flut | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls?state=open" | jq -r '.[]
| select(.head.ref == "renovate/all-minor-patch") | .body' | grep -cE '^\| \['
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/flut/pulls?state=closed" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"' | tail -2Expected:
INFO: PR created (repository=shop/flut, branch=renovate/all-minor-patch)
"result": "done",
6
#30 closed refs/pull/30/head | Update dependency uuid to v12 - autoclosed
#31 closed refs/pull/31/head | Update dependency uuid to v13 - autoclosed
One PR with six packages instead of 25 individual ones. The checkbox
also bypasses prConcurrentLimit: six PRs are open now.
Lesson:
- Clean up via the config, never by mass-closing. Renovate closes what it no longer needs and leaves no ignores behind.
- After a flood, the default
prHourlyLimitslows down the repair (“Rate-Limited” in the dashboard,Hourly PR Countin the debug log).
Save your progress? Sign up for free
Round 6: The ignored major
Situation: In shop/major, someone
closed the express 5 PR in the spring. Now the team wants to migrate
after all, but Renovate delivers nothing.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/pulls?state=all" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues/3/comments" | jq -r '.[].body' | grep -E '^###|rename' \
| cut -c1-88Expected:
#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#3 closed refs/pull/3/head | Update dependency express to v5
### Renovate Ignore Notification
If you accidentally closed this PR, or if you changed your mind: rename this PR to get a
Task: An open PR for express 5, created by Renovate. No config change.
Time budget: 5 min · Points: 10
Hint: The dashboard has a dedicated section for closed PRs.
Solution:
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
sed -n '/^## PR Closed/,/^## /p' | grep -E '^ - ' | cut -c1-88
D=$(curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .number')
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/issues/$D" | jq -r .body > box.md
sed -i '' 's|- \[ \] <!-- recreate-branch=renovate/express-5.x|- [x] <!-- recreate-branch=renovate/express-5.x|' box.md
jq -Rs '{body: .}' box.md > box.json
curl -s -u "$ADMIN" -X PATCH --json @box.json \
"$FORGEJO/api/v1/repos/shop/major/issues/$D" | jq -r .number
renovate shop/major | grep -E 'PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/major/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'Expected:
- [ ] <!-- recreate-branch=renovate/express-5.x -->[Update dependency express to v5](pu
INFO: PR created (repository=shop/major, branch=renovate/express-5.x)
"result": "done",
#1 open renovate/express-4.x | Update dependency express to v4.22.3
#2 open renovate/prettier-3.x | Update dependency prettier to v3.9.10
#5 open renovate/express-5.x | Update dependency express to v5
A new PR (#5); the old one stays closed. The way described in the comment also works: change the title of #3 via the API, and the block no longer applies.
Lesson:
- The block is tied to the title of the closed PR. It applies to all future 5.x versions, not just the one at the time.
- If you deliberately postpone a major, write that into the config
(
allowedVersions, Day 6) instead of closing PRs. Then the decision is visible and reviewable.
Save your progress? Sign up for free
Round 7: The private registry
Situation: shop/bezahlung uses
@shop/money from the Forgejo registry. Version 1.1.0 has
been published for days, but no PR comes.
grep -A2 'lookup failures' nacht.log | cut -c1-88
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -A2 dash.md 'Warning' | cut -c1-88Expected:
WARN: Package lookup failures (repository=shop/bezahlung)
"warnings": ["Failed to look up npm package @shop/money: no-result"],
"files": ["package.json"]
> ⚠️ **Warning**
>
> Renovate failed to look up the following dependencies: `Failed to look up npm package
Task: A PR for @shop/money 1.1.0, a
dashboard without a lookup warning. The token does not belong in the
repo and does not stay set in your shell.
Time budget: 10 min · Points: 10
Hint: no-result says little; the status
code of the registry request in the debug log says everything.
Solution:
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/raw/.npmrc"
LOG_LEVEL=debug renovate --dry-run=lookup shop/bezahlung > lookup.log 2>&1
grep -E 'statusCode=' lookup.log \
| sed -E 's/.*GET ([^ ]+) .*statusCode=(-?[0-9]+).*/\2 \1/' | sort -u
curl -s -H "Authorization: token $RENOVATE_TOKEN" \
"$FORGEJO/api/packages/shop/npm/@shop%2fmoney" | jq -c '{name, v: (.versions | keys)}'Expected:
@shop:registry=http://127.0.0.1:3000/api/packages/shop/npm/
401 http://127.0.0.1:3000/api/packages/shop/npm/@shop%2Fmoney
{"name":"@shop/money","v":["1.0.0","1.1.0"]}
Renovate queries the registry anonymously, but the org is
limited. With the bot token it works. As on Day 8, Renovate
gets a hostRules rule via the environment, here in a file
that you load only for the run in a subshell:
cat > wk-creds.sh <<'EOF'
# wk-creds.sh – hostRules for Forgejo's npm registry (not in the repo!)
export RENOVATE_HOST_RULES=$(jq -nc --arg t "$RENOVATE_TOKEN" \
--arg h "$FORGEJO/api/packages/" '[{matchHost: $h, hostType: "npm", token: $t}]')
EOF
chmod 600 wk-creds.sh
(source wk-creds.sh && renovate shop/bezahlung) | grep -E 'WARN|PR created|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/bezahlung/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
echo "RENOVATE_HOST_RULES in dieser Shell: ${RENOVATE_HOST_RULES:-leer}"Expected:
INFO: PR created (repository=shop/bezahlung, branch=renovate/shop-money-1.x)
"result": "done",
#2 open renovate/shop-money-1.x | Update dependency @shop/money to v1.1.0
RENOVATE_HOST_RULES in dieser Shell: leer
Lesson:
- Renovate does not automatically send the platform token to the
package registry, not even on the same host. Every registry needs its
own
hostRulesrule. matchHostwith anhttp://URL, because the lab runs without TLS (Day 8). In production,https://and a read token that can only doread:package. Lookup errors only show up in the dashboard and the log, never as an issue of their own.
Save your progress? Sign up for free
Round 8 (bonus): The lockfile goes on strike
Situation: shop/lager wanted “one PR
per package” and switched off a group to get it. Since then, three of
four PRs are red.
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
for b in react-18.x react-dom-18.x react-19.x react-dom-19.x; do
printf '%-16s %s\n' $b "$(checks lager renovate/$b | sed -n 2p)"; done
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/raw/renovate.json" | jq -c '{extends, ignorePresets}'Expected:
#1 open renovate/react-18.x | Update dependency react to v18.3.1
#2 open renovate/react-dom-18.x | Update dependency react-dom to v18.3.1
#3 open renovate/react-19.x | Update dependency react to v19
#4 open renovate/react-dom-19.x | Update dependency react-dom to v19
react-18.x
react-dom-18.x renovate/artifacts = failure (Artifact file update failure)
react-19.x renovate/artifacts = failure (Artifact file update failure)
react-dom-19.x renovate/artifacts = failure (Artifact file update failure)
{"extends":["config:recommended"],"ignorePresets":["group:monorepos"]}
Task: All open PRs in lager without
artifact errors. ignorePresets stays; the team still does
not want the big monorepo groups.
Time budget: 10 min · Points: 10 (bonus)
Hint: Read the “Artifact update problem” comment on one of the red PRs.
Solution:
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/issues/3/comments" | jq -r '.[].body' \
| grep -E '^###|npm error (While|Found|peer|Conflicting)' | cut -c1-88Expected:
### ⚠️ Artifact update problem
##### File name: package-lock.json
npm error While resolving: [email protected]
npm error Found: [email protected]
npm error peer react@"^18.2.0" from [email protected]
npm error Conflicting peer dependency: [email protected]
Via a peer dependency, react-dom requires the matching
react version. Individually, npm cannot resolve the
lockfile (ERESOLVE). Only react 18.3.1 happens
to still fit react-dom 18.2.0. The two belong in a targeted
group:
cd ~/renovate-lab/tag15
rm -rf lager
git clone -q forgejo:shop/lager.git
cd lager
cat > renovate.json <<'EOF'
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"],
"ignorePresets": ["group:monorepos"],
"prHourlyLimit": 0,
"packageRules": [
{
"description": "react und react-dom nur gemeinsam (Peer-Dependency)",
"matchPackageNames": ["react", "react-dom"],
"groupName": "react"
}
]
}
EOF
renovate-config-validator --no-global renovate.json >/dev/null
git add -A
git commit -qm "react und react-dom gruppieren"
git push -q
cd .. && renovate shop/lager | grep -E 'PR created|Autoclosing|WARN|"result"'
curl -s -u "$ADMIN" "$FORGEJO/api/v1/repos/shop/lager/pulls" \
| jq -r 'sort_by(.number)[] | "#\(.number) \(.state) \(.head.ref) | \(.title)"'
for b in react major-react; do printf '%-12s %s\n' $b \
"$(checks lager renovate/$b | head -1)"; doneExpected:
INFO: PR created (repository=shop/lager, branch=renovate/react)
INFO: PR created (repository=shop/lager, branch=renovate/major-react)
INFO: Autoclosing PR (repository=shop/lager)
INFO: Autoclosing PR (repository=shop/lager)
INFO: Autoclosing PR (repository=shop/lager)
INFO: Autoclosing PR (repository=shop/lager)
"result": "done",
#6 open renovate/react | Update react to v18.3.1
#7 open renovate/major-react | Update react to v19
react gesamt:
major-react gesamt:
No more renovate/artifacts status; the four old PRs are
autoclosed.
Lesson:
group:monorepos(inconfig:recommended) is not a convenience; it prevents exactly this error. Whoever switches it off has to group peer pairs themselves.- An artifact error does not break the PR, it makes it incomplete:
package.jsonis changed, the lockfile is not. Merging would be wrong, and Renovate says so in the comment.
Save your progress? Sign up for free
Scoring
Full points only within the time budget and without a hint. With a hint or up to double the time you get half, after that nothing.
| Round | Time budget | Points |
|---|---|---|
| 1 A missing comma | 5 min | 10 |
| 2 The rotated token | 10 min | 15 |
| 3 The regex sees nothing | 10 min | 15 |
| 4 The automerge that never comes | 15 min | 15 |
| 5 The PR flood | 15 min | 15 |
| 6 The ignored major | 5 min | 10 |
| 7 The private registry | 10 min | 10 |
| 8 The lockfile goes on strike (bonus) | 10 min | 10 |
| Total | 80 min | 100 |
With 70 points or more, you can run Renovate for an organization. For every missed round, write one line into the runbook: symptom, first command, solution.
Checkpoint
After all rounds, what counts is the clean final state, not just the
vanished symptom. O saves typing for the address of the org
repos.
R1 – kasse has a valid config, no “Action Required” and
open PRs:
cd ~/renovate-lab/tag15
O="$FORGEJO/api/v1/repos/shop"
curl -s -u "$ADMIN" "$O/kasse/raw/renovate.json" | jq -c .extends
curl -s -u "$ADMIN" "$O/kasse/issues?type=issues" | jq -r '.[].title'
curl -s -u "$ADMIN" "$O/kasse/pulls" | jq lengthExpected:
R15_R1
R2 – the stripped-down token is gone, the bot token is sufficient:
curl -s -u renovate-bot:lab-bot-pw-123 "$FORGEJO/api/v1/users/renovate-bot/tokens" \
| jq -c '.[] | {name, scopes}'Expected:
R15_R2
R3 – the Makefile tools are listed in the dashboard of
werkzeug:
curl -s -u "$ADMIN" "$O/werkzeug/issues?type=issues" \
| jq -r '.[] | select(.title == "Dependency Dashboard") | .body' > dash.md
grep -oE 'markdownlint-cli2|redocly' dash.md | sort -uExpected:
markdownlint-cli2
redocly
R4 – in automat the prettier PR is merged, Actions are
off, ci/test is required:
curl -s -u "$ADMIN" "$O/automat/pulls?state=closed" | jq -r '.[] | "\(.merged) \(.title)"'
curl -s -u "$ADMIN" "$O/automat" | jq .has_actions
curl -s -u "$ADMIN" "$O/automat/branch_protections" | jq -c '.[].status_check_contexts'Expected:
R15_R4
R5 – flut has at most six open PRs, a limit and
autoclosed leftovers:
curl -s -u "$ADMIN" "$O/flut/pulls" | jq length
curl -s -u "$ADMIN" "$O/flut/raw/renovate.json" | jq .prConcurrentLimit
curl -s -u "$ADMIN" "$O/flut/pulls?state=closed" | jq -r '.[].title' | grep -c autoclosedExpected:
R15_R5
R6 and R7 – the express major PR is back, bezahlung
finds @shop/money:
curl -s -u "$ADMIN" "$O/major/pulls" | jq -r '.[].head.ref' | grep express-5
curl -s -u "$ADMIN" "$O/bezahlung/pulls" | jq -r '.[].title'
echo "hostRules in der Shell: ${RENOVATE_HOST_RULES:-keine}"Expected:
R15_R67
R8 (bonus) – the open PRs in lager have no artifact
error:
for b in $(curl -s -u "$ADMIN" "$O/lager/pulls" | jq -r '.[].head.ref'); do
curl -s -u "$ADMIN" "$O/lager/commits/${b//\//%2F}/status" | jq -r .state
doneExpected:
R15_R8
Challenge
An early-warning system would have reported rounds 1, 4, 5, 7 and 8
before anyone complained. Write fruehwarnung.sh, which
reports five signals for all repos of the org shop: an open
“Action Required” issue, lookup errors in the dashboard, more than ten
open PRs, bot PRs with renovate/artifacts = failure and
automerge PRs with status pending. Test it against a fresh
setup.
Hint: GET /orgs/shop/repos returns the
org repos, and GET /repos/{o}/{r}/commits/{branch}/status
the combined status of a branch.
Solution
The file is also available at
code/tag15/challenge/fruehwarnung.sh.
cat > fruehwarnung.sh <<'EOF'
#!/usr/bin/env bash
# fruehwarnung.sh – warning signals from the competition for all repos of the org shop.
# Usage: ./fruehwarnung.sh [max-open-PRs] (default 10, like prConcurrentLimit)
LAB="${LAB:-$HOME/renovate-lab}"
source "$LAB/setup/env.sh"
MAX="${1:-10}"; found=0
api() { curl -sf -u "$ADMIN" "$@"; }
warn() { printf '%-10s %s\n' "$1" "$2"; found=$((found + 1)); }
for r in $(api "$FORGEJO/api/v1/orgs/shop/repos?limit=50" | jq -r '.[].name' | sort); do
A="$FORGEJO/api/v1/repos/shop/$r"
iss=$(curl -s -u "$ADMIN" "$A/issues?type=issues&state=open&limit=50")
jq -e '.[] | select(.title | startswith("Action Required"))' <<<"$iss" >/dev/null \
&& warn "$r" "Config-Fehler-Issue offen"
jq -r '.[] | select(.title == "Dependency Dashboard") | .body' <<<"$iss" \
| grep -q 'Failed to look up' && warn "$r" "Lookup-Fehler im Dashboard"
prs=$(curl -s -u "$ADMIN" "$A/pulls?state=open&limit=50")
n=$(jq length <<<"$prs"); [ "$n" -gt "$MAX" ] && warn "$r" "PR-Flut: $n offen"
for b in $(jq -r '.[] | select(.user.login == "renovate-bot") | .head.ref' <<<"$prs")
do
st=$(curl -s -u "$ADMIN" "$A/commits/${b//\//%2F}/status")
jq -e '.statuses[]? | select(.context == "renovate/artifacts"
and .status == "failure")' <<<"$st" >/dev/null \
&& warn "$r" "Artefakt-Fehler: $b"
jq -e --arg b "$b" '.[] | select(.head.ref == $b)
| .body | contains("**Automerge**: Enabled")' <<<"$prs" >/dev/null \
&& [ "$(jq -r .state <<<"$st")" = pending ] \
&& warn "$r" "Automerge wartet auf pending: $b"
done
done
echo "Warnsignale: $found"
EOF
chmod +x fruehwarnung.sh
./fruehwarnung.sh
./aufstellung.sh > /dev/null && ./fruehwarnung.shExpected:
Warnsignale: 0
automat Automerge wartet auf pending: renovate/prettier-3.x
bezahlung Lookup-Fehler im Dashboard
flut PR-Flut: 32 offen
kasse Config-Fehler-Issue offen
lager Artefakt-Fehler: renovate/react-dom-19.x
lager Artefakt-Fehler: renovate/react-19.x
lager Artefakt-Fehler: renovate/react-dom-18.x
Warnsignale: 7
After your repairs everything is quiet; after the fresh setup the script reports five of the eight rounds. Rounds 3 and 6 remain invisible: a regex that finds nothing and a deliberately closed PR look normal to any machine. Round 2 shows up in the exit code of the Renovate run, not in Forgejo.
Finally, you clean up the competition repos and the rotation token:
for r in kasse werkzeug automat flut major bezahlung lager; do
curl -s -u "$ADMIN" -X DELETE "$FORGEJO/api/v1/repos/shop/$r)"; done
curl -s -o /dev/null -w "%{http_code}\n" -u renovate-bot:lab-bot-pw-123 -X DELETE \
"$FORGEJO/api/v1/users/renovate-bot/tokens/renovate-rotation"
rm -f wk-token wk-creds.shExpected:
204
Save your progress? Sign up for free
Enterprise Note
- These drills belong in every quarter as a game day, on a staging instance with real repos. What gets measured is the time to detection, not just the time to repair.
- An early-warning system like
fruehwarnung.shruns after every Renovate run, plus alerts on exit code ≠ 0 and on"result"other thandonein the JSON logs (Day 9). - Token rotation as a runbook with a fixed scope list, a test run and deletion of the old token. On GitHub and GitLab, the same applies to app permissions and token scopes.
- Limits, groups and
dependencyDashboardApprovalcome from the org preset (Day 11). A repo can override them, but a review by the platform team will catch it. - Every round ends with a new test: the validator in CI (Round 1), an extract report with a minimum number of dependencies (Round 3), required checks via branch protection (Round 4).