Day 8: curl and jq
Goal: Make HTTP requests with
curlin a targeted way and evaluate them, read, filter and safely generate JSON withjq. At the end,notes-addcreates every text correctly, andnotes-exportwrites all notes as CSV. · Duration: approx. 75 min · Prerequisite: Day 7
Briefing
- curl speaks HTTP: method (
-X), headers (-H), data (-d), time limits (--max-time), retries (--retry). With-w, it prints the status code and timings without touching the content. - jq is
sedandawkfor JSON:.field,.[],select(),map(),length.-rprints text without quotes. - You never assemble JSON with
printfor"{\"text\":\"$x\"}", but withjq -n --arg. Only then are quotes, backslashes and line breaks in the text safe.
notes-add 'Er sagte "hallo"' ─► jq -n --arg t … '{text: $t}' ─► curl -d @- ─► 201
notes-export ─► curl /notes ─► jq -r '.[] | [.id, .text] | @csv' ─► notes.csv
Drill 1: Targeted curl
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag08 && cd ~/sh-lab/tag08
curl -s -i $NOTES_URL/healthz | sed -n '1p;4p'
curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download} B\n' \
$NOTES_URL/version
curl -s $NOTES_URL/version; echo
curl -s -o /dev/null -w 'DNS %{time_namelookup}s, gesamt %{time_total}s\n' \
$NOTES_URL/notes | sed -E 's/[0-9]+\.[0-9]+s/…s/g'
curl -sS --max-time 2 --retry 2 --retry-delay 1 http://127.0.0.1:9/ 2>&1 | tail -1Expected:
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
200 text/plain; charset=utf-8 6 B
0.4.0
DNS …s, gesamt …s
curl: (7) Failed to connect to 127.0.0.1 port 9 after 0 ms: Could not connect to se…
| Option | Effect |
|---|---|
-s / -S |
silent / still show errors |
-f |
HTTP errors (≥ 400) as exit code 22 |
-i |
include headers in the output |
-o file |
content to a file (/dev/null discards it) |
-w '…' |
print values after the request: %{http_code},
%{time_total} … |
--retry n |
retry up to n times on transient errors |
Save your progress? Sign up for free
Drill 2: Reading with jq
curl -s $NOTES_URL/notes > notes.json
jq 'length' notes.json
jq '.[0] | keys' notes.json
jq -r '.[0].text' notes.json
jq -c '[.[] | select(.text == "erste Notiz")] | length' notes.json
jq -r '.[].text' notes.json | grep -c '^Lastnotiz'
jq -c 'map(.text | length) | {"kürzeste": min, "längste": max}' notes.jsonExpected (the numbers depend on how often you have called
notes-last and notes-add):
28
[
"id",
"text"
]
erste Notiz
1
22
{"kürzeste":11,"längste":28}
.[0]is the first element,.[]are all of them,|passes the result on – as in the shell, only inside jq.select(condition)keeps only what matches.map(f)appliesfto every element.- Keys with umlauts or special characters go in quotes:
{"kürzeste": min}. -r(raw) prints text without JSON quotes – a must when the result goes into a shell variable or togrep.-cprints compactly on one line.
Save your progress? Sign up for free
Drill 3: Generating JSON safely
text='Er sagte "hallo" und ging\ – mit Tab: Ende'
printf '{"text":"%s"}\n' "$text" | jq . 2>&1 | head -1
jq -n --arg t "$text" '{text: $t}'
jq -nc --arg t "$text" --argjson prio 2 '{text: $t, prio: $prio, tags: ["a", "b"]}'Expected:
jq: parse error: Invalid numeric literal at line 1, column 25
{
"text": "Er sagte \"hallo\" und ging\\ – mit Tab:\tEnde"
}
{"text":"Er sagte \"hallo\" und ging\\ – mit Tab:\tEnde","prio":2,"tags":["a","b"]}
--arg name value passes text, --argjson
real JSON (numbers, true, lists). jq escapes
", \ and control characters itself. With that,
you rebuild notes-add:
cat > ~/sh-lab/werkzeug/bin/notes-add <<'EOF'
#!/usr/bin/env bash
# notes-add – creates a note and prints its ID: notes-add text of the note
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
(($# > 0)) || die "Aufruf: $PROG Text der Notiz" 2
key="${NOTES_API_KEY:?fehlt – source ~/sh-lab/setup/env.sh}"
antwort=$(jq -n --arg t "$*" '{text: $t}' \
| curl -fsS -A "notes-add" -X POST -H "X-API-Key: $key" \
-H 'Content-Type: application/json' -d @- "$(notes_url)/notes") \
|| die "Anlegen gescheitert"
jq -r '.id' <<<"$antwort"
EOF
shellcheck -x ~/sh-lab/werkzeug/bin/notes-add && echo sauber
id=$(notes-add 'Er sagte "hallo"')
curl -s $NOTES_URL/notes | jq -r --arg id "$id" '.[] | select(.id == $id) | .text'
grep 'notes-add$' "$NOTES_LOG" | tail -1 | awk '{print $4, $5, $7, $10}'Expected:
sauber
Er sagte "hallo"
"POST /notes 201 notes-add
-d @-reads the request body from stdin – the JSON comes from the pipe and never appears on a command line.<<<"$antwort"is a here string: the text becomes the stdin ofjq.notes-addnow prints only the ID. That makes it suitable forid=$(notes-add …).
Save your progress? Sign up for free
Drill 4: Reading metrics
notes-api provides metrics at /metrics in the Prometheus
text format – not JSON, so again a case for grep and
awk:
curl -s $NOTES_URL/metrics | grep -E '^notes_(db_up|build_info)'
curl -s $NOTES_URL/metrics \
| awk -F'[{}]' '/^notes_http_requests_total/ {split($3, w, " "); s += w[1]}
END {print "Anfragen seit Start:", s}' | sed -E 's/[0-9]+$/…/'Expected:
notes_build_info{store="postgres",version="0.4.0"} 1
notes_db_up 1
Anfragen seit Start: …
-F'[{}]' splits at curly braces: field 2 is the labels,
field 3 the value. split() splits field 3 once more at
spaces.
Save your progress? Sign up for free
Drill 5: notes-export
cat > ~/sh-lab/werkzeug/bin/notes-export <<'EOF'
#!/usr/bin/env bash
# notes-export – writes all notes as CSV to stdout
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
echo '"id","text"'
curl -fsS -A notes-export "$(notes_url)/notes" | jq -r '.[] | [.id, .text] | @csv'
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-export
shellcheck -x ~/sh-lab/werkzeug/bin/notes-export && echo sauber
notes-export > notes.csv
head -1 notes.csv
grep -F 'hallo' notes.csv | cut -d, -f2-Expected:
sauber
"id","text"
"Er sagte ""hallo"""
@csv puts every field in quotes and doubles quotes
inside the text – just as spreadsheet programs expect. For tabs there is
@tsv, for the shell @sh.
Save your progress? Sign up for free
Checkpoint
notes-add Kontrollpunkt Tag 8 | grep -cE '^[0-9a-f-]{36}$'Expected: an ID in UUID format.
1
curl -s $NOTES_URL/notes | jq -r '.[] | select(.text == "Kontrollpunkt Tag 8") | .text'Expected:
Kontrollpunkt Tag 8
jq -n --arg x 'a"b' '$x | length'Expected:
3
Challenge
Write notes-find word: it lists the texts of all notes
that contain the word, regardless of upper and lower case.
Note: In jq, test(pattern; "i") checks
a regular expression case-insensitively. The word goes in with
--arg, not into the filter text.
Solution
cat > ~/sh-lab/werkzeug/bin/notes-find <<'EOF'
#!/usr/bin/env bash
# notes-find – find notes containing a word: notes-find word
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
(($# == 1)) || die "Aufruf: $PROG wort" 2
curl -fsS -A notes-find "$(notes_url)/notes" \
| jq -r --arg w "$1" '.[] | select(.text | test($w; "i")) | .text'
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-find
shellcheck -x ~/sh-lab/werkzeug/bin/notes-find && echo sauber
notes-find HALLO
notes-find einkaufen | sortExpected:
sauber
Er sagte "hallo"
Einkaufen: Milch und Brot
Einkaufen: Milch und Brot
With "…$1…" in the filter, a word like
") | halt_error could change the filter – with
--arg, it always stays a value.
Save your progress? Sign up for free
Enterprise Note
- Always use time limits:
curlwithout--max-timewaits indefinitely in the worst case. In scripts,--connect-timeoutand--max-timebelong on every request. - Secrets not on the command line: Anyone can see
-H "Authorization: Bearer $TOKEN"withps. Better:curl -H @fileor a--configfile with permissions 600. - jq versions: Ubuntu 26.04 ships jq 1.8. Older
servers have 1.6 – test functions like
ltrimstr,getpathor@base32dbeforehand.