Skip to content
Hands-on IT IT Drills

Day 8: curl and jq

Goal: Make HTTP requests with curl in a targeted way and evaluate them, read, filter and safely generate JSON with jq. At the end, notes-add creates every text correctly, and notes-export writes all notes as CSV. · Duration: approx. 75 min · Prerequisite: Day 7

Briefing

  • curl speaks HTTP: method (-X), headers (-H), data (-d), time limits (--max-time), retries (--retry). With -w, it prints the status code and timings without touching the content.
  • jq is sed and awk for JSON: .field, .[], select(), map(), length. -r prints text without quotes.
  • You never assemble JSON with printf or "{\"text\":\"$x\"}", but with jq -n --arg. Only then are quotes, backslashes and line breaks in the text safe.
 notes-add 'Er sagte "hallo"' ─► jq -n --arg t … '{text: $t}' ─► curl -d @- ─► 201
 notes-export ─► curl /notes ─► jq -r '.[] | [.id, .text] | @csv' ─► notes.csv

Drill 1: Targeted curl

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag08 && cd ~/sh-lab/tag08
curl -s -i $NOTES_URL/healthz | sed -n '1p;4p'
curl -s -o /dev/null -w '%{http_code} %{content_type} %{size_download} B\n' \
  $NOTES_URL/version
curl -s $NOTES_URL/version; echo
curl -s -o /dev/null -w 'DNS %{time_namelookup}s, gesamt %{time_total}s\n' \
  $NOTES_URL/notes | sed -E 's/[0-9]+\.[0-9]+s/…s/g'
curl -sS --max-time 2 --retry 2 --retry-delay 1 http://127.0.0.1:9/ 2>&1 | tail -1

Expected:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
200 text/plain; charset=utf-8 6 B
0.4.0
DNS …s, gesamt …s
curl: (7) Failed to connect to 127.0.0.1 port 9 after 0 ms: Could not connect to se…
Option Effect
-s / -S silent / still show errors
-f HTTP errors (≥ 400) as exit code 22
-i include headers in the output
-o file content to a file (/dev/null discards it)
-w '…' print values after the request: %{http_code}, %{time_total} …
--retry n retry up to n times on transient errors

Save your progress? Sign up for free

Drill 2: Reading with jq

curl -s $NOTES_URL/notes > notes.json
jq 'length' notes.json
jq '.[0] | keys' notes.json
jq -r '.[0].text' notes.json
jq -c '[.[] | select(.text == "erste Notiz")] | length' notes.json
jq -r '.[].text' notes.json | grep -c '^Lastnotiz'
jq -c 'map(.text | length) | {"kürzeste": min, "längste": max}' notes.json

Expected (the numbers depend on how often you have called notes-last and notes-add):

28
[
  "id",
  "text"
]
erste Notiz
1
22
{"kürzeste":11,"längste":28}
  • .[0] is the first element, .[] are all of them, | passes the result on – as in the shell, only inside jq.
  • select(condition) keeps only what matches. map(f) applies f to every element.
  • Keys with umlauts or special characters go in quotes: {"kürzeste": min}.
  • -r (raw) prints text without JSON quotes – a must when the result goes into a shell variable or to grep. -c prints compactly on one line.

Save your progress? Sign up for free

Drill 3: Generating JSON safely

text='Er sagte "hallo" und ging\ – mit Tab: Ende'
printf '{"text":"%s"}\n' "$text" | jq . 2>&1 | head -1
jq -n --arg t "$text" '{text: $t}'
jq -nc --arg t "$text" --argjson prio 2 '{text: $t, prio: $prio, tags: ["a", "b"]}'

Expected:

jq: parse error: Invalid numeric literal at line 1, column 25
{
  "text": "Er sagte \"hallo\" und ging\\ – mit Tab:\tEnde"
}
{"text":"Er sagte \"hallo\" und ging\\ – mit Tab:\tEnde","prio":2,"tags":["a","b"]}

--arg name value passes text, --argjson real JSON (numbers, true, lists). jq escapes ", \ and control characters itself. With that, you rebuild notes-add:

cat > ~/sh-lab/werkzeug/bin/notes-add <<'EOF'
#!/usr/bin/env bash
# notes-add – creates a note and prints its ID: notes-add text of the note
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
(($# > 0)) || die "Aufruf: $PROG Text der Notiz" 2
key="${NOTES_API_KEY:?fehlt – source ~/sh-lab/setup/env.sh}"

antwort=$(jq -n --arg t "$*" '{text: $t}' \
  | curl -fsS -A "notes-add" -X POST -H "X-API-Key: $key" \
    -H 'Content-Type: application/json' -d @- "$(notes_url)/notes") \
  || die "Anlegen gescheitert"
jq -r '.id' <<<"$antwort"
EOF
shellcheck -x ~/sh-lab/werkzeug/bin/notes-add && echo sauber
id=$(notes-add 'Er sagte "hallo"')
curl -s $NOTES_URL/notes | jq -r --arg id "$id" '.[] | select(.id == $id) | .text'
grep 'notes-add$' "$NOTES_LOG" | tail -1 | awk '{print $4, $5, $7, $10}'

Expected:

sauber
Er sagte "hallo"
"POST /notes 201 notes-add
  • -d @- reads the request body from stdin – the JSON comes from the pipe and never appears on a command line.
  • <<<"$antwort" is a here string: the text becomes the stdin of jq.
  • notes-add now prints only the ID. That makes it suitable for id=$(notes-add …).

Save your progress? Sign up for free

Drill 4: Reading metrics

notes-api provides metrics at /metrics in the Prometheus text format – not JSON, so again a case for grep and awk:

curl -s $NOTES_URL/metrics | grep -E '^notes_(db_up|build_info)'
curl -s $NOTES_URL/metrics \
  | awk -F'[{}]' '/^notes_http_requests_total/ {split($3, w, " "); s += w[1]}
    END {print "Anfragen seit Start:", s}' | sed -E 's/[0-9]+$/…/'

Expected:

notes_build_info{store="postgres",version="0.4.0"} 1
notes_db_up 1
Anfragen seit Start: …

-F'[{}]' splits at curly braces: field 2 is the labels, field 3 the value. split() splits field 3 once more at spaces.

Save your progress? Sign up for free

Drill 5: notes-export

cat > ~/sh-lab/werkzeug/bin/notes-export <<'EOF'
#!/usr/bin/env bash
# notes-export – writes all notes as CSV to stdout
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
echo '"id","text"'
curl -fsS -A notes-export "$(notes_url)/notes" | jq -r '.[] | [.id, .text] | @csv'
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-export
shellcheck -x ~/sh-lab/werkzeug/bin/notes-export && echo sauber
notes-export > notes.csv
head -1 notes.csv
grep -F 'hallo' notes.csv | cut -d, -f2-

Expected:

sauber
"id","text"
"Er sagte ""hallo"""

@csv puts every field in quotes and doubles quotes inside the text – just as spreadsheet programs expect. For tabs there is @tsv, for the shell @sh.

Save your progress? Sign up for free

Checkpoint

notes-add Kontrollpunkt Tag 8 | grep -cE '^[0-9a-f-]{36}$'

Expected: an ID in UUID format.

1
curl -s $NOTES_URL/notes | jq -r '.[] | select(.text == "Kontrollpunkt Tag 8") | .text'

Expected:

Kontrollpunkt Tag 8
jq -n --arg x 'a"b' '$x | length'

Expected:

3

Challenge

Write notes-find word: it lists the texts of all notes that contain the word, regardless of upper and lower case.

Note: In jq, test(pattern; "i") checks a regular expression case-insensitively. The word goes in with --arg, not into the filter text.

Solution

cat > ~/sh-lab/werkzeug/bin/notes-find <<'EOF'
#!/usr/bin/env bash
# notes-find – find notes containing a word: notes-find word
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need curl jq
(($# == 1)) || die "Aufruf: $PROG wort" 2
curl -fsS -A notes-find "$(notes_url)/notes" \
  | jq -r --arg w "$1" '.[] | select(.text | test($w; "i")) | .text'
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-find
shellcheck -x ~/sh-lab/werkzeug/bin/notes-find && echo sauber
notes-find HALLO
notes-find einkaufen | sort

Expected:

sauber
Er sagte "hallo"
Einkaufen:   Milch und  Brot
Einkaufen: Milch und Brot

With "…$1…" in the filter, a word like ") | halt_error could change the filter – with --arg, it always stays a value.

Save your progress? Sign up for free

Enterprise Note

  • Always use time limits: curl without --max-time waits indefinitely in the worst case. In scripts, --connect-timeout and --max-time belong on every request.
  • Secrets not on the command line: Anyone can see -H "Authorization: Bearer $TOKEN" with ps. Better: curl -H @file or a --config file with permissions 600.
  • jq versions: Ubuntu 26.04 ships jq 1.8. Older servers have 1.6 – test functions like ltrimstr, getpath or @base32d beforehand.