■ space · prose lane

THE SIX, AS CARDS

the -p ladder, descending — journalctl -p warning..alert takes a range · -p err is the workhorse

WHY NOT JUST READ /var/log/syslog?

Old-school Linux logging wrote plain text files under /var/log — syslog, auth.log, messages. systemd replaced that with the journal: a binary, indexed log managed by systemd-journald. Plain grep cannot read it; journalctl is the only front door, and in exchange you get filtering by service, time, boot and severity without regex gymnastics.

The mental model: the journal stores everything, and journalctl queries it. A service does not need its own log file — anything it writes to stdout/stderr under systemd lands in the journal automatically. One caveat: on some minimal installs the journal is volatile (stored in /run, wiped on reboot) because /var/log/journal does not exist. Fix it once:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

verify: ls /var/log/journal → directory exists — boots accumulate instead of wiping

LINES, FOLLOWING, UNITS

-n limits output to the newest N lines (default is 10). --no-pager matters more than it looks: without it journalctl opens less and your script, pipe or ssh one-liner hangs waiting for a keypress — any command you pipe onward should carry it.

journalctl -n 100                      # last 100 lines of everything
journalctl -u nginx -n 100             # last 100 lines from one unit
journalctl -n 100 --no-pager           # print and exit — perfect for piping

-f is journalctl's tail -f — it streams new entries as they happen. This is the command to run in a second terminal while you restart a service in the first one: systemctl restart nginx in one pane, journalctl -u nginx -f in the other, and the cause of the crash usually announces itself within seconds.

journalctl -f                    # everything, live
journalctl -u sshd -f            # just the SSH daemon, live
journalctl -u ollama -f -n 50    # live, but start with the last 50 lines

-u filters by systemd unit — one, several, or the exact-match alternative:

journalctl -u nginx                    # one unit, all history
journalctl -u nginx -u redis           # several units at once
journalctl _SYSTEMD_UNIT=nginx.service # exact match

-u <service> shows nothing but the service clearly runs? Units can spawn helper units that keep the interesting output — a failing web app sometimes logs the real error under a different unit. Find the actual name first with systemctl list-units --type=service | grep -i <guess>. The same pattern works for timers (systemctl list-timers shows names) and for anything running under systemd --user, add --user: journalctl --user -u pipewire -n 50.

TIME AND SEVERITY

--since and --until take timestamps, but they also accept forgiving relative phrasing. Pair a window with a unit and you have incident triage in one line — "what did the API log between 09:00 and when it fell over?" — faster than any log aggregator for a single box:

journalctl --since "1 hour ago"
journalctl --since "2026-09-01 09:00" --until "2026-09-01 12:00"
journalctl --since today
journalctl --since yesterday --until now -u cron

-p filters by priority using the syslog ladder shown above. When a box is "acting weird", journalctl -p err -b --since today is the fastest sanity check — it answers "is anything actually failing?" without noise from routine info lines.

journalctl -p err -b                    # errors only, since this boot
journalctl -p warning..alert -u nginx   # a range of severities

BOOTS

The crash you rebooted away is still in the journal. Services that crash at startup produce log lines before your current boot, and journalctl silently shows only the current one. -b selects boots — this is the single most useful flag for "it broke and I rebooted it and now I can't see the error":

journalctl -b                 # current boot only
journalctl -b -1              # previous boot
journalctl -b -1 -u sshd      # why SSH died last time
journalctl --list-boots       # index of stored boots

THE DISK

Check what the journal costs, then cap it — one vacuum now, one config line forever:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M     # keep newest 500 MB
sudo journalctl --vacuum-time=30d      # keep newest 30 days

To make the cap permanent, set SystemMaxUse=500M under the [Journal] section in /etc/systemd/journald.conf, then sudo systemctl restart systemd-journald. Between the size cap and the boot filter, the journal stays a tool rather than a slow disk leak.

08WHICH LOG FIRST?

• journalctl — application and service behaviour. Anything systemd manages is here, indexed and filterable. Default first stop.
• journalctl -k / dmesg — kernel and hardware: OOM kills, USB resets, disk errors. If a process died mysteriously, look for the OOM killer here before blaming the app.
• /var/log/<app>/ — only for apps that do their own file logging (nginx access logs, PostgreSQL). Even then, startup errors usually still land in the journal.

The same workflow extends to local AI services — when an ollama serve unit stalls, journalctl -u ollama -n 100 shows the model load failure immediately. That is the whole cheat sheet: -u for unit, -f to follow, -n for lines, --since for time, -p for severity, -b for boots. Six flags cover nearly every log question a Linux box will ask you — and when the question is why a systemd service won't start, these flags are the whole investigation.

end of the cheat sheet · nothing fetched · everything local