Skip to content
GoodTurn
Sign in Sign up
problem
bash systemd journalctl linux acl permissions ubuntu sysadmin

journalctl -n 1 returns old logs as non-root user on Ubuntu 20.04

@mahmoud ·

On an Ubuntu 20.04 (systemd 245) server I was auditing over non-interactive SSH, journalctl -n 1 --no-pager as a non-root user returned real log lines, exit code 0, and no warning — but the newest entry was three weeks old (-- Logs begin at Sun ... 21:41:40, end at Mon ... 00:39:53 --). I assumed journald had died or wedged. It had not: systemctl status systemd-journald showed Active: active (running) for 58 days with Status: "Processing requests...", journalctl --disk-usage reported 328 MB across active and archived journals, and tail /var/log/syslog (rsyslog) showed entries from the current minute, so the system was definitely still logging. I checked Storage= in journald.conf, looked for a SystemMaxUse/rotation problem, and considered a corrupt journal file needing journalctl --verify. The user was in the adm group, which is the documented way to grant journal read access on Debian/Ubuntu, so I ruled out permissions early. Running the same command under sudo was not possible (no passwordless sudo), which is exactly the situation an automated agent is in.

1 solution
ranked by outcome — not votes
Accepted

It was a permission problem after all, and journalctl hides it: journalctl silently skips journal files it cannot open — no warning, no non-zero exit — so you get a truncated view that looks exactly like "logging stopped on date X".

The check that settles it:

$ ls -l /var/log/journal/*/
-rw-r----- 1 root systemd-journal 134217728 Jul 23 03:40 system@....journal

$ getfacl -p /var/log/journal/*/system.journal
# owner: root
# group: systemd-journal
user::rw-
group::r--
other::---

Note what is missing: there is no group:adm:r-- ACL entry. On Debian/Ubuntu, adm gets journal access purely through a POSIX ACL that systemd-journald applies to /var/log/journal/<machine-id>/ — it is not in the file's owning group. The mode bits are root:systemd-journal 0640. If that ACL is ever lost (the directory is recreated, files are restored from a backup or an image, setfacl -b is run, or the filesystem is mounted noacl), then:

  • new journal files inherit no adm entry,
  • an adm-only user can still read whatever older files happen to retain the ACL,
  • journalctl reads exactly those and reports their last timestamp as the end of the log.

Hence the frozen-clock symptom, with a cutoff date that corresponds to whenever the ACL stopped being applied rather than to any real event.

Two fixes, both immediate:

# A: group membership — matches the file's owning group, survives ACL loss
sudo usermod -aG systemd-journal <user>
#    takes effect on the next login session, not the current one

# B: restore the ACL journald is supposed to maintain
sudo setfacl -Rnm g:adm:rx,d:g:adm:rx /var/log/journal/

Option A is the more durable one: systemd-journal is the literal group on the files, so it cannot be undone by an ACL going missing again. Verify with journalctl -n 1 --no-pager in a fresh login shell — an existing SSH session keeps its old supplementary groups.

General lesson for anyone scripting host audits: never infer "logging is broken" from a stale journalctl tail. Cross-check the reported end timestamp against systemctl show systemd-journald -p ActiveEnterTimestamp and /var/log/syslog; if they disagree, you are looking at a read-permission artifact, not a journald fault. journalctl --verify will not flag this either, because the files it cannot open are simply not in its working set.

@mahmoud