What is the difference between journalctl and traditional syslog?

journalctl reads from the systemd journal, a binary log format that stores structured metadata (unit name, PID, priority, boot ID) alongside each message. Traditional syslog stores plain text files in /var/log/. The journal enables powerful filtering by any metadata field, while syslog requires grep and regex parsing. Both can run side by side — rsyslog or syslog-ng can import journal entries for compatibility.

Are systemd journal logs persistent across reboots by default?

It depends on the distribution. On Ubuntu and Debian, journald defaults to persistent storage if /var/log/journal/ exists. On some distributions, the default is volatile (stored only in /run/log/journal/, lost on reboot). To guarantee persistence, create the directory with 'sudo mkdir -p /var/log/journal' and restart systemd-journald.

How do I limit the disk space used by systemd journal logs?

Edit /etc/systemd/journald.conf and set SystemMaxUse (maximum disk usage, e.g., 500M), SystemMaxFileSize (maximum individual file size), and MaxRetentionSec (maximum age, e.g., 1month). After editing, restart the service with 'sudo systemctl restart systemd-journald'. You can also run 'sudo journalctl --vacuum-size=500M' for immediate cleanup.

Can I export journalctl output in JSON format for log analysis tools?

Yes. Use 'journalctl -o json' for one JSON object per line, or 'journalctl -o json-pretty' for human-readable JSON. This is useful for piping into tools like jq, sending to Elasticsearch, or processing with custom scripts. Other output formats include short, verbose, cat, and export.

Mastering journalctl: Query and Analyze Linux System Logs

When something goes wrong on a Linux server, your first instinct is to check the logs. On any modern Linux distribution running systemd, the primary tool for that job is journalctl. It provides a single interface to query structured logs from every service, the kernel, and the boot process itself — all indexed and filterable by time, priority, service unit, and dozens of other fields.

This guide covers everything you need to use journalctl effectively for daily administration and incident troubleshooting, from basic queries to advanced filtering, persistent storage configuration, and integration with external log analysis tools.

Prerequisites

Before you begin, make sure you have:

A Linux system running systemd (Ubuntu 20.04+, Debian 11+, RHEL/CentOS 8+, Fedora, Arch, or similar)
Terminal access with sudo privileges
Basic familiarity with systemd services (systemctl start, systemctl status)

Understanding the systemd Journal

The systemd journal is a centralized logging system managed by the systemd-journald service. Unlike traditional syslog, which writes plain text to files in /var/log/, the journal stores entries in a structured binary format that includes metadata with every log line:

Timestamp with microsecond precision
Boot ID — which boot session produced the entry
Unit name — which systemd service generated it
PID and UID — process and user identifiers
Priority level — from emergency (0) to debug (7)
Machine ID — useful in multi-host setups

This structure is what makes journalctl so powerful: you can filter on any combination of these fields without writing complex grep patterns.

Check that journald is running on your system:

systemctl status systemd-journald

View how much disk space the journal currently uses:

journalctl --disk-usage

Basic journalctl Usage

View All Logs

journalctl

This opens the full journal in a pager (usually less). Press q to exit, / to search, and G to jump to the end (most recent entries).

View Most Recent Entries

# Show the last 50 lines
journalctl -n 50

# Show the last 100 lines with no pager (direct output)
journalctl -n 100 --no-pager

Follow Logs in Real Time

journalctl -f

This works like tail -f but for the entire journal. Press Ctrl+C to stop. You can combine -f with any filter:

# Follow only nginx logs
journalctl -f -u nginx.service

Reverse Order (Newest First)

journalctl -r

Filtering by Time

Time-based filtering is one of journalctl’s most useful features for incident investigation.

Absolute Time Ranges

# Logs since a specific date and time
journalctl --since "2025-12-26 08:00:00"

# Logs between two timestamps
journalctl --since "2025-12-25 18:00" --until "2025-12-26 06:00"

# Logs from a specific date (whole day)
journalctl --since "2025-12-25" --until "2025-12-26"

Relative Time Ranges

# Logs from the last hour
journalctl --since "1 hour ago"

# Logs from the last 30 minutes
journalctl --since "30 min ago"

# Since yesterday
journalctl --since yesterday

# Today only
journalctl --since today

Current Boot vs. Previous Boots

# Logs from the current boot only
journalctl -b

# Logs from the previous boot
journalctl -b -1

# Logs from two boots ago
journalctl -b -2

# List all available boots
journalctl --list-boots

The --list-boots output shows boot IDs and timestamps, which is invaluable for investigating what happened before an unexpected reboot.

Filtering by Service Unit

This is where journalctl saves the most time compared to traditional log analysis.

Single Service

# All logs from nginx
journalctl -u nginx.service

# All logs from SSH daemon
journalctl -u sshd.service

# All logs from PostgreSQL
journalctl -u postgresql.service

Multiple Services

# Logs from both nginx and PHP-FPM
journalctl -u nginx.service -u php8.1-fpm.service

Combine Service and Time Filters

# nginx errors in the last 2 hours
journalctl -u nginx.service --since "2 hours ago" -p err

User Services

# Logs from a user-level service
journalctl --user -u myapp.service

Filtering by Priority

Syslog priority levels range from 0 (emergency) to 7 (debug):

Priority	Keyword	Description
0	emerg	System is unusable
1	alert	Immediate action required
2	crit	Critical conditions
3	err	Error conditions
4	warning	Warning conditions
5	notice	Normal but significant
6	info	Informational
7	debug	Debug-level messages

Filter by Maximum Priority

# Show errors and above (0-3)
journalctl -p err

# Show warnings and above (0-4)
journalctl -p warning

# Show critical and above (0-2)
journalctl -p crit

Priority Range

# Show only warnings and errors (exclude critical/emergency)
journalctl -p warning..err

Combine with Other Filters

# Errors from nginx in the last 24 hours
journalctl -u nginx.service -p err --since "24 hours ago"

Filtering by Other Fields

journalctl can filter on any field stored in the journal. List available fields:

journalctl --fields

Common Field Filters

# Logs from a specific PID
journalctl _PID=1234

# Logs from a specific user
journalctl _UID=1000

# Logs from the kernel
journalctl -k
# or equivalently:
journalctl _TRANSPORT=kernel

# Logs from a specific binary
journalctl /usr/sbin/sshd

# Logs from a specific hostname (in multi-host setups)
journalctl _HOSTNAME=webserver01

Combining Field Filters

When you specify multiple different fields, they are combined with AND logic:

# Logs from UID 1000 AND priority error
journalctl _UID=1000 -p err

When you specify the same field multiple times, they are combined with OR logic:

# Logs from PID 1234 OR PID 5678
journalctl _PID=1234 _PID=5678

Output Formats

journalctl supports several output formats for different use cases:

# Default short format (similar to syslog)
journalctl -o short

# Short format with ISO timestamps
journalctl -o short-iso

# Full structured output
journalctl -o verbose

# JSON format (one object per line)
journalctl -o json

# Pretty-printed JSON
journalctl -o json-pretty

# Raw message only (no metadata)
journalctl -o cat

# Binary export format (for journalctl --merge)
journalctl -o export

JSON Output for Log Analysis

The JSON format is particularly useful for piping to analysis tools:

# Extract all nginx errors as JSON
journalctl -u nginx.service -p err -o json --no-pager | jq '.'

# Get specific fields
journalctl -u sshd.service -o json --no-pager | jq '{timestamp: .__REALTIME_TIMESTAMP, message: .MESSAGE, pid: ._PID}'

Persistent Storage Configuration

By default, some distributions store journal logs only in memory (/run/log/journal/), which means they are lost on reboot.

Enable Persistent Storage

# Create the persistent storage directory
sudo mkdir -p /var/log/journal

# Set correct ownership
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Restart journald
sudo systemctl restart systemd-journald

Configure Retention Limits

Edit /etc/systemd/journald.conf:

[Journal]
# Maximum disk space for journal files
SystemMaxUse=500M

# Maximum size of individual journal files
SystemMaxFileSize=50M

# Maximum time to keep entries
MaxRetentionSec=1month

# Maximum disk space for runtime (volatile) journal
RuntimeMaxUse=100M

After editing, restart the service:

sudo systemctl restart systemd-journald

Manual Cleanup

# Remove entries older than 2 weeks
sudo journalctl --vacuum-time=2weeks

# Reduce journal to a maximum of 500MB
sudo journalctl --vacuum-size=500M

# Keep only the most recent 5 journal files
sudo journalctl --vacuum-files=5

Verify Settings

# Check current disk usage
journalctl --disk-usage

# Show journal configuration
systemd-analyze cat-config systemd/journald.conf

Kernel Messages

For kernel-level troubleshooting, journalctl replaces the dmesg command with better filtering:

# All kernel messages
journalctl -k

# Kernel messages from the current boot
journalctl -k -b

# Kernel errors only
journalctl -k -p err

# Kernel messages from a previous boot
journalctl -k -b -1

This is essential for diagnosing hardware issues, driver problems, filesystem errors, and OOM (out-of-memory) kills.

Practical Troubleshooting Examples

Investigate a Service Crash

# Check when the service last failed
systemctl status myapp.service

# View the last 200 lines before the crash
journalctl -u myapp.service -n 200 --no-pager

# Check for OOM kills around the same time
journalctl -k --since "1 hour ago" | grep -i "oom\|killed process"

# All SSH authentication messages
journalctl -u sshd.service | grep -i "failed\|invalid\|accepted"

# Failed SSH attempts in the last 24 hours
journalctl -u sshd.service --since "24 hours ago" -p warning

Monitor Disk and Filesystem Issues

# Filesystem errors from the kernel
journalctl -k | grep -i "ext4\|xfs\|btrfs\|i/o error\|read-only"

# Disk-related messages
journalctl -k | grep -i "sd[a-z]\|nvme\|ata"

Check Boot Problems

# Full log from the last failed boot
journalctl -b -1 -p err

# List all boots to find the problematic one
journalctl --list-boots

# Show only failed service starts during boot
journalctl -b -p err | grep "Failed to start"

Integration with External Tools

Forward to rsyslog

If you need traditional log files alongside the journal, configure rsyslog to read from the journal. In /etc/rsyslog.conf:

module(load="imjournal")

Export for Elasticsearch/Loki

# Export as JSON for ingestion
journalctl -o json --since "1 hour ago" --no-pager > /tmp/journal-export.json

# Stream to a remote endpoint
journalctl -f -o json | curl -X POST -H "Content-Type: application/json" -d @- http://logserver:9200/journal/_bulk

Use with systemd-journal-remote

For centralized logging across multiple servers, systemd provides systemd-journal-remote:

# Install on the central server
sudo apt install systemd-journal-remote

# Enable the receiver
sudo systemctl enable --now systemd-journal-remote.socket

Troubleshooting

Journal Files Are Corrupt

# Verify journal integrity
journalctl --verify

# If corrupt, rotate and start fresh
sudo journalctl --rotate
sudo journalctl --vacuum-time=1s

No Logs from a Specific Service

Verify the service uses standard output/error:

systemctl show myapp.service | grep -i "standard\|log"

Make sure the unit file redirects output to the journal:

[Service]
StandardOutput=journal
StandardError=journal

journalctl Is Slow on Large Journals

# Use --since to limit the search range
journalctl --since "1 hour ago" -u myapp.service

# Reduce journal size
sudo journalctl --vacuum-size=200M

Summary

journalctl is the standard tool for querying logs on systemd-based Linux systems. Key takeaways:

Use -u to filter by service, -p to filter by priority, and --since/--until for time ranges
Combine filters for targeted searches: journalctl -u nginx -p err --since "1 hour ago"
Use -b to scope queries to the current or previous boot sessions
Enable persistent storage with mkdir -p /var/log/journal if your distribution defaults to volatile
Configure retention limits in /etc/systemd/journald.conf to prevent disk exhaustion
Use -o json for integration with log analysis pipelines like Elasticsearch or Loki
Use journalctl -f instead of tail -f for real-time log monitoring with filtering

For related topics, see our guides on Logrotate: Manage Log Files on Linux and Elasticsearch Setup for Log Analysis.