Mastering systemd: Service Management on Modern Linux

systemd is the init system and service manager that powers virtually every modern Linux distribution, including Ubuntu, Debian, Fedora, CentOS, and Arch Linux. It replaced the traditional SysVinit system to provide parallel service startup, dependency management, on-demand activation, and centralized logging. Whether you are managing a single-server deployment or a fleet of production machines, understanding systemd is essential for effective Linux administration. This guide covers systemctl commands, custom unit files, timers, journal management, boot analysis, and resource control with cgroups.

Prerequisites

Before you begin, make sure you have:

• Ubuntu Server 20.04, 22.04, or 24.04 (or any systemd-based distribution)
• Terminal access with sudo privileges
• Basic familiarity with the Linux command line
• A text editor (nano, vim, or your preference)

What Is systemd?

systemd is a system and service manager for Linux. It runs as PID 1 — the first process started by the kernel — and is responsible for bootstrapping the entire user space, managing services throughout their lifecycle, and shutting down the system cleanly.

Core components of systemd include:

• systemctl — the primary command-line tool for controlling services and inspecting system state
• journald — a structured logging system that collects logs from all services, the kernel, and early boot
• systemd-analyze — a tool for profiling and optimizing boot performance
• systemd-resolved — a DNS resolver service
• systemd-networkd — a network configuration manager
• systemd-logind — a login and session manager
• systemd-tmpfiles — a tool for managing temporary files and directories

Verify that your system runs systemd:

systemctl --version

Expected output:

systemd 252 (252.22-1ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT ...

Check the overall system state:

systemctl status

This shows a tree of running services, their PIDs, and memory usage.

systemd vs SysVinit

Understanding what changed from SysVinit helps clarify why systemd works the way it does:

Feature	SysVinit	systemd
Service definition	Shell scripts in /etc/init.d/	Declarative unit files in /etc/systemd/system/
Startup order	Sequential (one after another)	Parallel (based on dependencies)
Dependency handling	Manual with runlevel scripts	Automatic with Requires, After, Wants
Process tracking	PID files (unreliable)	cgroups (kernel-level tracking)
Logging	syslog / separate log files	journald (structured binary journal)
On-demand activation	Not built-in	Socket activation, D-Bus activation
Resource control	None	cgroups (CPU, memory, I/O limits)
Boot targets	Runlevels (0-6)	Targets (multi-user.target, graphical.target)

The key advantage of systemd is reliability: it tracks processes via cgroups so it always knows which processes belong to a service, even if they fork. PID files used by SysVinit were a common source of stale state and service management failures.

Managing Services with systemctl

systemctl is the command you will use most often. It communicates with the systemd manager to control services, inspect their status, and manage their boot behavior.

Starting and Stopping Services

# Start a service
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service (stop then start)
sudo systemctl restart nginx

# Reload configuration without restarting (if supported)
sudo systemctl reload nginx

# Reload or restart (reload if supported, otherwise restart)
sudo systemctl reload-or-restart nginx

Checking Service Status

sudo systemctl status nginx

Example output:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-01-29 10:15:30 UTC; 2h ago
       Docs: man:nginx(8)
    Process: 1234 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1240 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1241 (nginx)
      Tasks: 3 (limit: 4647)
     Memory: 8.2M
        CPU: 52ms
     CGroup: /system.slice/nginx.service
             ├─1241 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─1242 "nginx: worker process" "" "" "" "" "" "" "" "" ""
             └─1243 "nginx: worker process" "" "" "" "" "" "" "" "" ""

Key fields:

• Loaded — whether the unit file is loaded and where it is located
• Active — the current state (active, inactive, failed)
• Main PID — the primary process ID for the service
• CGroup — all processes belonging to this service

Enabling and Disabling Services

Enabling a service means it will start automatically at boot. Disabling removes this behavior.

# Enable a service to start at boot
sudo systemctl enable nginx

# Disable a service from starting at boot
sudo systemctl disable nginx

# Enable and start in one command
sudo systemctl enable --now nginx

# Disable and stop in one command
sudo systemctl disable --now nginx

Checking Enabled State

systemctl is-enabled nginx
# Output: enabled

systemctl is-active nginx
# Output: active

systemctl is-failed nginx
# Output: active (not failed)

Listing Services

# List all active services
systemctl list-units --type=service

# List all services (including inactive)
systemctl list-units --type=service --all

# List enabled services
systemctl list-unit-files --type=service --state=enabled

# List failed services
systemctl list-units --type=service --state=failed

Understanding Unit Files

systemd manages everything through unit files — declarative configuration files that describe services, mount points, devices, sockets, timers, and more.

Unit Types

Unit Type	Extension	Purpose
Service	.service	Manages a daemon or one-shot process
Timer	.timer	Schedules service activation (cron replacement)
Socket	.socket	Manages socket-based activation
Mount	.mount	Manages filesystem mount points
Target	.target	Groups units for synchronization
Path	.path	Monitors filesystem paths for changes
Slice	.slice	Manages cgroup resource partitions
Device	.device	Manages kernel device nodes

Unit File Locations

Unit files are loaded from multiple directories with specific precedence:

Directory	Purpose	Priority
/etc/systemd/system/	Local administrator overrides	Highest
/run/systemd/system/	Runtime units (non-persistent)	Medium
/lib/systemd/system/	Distribution-provided units	Lowest

Always create or modify units in /etc/systemd/system/. Never edit files in /lib/systemd/system/ directly, as package updates will overwrite your changes.

Inspecting Unit Files

# View the full unit file for a service
systemctl cat nginx.service

# Show all properties of a service (effective configuration)
systemctl show nginx.service

# Show a specific property
systemctl show nginx.service -p MainPID
systemctl show nginx.service -p ActiveState

# List all dependencies of a service
systemctl list-dependencies nginx.service

Creating Custom Service Units

One of the most practical systemd skills is writing your own service unit files. This allows you to manage any application — from a Node.js server to a Go binary to a Python script — as a proper system service.

Basic Service Unit

Create a file at /etc/systemd/system/myapp.service:

[Unit]
Description=My Custom Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Section Breakdown

[Unit] — metadata and dependencies:

• Description — human-readable description shown in status output
• Documentation — URL for documentation
• After — start this unit after the listed units
• Wants — weak dependency (does not fail if the dependency fails)
• Requires — strong dependency (fails if the dependency fails)

[Service] — service behavior:

• Type — how systemd determines the service is ready (simple, forking, oneshot, notify, idle)
• User / Group — run the service as this user/group
• WorkingDirectory — set the working directory
• ExecStart — the command to start the service
• ExecStartPre — commands to run before the main process
• ExecStartPost — commands to run after the main process starts
• ExecStop — the command to stop the service (defaults to SIGTERM)
• ExecReload — the command to reload configuration
• Restart — when to restart (no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, always)
• RestartSec — time to wait before restarting

[Install] — installation behavior:

• WantedBy — the target that should include this service (usually multi-user.target)

Service Types Explained

# Simple (default): ExecStart is the main process
Type=simple

# Forking: the process forks and the parent exits (traditional daemons)
Type=forking
PIDFile=/run/myapp.pid

# Oneshot: the process runs to completion then exits
Type=oneshot
RemainAfterExit=yes

# Notify: the process sends a notification when ready
Type=notify

# Idle: like simple but waits until all jobs are dispatched
Type=idle

Activating the Service

After creating or modifying a unit file, reload the systemd daemon and manage the service:

# Reload systemd to read new/modified unit files
sudo systemctl daemon-reload

# Start the service
sudo systemctl start myapp

# Check the status
sudo systemctl status myapp

# Enable at boot
sudo systemctl enable myapp

Overriding Existing Units

Instead of editing vendor-provided unit files, use drop-in overrides:

# Create an override directory and file
sudo systemctl edit nginx.service

This opens an editor for /etc/systemd/system/nginx.service.d/override.conf. Add only the directives you want to change:

[Service]
LimitNOFILE=65535

To view the effective configuration after overrides:

systemctl cat nginx.service

systemd Timers

systemd timers are a modern replacement for cron jobs. They offer persistent scheduling (catching up on missed runs), per-timer logging, and integration with cgroups resource control.

Timer Unit Structure

A timer requires two files: a .timer unit and a corresponding .service unit with the same name.

Create /etc/systemd/system/backup.timer:

[Unit]
Description=Run backup every day at 2:00 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Create /etc/systemd/system/backup.service:

[Unit]
Description=System Backup Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=root
StandardOutput=journal
StandardError=journal

Timer Types

Calendar-based (OnCalendar):

# Every day at midnight
OnCalendar=daily

# Every Monday at 3 AM
OnCalendar=Mon *-*-* 03:00:00

# Every 6 hours
OnCalendar=*-*-* 00/6:00:00

# First day of every month
OnCalendar=*-*-01 00:00:00

# Every 15 minutes
OnCalendar=*-*-* *:00/15:00

Monotonic (relative to an event):

# 5 minutes after boot
OnBootSec=5min

# 1 hour after the timer is activated
OnActiveSec=1h

# 30 minutes after the last run completed
OnUnitActiveSec=30min

Managing Timers

# Reload, enable, and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# List all active timers
systemctl list-timers --all

# Check the timer status
systemctl status backup.timer

# Test the associated service manually
sudo systemctl start backup.service

Verifying Calendar Expressions

Use systemd-analyze calendar to test your timer expressions:

systemd-analyze calendar "Mon *-*-* 03:00:00"

Output:

  Original form: Mon *-*-* 03:00:00
Normalized form: Mon *-*-* 03:00:00
    Next elapse: Mon 2026-02-02 03:00:00 UTC
       (in UTC): Mon 2026-02-02 03:00:00 UTC
       From now: 3 days left

Journal and Log Management with journalctl

journald is the systemd logging component. It collects logs from all services, the kernel, and the early boot process into a structured binary journal that supports fast searches, filtering, and export.

Basic journalctl Usage

# View all logs (oldest first)
journalctl

# View the most recent entries (newest first)
journalctl -r

# Follow logs in real time
journalctl -f

# Show logs since last boot
journalctl -b

# Show logs from the previous boot
journalctl -b -1

Filtering by Service

# Logs for a specific service
journalctl -u nginx.service

# Follow a specific service in real time
journalctl -u nginx.service -f

# Logs for multiple services
journalctl -u nginx.service -u php-fpm.service

Filtering by Time

# Logs since a specific time
journalctl --since "2026-01-29 10:00:00"

# Logs in a time range
journalctl --since "2026-01-29 10:00:00" --until "2026-01-29 12:00:00"

# Logs from the last hour
journalctl --since "1 hour ago"

# Logs from today
journalctl --since today

Filtering by Priority

journalctl supports syslog priority levels:

# Show only errors and above
journalctl -p err

# Show warnings and above
journalctl -p warning

# Show critical and above
journalctl -p crit

Priority Level	Keyword	Numeric
Emergency	emerg	0
Alert	alert	1
Critical	crit	2
Error	err	3
Warning	warning	4
Notice	notice	5
Info	info	6
Debug	debug	7

Output Formats

# JSON output (for parsing)
journalctl -u nginx.service -o json-pretty

# Short output with precise timestamps
journalctl -u nginx.service -o short-precise

# Verbose output with all fields
journalctl -u nginx.service -o verbose

Managing Journal Size

# Check current disk usage
journalctl --disk-usage

# Retain only the last 2 weeks of logs
sudo journalctl --vacuum-time=2weeks

# Limit journal size to 500 MB
sudo journalctl --vacuum-size=500M

# Remove archived journals
sudo journalctl --rotate
sudo journalctl --vacuum-time=1s

For persistent configuration, edit /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=2week

Then restart journald:

sudo systemctl restart systemd-journald

Analyzing Boot Performance

systemd provides built-in tools to measure and optimize boot times.

Overall Boot Time

systemd-analyze

Output:

Startup finished in 3.245s (kernel) + 8.712s (userspace) = 11.957s
graphical.target reached after 8.510s in userspace.

Service-by-Service Breakdown

# List services sorted by startup time
systemd-analyze blame

Output:

5.012s NetworkManager-wait-online.service
1.234s snapd.service
  892ms docker.service
  456ms nginx.service
  234ms ssh.service
  ...

Critical Chain

The critical chain shows the sequence of units that determined the total boot time:

systemd-analyze critical-chain

Output:

graphical.target @8.510s
└─multi-user.target @8.508s
  └─nginx.service @8.052s +456ms
    └─network-online.target @8.050s
      └─NetworkManager-wait-online.service @3.038s +5.012s
        └─NetworkManager.service @2.890s +145ms
          └─basic.target @2.880s

SVG Boot Chart

Generate a visual timeline of the boot process:

systemd-analyze plot > boot-chart.svg

Open the SVG file in a web browser to see a detailed graphical representation of service startup times and dependencies.

Identifying Slow Services

# Find services that are slow to start
systemd-analyze blame | head -20

# Check if a specific service is delaying boot
systemd-analyze critical-chain nginx.service

Common boot optimization strategies:

• Disable services you do not need (sudo systemctl disable servicename)
• Mask services you never want to run (sudo systemctl mask servicename)
• Investigate NetworkManager-wait-online.service (a common boot delay source)
• Use socket activation for services that are not needed immediately

Resource Control with Cgroups

systemd uses Linux control groups (cgroups) to manage and limit the resources available to each service. This prevents a single misbehaving service from consuming all system resources.

Viewing Current Resource Usage

# Show resource usage by slice
systemd-cgtop

Setting Memory Limits

Add resource directives to the [Service] section of a unit file:

[Service]
MemoryMax=512M
MemoryHigh=400M

• MemoryMax — hard limit; the OOM killer intervenes if exceeded
• MemoryHigh — soft limit; the kernel throttles the service when exceeded

Setting CPU Limits

[Service]
CPUQuota=50%
CPUWeight=100

• CPUQuota — limits the service to a percentage of a single CPU core (200% = two cores)
• CPUWeight — relative weight for CPU time sharing (default is 100)

Setting I/O Limits

[Service]
IOWeight=50
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M

Applying Limits at Runtime

You can apply resource limits temporarily without modifying unit files:

# Limit memory for a running service
sudo systemctl set-property nginx.service MemoryMax=512M

# Limit CPU for a running service
sudo systemctl set-property nginx.service CPUQuota=50%

# Make the change persistent
sudo systemctl set-property nginx.service MemoryMax=512M --runtime=false

Viewing Effective Limits

systemctl show nginx.service -p MemoryMax -p CPUQuota -p IOWeight

systemctl Commands Reference

Command	Description
systemctl start <unit>	Start a unit
systemctl stop <unit>	Stop a unit
systemctl restart <unit>	Restart a unit
systemctl reload <unit>	Reload unit configuration
systemctl enable <unit>	Enable unit at boot
systemctl disable <unit>	Disable unit at boot
systemctl enable --now <unit>	Enable and start immediately
systemctl status <unit>	Show unit status
systemctl is-active <unit>	Check if unit is running
systemctl is-enabled <unit>	Check if unit is enabled
systemctl is-failed <unit>	Check if unit has failed
systemctl list-units	List all loaded units
systemctl list-unit-files	List all unit files
systemctl list-timers	List active timers
systemctl daemon-reload	Reload unit file changes
systemctl cat <unit>	Show unit file contents
systemctl show <unit>	Show all unit properties
systemctl edit <unit>	Create drop-in override
systemctl mask <unit>	Prevent unit from starting
systemctl unmask <unit>	Remove mask from unit
systemctl list-dependencies <unit>	Show unit dependencies
systemctl set-property <unit> <key>=<val>	Set runtime property
journalctl -u <unit>	View unit logs
journalctl -u <unit> -f	Follow unit logs
systemd-analyze blame	Show boot time per service
systemd-analyze critical-chain	Show critical boot path

Troubleshooting Failed Services

When a service fails, follow this systematic approach to diagnose and resolve the issue.

Step 1: Check the Status

sudo systemctl status myapp.service

Look for the Active line. Common states:

• active (running) — service is running normally
• inactive (dead) — service is stopped
• failed — service crashed or failed to start
• activating (auto-restart) — service is restarting

Step 2: Read the Logs

# Full logs for the service
journalctl -u myapp.service --no-pager

# Last 50 lines
journalctl -u myapp.service -n 50

# Errors only
journalctl -u myapp.service -p err

Step 3: Validate the Unit File

# Check for syntax errors
systemd-analyze verify /etc/systemd/system/myapp.service

Step 4: Test the Command Manually

Run the ExecStart command manually as the service user to see if the application itself works:

sudo -u myapp /opt/myapp/bin/myapp --config /etc/myapp/config.yaml

Step 5: Check Common Issues

Permission errors:

# Verify the binary is executable
ls -la /opt/myapp/bin/myapp

# Verify the user exists
id myapp

# Check directory permissions
ls -la /opt/myapp/

Missing dependencies:

# Check for missing shared libraries
ldd /opt/myapp/bin/myapp

Port conflicts:

# Check if the port is already in use
ss -tlnp | grep :8080

Step 6: Reset a Failed Service

# Reset the failed state
sudo systemctl reset-failed myapp.service

# Try starting again
sudo systemctl start myapp.service

Common Error Patterns

Symptom	Likely Cause	Solution
code=exited, status=203/EXEC	Binary not found or not executable	Verify ExecStart path and permissions
code=exited, status=217/USER	Service user does not exist	Create the user with useradd
code=exited, status=200/CHDIR	WorkingDirectory does not exist	Create the directory
code=exited, status=1/FAILURE	Application error	Check journalctl logs
Service keeps restarting	Crash loop	Check RestartSec, review application logs
code=killed, signal=KILL	OOM killer	Increase MemoryMax or optimize the application

Summary

systemd is the backbone of service management on modern Linux systems. It provides a unified, declarative approach to managing services, scheduling tasks, controlling resources, and analyzing system behavior. Mastering systemctl, unit files, timers, and journalctl gives you complete control over how your Linux servers operate.

Key takeaways:

• Use systemctl for all service management tasks: start, stop, enable, disable, and status
• Write custom unit files in /etc/systemd/system/ to manage your own applications as proper system services
• Replace cron jobs with systemd timers for persistent scheduling, per-task logging, and resource control
• Use journalctl for powerful log filtering by service, time range, and priority level
• Profile boot performance with systemd-analyze and disable unnecessary services to speed up startup
• Apply cgroups resource limits (MemoryMax, CPUQuota) to prevent services from starving the system
• Use drop-in overrides (systemctl edit) instead of modifying vendor unit files

For a broader approach to securing your Linux servers, see our Linux Server Security Checklist and UFW Firewall Configuration Guide.