What is a systemd unit file?

A systemd unit file is a declarative configuration file that defines how a service, mount, device, socket, or timer should behave. Unit files are stored in /etc/systemd/system/ for custom units and /lib/systemd/system/ for distribution defaults, using INI-style sections like [Unit], [Service], and [Install].

How do I check why a systemd service failed?

Run 'systemctl status servicename' to see the current state and recent log lines. For detailed logs use 'journalctl -u servicename -e' to jump to the end, or add '-p err' to filter errors only. The 'systemctl list-units --failed' command shows all failed units at once.

Can systemd timers replace cron jobs?

Yes, systemd timers can fully replace cron. They offer persistent scheduling that catches missed runs after downtime, calendar and monotonic expressions, per-timer logging via journalctl, randomized delays to avoid thundering herd, and dependency-aware execution. Each timer requires a paired .timer and .service unit file.

How do I create a service that restarts automatically on failure?

Add 'Restart=on-failure' and 'RestartSec=5' in the [Service] section of your unit file. Use 'StartLimitIntervalSec=300' and 'StartLimitBurst=5' in the [Unit] section to prevent infinite restart loops. The service will restart up to 5 times within 300 seconds before systemd stops trying.

Systemd Services Management on Linux

Managing services on modern Linux distributions means working with systemd, the init system and service manager that has become the standard across most distributions. This guide covers everything you need to know about creating custom service units, managing dependencies between services, analyzing logs with journalctl, replacing cron with systemd timers, and diagnosing problems when services fail to start or crash unexpectedly.

Prerequisites

A Linux distribution running systemd (Ubuntu 16.04+, CentOS 7+, Debian 8+, Fedora 15+)
Terminal access with sudo privileges
Basic understanding of Linux file permissions and process management
A text editor installed (vim, nano, or similar)

Understanding Systemd Units

Systemd organizes everything into units — resources that the system knows how to manage. The most common unit types are:

Unit Type	Extension	Purpose
Service	`.service`	Daemon processes and one-shot tasks
Timer	`.timer`	Scheduled execution (replaces cron)
Socket	`.socket`	IPC and network socket activation
Mount	`.mount`	Filesystem mount points
Target	`.target`	Groups of units (like runlevels)

Unit files live in three locations, with descending priority:

/etc/systemd/system/      # Admin custom units (highest priority)
/run/systemd/system/       # Runtime units
/lib/systemd/system/       # Distribution default units

To list all loaded units and their states:

systemctl list-units --type=service
systemctl list-units --type=service --state=failed

Creating Custom Service Units

A well-structured service unit file has three sections. Here is a complete example for a Node.js application:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js Application
Documentation=https://example.com/docs
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
Environment=PORT=3000
ExecStartPre=/usr/bin/node --check /opt/myapp/server.js
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp/data
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Key directives in the [Service] section:

Type: simple (default, process stays in foreground), forking (for daemons that fork), oneshot (runs once and exits), notify (signals readiness via sd_notify)
ExecStartPre: Command to run validation before starting the main process
Restart: on-failure, always, on-abnormal, or no
RestartSec: Delay between restart attempts in seconds

After creating the file, load and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service

Managing Dependencies

Systemd uses several directives to express relationships between units:

[Unit]
# Ordering (when to start, not whether)
After=network-online.target    # Start after network is ready
Before=nginx.service           # Start before nginx

# Dependency strength
Requires=postgresql.service    # Hard dependency — fail if postgres fails
Wants=redis.service            # Soft dependency — continue if redis fails
BindsTo=docker.service         # Tied lifecycle — stop when docker stops

# Conflict resolution
Conflicts=iptables.service     # Cannot run alongside iptables

The difference between After and Requires is critical. After controls ordering (sequencing) while Requires controls activation (whether to pull in a dependency). You usually need both:

Requires=postgresql.service
After=postgresql.service

Using Requires alone does not guarantee ordering — both services may start simultaneously. Using After alone does not pull in the dependency — it only orders them if both happen to be starting.

To visualize the dependency tree for a service:

systemctl list-dependencies myapp.service
systemctl list-dependencies myapp.service --reverse

Journalctl Log Analysis

Journalctl is the tool for querying the systemd journal. Here are the most useful patterns:

# Follow logs for a specific service in real time
journalctl -u myapp.service -f

# Show logs since last boot
journalctl -u myapp.service -b

# Filter by time range
journalctl -u myapp.service --since "2026-02-20 08:00" --until "2026-02-20 18:00"

# Show only errors and above
journalctl -u myapp.service -p err

# Output in JSON format for parsing
journalctl -u myapp.service -o json-pretty --no-pager

# Show kernel messages related to OOM kills
journalctl -k --grep="Out of memory"

# Check disk usage of journal
journalctl --disk-usage

# Vacuum old logs to free space
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=500M

Priority levels follow syslog convention: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). Using -p err shows err and everything more severe.

To configure persistent journal storage (survives reboots):

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Systemd Timers

Systemd timers replace cron with better logging, dependency management, and reliability. A timer consists of two files — the .timer and its paired .service:

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily database backup timer

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=900
Persistent=true
Unit=backup.service

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Database backup job

[Service]
Type=oneshot
User=backup
ExecStart=/opt/scripts/backup-db.sh
StandardOutput=journal

Common OnCalendar expressions:

Expression	Meaning
`--* 02:00:00`	Daily at 2 AM
`Mon --* 09:00:00`	Every Monday at 9 AM
`--01 00:00:00`	First day of every month
`--* *:00/15:00`	Every 15 minutes
`hourly`	Every hour (shorthand)

Enable and manage timers:

sudo systemctl enable --now backup.timer
systemctl list-timers --all
systemctl status backup.timer

The Persistent=true directive ensures that if the system was off when the timer should have fired, it runs the job immediately on next boot.

Comparison: Systemd vs SysVinit vs Upstart

Feature	systemd	SysVinit	Upstart
Parallel startup	Yes	No	Partial
Dependency management	Declarative directives	Manual ordering with numbers	Event-based
Service supervision	Built-in restart policies	None (requires external tools)	Respawn directive
Logging	journald (structured, indexed)	syslog (plain text files)	syslog
Socket activation	Yes	No	No
Resource control	cgroups integration	None	None
Timer/scheduling	Built-in timers	Requires cron	Requires cron
Configuration format	INI-style unit files	Shell scripts	Stanza-based conf files
Boot analysis	systemd-analyze	None	None
Adoption status	Default on most distros	Legacy systems	Deprecated

Real-World Scenario

You have a production server running a Python API that depends on PostgreSQL and Redis. The API must start after both databases are ready, restart on crashes with a back-off strategy, and run a cleanup job every 6 hours.

Service unit (/etc/systemd/system/api.service):

[Unit]
Description=Production Python API
After=network-online.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service

[Service]
Type=simple
User=apiuser
WorkingDirectory=/opt/api
Environment=PYTHONUNBUFFERED=1
ExecStart=/opt/api/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:create_app()
Restart=on-failure
RestartSec=10
StartLimitIntervalSec=300
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

Cleanup timer (/etc/systemd/system/api-cleanup.timer):

[Unit]
Description=API cleanup every 6 hours

[Timer]
OnBootSec=15min
OnUnitActiveSec=6h
Persistent=true

[Install]
WantedBy=timers.target

This setup ensures the API only starts when PostgreSQL is running (hard dependency) and optionally uses Redis (soft dependency). The restart policy allows 5 restart attempts within 5 minutes before giving up.

Gotchas and Edge Cases

ExecStart must use absolute paths: Relative paths cause silent failures. Always use /usr/bin/node, not node
Type=forking needs PIDFile: If your daemon forks, systemd must track the child PID. Set PIDFile=/run/myapp.pid and ensure your app writes it
Environment files vs inline: For many variables, use EnvironmentFile=/etc/myapp/env instead of multiple Environment= lines. The file format is KEY=value (no export)
User services vs system services: User units in ~/.config/systemd/user/ run without sudo but only while the user is logged in (unless loginctl enable-linger is set)
Reload vs restart daemon-reload: systemctl restart myapp restarts the service process. systemctl daemon-reload reloads unit file definitions. After editing a unit file you need daemon-reload first
WantedBy target matters: Use multi-user.target for headless servers, graphical.target for desktop services. Using the wrong target means the service will not start at boot

Troubleshooting

When a service fails to start, follow this diagnostic workflow:

# 1. Check current status and recent logs
systemctl status myapp.service

# 2. Get detailed failure information
journalctl -u myapp.service -e --no-pager -n 50

# 3. Validate unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service

# 4. Check for ordering cycles
systemd-analyze critical-chain myapp.service

# 5. Test the ExecStart command manually as the service user
sudo -u appuser /usr/bin/node /opt/myapp/server.js

# 6. Check SELinux or AppArmor denials
sudo ausearch -m avc -ts recent 2>/dev/null || sudo journalctl -k --grep="apparmor"

# 7. Verify file permissions
ls -la /opt/myapp/
namei -l /opt/myapp/server.js

Common failure reasons and fixes:

Symptom	Cause	Fix
`code=exited, status=203/EXEC`	Binary not found or not executable	Check path with `which`, set `chmod +x`
`code=exited, status=217/USER`	User specified in `User=` does not exist	Create user with `useradd -r -s /sbin/nologin`
`Start request repeated too quickly`	Restart loop hit StartLimitBurst	Increase `RestartSec`, fix the underlying crash, then `systemctl reset-failed`
`code=exited, status=200/CHDIR`	WorkingDirectory does not exist	Create the directory and fix ownership
Service starts but port not accessible	Firewall or wrong bind address	Check `ss -tlnp`, verify `firewall-cmd` or `ufw` rules

Summary

Systemd unit files use declarative INI-style sections ([Unit], [Service], [Install]) to define service behavior
Custom units go in /etc/systemd/system/ — always run daemon-reload after changes
Use Requires= plus After= together for hard dependencies with correct ordering
Journalctl provides powerful filtering by unit, priority, time range, and output format
Systemd timers offer better reliability than cron with persistent scheduling and per-job logging
Security hardening directives like ProtectSystem=strict and PrivateTmp=true should be standard in production units
Troubleshoot with systemctl status, journalctl -u, and systemd-analyze verify