What is a systemd unit file?

A systemd unit file is a plain-text configuration file that describes a service, timer, socket, or other resource managed by systemd. It has three main sections: [Unit] for metadata, [Service] for execution details, and [Install] for boot targets.

How do I make a systemd service start automatically on boot?

Run sudo systemctl enable your-service.service after creating the unit file. This creates a symlink in the appropriate target directory so systemd starts the service when the system reaches the configured WantedBy target, usually multi-user.target.

How do I view logs for a systemd service?

Use journalctl -u your-service.service to view all logs for the service. Add -f to follow logs in real time, -n 50 to show the last 50 lines, or --since today to filter by date.

What is the difference between Restart=always and Restart=on-failure?

Restart=always restarts the service regardless of exit code, including clean exits. Restart=on-failure restarts only when the process exits with a non-zero code, is killed by a signal, or hits a timeout — not on a clean exit or explicit systemctl stop.

Create Custom systemd Services on Linux: Complete Guide

TL;DR — Quick Summary

Master custom systemd services on Linux: unit file anatomy, Node.js and Python examples, restart policies, resource limits, security hardening, and timer units.

Keeping long-running processes alive on a Linux server used to mean wrestling with SysVinit scripts, supervisor configs, or fragile shell hacks. systemd changed all of that. As the default init system on every major Linux distribution — Ubuntu, Debian, RHEL, Fedora, Arch — systemd gives you a unified, declarative way to define exactly how any process should start, stop, restart, log, and interact with the rest of the system.

This guide walks through everything you need to create production-ready custom systemd services: the anatomy of a unit file, concrete examples for Node.js and Python applications, service types, restart policies, resource limits, security hardening directives, socket activation, timer units as a cron replacement, and how to debug failures with journalctl. A comparison table at the end shows how systemd stacks up against init.d, supervisord, and PM2.

Prerequisites

Before you begin, make sure you have:

A Linux system running systemd (Ubuntu 16.04+, Debian 8+, CentOS 7+, RHEL 7+, Fedora 15+, or any modern distribution)
A terminal with sudo or root access
Basic familiarity with the command line and a text editor (nano or vim)
An application you want to run as a service (Node.js, Python, Go binary, etc.)

Verify systemd is running:

systemctl --version
# Should output: systemd 249 (or similar version number)

Unit File Anatomy

Every service systemd manages is described by a unit file — a plain-text INI-style configuration file. Custom service unit files live in /etc/systemd/system/. The filename ends in .service.

A complete unit file has three sections:

[Unit]
Description=My Application Service
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The [Unit] Section

Directive	Purpose
`Description=`	Human-readable name shown in `systemctl status` output
`Documentation=`	URL or man page reference
`After=`	Start this unit after the listed units are active
`Before=`	Start this unit before the listed units
`Wants=`	Soft dependency — listed units are started but failure is tolerated
`Requires=`	Hard dependency — if listed unit fails, this unit fails too
`BindsTo=`	Like Requires= but also stops this unit when the dependency stops

The [Service] Section

This is where most of the configuration lives. Key directives:

Directive	Purpose
`Type=`	Process lifecycle model (simple, forking, oneshot, notify, idle)
`User=` / `Group=`	Run the process as this user/group
`WorkingDirectory=`	Set the working directory before executing
`ExecStart=`	The command to run (must be an absolute path)
`ExecStartPre=`	Commands to run before ExecStart
`ExecStartPost=`	Commands to run after ExecStart succeeds
`ExecStop=`	Custom stop command (defaults to SIGTERM)
`ExecReload=`	Command to reload without restarting
`Restart=`	When to automatically restart
`RestartSec=`	Delay between restart attempts
`Environment=`	Set environment variables inline
`EnvironmentFile=`	Load environment variables from a file
`StandardOutput=`	Where to send stdout (journal, file:, append:, null)
`StandardError=`	Where to send stderr

The [Install] Section

Directive	Purpose
`WantedBy=`	Which target enables this service (usually `multi-user.target`)
`RequiredBy=`	Hard version of WantedBy
`Alias=`	Additional names for this unit

The [Install] section only matters when you run systemctl enable. It determines which systemd target pulls in your service at boot.

Creating a Node.js Service

Let us say you have a Node.js API running at /opt/myapi/server.js that listens on port 3000. Here is a complete, production-ready unit file.

Step 1: Create a dedicated user

sudo useradd --system --no-create-home --shell /usr/sbin/nologin nodeapi
sudo chown -R nodeapi:nodeapi /opt/myapi

Step 2: Create the unit file

sudo nano /etc/systemd/system/myapi.service

[Unit]
Description=My Node.js API Service
Documentation=https://github.com/example/myapi
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=nodeapi
Group=nodeapi
WorkingDirectory=/opt/myapi

# Load secrets from a protected env file
EnvironmentFile=/etc/myapi/environment

ExecStart=/usr/bin/node /opt/myapi/server.js
ExecStartPre=/usr/bin/node --check /opt/myapi/server.js

Restart=on-failure
RestartSec=5s
TimeoutStopSec=10s

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapi

# Resource limits
MemoryMax=512M
CPUQuota=50%
LimitNOFILE=65536

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
PrivateTmp=true
ReadWritePaths=/opt/myapi/logs /var/lib/myapi

[Install]
WantedBy=multi-user.target

Step 3: Create the environment file

sudo mkdir -p /etc/myapi
sudo nano /etc/myapi/environment

NODE_ENV=production
PORT=3000
DATABASE_URL=postgresql://user:pass@localhost/mydb
JWT_SECRET=your-secret-here

# Secure the file — only root can read it
sudo chmod 600 /etc/myapi/environment
sudo chown root:root /etc/myapi/environment

Step 4: Enable and start

sudo systemctl daemon-reload
sudo systemctl enable myapi.service
sudo systemctl start myapi.service
sudo systemctl status myapi.service

Creating a Python Service

For a Python background worker at /opt/worker/worker.py:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin pyworker
sudo chown -R pyworker:pyworker /opt/worker

sudo nano /etc/systemd/system/pyworker.service

[Unit]
Description=Python Background Worker
After=network.target redis.service
Requires=redis.service

[Service]
Type=simple
User=pyworker
Group=pyworker
WorkingDirectory=/opt/worker

# Use the virtualenv Python binary directly
ExecStart=/opt/worker/venv/bin/python -u worker.py
ExecStartPre=/opt/worker/venv/bin/python -c "import redis; redis.Redis().ping()"

EnvironmentFile=/etc/worker/environment

Restart=on-failure
RestartSec=10s
StartLimitIntervalSec=60s
StartLimitBurst=3

StandardOutput=journal
StandardError=journal
SyslogIdentifier=pyworker

MemoryMax=256M
CPUQuota=25%
LimitNOFILE=4096

NoNewPrivileges=true
ProtectSystem=strict
PrivateTmp=true
ReadWritePaths=/opt/worker/data /tmp/worker

[Install]
WantedBy=multi-user.target

Note the -u flag on the Python command — this disables output buffering so logs appear in journald immediately rather than in chunks.

Service Types

The Type= directive tells systemd how to track whether your service has finished starting. Choosing the wrong type leads to dependency ordering failures.

Type	When to Use	How systemd Tracks Readiness
`simple`	Most modern apps. ExecStart is the main process.	Considers the unit started as soon as ExecStart forks
`exec`	Like simple, but waits for exec to succeed	Waits until the binary executes successfully
`forking`	Traditional daemons that fork and exit the parent	Waits for the parent process to exit
`oneshot`	Scripts that run once and exit (like initialization tasks)	Waits for ExecStart to exit before marking active
`notify`	Apps that send a readiness notification via `sd_notify()`	Waits for the notification before marking active
`idle`	Delay execution until all active jobs finish	Like simple, but starts after the boot job queue drains

For most Node.js, Python, Go, and Java applications use Type=simple. Use Type=notify if your app integrates with the systemd notification protocol (recommended for services that need time to initialize before accepting connections).

Restart Policies and Resource Limits

Restart Policies

# Restart whenever the service exits (any reason including clean exit)
Restart=always

# Restart only on failure (non-zero exit, signal, timeout) — NOT on clean exit
Restart=on-failure

# Restart on failure + watchdog timeout + abnormal signals
Restart=on-abnormal

# Never restart
Restart=no

Combined with rate limiting to prevent a crashing service from hammering the system:

Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=5

This configuration allows up to 5 restarts in 60 seconds before systemd gives up and marks the service as failed.

Resource Limits

Control how much of the system a service can consume:

[Service]
# Memory
MemoryMax=1G          # Hard cap — process is killed if exceeded
MemoryHigh=800M       # Soft limit — kernel throttles memory reclaim
MemorySwapMax=0       # Disable swap for this service

# CPU
CPUQuota=200%         # Limit to 2 full CPU cores (200% = 2 cores)
CPUWeight=100         # Relative scheduling weight (default 100)

# File descriptors
LimitNOFILE=65536     # Maximum open file descriptors

# Processes
LimitNPROC=512        # Maximum number of processes/threads

# Disk I/O (cgroup v2)
IOWeight=50           # Relative I/O weight (default 100)

Security Hardening

systemd provides powerful security directives that sandbox your service at the kernel level — no container runtime required.

[Service]
# Prevent the process from gaining new privileges via setuid/setgid
NoNewPrivileges=true

# Mount /usr, /boot, /etc as read-only for this service
ProtectSystem=strict

# Give the service its own private /tmp (not shared with other processes)
PrivateTmp=true

# Create an unprivileged dynamic user (no need to useradd manually)
DynamicUser=true

# Prevent access to /home directories
ProtectHome=true

# Make /proc read-only and hide processes of other users
ProtectProc=invisible
ProcSubset=pid

# Restrict which kernel syscalls the service can make (minimizes attack surface)
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Prevent writing to kernel variables
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true

# Restrict network access (uncomment to allow only specific address families)
# RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Restrict available capabilities
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

Note on DynamicUser=true: When you use DynamicUser=true, systemd creates a temporary unprivileged user at service start and removes it at service stop. This is ideal for stateless services. For stateful services that need persistent file ownership, create a dedicated system user with useradd --system instead.

You can analyze the security score of any running service:

systemd-analyze security myapi.service

This outputs a table of directives with their security impact and a composite score from 0 (fully sandboxed) to 10 (completely exposed).

Socket Activation

Socket activation lets systemd create and hold the listening socket. The service only starts when a connection arrives — great for reducing boot-time resource consumption and enabling zero-downtime restarts.

Create two files:

sudo nano /etc/systemd/system/myapi.socket

[Unit]
Description=My API Socket

[Socket]
ListenStream=3000
Accept=false
SocketUser=nodeapi

[Install]
WantedBy=sockets.target

sudo nano /etc/systemd/system/myapi.service

[Unit]
Description=My API Service (socket-activated)
Requires=myapi.socket
After=myapi.socket

[Service]
Type=notify
User=nodeapi
ExecStart=/usr/bin/node /opt/myapi/server.js
StandardOutput=journal

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now myapi.socket

The service starts on first connection, and during a restart the socket remains open so no incoming connections are dropped.

Timer Units: systemd as a Cron Alternative

systemd timers replace cron jobs with better logging, dependency management, and catch-up behavior.

A timer unit pairs with a service unit of the same base name. To run a backup job every night at 2 AM:

sudo nano /etc/systemd/system/nightly-backup.service

[Unit]
Description=Nightly Database Backup

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup-db.sh
StandardOutput=journal
StandardError=journal

sudo nano /etc/systemd/system/nightly-backup.timer

[Unit]
Description=Run Nightly Database Backup at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now nightly-backup.timer

# Verify the timer is scheduled
systemctl list-timers --all

Key timer directives:

Directive	Example	Meaning
`OnCalendar=`	`Mon --* 09:00`	Run every Monday at 9 AM
`OnBootSec=`	`10min`	Run 10 minutes after boot
`OnUnitActiveSec=`	`1h`	Run 1 hour after last activation
`Persistent=true`	—	Run missed jobs after system wake/boot
`RandomizedDelaySec=`	`5min`	Add random delay to avoid thundering herd

systemd vs Other Process Managers

Feature	systemd	SysVinit / init.d	supervisord	PM2
Ships with OS	Yes (all major distros)	Yes (legacy)	No (pip install)	No (npm install)
Dependency ordering	Full graph	Manual ordering	Limited	No
Socket activation	Yes	No	No	No
Cgroup resource limits	Yes (native)	No	No	No
Kernel-level sandboxing	Yes (extensive)	No	No	No
Centralized logging	journald (structured)	syslog / files	Files only	PM2 log files
Timer / scheduled jobs	Yes (timer units)	No	No	Yes (cron-like)
Cluster mode	No	No	Yes	Yes
Zero-downtime reload	With socket activation	No	Yes	Yes
Node.js ecosystem integration	Good	Poor	Good	Excellent
Python support	Excellent	Manual	Excellent	No
GUI / web dashboard	No	No	Yes (web UI)	Yes (pm2-web)
Learning curve	Moderate	Low (simple shell)	Low	Very low

When to choose systemd: System-level services, services that must start at boot before login, anything requiring resource isolation, security-sensitive processes.

When to choose PM2: Node.js-only shops that want cluster mode and a web dashboard without learning systemd.

When to choose supervisord: Heterogeneous environments where systemd is unavailable (containers, legacy systems), or when you need a simple Python-configurable process manager.

Real-World Scenario

You have a production server running a Python FastAPI application. It processes jobs from a Redis queue and must restart automatically if it crashes, never consume more than 512 MB of RAM, and write logs to journald. The Redis service must be available before it starts.

Create /etc/systemd/system/fastapi-worker.service:

[Unit]
Description=FastAPI Job Worker
After=network.target redis.service
Requires=redis.service
StartLimitIntervalSec=120s
StartLimitBurst=4

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/fastapi-worker

ExecStart=/opt/fastapi-worker/venv/bin/uvicorn main:app --host 0.0.0.0 --port 8000 --workers 2
ExecStartPre=/opt/fastapi-worker/venv/bin/python -c "import redis; redis.Redis(host='localhost').ping()"
ExecReload=/bin/kill -HUP $MAINPID

EnvironmentFile=/etc/fastapi-worker/environment

Restart=on-failure
RestartSec=8s
TimeoutStartSec=30s
TimeoutStopSec=15s

StandardOutput=journal
StandardError=journal
SyslogIdentifier=fastapi-worker

MemoryMax=512M
CPUQuota=100%
LimitNOFILE=32768

NoNewPrivileges=true
ProtectSystem=strict
PrivateTmp=true
ProtectHome=true
ReadWritePaths=/opt/fastapi-worker/data

[Install]
WantedBy=multi-user.target

Deploy with:

sudo systemctl daemon-reload
sudo systemctl enable --now fastapi-worker.service
journalctl -u fastapi-worker.service -f

Gotchas and Edge Cases

ExecStart must be an absolute path. You cannot use shell built-ins, pipes, or redirects directly in ExecStart. Wrap complex commands in a script.

Environment variable expansion in ExecStart. systemd does expand $VAR in ExecStart when the variable is set via Environment= or EnvironmentFile=, but does NOT source shell initialization files. Your ~/.bashrc is never loaded.

Percent signs in unit files. In unit files, % is a specifier character. To use a literal percent sign, escape it as %%.

DynamicUser and file permissions. When using DynamicUser=true, the allocated UID changes between restarts. Any files written by the service will be owned by that UID and may become inaccessible. Use StateDirectory=, CacheDirectory=, or LogsDirectory= directives to let systemd manage persistent directories properly.

After= vs Requires=. After= only controls ordering — it does not create a dependency. Requires= creates the dependency but does not control ordering. You almost always want both together.

Type=forking and PIDFile. If your daemon forks, you must also set PIDFile= so systemd can track the correct main process. Without it, systemd loses track of the daemon and will be unable to correctly report status or send signals.

StartLimitBurst without StartLimitIntervalSec. The StartLimitBurst and StartLimitIntervalSec directives moved from [Service] to [Unit] in systemd 229. On older systems they must be in [Service]. On modern systems, put them in [Unit] to avoid deprecation warnings.

Troubleshooting

Service fails to start

# Show the last status and recent log lines
sudo systemctl status myapi.service

# Show full logs since last boot
journalctl -u myapi.service -b

# Show logs from the last 10 minutes
journalctl -u myapi.service --since "10 minutes ago"

# Watch logs in real time
journalctl -u myapi.service -f

Service starts but crashes immediately

# Show all log entries including the crash
journalctl -u myapi.service -n 100 --no-pager

# Check if the binary exists and is executable
ls -la /usr/bin/node
ls -la /opt/myapi/server.js

# Test the ExecStart command manually as the service user
sudo -u nodeapi /usr/bin/node /opt/myapi/server.js

Service is stuck in activating state

# Check if ExecStartPre is failing
journalctl -u myapi.service --since "5 minutes ago"

# Reset failed state and try again
sudo systemctl reset-failed myapi.service
sudo systemctl start myapi.service

Permission denied errors

# Verify the user exists
id nodeapi

# Check file ownership
ls -la /opt/myapi/

# Check if ProtectSystem=strict is blocking writes
# Add the path to ReadWritePaths= in the unit file

Check security exposure of a running service

systemd-analyze security myapi.service
systemd-analyze verify /etc/systemd/system/myapi.service

Summary

systemd is the right tool for managing any long-running process on a modern Linux server. Key takeaways:

Unit files live in /etc/systemd/system/ and have three sections: [Unit], [Service], and [Install]
Always run systemctl daemon-reload after editing a unit file
Use Type=simple for most modern apps; use Type=notify for apps that signal readiness
Store secrets in a separate EnvironmentFile= with mode 600, not inline in the unit file
Set restart policies with Restart=on-failure and StartLimitBurst to survive transient crashes without hammering the system
Apply security directives — NoNewPrivileges, ProtectSystem, PrivateTmp — to every production service
Use DynamicUser=true for stateless services to avoid manual user management
Replace cron jobs with timer units for better logging and catch-up behavior
Debug with journalctl — journalctl -u service-name -f is your first stop for any failure