Systemd Unit Files Explained: A Practical Guide

If you administer Linux servers, you interact with systemd every day whether you realize it or not. Every time you run systemctl restart nginx or check journalctl -f, that's systemd. It's the init system and service manager on virtually every mainstream Linux distribution — Ubuntu, Debian, Fedora, CentOS Stream, RHEL, Arch, and more.

Most sysadmins learn just enough systemd to start and stop services. This guide goes further: you'll understand what's actually happening when you run those commands, how to write your own unit files from scratch, how to read logs intelligently, and how to debug services when they fail to start.

What Is systemd?

systemd is the first process that runs when Linux boots. It gets assigned PID 1 — Process ID 1, the root of every other process on the system. From there, it's responsible for starting every service, mounting every filesystem, and managing the system for its entire uptime.

Before systemd, most Linux distributions used SysVinit — a collection of shell scripts in /etc/init.d/. Those scripts had to handle start, stop, restart, and status logic manually. Each one was different. Dependency declaration was clunky. Services had to start one at a time in sequence. If you've ever seen the old red/green boot messages scrolling by on a Linux system, that was SysVinit.

systemd replaced that model with something declarative. Instead of writing a shell script that describes how to start a service, you write a unit file that declares what the service is — and systemd figures out the rest. Dependencies are resolved automatically, independent services start in parallel, and failures are handled consistently.

The practical results: faster boot times, standardized logging, consistent service management commands across every distribution, and automatic service restart on failure.

Unit Types

In systemd, everything it manages is called a "unit." Units are defined in configuration files called unit files, and they come in several types identified by their file extension:

Extension	What It Manages	Example
.service	A process or daemon	nginx.service, sshd.service
.timer	A scheduled task (like cron)	backup.timer
.socket	A network or IPC socket	ssh.socket
.mount	A filesystem mount point	mnt-data.mount
.target	A group of units (like a runlevel)	multi-user.target
.path	File/directory monitoring	config-watch.path
.slice	Resource limit groups	user.slice

For most sysadmin work, you'll deal primarily with .service and .timer units. The others exist and are useful in specific situations, but you won't need them on day one.

Where Unit Files Live

systemd looks for unit files in several locations, with a priority order:

/etc/systemd/system/      # Your custom files and overrides (highest priority)
/run/systemd/system/      # Runtime units (generated, temporary)
/lib/systemd/system/      # Package-installed units (lowest priority)
/usr/lib/systemd/system/  # Same as above on some distributions

The most important thing to understand here: put your custom files in /etc/systemd/system/. Files there take precedence over anything in /lib/systemd/system/ and — critically — they won't be overwritten when you update packages.

When you install a package like nginx, its unit file goes to /lib/systemd/system/nginx.service. If you want to customize how nginx starts, you don't edit that file directly — you create an override in /etc/systemd/system/ instead. This is the right way to do it and we'll cover the technique below.

Anatomy of a Service Unit File

Unit files use a simple INI-style format — sections marked with square brackets, directives as key=value pairs. A typical service unit has three sections:

[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network.target

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Let's go through each section:

[Unit] — What this unit is

Description is a human-readable name that shows up in systemctl status output. Make it descriptive.

After tells systemd to start this unit only after the specified units have started. network.target is the most common — it means "wait until networking is up." This is an ordering directive, not a dependency — if network.target fails, this service still tries to start.

Documentation is optional but good practice. It accepts URLs and man page references.

[Service] — How to run it

Type tells systemd how the process behaves when it starts. The most important options:

• simple — The process started by ExecStart is the main service process. systemd considers it started immediately. This is the default and right for most modern applications.
• forking — The process forks a child and exits. systemd waits for the fork. This is legacy behavior from old-style daemons. Use it when you have to, not by default.
• oneshot — The process runs to completion and exits. systemd waits for it to finish before considering the unit active. Good for scripts and initialization tasks.
• notify — Like simple, but the service signals systemd when it's actually ready using the sd_notify() API. Gives more accurate status tracking. Some packages like nginx and PostgreSQL support this.

When in doubt, use simple.

User and Group control which account the service runs as. Always run services as a dedicated non-root user when possible. This limits the damage if the service is compromised.

WorkingDirectory sets the current directory for the process. Use this instead of relying on relative paths in your application.

ExecStart is the command that starts your service. Always use absolute paths — systemd doesn't use your shell's PATH. /opt/myapp/bin/myapp, not myapp.

Restart controls when systemd restarts the service automatically:

• no — never restart (the default if omitted)
• on-failure — restart if the process exits with a non-zero code or is killed by a signal
• always — always restart, even on clean exit
• on-abnormal — restart on signals, watchdog timeout, or non-zero exit (but not clean exit)

on-failure is usually what you want for a service you want to keep running. RestartSec=5 adds a 5-second delay before restarting, which prevents rapid restart loops from hammering the system.

[Install] — How to enable it

WantedBy controls which target this service gets linked to when you run systemctl enable. multi-user.target is the right choice for almost every server service — it's the equivalent of the old runlevel 3 (multi-user, non-graphical).

The [Install] section only matters when you run systemctl enable. It doesn't affect whether the service starts now — it only controls whether it starts at boot. A service with no [Install] section can still be started manually; it just won't start automatically.

Writing Your First Unit File

Let's walk through a real example: turning a Python web application into a proper systemd service.

Assume you have a Flask application at /opt/myapp/app.py that you normally run with Gunicorn. Here's the process:

Step 1: Create a dedicated user for the service

sudo useradd --system --shell /sbin/nologin --home-dir /opt/myapp myapp

The --system flag creates a system account (lower UID range, no password). --shell /sbin/nologin prevents anyone from logging in as this user. This is standard practice for service accounts.

Step 2: Write the unit file

sudo nano /etc/systemd/system/myapp.service

[Unit]
Description=My Flask Application
Documentation=https://github.com/me/myapp
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/environment
ExecStart=/opt/myapp/venv/bin/gunicorn -w 4 -b 127.0.0.1:8000 app:app
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

A few things worth noting here:

EnvironmentFile points to a file containing environment variables (like DATABASE_URL=... or SECRET_KEY=...). This keeps sensitive config out of the unit file itself. The file should be owned by root, readable only by root and the service user.

ExecReload lets you send a HUP signal to reload config without restarting the process. Not all applications support this, but it's worth including when they do.

StandardOutput=journal and StandardError=journal send both stdout and stderr to the systemd journal. This means everything the application prints goes to journalctl. This is actually the default, but being explicit is good documentation.

Step 3: Tell systemd the file exists

sudo systemctl daemon-reload

This step is critical and frequently forgotten. Any time you create or modify a unit file, you must run daemon-reload before trying to start or enable the service. Without it, systemd is still using its cached version of your unit files.

Step 4: Start and enable it

sudo systemctl start myapp
sudo systemctl enable myapp

Or combine both in one command:

sudo systemctl enable --now myapp

Step 5: Verify it's running

sudo systemctl status myapp

Managing Services with systemctl

Here are the systemctl commands you'll use most often:

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Reload config without full restart (if supported)
systemctl reload nginx

# Enable at boot / disable
systemctl enable nginx
systemctl disable nginx

# Enable AND start in one command
systemctl enable --now nginx

# Check status
systemctl status nginx

# List all running services
systemctl list-units --type=service

# List all services including stopped ones
systemctl list-units --type=service --all

# Show only failed services
systemctl list-units --type=service --state=failed

# Check if a service is running
systemctl is-active nginx

# Check if enabled at boot
systemctl is-enabled nginx

An important distinction: enable and start are separate operations. enable creates symlinks so the service starts at boot. start starts it right now. Running enable without start means the service will start next boot but not immediately. Running start without enable starts it now but it won't survive a reboot. Usually you want both.

Reading Logs with journalctl

journald is systemd's logging system. It captures everything services write to stdout and stderr, plus kernel messages, boot logs, and more — all in a structured, indexed journal. The tool for reading it is journalctl.

# Follow all logs in real time (like tail -f /var/log/syslog)
journalctl -f

# Logs for a specific service
journalctl -u nginx

# Follow a specific service in real time
journalctl -u nginx -f

# Last 50 lines for a service
journalctl -u nginx -n 50

# Logs since the last boot
journalctl -b

# Logs from the previous boot (useful when something crashed on reboot)
journalctl -b -1

# Logs since a specific time
journalctl --since "2026-05-15 10:00:00"

# Last hour
journalctl --since "1 hour ago"

# Only errors and worse
journalctl -p err

# Only for a specific service, last hour
journalctl -u myapp --since "1 hour ago"

One thing that catches people: by default journalctl opens in a pager (like less). Press G to jump to the end, or use -f to follow in real time. Add --no-pager to pipe the output elsewhere.

Dependencies and Ordering

systemd gives you several directives to control how units relate to each other. This is one of the things that makes it significantly more powerful than old init scripts.

Ordering directives (when to start, not whether to start):

• After= — start after the listed units. Most common. After=network.target means "wait until networking is up."
• Before= — start before the listed units. Less common, useful when your service needs to be running before something else starts.

Dependency directives (whether to start, not just when):

• Wants= — start the listed units alongside this one, but don't fail if they don't start. The most common and most forgiving dependency type. Use this for most cases.
• Requires= — like Wants, but if the required unit fails or stops, this unit also stops. Strict dependency. Use it when your service genuinely cannot function without the other.
• BindsTo= — even stricter than Requires. If the other unit stops for any reason, this unit immediately stops too.

A common pattern: a web application that needs a database to be running first.

[Unit]
Description=My Web Application
After=network.target postgresql.service
Requires=postgresql.service

This means: start after networking and PostgreSQL are up, and stop if PostgreSQL stops.

Note that After and Requires serve different purposes and are often combined. Requires=postgresql.service alone doesn't guarantee PostgreSQL starts first — it just makes this service dependent on it. You need After=postgresql.service to also enforce the ordering.

Security Hardening

systemd has a surprisingly rich set of security directives you can add to unit files to limit what a service can do. These are worth knowing even for homelab use — they significantly reduce the impact of a compromised service.

[Service]
# Run as a non-root user
User=myapp
Group=myapp

# Make the filesystem mostly read-only
ProtectSystem=strict
# But allow writing to these directories
ReadWritePaths=/var/lib/myapp /var/log/myapp

# Prevent the service from seeing /home
ProtectHome=true

# Give the service its own private /tmp
PrivateTmp=true

# Prevent privilege escalation
NoNewPrivileges=true

# Restrict which system calls the service can make
SystemCallFilter=@system-service

You don't need to use all of these, but User=, NoNewPrivileges=true, and PrivateTmp=true are low-effort and high-value. They cost nothing in terms of functionality for most services and significantly reduce the blast radius of a vulnerability.

To check how hardened a service already is, run:

systemd-analyze security nginx.service

This gives you a security score and lists which protections are enabled and missing. It won't tell you what's appropriate for every service, but it's a useful starting point.

Systemd Timers: Replacing Cron

systemd timers are an alternative to cron for scheduling recurring tasks. They're not necessarily better than cron for simple cases, but they have a few advantages: they log to the journal (so you can see output with journalctl), they integrate with systemd's dependency system, and they can be paused, started, and inspected with the same tools as services.

A timer requires two files: a .service file for the task and a .timer file for the schedule. Example — a daily backup script:

# /etc/systemd/system/backup.service
[Unit]
Description=Daily Backup

[Service]
Type=oneshot
User=backup
ExecStart=/opt/backup/run-backup.sh

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

OnCalendar uses a calendar expression format. Common examples:

OnCalendar=daily           # Every day at midnight
OnCalendar=weekly          # Every Monday at midnight
OnCalendar=*-*-* 02:00:00  # Every day at 2 AM
OnCalendar=Mon *-*-* 09:00 # Every Monday at 9 AM
OnCalendar=*:0/15          # Every 15 minutes

Persistent=true means if the system was off when the timer was supposed to fire, it will run the job immediately the next time the system boots. Useful for backup scripts that should never be skipped.

Enable and start the timer (not the service — the timer manages the service):

sudo systemctl enable --now backup.timer

# List all active timers and when they next fire
systemctl list-timers

Debugging Failed Services

Services fail. Here's the diagnostic sequence that resolves most issues:

Step 1: Check the status

systemctl status myapp

The output tells you the current state, the exit code if it failed, and the last few log lines. Look for the Active: line — it will say failed with a reason, or activating if it's stuck starting up.

Step 2: Read the full logs

journalctl -u myapp -n 100 --no-pager

This shows the last 100 lines of logs for the service. The actual error is almost always in here.

Step 3: Interpret the exit code

systemd reports exit codes as status=N/DESCRIPTION. Common ones:

• 203/EXEC — systemd couldn't execute the binary in ExecStart. Usually means the path is wrong, the file isn't executable, or the file doesn't exist.
• 217/USER — the User= specified in the unit file doesn't exist. Create the user with useradd --system myapp.
• 1/FAILURE — the program exited with code 1. This is the application itself reporting an error — check the logs for what the application said.
• 143 or 137 — the process was killed by a signal (SIGTERM or SIGKILL respectively). Could be systemd stopping it, OOM killer, or something else.

Step 4: Test the command manually

Copy the ExecStart command and run it yourself as the service user:

sudo -u myapp /opt/myapp/bin/myapp --config /etc/myapp/config.yaml

If it fails in your terminal, you'll see the error directly. If it works in your terminal but not as a service, the issue is usually a missing environment variable, a permission problem, or a working directory issue.

Step 5: Check boot performance

If your system is slow to boot, these tools help identify which services are taking too long:

# Overall boot time breakdown
systemd-analyze

# Show which units took longest
systemd-analyze blame

# Show the critical path (what's actually holding up boot)
systemd-analyze critical-chain

Common Real-World Patterns

Override a package's unit file without editing it directly

The right way to customize a package-provided unit file is with a "drop-in" override, not by editing the original:

sudo systemctl edit nginx

This opens an editor and creates a file at /etc/systemd/system/nginx.service.d/override.conf. Add only the directives you want to change:

[Service]
LimitNOFILE=65535

When nginx's package updates and overwrites /lib/systemd/system/nginx.service, your override survives untouched because it's a separate file.

Restart a service if it stays down too long

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

StartLimitBurst=3 and StartLimitIntervalSec=60 together mean: if the service fails to start 3 times within 60 seconds, stop trying. This prevents a crash-looping service from consuming all your resources. Without this, a service set to Restart=on-failure with a fast failure will restart hundreds of times per minute.

Run a service after a network connection is actually established

There's a subtle but important distinction on servers:

• network.target — networking services have started, but may not have connectivity yet
• network-online.target — at least one network interface has an IP and connectivity

If your service needs actual internet or LAN access to start (not just for the network stack to be initialized), use:

[Unit]
After=network-online.target
Wants=network-online.target

Pass environment variables to a service

Two ways to do this. Inline in the unit file:

[Service]
Environment="NODE_ENV=production"
Environment="PORT=3000"

Or from a file (better for secrets, since the unit file itself is world-readable):

[Service]
EnvironmentFile=/etc/myapp/environment

The environment file format is simple key=value pairs, one per line. Set its permissions to be readable only by root and the service user:

sudo chmod 640 /etc/myapp/environment
sudo chown root:myapp /etc/myapp/environment

systemd is one of those tools that rewards time invested in understanding it. Once you know the structure of unit files and the systemctl/journalctl workflow, you can manage any service on any systemd-based Linux system confidently — regardless of whether it's a homelab VM, a cloud VPS, or a production server.