How to Debug a systemd Service That Fails to Start 2026

It is 3 AM and your monitoring system fires an alert. A critical service went down, and every restart attempt fails. You SSH into the server, run systemctl status, and stare at a wall of text with cryptic exit codes.

If you have been there, you know the frustration. The good news is that systemd gives you a structured way to diagnose almost any startup failure. The bad news is that most engineers never learn the full workflow. They jump between random Stack Overflow answers without understanding what the output actually tells them.

This guide walks through a complete method for debugging a systemd service that fails to start using journalctl and exit codes. You will learn how to read systemctl status output, filter logs with journalctl, decode exit codes in the 200 to 242 range, and fix the seven most common failure causes. By the end, you will have a repeatable workflow that works on any Linux distribution running systemd.

How to Debug a systemd Service That Fails to Start: The Core Workflow

Every systemd debugging session follows the same four-step pattern. Master this sequence and you will cut your troubleshooting time dramatically.

Step 1: Run systemctl status myapp.service to see the Result type and exit code.

Step 2: Pull logs with journalctl -u myapp.service -b to find the actual error message.

Step 3: Match the exit code to a root cause category (exec failure, permissions, crash, timeout).

Step 4: Apply the fix, run systemctl daemon-reload, and restart.

I have used this exact workflow on hundreds of production incidents. It works because each step narrows the problem space. You never guess. You read what systemd tells you and follow the trail.

Let me break down each step in detail.

Step 1: Reading systemctl status Output

The systemctl status command is your first stop. It gives you a snapshot of the service state, the last exit information, and recent log lines.

Run this command:

systemctl status myapp.service

The output contains several fields that tell you exactly what happened. Here are the ones that matter most for debugging.

Active: Shows the current state. active (running) means the service is up. failed means it crashed or could not start. inactive (dead) means it was never started or was stopped cleanly.

Main PID: The process ID of the service. If it shows a PID and then code=killed, the process was terminated by a signal. If the PID is missing entirely, the process never started.

Process: Shows the ExecStart line execution result. This is where you see the exit code in detail. For example, status=203/EXEC means systemd could not execute the binary at all.

Result: This is the most important field for diagnosis. It tells you the category of failure:

• Result: exit-code – The process ran but exited with a non-zero code. The application itself failed.

• Result: signal – The process was killed by a signal like SIGSEGV or SIGTERM. Look at the signal name.

• Result: timeout – The service did not signal readiness within the configured time limit.

• Result: oom-kill – The kernel killed the process because it ran out of memory.

• Result: core-dump – The process crashed and produced a core dump.

Here is a real-world example of status output for a failed service:

myapp.service - My Application
Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
Active: failed (Result: exit-code) since Mon 2026-08-04 03:12:44 UTC
Process: 15432 ExecStart=/opt/myapp/bin/start.sh (code=exited, status=203/EXEC)

In this case, status=203/EXEC immediately tells us systemd could not execute the binary. We know to check the ExecStart path next.

One command I also run alongside status is systemctl cat myapp.service. It prints the full unit file as systemd sees it, including any drop-in overrides. This is critical because the file on disk might not match what systemd actually loaded.

Step 2: Pulling Logs with journalctl

Once you know the Result type and exit code from systemctl status, the next step is reading the detailed logs. journalctl is the tool for this.

Here are the flags I use most often when debugging failed services.

journalctl -u myapp.service -b – Shows all logs for the service from the current boot. The -u flag filters by unit name, and -b limits output to the current boot. This is the command I start with 90 percent of the time.

journalctl -u myapp.service -b -1 – Shows logs from the previous boot. Useful when a service failed before a reboot and you need the old logs.

journalctl -u myapp.service -e – Jumps to the end of the log. The -e flag shows the most recent entries first, which is usually what you want during an incident.

journalctl -u myapp.service -f – Follows the log in real time, similar to tail -f. I use this while restarting a service to watch for errors as they happen.

journalctl -u myapp.service -p err – Filters to error-level messages only. The -p flag accepts priority levels from 0 (emerg) to 7 (debug). Using -p err shows only error and higher severity, cutting through noise.

journalctl -u myapp.service --since "1 hour ago" – Limits output to a time range. Helpful when the service has been running for days and you only want recent entries.

journalctl -u myapp.service --grep "error" – Filters log lines matching a pattern. This uses regular expressions, so you can search for patterns like --grep "FATAL|panic|traceback".

For a comprehensive view of everything systemd logged about the service, combine flags:

journalctl -u myapp.service -b -p err -e

This shows error-level messages from the current boot, starting from the most recent. It is my go-to command for incident response.

What to do when journalctl shows no output: If journalctl -u myapp.service returns nothing, the journal may have rotated since the failure. Try journalctl -u myapp.service --no-pager --all or check journalctl --vacuum-time settings. Another common cause is that the service sends output to a file instead of stdout or syslog. In that case, check the application’s own log files in /var/log/ or wherever your app writes.

Understanding systemd Exit Codes (200 to 242)

Exit codes in the 200 to 242 range are special. They do not come from your application. They come from systemd itself, indicating that systemd could not start or manage the process. Understanding these codes is the fastest way to identify the root cause.

Here are the most common systemd exit codes you will encounter:

• 200/CHDIR – The WorkingDirectory specified in the unit file does not exist or is not accessible. Fix: create the directory or correct the path.

• 203/EXEC – systemd could not execute the binary specified in ExecStart. The path is wrong, the binary is missing, or it lacks execute permissions. This is the single most common exit code I see.

• 206/SETGROUP – The group specified in Group= does not exist on the system.

• 207/SETUSER – The user specified in User= does not exist.

• 208/CHROOT – The RootDirectory path does not exist.

• 210/SETSID – Failed to create a new session.

• 211/EXEC – Another variant of execution failure, often related to shell wrapper issues.

• 214/CHDIR – Another variant of directory change failure.

• 216/GROUP – Group configuration error.

• 217/USER – The User= directive references a user that does not exist on the system. Fix: create the user or correct the username.

• 219/SETGROUP – Additional group setting failure.

• 224/STDIN – Failed to set up standard input.

• 227/RLIMIT – Failed to set resource limits.

• 228/SETCAPABILITIES – Failed to set process capabilities.

• 232/CHDIR – Yet another directory change failure variant.

Exit codes below 200 come from your application. For example, status=1 means the application exited with code 1. You need to check the application’s own error handling to understand why.

Exit codes above 128 indicate the process was killed by a signal. The signal number is the exit code minus 128. For example, status=139 means signal 11 (SIGSEGV), and status=137 means signal 9 (SIGKILL), often from an OOM killer.

Here is a quick reference for translating exit codes:

• Code 0 – Clean exit. Not an error.

• Code 1 to 125 – Application-specific error. Check the app logs.

• Code 126 – Command found but not executable (permissions).

• Code 127 – Command not found (bad path).

• Code 128 + N – Killed by signal N (e.g., 137 = SIGKILL, 139 = SIGSEGV, 134 = SIGABRT).

• Code 200 to 242 – systemd management failure (bad path, bad user, bad directory).

The 7 Most Common Causes of a Failed systemd Service

Based on my experience and forum discussions across Reddit, Stack Exchange, and Server Fault, almost every systemd startup failure falls into one of these seven categories. Let me walk through each one with real examples and fixes.

1. Bad ExecStart Path or Syntax

This is the number one cause of startup failures. The ExecStart= line points to a binary that does not exist, has wrong permissions, or uses incorrect syntax.

A common mistake is using relative paths. systemd requires absolute paths for ExecStart. Writing ExecStart=node app.js fails with code 203. You need ExecStart=/usr/bin/node /opt/myapp/app.js.

Another mistake is shell syntax in ExecStart. systemd does not run commands through a shell by default. Pipes, environment variable expansion, and redirections do not work unless you wrap the command:

ExecStart=/bin/bash -c 'node app.js | tee /var/log/myapp.log'

Run systemctl cat myapp.service to verify what systemd actually loaded.

2. WorkingDirectory and User Issues

If WorkingDirectory= points to a path that does not exist, you get exit code 200/CHDIR. This happens when you deploy to a new server and forget to create the directory.

Fix it by creating the directory or correcting the path:

sudo mkdir -p /opt/myapp
sudo chown myapp:myapp /opt/myapp

Similarly, if User= references a system user that does not exist, you get 217/USER. Create the user with useradd -r myapp or change the directive to an existing user.

3. Type= Mismatch

The Type= directive tells systemd how the service starts. Getting this wrong causes confusing failures where the service runs briefly then gets killed.

The most common mismatch: using Type=simple for a daemon that forks. With Type=simple, systemd considers the service started as soon as the main process launches. If the process forks and the parent exits, systemd thinks the service died and kills the child.

Use Type=forking for traditional daemons that fork into the background. Use Type=simple for processes that stay in the foreground. Use Type=notify for modern services that support sd_notify readiness signaling.

4. Dependency and Ordering Problems

Your service might start before its dependencies are ready. A database client service that starts before PostgreSQL will fail. A web app that starts before the network is up will fail.

Check dependencies with systemctl list-dependencies myapp.service. This shows the full dependency tree.

Fix ordering with After= and Requires= directives:

After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

Use After= for ordering (start after, but do not fail if the dependency is absent). Use Requires= for hard dependencies (fail if the dependency fails). Use Wants= for soft dependencies (try to start, but do not fail if absent).

The network-online.target is especially important. The regular network.target does not guarantee the network is configured. Many services fail because they start after network.target but before the network is actually ready.

5. Permissions and SELinux Denials

File permission issues are a frequent cause of failures. The service user might not have read access to a config file or execute permission on a script.

Check permissions by running the command manually as the service user:

sudo -u myapp /opt/myapp/bin/start.sh

If this works but systemd still fails, SELinux might be blocking it. Check for AVC denials:

sudo ausearch -m avc -ts recent

Or check the audit log directly:

sudo journalctl -t setroubleshoot

If you see AVC denials, you need to adjust SELinux policy or file contexts. Running restorecon -Rv /opt/myapp/ fixes many label issues.

6. Environment Variable Problems

The EnvironmentFile= directive loads environment variables from a file. If the file is missing and not prefixed with a minus sign, the service fails to start.

Fix optional environment files by prefixing with a minus:

EnvironmentFile=-/etc/default/myapp

The minus tells systemd to ignore the file if it does not exist. Without it, a missing file causes a startup failure.

Also watch for syntax errors in environment files. Each line should be KEY=value without spaces around the equals sign. Do not use export in environment files.

7. Restart Loops and start-limit-hit

When a service keeps failing and restarting, systemd eventually stops trying. You see the message start request repeated too quickly and the status shows start-limit-hit.

This is controlled by StartLimitBurst and StartLimitIntervalSec. By default, systemd allows 5 restarts within 10 seconds before hitting the limit.

To recover from this state, reset the failure counter:

sudo systemctl reset-failed myapp.service

Then fix the underlying issue before restarting. If your service genuinely needs more restart attempts, adjust the limits in the unit file:

StartLimitBurst=10
StartLimitIntervalSec=60

Fixing Type= Mismatches: simple vs forking vs notify

The Type= directive confuses more engineers than any other systemd setting. Getting it wrong causes symptoms that look like crashes but are actually systemd misinterpreting the service behavior.

Here is how to choose the right Type for your service.

Type=simple (default): Use this for processes that run in the foreground. systemd considers the service started as soon as the main process begins. This is correct for most modern applications written in Node.js, Python, Go, or Java that do not fork.

Type=forking: Use this for traditional Unix daemons that fork a child process and exit the parent. The original process exits, and the child continues running in the background. If you use this type, you should also set PIDFile= so systemd can track the child process. Without PIDFile=, systemd might lose track of the service.

Type=notify: Use this for services that support sd_notify readiness signaling. The service sends a READY=1 message to systemd when it is fully initialized and ready to serve. This is the most accurate type for modern services that support it. Many databases, container runtimes, and server applications support Type=notify natively.

Type=oneshot: Use this for scripts that do a single task and exit. systemd considers the service started when the process exits with code 0. Useful for setup scripts, backup tasks, and configuration initialization.

Type=exec: Similar to Type=simple but systemd waits until the main binary has been executed successfully before considering the service started. This catches path and permission errors at start time rather than after systemd reports success.

The most common mistake I see: a developer writes a Python script using Type=simple, but the script forks using os.fork() or runs a daemon library. The parent exits immediately, systemd marks the service as failed, and the developer is confused because the process is actually running.

The fix is either to remove the forking behavior from the application or switch to Type=forking with a proper PIDFile=.

Using Drop-in Overrides to Fix Unit Configuration

You should almost never edit a unit file in /lib/systemd/system/ directly. Package updates will overwrite your changes. Instead, use drop-in overrides.

The systemctl edit command creates a drop-in override file that layers on top of the original unit file:

sudo systemctl edit myapp.service

This opens an editor where you add only the directives you want to change or add. For example, to add environment variables and change the restart behavior:

[Service]
Environment=NODE_ENV=production
Environment=PORT=3000
Restart=always
RestartSec=5

Save and exit the editor. systemd creates the override file at /etc/systemd/system/myapp.service.d/override.conf.

After any unit file change, you must run daemon-reload:

sudo systemctl daemon-reload

This tells systemd to re-read all unit files. Without it, systemd uses the cached configuration and your changes have no effect. Forgetting daemon-reload is one of the most common mistakes I see, especially among engineers new to systemd.

To clear an override, use:

sudo systemctl revert myapp.service

To see the effective configuration including overrides:

systemctl cat myapp.service

Breaking Restart Loops and start-limit-hit

Restart loops happen when a service fails, systemd restarts it, and it fails again. Eventually, systemd hits the start rate limit and stops trying entirely.

The error message looks like this:

Job for myapp.service failed because start-limit-hit exceeded.

Or in status output:

Active: failed (Result: start-limit-hit)

When you see this, systemd has given up. The underlying problem is still there, but now you also have to clear the limit before you can test your fix.

Here is the recovery sequence:

Step 1: Clear the failed state:

sudo systemctl reset-failed myapp.service

Step 2: Fix the underlying issue (bad path, permissions, config error).

Step 3: Test manually before letting systemd manage restarts:

sudo systemctl start myapp.service

Step 4: Monitor with journalctl:

journalctl -u myapp.service -f

If the service legitimately needs more restart attempts, increase the limits. Put these in the [Unit] section of your override:

[Unit]
StartLimitBurst=10
StartLimitIntervalSec=120

This allows 10 restart attempts within 120 seconds. But be careful: if your service is genuinely broken, more restart attempts just burn CPU and fill logs. Fix the root cause first.

Another useful command for finding all failed services on a system:

systemctl --failed

This lists every failed unit, not just the one you are debugging. I run this during incident triage to see if the failure is isolated or systemic.

Advanced Tools: systemd-analyze, coredumpctl, and SELinux

Once you have mastered the basic workflow, these advanced tools help with tricky failures that resist standard debugging.

systemd-analyze verify

Before you even start the service, check your unit file for syntax errors:

systemd-analyze verify /etc/systemd/system/myapp.service

This catches typos, invalid directives, and configuration errors without actually running the service. I run this every time I create or modify a unit file. It has saved me from countless silly mistakes.

systemd-analyze blame

To identify services that are slow to start:

systemd-analyze blame

This shows each service and how long it took to initialize, sorted from slowest to fastest. A service that takes 30 seconds might be hitting a timeout or waiting for a dependency that never becomes ready.

systemd-analyze critical-chain

To see the dependency chain that blocks boot completion:

systemd-analyze critical-chain myapp.service

This shows the chain of services that must complete before your service can start. If a service in the chain is slow or failing, your service waits indefinitely.

Core Dump Analysis with coredumpctl

If your service crashes with a segfault or abort, check for core dumps:

coredumpctl list

This lists all available core dumps. Find the one for your service and get details:

coredumpctl info myapp

To open the core dump in GDB for stack trace analysis:

coredumpctl debug myapp

This launches GDB with the core file and the executable loaded, ready for analysis. The stack trace will show exactly where the crash occurred.

SELinux AVC Denial Checking

On systems with SELinux in enforcing mode, denials are a common cause of silent failures. The service appears configured correctly, but SELinux blocks access to files, ports, or resources.

Search for recent AVC denials:

sudo ausearch -m avc -ts recent

Or use the faster journalctl approach:

sudo journalctl -t setroubleshoot --since "1 hour ago"

If you find denials, you can generate a custom policy module or fix file contexts with restorecon. Never disable SELinux as a debugging step. Instead, temporarily set it to permissive mode:

sudo setenforce 0

If the service works in permissive mode, you have confirmed it is an SELinux issue. Generate an audit2allow policy, then re-enable enforcing:

sudo audit2allow -a -M myapp-policy
sudo semodule -i myapp-policy.pp
sudo setenforce 1

Emergency and Rescue Targets

If a critical service failure prevents booting, you can boot into rescue or emergency mode. These targets provide a minimal environment for recovery.

Boot into rescue mode (single-user with some services):

sudo systemctl rescue

Boot into emergency mode (minimal shell, no services):

sudo systemctl emergency

If the system will not boot at all, add systemd.unit=emergency.target to the kernel command line in GRUB. This drops you into a root shell where you can fix the broken service.

For early boot debugging, enable the debug shell:

sudo systemctl enable debug-shell.service

This opens a root shell on tty9 early in the boot process. Disable it after debugging, as it is a security risk.

Quick Command Reference Cheatsheet

Here is a summary of every command covered in this guide, organized by when you need them. Bookmark this for your next incident.

Diagnosis commands:

• systemctl status myapp.service – Current state, exit code, recent logs

• systemctl cat myapp.service – Full unit file including overrides

• systemctl --failed – List all failed services

• systemctl list-dependencies myapp.service – Dependency tree

• journalctl -u myapp.service -b -e – Recent logs from current boot

• journalctl -u myapp.service -p err – Error-level messages only

Fix and recovery commands:

• sudo systemctl daemon-reload – Re-read unit files after changes

• sudo systemctl reset-failed myapp.service – Clear start-limit-hit

• sudo systemctl edit myapp.service – Create drop-in override

• systemd-analyze verify /etc/systemd/system/myapp.service – Check syntax

Advanced debugging commands:

• systemd-analyze blame – Slowest services at boot

• systemd-analyze critical-chain myapp.service – Dependency chain

• coredumpctl list – Available core dumps

• sudo ausearch -m avc -ts recent – SELinux denials

Frequently Asked Questions

Conclusion

Debugging a systemd service that fails to start becomes systematic once you know the workflow. Start with systemctl status to read the Result type and exit code. Move to journalctl for detailed logs. Decode the exit code to narrow the root cause. Apply the fix, reload the daemon, and restart.

The exit codes in the 200 to 242 range are your fastest diagnostic tool. A 203/EXEC means a bad binary path. A 200/CHDIR means a missing working directory. A 217/USER means a non-existent user. Each code points directly to a fix.

The seven common causes cover almost every failure you will encounter: bad ExecStart, WorkingDirectory or User issues, Type mismatches, dependency problems, permissions and SELinux, environment variable mistakes, and restart loops. When you hit start-limit-hit, run systemctl reset-failed to clear the state and try again.

For advanced scenarios, reach for systemd-analyze verify to check syntax before starting, coredumpctl for crash analysis, and ausearch -m avc for SELinux denials. These tools catch issues that basic status and log analysis miss.

Remember the two most common mistakes: forgetting systemctl daemon-reload after editing unit files, and using Type=simple for services that fork. Fix those two habits and you will solve a large percentage of systemd startup failures before they happen.

Next time a service fails at 3 AM, you will know exactly what to do. Run the status command, read the exit code, pull the journal, and follow the evidence. No guesswork required.