Home/Linux Administration/Processes & Services
⚙️

Level 5 of 24

Processes & Services

PIDs, signals, systemd units, and keeping services alive.

A running Linux system is, at any moment, just a tree of processes started by systemd (PID 1) and each other. This level covers inspecting, controlling, and permanently managing processes — including turning your own program into a proper, auto-restarting systemd service.

Prerequisites

  • • Level 4: Users & Permissions

By the end of this level, you can

  • ✓ Identify and control a runaway or misbehaving process
  • ✓ Explain process states, signals, and what kill actually does
  • ✓ Read systemctl status and journalctl output to diagnose a service
  • ✓ Create and manage a custom systemd service

Processes, PIDs, and process states

ps, top, and understanding what a process actually is. · 9 min

A process is a running instance of a program, and every process has a unique Process ID (PID) assigned by the kernel, plus a Parent Process ID (PPID) identifying which process started it. PID 1 is always the first process the kernel starts at boot — on virtually all modern distributions, that's `systemd`, which then starts every other process, directly or indirectly, making every process on the system a descendant of PID 1.

A process moves through states over its lifetime: Running (actively executing or ready to), Sleeping (waiting for something — input, a timer, a lock — the most common state for an idle process), Stopped (paused, typically by a signal, resumable), and Zombie (the process has finished executing but its parent hasn't yet collected its exit status — a genuine zombie is harmless in small numbers but indicates the parent process has a bug if they accumulate). An orphan process is one whose parent exited first; it gets automatically re-parented to PID 1 (or a subreaper) so it isn't left dangling.

`ps` gives a snapshot of processes at the moment you run it; `top` (or the friendlier `htop`, often installed separately) gives a live, continuously updating view sorted by resource usage — your default tool for "what is actually using CPU/memory right now on this box".

CommandPurposeExample
ps auxList all running processes with detailed info (user, CPU%, memory%, command)—
ps -efAlternative process listing format, showing PPID clearly—
topLive, continuously updating process viewer sorted by CPU usage by default—
htopA more user-friendly, colorized live process viewer (often needs installing separately)—
pgrepFind PIDs by process namepgrep nginx
pidofFind the PID(s) of a running program by exact namepidof sshd

Takeaway: Every process on the system is a descendant of PID 1 (systemd on modern distros) — `ps -ef` showing PPID is how you trace that ancestry when you need to understand what actually launched a misbehaving process.

Signals, kill, and job control

What kill actually sends, why SIGKILL is a last resort, and running jobs in the background. · 9 min

`kill` doesn't inherently mean "terminate" — it sends a signal to a process, and different signals request different behavior. `SIGTERM` (signal 15, the default `kill` sends) politely asks a process to shut down, giving it a chance to close files, save state, and exit cleanly — a well-written program handles SIGTERM by cleaning up before exiting. `SIGKILL` (signal 9) is not a request; the kernel terminates the process immediately with no chance for cleanup, which is why it's a last resort for a process that's truly stuck and not responding to SIGTERM, not a routine default.

`SIGHUP` (signal 1) is used by many services as a signal to reload their configuration without fully restarting — this predates systemd, from an era when a terminal hangup literally sent this signal to processes attached to it. `SIGSTOP` pauses a process (not gracefully cancelable by the process itself, unlike SIGTERM) and `SIGCONT` resumes it — this is how job control works under the hood.

Job control lets you manage multiple processes within one shell session: `command &` starts a command in the background immediately; `Ctrl+Z` suspends a running foreground command; `bg` resumes a suspended job in the background; `fg` brings a background or suspended job back to the foreground; `jobs` lists what's running in the current session. `nohup command &` goes a step further — it starts a command in the background that will keep running even after you log out and your shell session ends, which plain `&` alone does not guarantee.

CommandPurposeExample
kill PIDSend SIGTERM (polite shutdown request) to a process by PIDkill 4821
kill -9 PIDSend SIGKILL (immediate, un-ignorable termination)kill -9 4821
killall nameSend a signal to all processes matching a namekillall -9 stuck-process
nice -n 10 commandStart a command with lower scheduling priority (higher niceness = lower priority)—
reniceChange the priority of an already-running processrenice -n 5 -p 4821
jobs / bg / fg / nohupJob control — list, background, foreground, and detach jobs from the current shell session—

Common Mistakes

  • ⚠ Reaching for `kill -9` as the first attempt instead of a plain `kill` (SIGTERM) — this skips any graceful shutdown/cleanup the process would otherwise perform
  • ⚠ Assuming `command &` alone survives logging out — it usually doesn't; use `nohup` (or a proper systemd service) for anything that needs to outlive your session

Takeaway: Always try plain `kill` (SIGTERM) before `kill -9` — SIGKILL skips any cleanup the process would otherwise do, and should be reserved for a process that's genuinely unresponsive to the polite request.

systemd: managing and creating services

systemctl, journalctl, and writing your own unit file. · 13 min

systemd manages services as "units" — most commonly `.service` units, though timer, socket, and mount units also exist. `systemctl status <name>` is the single most useful diagnostic command on a modern Linux system: it shows whether a service is running, its recent log output, its PID, and its enabled/disabled state, all in one view. `systemctl start/stop/restart <name>` control a service immediately for the current boot; `systemctl enable/disable <name>` control whether it starts automatically on future boots — these are independent: a service can be running now but not enabled for next boot, or enabled for next boot but not currently running.

`journalctl` reads systemd's centralized logging (the journal), which captures both systemd's own messages and anything a service writes to standard output/error. `journalctl -u <service-name>` filters to just that service's logs — almost always your first move after `systemctl status` shows a problem. `journalctl -f` follows the log live, like `tail -f` for the entire system journal, and `journalctl --since "10 minutes ago"` scopes to a time window, which is invaluable when correlating a service failure with a specific incident time.

Writing your own unit file is how you turn any script or long-running program into a properly managed service: a `.service` file placed in `/etc/systemd/system/` defines what to run (`ExecStart`), when to start it relative to other units (`After`, `Wants`), and how to handle it exiting (`Restart=on-failure` is a common, genuinely useful setting that makes systemd automatically restart a crashed service). After creating or editing a unit file, `systemctl daemon-reload` is required to make systemd re-read it before `start`/`enable` will pick up the change — a step that's easy to forget and produces confusing "nothing happened" results if skipped.

CommandPurposeExample
systemctl statusShow a service's current state, recent logs, and PIDsystemctl status nginx
systemctl start/stop/restartControl a service for the current bootsudo systemctl restart nginx
systemctl enable/disableControl whether a service starts automatically on boot—
systemctl daemon-reloadRe-read unit files after creating or editing one — required before changes take effect—
journalctl -u nameView logs for a specific servicejournalctl -u nginx -f

A minimal custom systemd service unit

# /etc/systemd/system/my-app.service
[Unit]
Description=My Custom Application
After=network.target

[Service]
ExecStart=/usr/local/bin/my-app
Restart=on-failure
User=appuser

[Install]
WantedBy=multi-user.target
🧪

Hands-On Lab

Build a custom systemd service

Objectives

  • ✓ Write a minimal script to run as a service
  • ✓ Create a systemd unit file for it
  • ✓ Start, enable, and verify the service
  • ✓ Inspect its logs via journalctl

Instructions

  1. Create a script at `/usr/local/bin/heartbeat.sh` containing: `#!/bin/bash` followed by `while true; do echo "heartbeat: $(date)"; sleep 10; done` — make it executable with `chmod +x`.
  2. Create a unit file at `/etc/systemd/system/heartbeat.service` following the example structure above, with `ExecStart=/usr/local/bin/heartbeat.sh`.
  3. Run `sudo systemctl daemon-reload`, then `sudo systemctl start heartbeat`.
  4. Check `sudo systemctl status heartbeat` and confirm it shows "active (running)".
  5. Run `sudo journalctl -u heartbeat -f` and confirm you see a new "heartbeat" line every 10 seconds.
  6. Enable it with `sudo systemctl enable heartbeat` so it survives a reboot.
Hints (2)
  • If the service fails immediately, check that the script has execute permission and the shebang line is correct.
  • Forgot to run `daemon-reload` after creating the file? That's the most common reason a brand-new unit "doesn't exist" to systemctl.

Quick Check

A service is currently running, but the server just rebooted and it did NOT come back up. What was most likely missed?

Takeaway: `systemctl start` and `systemctl enable` are two separate, independent switches — a service can be running right now but vanish on next reboot if it was never enabled.

Sponsor / Advertisement