Systemd Complete Guide: From Beginner to Advanced

Master systemd, Linux’s powerful service manager, with this comprehensive beginner-to-advanced guide. Learn to create production-ready services, convert PM2 applications, set up Node.js + Python hybrid architectures, configure Nginx + SSL, and even work with Apache2.

🧠 What is Systemd? (Beginner Level)

Systemd is Linux’s built-in service manager that handles application lifecycle management automatically. It replaces traditional process managers like PM2, forever, nohup, and screen with a more robust, native solution.

Core Capabilities

Systemd automatically:

• Starts your applications when the server boots
• Restarts applications if they crash
• Prevents infinite crash loops with built-in protection
• Runs applications in the background
• Manages application logs through journalctl
• Handles service dependencies and startup order

Why Systemd Over PM2?

Systemd Advantages:

• Linux Native: Built into every modern Linux distribution
• Best for Python: Designed for all applications, not just Node.js
• Crash Protection: Advanced restart policies and crash loop prevention
• Auto-restart on boot: Automatic service startup after system reboot
• No Node.js dependency: No need to install Node.js just for process management
• Industry standard: The professional choice for production deployments

PM2 Limitations:

• Requires Node.js runtime
• Limited crash protection features
• Manual boot configuration needed
• Primarily designed for JavaScript applications

🏗️ How Systemd Works (Mental Model)

System Boot
    ↓
systemd starts
    ↓
systemd reads *.service files
    ↓
systemd runs your applications

Each application is described in a .service file that tells systemd how to start, stop, and manage it. Once configured, systemd handles everything automatically.

📁 Where Systemd Files Live

Custom service files go in:

/etc/systemd/system/

Example service files:

• myapp.service
• fastapi.service
• node-api.service
• redis.service

🧩 Your First Systemd Service (Beginner)

Step 1: Create Service File

sudo nano /etc/systemd/system/fastapi.service

Step 2: Minimal Working Service

[Unit]
Description=FastAPI App

[Service]
ExecStart=/usr/bin/python3 app.py

[Install]
WantedBy=multi-user.target

✅ This works! But it’s not production-ready yet.

🧠 Understanding Every Section (Important)

[Unit] Section (WHEN & ORDER)

[Unit]
Description=FastAPI App
After=network.target

Configuration Explanation:

• Description: Human-readable name for the service
• After=network.target: Start after network is available

[Service] Section (HOW TO RUN)

[Service]
ExecStart=/usr/bin/python3 app.py

Configuration Explanation:

• ExecStart: Command to run your application
• Absolute path: REQUIRED for systemd (no relative paths)

Check your paths:

which python3
which node

[Install] Section (AUTO-START)

[Install]
WantedBy=multi-user.target

Means: “Start this service on normal system boot”

🔁 Restart & Crash Protection (Intermediate)

Auto-restart Configuration

[Service]
Restart=on-failure
RestartSec=5

Restart Options:

• Restart=on-failure: Restart only when application crashes
• RestartSec=5: Wait 5 seconds before restarting

Crash Loop Protection (CRITICAL)

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=5

Meaning: Maximum 5 restart attempts within 60 seconds. If exceeded, systemd stops trying to prevent infinite crash loops.

👤 Users & Security (Intermediate)

NEVER Run Production Apps as Root

Create a dedicated user:

sudo adduser deploy

Configure service to run as that user:

[Service]
User=deploy

Benefits:

• Enhanced security
• Prevents full system damage if compromised
• Better resource isolation

📁 WorkingDirectory & Virtual Environments

[Service]
WorkingDirectory=/var/www/fastapi
ExecStart=/var/www/fastapi/venv/bin/gunicorn main:app

Why this matters:

• Correct relative file paths
• Isolated Python dependencies
• Predictable execution environment

🌱 Environment Variables

Inline Environment Variables

[Service]
Environment=PORT=8000
Environment=ENV=production
Environment=DATABASE_URL=postgresql://user:pass@localhost/db

Environment File

[Service]
EnvironmentFile=/etc/fastapi.env

Create the environment file:

sudo nano /etc/fastapi.env

PORT=8000
ENV=production
DATABASE_URL=postgresql://user:pass@localhost/db

🧠 Logging with Systemd

Systemd automatically captures all application output. No separate log files needed.

View Logs

# View all logs for a service
journalctl -u fastapi

# Follow live logs
journalctl -u fastapi -f

# View last 100 lines
journalctl -u fastapi -n 100

# View logs since last boot
journalctl -u fastapi -b

Essential Commands (MUST KNOW)

Service Management:

• systemctl start app - Start the service
• systemctl stop app - Stop the service
• systemctl restart app - Restart the service
• systemctl status app - Check service status
• systemctl enable app - Enable auto-start on boot
• systemctl disable app - Disable auto-start
• systemctl daemon-reload - Reload configuration changes
• systemctl reset-failed app - Clear failed state

🧪 Full Production FastAPI Example (Advanced)

[Unit]
Description=FastAPI Backend
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
User=deploy
WorkingDirectory=/var/www/fastapi
ExecStart=/var/www/fastapi/venv/bin/gunicorn \
  -k uvicorn.workers.UvicornWorker \
  main:app \
  --bind 0.0.0.0:8000
Restart=on-failure
RestartSec=5
Environment=ENV=production

[Install]
WantedBy=multi-user.target

Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable fastapi
sudo systemctl start fastapi
sudo systemctl status fastapi

🔄 Convert PM2 App to Systemd (Node.js Example)

Current PM2 Setup

pm2 start server.js --name node-api

Step-by-Step Conversion

Step 1: Find Node.js path

which node
# Output: /usr/bin/node

Step 2: Create systemd service

sudo nano /etc/systemd/system/node-api.service

Step 3: Service configuration

[Unit]
Description=Node.js API
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
User=deploy
WorkingDirectory=/var/www/node-api
ExecStart=/usr/bin/node server.js
Restart=on-failure
RestartSec=5
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Step 4: Enable and start

sudo systemctl daemon-reload
sudo systemctl enable node-api
sudo systemctl start node-api

Step 5: Monitor logs

journalctl -u node-api -f

✔ PM2 no longer needed for this application

🏗️ Node.js + Python Hybrid Setup (Best Practice)

Architecture Overview

Nginx (Reverse Proxy)
├── Node.js API (systemd)
└── FastAPI Backend (systemd)

FastAPI Systemd Service

[Unit]
Description=FastAPI Backend
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
User=deploy
WorkingDirectory=/var/www/fastapi
ExecStart=/var/www/fastapi/venv/bin/gunicorn \
  -k uvicorn.workers.UvicornWorker \
  main:app \
  --bind 127.0.0.1:8000
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Node.js Systemd Service

[Unit]
Description=Node.js Backend
After=network.target

[Service]
User=deploy
WorkingDirectory=/var/www/node
ExecStart=/usr/bin/node server.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

Benefits of Hybrid Setup

• Both services run independently
• Each has its own restart policy
• Both auto-start on boot
• Separate logging and monitoring
• Language-specific optimizations

🌐 Nginx + SSL Configuration (Production Standard)

Install Nginx

sudo apt update
sudo apt install nginx -y

Nginx Reverse Proxy Configuration

sudo nano /etc/nginx/sites-available/app

server {
    listen 80;
    server_name example.com;

    # FastAPI backend
    location /api/ {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Node.js frontend
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable Site

sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Add SSL with Let’s Encrypt

sudo apt install certbot python3-certbot-nginx -y
sudo certbot --nginx -d example.com

Auto-renew SSL

sudo certbot renew --dry-run

✔ HTTPS enabled with automatic renewal

🔄 Apache2 Example (Alternative to Nginx)

Apache2 is older but still widely used. Here’s how to configure it with systemd services.

Install Apache2

sudo apt install apache2 -y

Enable Required Modules

sudo a2enmod proxy proxy_http ssl
sudo systemctl restart apache2

Apache VirtualHost Configuration

sudo nano /etc/apache2/sites-available/fastapi.conf

<VirtualHost *:80>
    ServerName example.com
    
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/
    
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Enable Site

sudo a2ensite fastapi.conf
sudo systemctl reload apache2

Add SSL with Let’s Encrypt

sudo certbot --apache -d example.com

Nginx vs Apache2 Comparison

Nginx Advantages:

• Performance: ⭐⭐⭐⭐⭐ - Faster and more efficient
• Reverse Proxy: ⭐⭐⭐⭐⭐ - Superior proxy capabilities
• Modern Setups: ✅ - Preferred for new projects
• Documentation: ⭐⭐⭐⭐⭐ - Extensive and up-to-date
• Community: ⭐⭐⭐⭐⭐ - Large and active community

Apache2 Characteristics:

• Performance: ⭐⭐⭐ - Good but slower than Nginx
• Reverse Proxy: ⭐⭐⭐ - Capable but less efficient
• Modern Setups: ⚠️ - Older technology
• Documentation: ⭐⭐⭐⭐ - Comprehensive but dated
• Community: ⭐⭐⭐⭐ - Mature but smaller

Recommendation: Use Nginx unless you specifically need Apache2.

🏁 Final Recommended Production Stack

Nginx + SSL
├── Node.js (systemd)
└── FastAPI (systemd)

Why This Stack Works

• systemd: Native Linux service management
• Nginx: High-performance reverse proxy
• SSL: Automatic security with Let’s Encrypt
• Separation: Each service runs independently
• Monitoring: Built-in logging and restart capabilities

🔧 Advanced Systemd Techniques

Service Dependencies

[Unit]
Description=FastAPI Backend
After=network.target mysql.service
Requires=mysql.service

Resource Limits

[Service]
MemoryMax=512M
CPUQuota=50%

Health Checks

[Service]
ExecStartPost=/bin/bash -c 'curl -f http://localhost:8000/health || exit 1'

Multiple Instances

[Unit]
Description=FastAPI Worker %i

[Service]
ExecStart=/var/www/fastapi/venv/bin/gunicorn \
  -k uvicorn.workers.UvicornWorker \
  main:app \
  --bind 127.0.0.1:800%i

📊 Monitoring and Troubleshooting

Check Service Status

systemctl status fastapi

View Resource Usage

systemd-cgtop

Analyze Boot Performance

For comprehensive boot optimization, check out our systemd-analyze blame guide to identify slow services and optimize startup times.

Common Issues and Solutions

Service fails to start:

journalctl -u fastapi -n 50

Service not found:

sudo systemctl daemon-reload

Permission denied:

sudo chown -R deploy:deploy /var/www/fastapi

🚀 Best Practices Summary

DO ✅

• Research services before disabling
• Test changes in development first
• Use dedicated users for applications
• Set up proper logging and monitoring
• Configure crash protection
• Use absolute paths
• Document all configurations

DON’T ❌

• Run applications as root
• Ignore service dependencies
• Skip crash loop protection
• Forget to reload daemon after changes
• Use relative paths in ExecStart
• Ignore security implications

🎯 Quick Reference Commands

Service Management

# Start service
sudo systemctl start app

# Stop service  
sudo systemctl stop app

# Restart service
sudo systemctl restart app

# Enable auto-start
sudo systemctl enable app

# Check status
sudo systemctl status app

# View logs
journalctl -u app -f

Configuration Updates

# After editing service file
sudo systemctl daemon-reload
sudo systemctl restart app

Troubleshooting

# Check service logs
journalctl -u app -n 100

# Check boot performance
systemd-analyze blame

# Reset failed service
sudo systemctl reset-failed app

🔗 Internal Links

• Systemd Analyze Blame Guide - Optimize Linux boot performance
• Edge AI IoT Integration - Deploy AI applications on edge devices
• Web Performance Practical Guide - Optimize application performance

FAQ

What is systemd and why should I use it?

Systemd is Linux’s native service manager that automatically handles application lifecycle management. Use it because it’s built into Linux, provides better crash protection than PM2, handles auto-restarts on boot, and is the industry standard for production deployments.

How do I convert a PM2 application to systemd?

Find your application’s executable path with which node or which python3, create a .service file in /etc/systemd/system/, configure the ExecStart with the absolute path, set up restart policies, and enable the service with systemctl enable.

Can I run Node.js and Python applications together?

Yes, you can run multiple services simultaneously. Create separate .service files for each application, use different ports (e.g., Node.js on 3000, FastAPI on 8000), and use Nginx as a reverse proxy to route traffic to the appropriate service.

What’s the difference between Nginx and Apache2 with systemd?

Both work as reverse proxies, but Nginx generally offers better performance and is more modern. Apache2 is still viable but typically slower. Choose Nginx for new projects unless you have specific Apache2 requirements.

How do I add SSL to my systemd services?

Use Let’s Encrypt with Certbot. For Nginx: sudo certbot --nginx -d example.com. For Apache2: sudo certbot --apache -d example.com. The SSL certificates are managed by the web server, not systemd directly.

How do I debug a failing systemd service?

Use journalctl -u service-name -f to view real-time logs, systemctl status service-name to check the current state, and sudo systemctl daemon-reload after making configuration changes.

Is systemd better than PM2 for Node.js applications?

Yes, systemd is generally better for production Node.js applications because it’s Linux-native, provides superior crash protection, handles auto-restarts on boot more reliably, and doesn’t require Node.js to be installed just for process management. PM2 can still be useful for development environments.

What are the essential systemd commands I should know?

The most important commands are: systemctl start/stop/restart service-name, systemctl enable/disable service-name for auto-start, systemctl status service-name for checking state, journalctl -u service-name -f for logs, and systemctl daemon-reload after configuration changes.

How do I set environment variables in systemd services?

You can set environment variables inline using Environment=KEY=value lines in the [Service] section, or use EnvironmentFile=/path/to/env-file to load variables from a separate file. This is better than using .env files directly in your application.

What is crash loop protection in systemd?

Crash loop protection prevents infinite restart cycles using StartLimitIntervalSec and StartLimitBurst settings. For example, StartLimitIntervalSec=60 and StartLimitBurst=5 means systemd will stop trying to restart after 5 failed attempts within 60 seconds.

Conclusion

Systemd is the definitive solution for Linux service management, offering superior reliability, security, and performance compared to alternatives like PM2. By mastering systemd, you gain production-grade deployment capabilities that scale from simple applications to complex microservices architectures.

Start with basic service files, gradually add advanced features like resource limits and health checks, and always prioritize security by running applications as dedicated users. Combined with Nginx for reverse proxy and Let’s Encrypt for SSL, you have a complete, production-ready deployment stack.

The key is to start simple, test thoroughly, and iterate toward more complex configurations as your needs grow. With systemd managing your services, you can focus on building great applications while letting Linux handle the operational complexity.