Master syslog in Linux: Setup, Monitoring, and Log Analysis

Introduction / Why This Matters

Syslog is the standard protocol for collecting and storing system logs in Unix-like operating systems. Understanding syslog is critical for system administrators and developers: it enables system monitoring, problem diagnosis, security tracking, and performance analysis. In this guide, you will learn about syslog architecture, how to configure the rsyslog daemon, and how to work with logs in practice. Upon completion, you will be able to effectively track events in Linux, configure logging for your applications, and maintain order in system journals.

Prerequisites / Preparation

Before you begin, ensure you have:

• Access to a Linux system (Ubuntu, CentOS, Debian, or another) with superuser (sudo) privileges.
• The rsyslog package installed (it is included by default in most distributions; if not, install it using the instructions below).
• Basic terminal skills and experience editing text files (e.g., with nano or vim).
• Understanding of fundamental Linux commands (e.g., ls, cat, grep).

Step 1: Syslog Architecture

Syslog consists of several key components that work together to collect, route, and store logs:

• Syslog daemon (e.g., rsyslog or syslog-ng): a background process that accepts messages from applications, the kernel, and network sources, then filters and directs them to specified destinations (files, databases, remote servers).
• Configuration files: define message processing rules. The main file is /etc/rsyslog.conf, and additional configurations can be placed in /etc/rsyslog.d/ (files with a .conf extension).
• Logging levels: each message has:
  • Facility (source category): e.g., auth (authentication), cron (cron jobs), local0–local7 (user-defined categories), kern (kernel).
  • Severity (importance level): from debug (debugging) to emerg (critical error). The order is: debug, info, notice, warning, err, crit, alert, emerg.
• Transport: syslog supports network transmission (UDP/TCP, typically port 514) and local storage in files.

💡 Tip: Modern distributions use rsyslog by default—a fast, extensible implementation with TLS support, filtering, and modules. For understanding the basics, it is sufficient.

Step 2: Installing and Starting rsyslog

Although rsyslog is often pre-installed, check its presence and install it if necessary.

For Debian/Ubuntu:

sudo apt update
sudo apt install rsyslog

For RHEL/CentOS (8+):

sudo dnf install rsyslog  # or sudo yum on older versions

After installation, start the daemon and enable it to run at boot:

sudo systemctl enable --now rsyslog

Check the status:

systemctl status rsyslog

You should see active (running). If the daemon is not running, execute sudo systemctl start rsyslog.

Step 3: Configuration Basics

The main configuration file is /etc/rsyslog.conf. Open it for viewing (e.g., sudo nano /etc/rsyslog.conf). Pay attention to the following elements:

• Modules: at the beginning of the file, modules may be loaded (e.g., module(load="imuxsock") for local socket support).
• Global directives: such as $FileOwner syslog, $FileGroup adm (owners of log files).
• Rules (selectors and actions): lines like:

  *.info;mail.none;authpriv.none;cron.none /var/log/messages
  

  This rule means: all messages with severity info and higher (from all facilities except mail, authpriv, cron) should be written to the file /var/log/messages.

Rule syntax: <selector> <action>

• Selector: facility.severity (e.g., auth.err), wildcards * (all facilities) and combinations via ; can be used.
• Action: file path (starts with /), remote server (@server for UDP, @@server for TCP), command (via |), database, etc.

⚠️ Important: After any configuration changes, restart rsyslog: sudo systemctl restart rsyslog. To check syntax, use sudo rsyslogd -N1 (test run without actually starting).

Step 4: Creating a Custom Logging Rule

Suppose you want to log messages from your application, which uses facility local0, to a separate file /var/log/myapp.log.

1. Create a separate configuration file in /etc/rsyslog.d/ (recommended for user rules to avoid modifying the main file):

   sudo nano /etc/rsyslog.d/myapp.conf
   

2. Add the line:

   local0.* /var/log/myapp.log
   

   
   This rule redirects all messages with facility local0 (any severity) to the specified file.
3. Create the log file and set correct permissions (rsyslog usually creates the file on first access, but you can pre-create it):

   sudo touch /var/log/myapp.log
   sudo chown syslog:adm /var/log/myapp.log  # owner and group may differ (e.g., `root:syslog`)
   sudo chmod 640 /var/log/myapp.log
   

4. Restart rsyslog:

   sudo systemctl restart rsyslog
   

Step 5: Testing with the logger Command

Use the logger utility to send test messages to syslog. It allows specifying facility and severity.

Send a message with facility local0 and severity info:

logger -p local0.info "Test message from my application"

Check if it appeared in the log:

tail -f /var/log/myapp.log

You should see a line similar to:

Feb 15 10:30:00 hostname username: Test message from my application

If the message did not appear:

• Ensure the rule in myapp.conf is correct (no typos).
• Verify rsyslog has been restarted.
• Confirm your application actually uses facility local0 (you may need to configure the application itself, e.g., in a configuration file or code).

Step 6: Viewing, Analyzing, and Rotating Logs

Viewing and Analyzing Logs

For monitoring and searching logs, use:

• tail: follow the latest lines in real time.

  tail -f /var/log/myapp.log  # follow your log
  tail -f /var/log/syslog | grep "error"  # filter by keyword
  

• grep: search by pattern.

  grep "local0" /var/log/syslog
  

• journalctl: if the system uses systemd-journald, it can show both its own logs and those forwarded from rsyslog (if configured).

  journalctl -u rsyslog -f  # follow rsyslog's own logs
  journalctl -f | grep "local0"  # filter by facility
  

💡 Tip: For advanced analysis (statistics, graphs), consider tools like awk, sed, or systems like ELK (Elasticsearch, Logstash, Kibana) or Graylog.

Log Rotation

Over time, log files can grow to enormous sizes. Rotation automatically archives old logs and creates new ones.

Rsyslog often integrates with logrotate. Configuration for standard logs is typically in /etc/logrotate.d/rsyslog:

/var/log/syslog
/var/log/mail.log
/var/log/kern.log
/var/log/auth.log
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Key parameters:

• rotate 4: keep 4 archived files (e.g., syslog.1, syslog.2.gz, etc.).
• weekly: rotate weekly (can be daily, monthly).
• compress: gzip compress archives.
• postrotate: script after rotation that notifies rsyslog about renamed files (important for correct continued logging).

For your custom log /var/log/myapp.log, add a separate block to the same file or create a new one in /etc/logrotate.d/:

/var/log/myapp.log {
    rotate 12
    monthly
    compress
    delaycompress
    missingok
    notifempty
    create 640 syslog adm
}

Test rotation manually:

sudo logrotate -f /etc/logrotate.conf  # force rotation

Verification

After completing all steps, ensure the configuration works correctly:

1. Rsyslog daemon is active: systemctl is-active rsyslog should return active.
2. Log file exists and is accessible: ls -l /var/log/myapp.log.
3. Test message is written: run logger -p local0.info "Another test message" again and check tail -f /var/log/myapp.log.
4. Rotation is configured: check logrotate configuration (sudo cat /etc/logrotate.d/rsyslog and your file) and ensure parameters meet requirements.
5. No errors in rsyslog logs: grep -i error /var/log/syslog (or journalctl -u rsyslog).

If all points are satisfied, you have successfully configured syslog for your application and learned how to work with system logs.

Troubleshooting

• Access error when writing to log file: Ensure rsyslog has write permissions. Check file ownership and permissions (ls -l /var/log/myapp.log). Log files should typically belong to group adm or syslog and have permissions 640. If the file was created manually, use sudo chown syslog:adm /var/log/myapp.log and sudo chmod 640 /var/log/myapp.log.
• Messages not reaching the log: Check configuration file syntax with sudo rsyslogd -N1 (output will show errors). Ensure your application uses facility local0 (sometimes it must be explicitly set in the application's configuration, e.g., local0 in syslog settings). Also verify the rule is loaded (the file in /etc/rsyslog.d/ has a .conf extension).
• Conflict with systemd-journald: If journald is configured to store logs (Storage=persistent in /etc/systemd/journald.conf), it may intercept messages before rsyslog. Ensure ForwardToSyslog=yes is set in journald.conf (default is often yes). After changes, restart journald: sudo systemctl restart systemd-journald.
• Logs not rotating: Check that logrotate configuration includes your file (file in /etc/logrotate.d/). Ensure the postrotate script correctly notifies rsyslog (the path /usr/lib/rsyslog/rsyslog-rotate may differ; on some systems it is /usr/sbin/rsyslog-rotate). Run sudo logrotate -d /etc/logrotate.conf for a test run without applying changes.
• Too many unnecessary messages in logs: Configure more precise selectors. For example, use local0.warn instead of local0.* to log only warnings and errors. Or exclude unwanted facilities via ; (as in the /var/log/messages example).