Managing Logs in Linux: A Complete Guide to Viewing and Configuration

Introduction / Why This Matters

Logs in Linux are a detailed journal of system and application events. They are indispensable for diagnosing problems, security auditing, and monitoring. Without proper management, logs can grow uncontrollably, filling all available disk space and causing failures. This guide will help you master the full log management cycle: from viewing and monitoring to configuring automatic rotation, ensuring system stability and the preservation of important records.

Requirements / Preparation

Before you begin, ensure you have:

1. Access to a Linux terminal (Ubuntu, CentOS, Debian, or another distribution).
2. Superuser (sudo) privileges for some operations (viewing system logs, clearing, configuring logrotate).
3. Installed basic utilities: coreutils (contains tail, less, truncate), logrotate (usually pre-installed). For systemd-based systems, journalctl is available.
4. An understanding of which logging system is in use:
   • Traditional syslog: logs are text files in /var/log/ (e.g., /var/log/syslog, /var/log/auth.log).
   • systemd-journald: a binary journal managed via journalctl. Can be stored in /var/log/journal/ or in memory.

Step-by-Step Guide

Step 1: Exploring Log Structure and Location

First, you need to understand where logs are located in your system and how they are organized.

1. View the contents of the /var/log/ directory:

   ls -lh /var/log/
   

   
   You will see many files and directories: syslog, auth.log, kern.log, apache2/ (web server logs), nginx/, as well as journal/ directories (if journald is used) and .gz archives (compressed old logs).
2. Determine if journald is in use:

   systemctl status systemd-journald
   

   
   If the service is active (active (running)), the main logs can be read via journalctl, and text files in /var/log/ may be secondary or contain only critical messages.
3. Check which processes write to logs. For example, for nginx:

   ps aux | grep nginx
   

   
   In the output, look for the error_log parameter, which will indicate the path to its logs.

Step 2: Viewing Logs with tail and less

These two commands are the foundation for working with text logs (syslog).

• tail — shows the end of a file. Ideal for viewing recent events.

  # Last 10 lines of the system log
  sudo tail /var/log/syslog
  
  # Last 50 lines of the authentication log
  sudo tail -n 50 /var/log/auth.log
  
  # Follow new entries in real-time (Ctrl+C to exit)
  sudo tail -f /var/log/syslog
  

• less — an interactive viewer for large files. Allows scrolling, searching, and switching between files.

  sudo less /var/log/syslog
  

  
  Navigation in less:
  • Space / PgDown — next page.
  • b / PgUp — previous page.
  • /text — forward search. n — next match.
  • ?text — backward search.
  • q — quit.
• For journald, use journalctl:

  # Show all journal entries (like less)
  sudo journalctl
  
  # Last 50 entries
  sudo journalctl -n 50
  
  # Filter by service (e.g., nginx)
  sudo journalctl -u nginx.service
  
  # View since last boot
  sudo journalctl -b
  
  # Real-time monitoring (like tail -f)
  sudo journalctl -f
  

Step 3: Real-Time Log Monitoring

For live tracking of events (e.g., while debugging), use the follow mode.

1. For text logs (syslog):

   sudo tail -f /var/log/syslog | grep --line-buffered "ERROR"
   

   
   The | grep --line-buffered construct filters only lines containing "ERROR" in real-time.
2. For journald:

   # Follow all new entries
   sudo journalctl -f
   
   # Follow entries for a specific service
   sudo journalctl -u ssh.service -f
   
   # Follow entries with a specific priority (e.g., err)
   sudo journalctl -p err -f
   

3. Combined monitoring (if you need to watch both sources):

   # In two separate terminal windows, run:
   sudo tail -f /var/log/syslog
   sudo journalctl -f
   

Step 4: Cleaning Up Old and Unnecessary Logs

Caution: Cleaning system logs can remove information necessary for incident investigation. Only delete logs you are confident are no longer needed.

1. Safe truncation (clearing) of a specific file (the file remains, but its contents are zeroed out). This is the fastest way to free space.

   # Truncate the file to zero size
   sudo truncate -s 0 /var/log/syslog
   
   # Alternative method (also zeros the file)
   sudo echo > /var/log/syslog
   

2. Deleting old archived logs (.gz):

   # Delete all compressed logs older than 30 days in /var/log/
   sudo find /var/log/ -name "*.gz" -type f -mtime +30 -delete
   
   # Delete all log files except the main ones (caution!)
   sudo find /var/log/ -type f ! -name 'syslog' ! -name 'auth.log' -delete
   

3. Cleaning the journald journal (if it is used and consumes a lot of space):

   # Rotate the journal (similar to log rotation)
   sudo journalctl --rotate
   sudo journalctl --vacuum-time=1d  # Keep only entries from the last day
   
   # Clean the journal, freeing a specific amount of space (e.g., 100M)
   sudo journalctl --vacuum-size=100M
   

Step 5: Configuring Automatic Rotation with logrotate

logrotate is the standard tool for automatic log rotation, compression, and deletion. It is usually configured by default, but additional configuration may be needed for applications.

1. Main configuration file: /etc/logrotate.conf. It contains global settings (often include /etc/logrotate.d;).
2. Configurations for specific logs are placed in /etc/logrotate.d/. For example, for nginx, there might be a file named nginx.
3. Example configuration for your application (create the file /etc/logrotate.d/myapp):

   sudo nano /etc/logrotate.d/myapp
   

   
   Insert the configuration (replace /var/log/myapp/app.log with the path to your log):

   /var/log/myapp/*.log {
       daily              # Rotate daily
       rotate 7           # Keep 7 archives (7 days)
       compress           # Compress rotated logs (gzip)
       delaycompress      # Compress not immediately, but on the next rotation (so the current log remains readable)
       missingok          # Do not throw an error if the log file is missing
       notifempty         # Do not rotate empty files
       create 644 root root # Create a new log file with these permissions after rotation
       sharedscripts      # Run scripts (postrotate) once for all files in the block
       postrotate
           # Command to notify the application about rotation (e.g., reopen the file)
           # systemctl reload myapp.service > /dev/null 2>&1 || true
       endscript
   }
   

4. Test the configuration (without actually executing):

   sudo logrotate -d /etc/logrotate.conf
   

   
   The -d (debug) flag will show what actions logrotate plans to take.
5. Force a logrotate run (for testing or a manual scenario):

   sudo logrotate -f /etc/logrotate.conf
   

6. For journald, rotation is configured in /etc/systemd/journald.conf:

   [Journal]
   SystemMaxUse=100M   # Maximum size of the journal on disk
   SystemMaxFileSize=50M # Maximum size of a single journal file
   MaxRetentionSec=1month # Keep logs no longer than one month
   

   
   After making changes, restart the service: sudo systemctl restart systemd-journald.

Verification

1. After configuring logrotate:
   • Ensure compressed archives (.gz) of your application appear in /var/log/.
   • Check that the application continues writing to the new (non-archived) log file.
   • Look at file dates: ls -lh /var/log/myapp/.
2. For journald:

   sudo journalctl --disk-usage  # Shows current journal size
   sudo journalctl --list-boots  # Shows available boots (rotations)
   

3. General monitoring:

   df -h /var/log  # Ensure the partition is not full
   sudo du -sh /var/log/*  # See which logs are the largest
   

Potential Issues

• "Permission denied" when viewing/clearing logs:
  • Solution: Always use sudo for system logs. Check file permissions (ls -l /var/log/syslog). If an application writes to a log under its own user, ensure that user has write permissions.
• The /var/ disk is full, system is unstable:
  • Solution: Immediately clear the largest logs (Step 4). Then configure aggressive rotation via logrotate or journald (reduce rotate and SystemMaxUse). Consider moving /var/log to a separate partition.
• Application logs are not rotating via logrotate:
  • Check:
    1. The path to the log file in the /etc/logrotate.d/ config is correct.
    2. logrotate is being run (usually via cron: grep logrotate /etc/crontab /etc/cron.*/*).
    3. There are no typos in the config (check with logrotate -d).
    4. Your configuration is not overriding settings from the main logrotate.conf.
• Application stopped writing to log after rotation:
  • Solution: Most often, the application needs to be notified about the file change. In the postrotate script block, add a command to reopen the log (e.g., kill -USR1 <pid> or systemctl reload service). This is standard practice for many daemons (nginx, apache).
• Journald does not free space after vacuum:
  • Check if the journal is mounted in tmpfs (/run/log/journal). In this case, it is stored in memory and is cleared on reboot. For permanent storage on disk, ensure the /var/log/journal/ directory exists (sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald).