Linux System Resource Monitoring: A Comprehensive Guide

Introduction / Why This Is Needed

System resource monitoring is the foundation for diagnosing performance issues in Linux. Whether it's a server that's "sluggish" or a workstation with high memory consumption, understanding how CPU, RAM, disks, and network are used allows you to quickly pinpoint bottlenecks. This guide will introduce you to the basic and advanced tools available in most distributions. After completing this, you will be able to assess system status in real-time and collect historical data for analysis.

Requirements / Preparation

1. Access to a Linux terminal (locally or via SSH).
2. Superuser privileges (sudo) for installing packages and running some utilities (iotop, nmon, sar).
3. Basic commands (top, free, df) are usually pre-installed. For extended functionality, install:
   • htop — improved process monitoring.
   • iotop — disk I/O monitoring.
   • sysstat — suite of utilities including sar for historical data.
   • glances — cross-platform monitoring.
   • nmon — comprehensive monitoring of all resources (optional).

Installation on Ubuntu/Debian:

sudo apt update
sudo apt install htop iotop sysstat glances nmon

Installation on CentOS/RHEL/Fedora:

sudo yum install htop iotop sysstat glances nmon

Step-by-Step Guide

Step 1: Basic real-time monitoring with top

top is the standard utility present in any system. It shows active processes and overall load.

1. Run: top
2. Key metrics in the header:
   • %Cpu(s) — CPU core usage. us (user) — user processes, sy (system) — kernel, id (idle) — idle.
   • KiB Mem — RAM usage: total, used, free, buff/cache.
   • KiB Swap — swap usage.
3. Controls:
   • Press 1 — expand load per CPU core.
   • M — sort processes by memory.
   • P — sort by CPU (default).
   • k — terminate a process (requires PID).
   • q — quit.

⚠️ Important: top updates every 3 seconds. To change the interval, run top -d 5 (5 seconds).

Step 2: Enhanced interactive monitoring with htop

htop is a colorful, more convenient alternative to top with mouse support.

1. Run: htop
2. Advantages:
   • Color indicators: red — high CPU load, green — low.
   • Tree view of processes (F5).
   • Horizontal and vertical scrolling of the process list.
   • Quick process termination/prioritization via F9/F10.
3. Quick keys:
   • F2 — settings (display metrics, colors).
   • F3 — search for a process.
   • F4 — filter by name.
   • F6 — sort by any column.
   • F9 — kill a process (after selecting signal, usually SIGTERM).

Step 3: Universal overview of all resources with glances

glances is a cross-platform monitoring tool that aggregates CPU, memory, disk, network, container, and process data in one interface.

1. Run: glances
2. Main sections:
   • CPU — total and per-core load, frequencies.
   • MEM — RAM and swap usage.
   • LOAD — average load over 1, 5, 15 minutes.
   • NETWORK — incoming/outgoing traffic per interface.
   • DISK I/O — read/write speed.
   • PROCESSES — process list sorted by CPU/memory.
3. Interactive keys:
   • c — show/hide CPU.
   • m — memory.
   • d — disks.
   • n — network.
   • f — process filter (e.g., f python).
   • h — help.
4. Remote monitoring:
   On the target host: glances -s (starts server).
   On the local machine: glances -c <IP_address>.

Step 4: In-depth disk I/O analysis with iotop

iotop is similar to top but filters processes by disk I/O.

1. Run with sudo: sudo iotop
2. Columns:
   • DISK READ / DISK WRITE — read/write speed (K/s, M/s).
   • SWAPIN — process swap usage.
   • IO> — percentage of time the process spends on I/O operations.
3. Filtering:
   • a — show all processes (including kernel threads).
   • o — show only processes performing I/O.
   • p — specific PID (press p then enter).
4. Exit: q.

Step 5: Comprehensive monitoring of all resources with nmon

nmon (Nigel's Monitor) is a powerful tool for collecting data on CPU, memory, disks, network, virtual memory, and filesystems.

1. Run: nmon
2. Main screens (single-key switching):
   • c — CPU (total and per-core load, frequencies).
   • m — memory and swap.
   • d or dsk — disk devices and statistics.
   • n — network interfaces (incoming/outgoing traffic).
   • j — filesystems (inode usage, occupancy).
   • t — top processes by CPU/memory.
   • v — virtual memory (large pages).
3. Recording a session for analysis:
   nmon -f -s 2 -c 100 — record data every 2 seconds, 100 times. File saved as hostname_YYYYMMDD_HHMMSS.nmon.
   To view the recording: nmon -f -t -s 2 -c 100 (graphs) or use an online analyzer (e.g., nmon2rrd).
4. Exit: q.

Step 6: Historical analysis and statistics collection with sar

sar (System Activity Reporter) is part of the sysstat package. It collects system activity in the background and stores data in /var/log/sysstat/.

1. Activate data collection (if not active):

   sudo systemctl enable --now sysstat
   

   By default, sysstat collects data every 10 minutes.
2. View historical data:
   • Current data (if collection is active): sar -u (CPU), sar -r (memory), sar -b (disk I/O), sar -n DEV (network).
   • Data for a specific day: sar -u -f /var/log/sysstat/sa17 (where sa17 is the file for the 17th).
3. Key options:
   • -u — CPU usage.
   • -r — memory and swap usage.
   • -b — block transfer and read (disk).
   • -n DEV — network interface statistics.
   • -P ALL — load per all CPU cores.
   • -s HH:MM:SS — start time.
   • -e HH:MM:SS — end time.
4. Example: View average CPU load for today from 9:00 to 12:00:

   sar -u -s 09:00:00 -e 12:00:00
   

Verification

Successful monitoring is determined by your ability to:

1. In real-time, identify which process/core is loading the CPU (top, htop, glances).
2. See memory and swap usage, as well as processes consuming the most RAM (htop, glances, free -h).
3. Assess disk activity and find "gluttonous" processes (iotop, glances).
4. Obtain historical data for analyzing peak loads (sar).
5. Capture a complete system snapshot with one screenshot/session (nmon).

If all tools launch and display current data (e.g., CPU values change in top), monitoring is working.

Possible Issues

1. Utility not found (command not found)

Cause: Package not installed.
Solution: Install the corresponding package (see Requirements). For iotop and nmon, sudo may be required.

2. iotop runs but process list is empty or no permissions

Cause: iotop requires root privileges to read I/O statistics.
Solution: Run via sudo iotop. If the error persists, check if the blkio kernel module is loaded (usually loaded automatically).

3. sar does not show data for today

Cause: Data collection is not active or files for the current day haven't been generated yet (default collection at 23:50).
Solution:

• Check status: sudo systemctl status sysstat.
• Run collection manually: sudo sar -u 1 3 (collect every second, 3 times).
• For immediate write to today's file: sudo sar -u -o /var/log/sysstat/sa$(date +%d).

4. nmon displays only partial data or no network interfaces

Cause: The tool depends on available system metrics.
Solution: Ensure necessary files exist in /proc (e.g., /proc/net/dev for network). Update the kernel or install additional packages (e.g., sysstat for some metrics).

5. High CPU load from the monitoring utilities themselves

Cause: Some utilities (especially glances with many plugins) can consume resources.
Solution:

• In glances, disable unnecessary plugins (config ~/.config/glances/glances.ini).
• In top/htop, ensure you haven't activated very frequent updates (interval less than 0.5 sec).
• For long-term collection, use sar (minimal overhead).

6. Insufficient permissions to terminate a process (in htop/top)

Cause: Trying to terminate a process belonging to another user (especially root).
Solution: Use sudo to run the utility (sudo htop) or only terminate your own processes.