Setting up an SSH Server on Linux: A Complete Security Guide

Introduction / Why This Is Needed

SSH (Secure Shell) is the standard for secure remote server management and data transfer. By default, after installation, the SSH server (sshd daemon) runs with settings that are insecure for public access: port 22 is open, superuser root login with a password is allowed, and there is no protection against brute-force attacks.

This guide will help you transform a vulnerable SSH server into a secure gateway. You will learn how to:

• Install and run the service.
• Change the default port.
• Restrict the set of allowed users.
• Critically important: disable password authentication in favor of SSH keys.
• Block direct root login.
• Add brute-force protection with Fail2ban.
• Configure the firewall correctly.

After completing these steps, the risk of unauthorized access to your server will be reduced by an order of magnitude.

Prerequisites / Preparation

Before you begin, ensure you have:

1. Server access with sudo privileges. You must be able to execute commands as the superuser.
2. OpenSSH client installed on your local machine (ssh).
3. A backup of the current configuration file. Create it manually:

   sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%F)
   

4. An alternative way to access the server (console via hosting provider control panel, IPMI/KVM), in case you lock yourself out.
5. Your SSH public key already added to ~/.ssh/authorized_keys on the target server. If not — add it before disabling password authentication.

Step 1: Installing and Verifying the OpenSSH Server Service

If the SSH server is not yet installed, start here.

For Ubuntu/Debian:

sudo apt update
sudo apt install -y openssh-server

For CentOS/RHEL/Rocky/AlmaLinux:

sudo dnf install -y openssh-server
# Or for older versions:
# sudo yum install -y openssh-server

After installation, the service usually starts automatically. Let's check the status:

sudo systemctl status ssh
# On CentOS/RHEL the service name might be `sshd`
sudo systemctl status sshd

You should see active (running). If not, start it and enable autostart:

sudo systemctl start ssh
sudo systemctl enable ssh
# Or for sshd:
# sudo systemctl start sshd
# sudo systemctl enable sshd

Ensure the daemon is listening on port 22 (default):

sudo ss -tlnp | grep :22

The output should contain a line with LISTEN and the sshd process.

Step 2: Backing Up and Editing the Main Configuration

All core settings are stored in /etc/ssh/sshd_config. Never edit this file without a backup first.

Open the file in a text editor (e.g., sudo nano /etc/ssh/sshd_config or sudo vim /etc/ssh/sshd_config) and make the following changes. Uncomment (remove the # at the start of the line) the necessary parameters and set the values.

Key Directives for Security:

# 1. Change the default port (e.g., to 2222). This complicates automated scans.
Port 2222

# 2. Prohibit root user login.
PermitRootLogin no

# 3. Restrict which users can connect.
# List logins separated by spaces.
AllowUsers your_username another_user

# 4. Enable SSH key authentication and DISABLE password authentication.
PubkeyAuthentication yes
PasswordAuthentication no

# 5. Disable empty passwords (if PasswordAuthentication is yes for some reason).
PermitEmptyPasswords no

# 6. Limit the number of simultaneous sessions (optional but useful).
MaxSessions 2
MaxAuthTries 3

# 7. Configure timeouts (closes inactive connections).
ClientAliveInterval 300
ClientAliveCountMax 2

Important: If you change the port (Port), do not delete the line for port 22 if you plan to keep the old port temporarily for debugging. It's better to comment it out (#Port 22) so the old port is not listening. After a successful connection test on the new port, you can delete the Port 22 line.

Step 3: Configuring the Firewall

Changing the port is useless if the firewall doesn't open the new one. Configure it according to your distribution.

For Ubuntu/Debian (UFW):

# Allow the new port (in our example 2222/tcp)
sudo ufw allow 2222/tcp

# Remove the old rule for port 22 if it exists
sudo ufw delete deny 22/tcp 2>/dev/null || true
sudo ufw status verbose

For CentOS/RHEL/Rocky (firewalld):

# Add the service/port to the public zone
sudo firewall-cmd --permanent --add-port=2222/tcp
# Remove the old port if needed
sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

Verify that the port is actually open from the outside (from another machine):

nc -zv your_server_IP 2222
# Or
nmap -p 2222 your_server_IP

Step 4: Installing and Configuring Fail2ban for Brute-Force Protection

Fail2ban scans logs of services (including SSH) and blocks IP addresses that make too many failed login attempts.

Installation:

# Ubuntu/Debian
sudo apt install -y fail2ban

# CentOS/RHEL/Rocky (EPEL repository may be required)
sudo dnf install -y fail2ban

Basic Configuration:

Fail2ban works with "jails". Create a local config that won't be overwritten on update:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Edit /etc/fail2ban/jail.local:

[DEFAULT]
# Ban time (seconds). 1h = 3600
bantime = 3600
# Number of attempts before a ban
maxretry = 5
# Backend to use (auto-detection)
backend = systemd

# SSH-specific settings (on your port!)
[sshd]
enabled = true
port = 2222  # Specify your port! If you leave it as `ssh`, fail2ban will determine the port from sshd_config.
filter = sshd
logpath = /var/log/auth.log  # For Ubuntu/Debian
# For CentOS/RHEL: logpath = /var/log/secure

For CentOS/RHEL, also set logpath = /var/log/secure.

Start and enable Fail2ban:

sudo systemctl start fail2ban
sudo systemctl enable fail2ban

Check the jail status:

sudo fail2ban-client status sshd

The output will show the current number of banned IPs.

Step 5: Applying Changes and Testing the Connection

Critically important stage. Do not disconnect your current active SSH session until testing is complete!

1. Check the SSH configuration syntax:

   sudo sshd -t
   

   
   If the output is empty — the syntax is correct.
2. Reload the SSH service:

   sudo systemctl reload ssh  # or sshd
   

3. From a NEW terminal (do not close the old one!) try to connect to the server:

   # Specify your port (-p) and username
   ssh -p 2222 your_username@your_server_IP
   

   
   The connection should succeed only with an SSH key. Password entry should end with the error Permission denied (publickey).
4. If the connection was successful — now you can disconnect the old session (if it was using a password) and work only through the new, secure one.
5. Verify that the old port 22 is no longer listening (if you removed it from the config):

   sudo ss -tlnp | grep :22
   

   
   The command should produce no output.

Verifying the Result

1. Connection works only with an SSH key.
2. Attempted root login fails with an error.
3. Connection on the old port 22 (if removed from config) is refused.
4. Fail2ban is active: try to make 6 failed login attempts (e.g., enter a wrong password 6 times, if PasswordAuthentication is temporarily enabled for some reason). Then check the list of banned IPs:

   sudo fail2ban-client status sshd
   

5. Firewall opens only the required port (2222) and blocks 22.

Potential Issues

❌ Error Connection refused or No route to host

• Cause: The firewall does not allow the new port or the SSH service is not running.
• Solution: Check the service status (systemctl status ssh) and firewall rules (sudo ufw status or sudo firewall-cmd --list-all). Ensure you opened the port on the external interface.

❌ Error Permission denied (publickey,password) during test connection

• Cause 1: Your public key (~/.ssh/id_rsa.pub) is not added to ~/.ssh/authorized_keys on the server.
• Solution: Add the key using ssh-copy-id -p 2222 your_username@server or manually.
• Cause 2: Incorrect permissions on files/directories on the server.
• Solution: On the server, run:

  chmod 700 ~/.ssh
  chmod 600 ~/.ssh/authorized_keys
  

• Cause 3: You enabled PasswordAuthentication no but are trying to connect with a password.
• Solution: Connect with a key. If the key is lost — you will need console access to recover it.

❌ I Locked Myself Out (Fail2ban or Wrong Port)

• Solution: You absolutely need alternative access (hosting console, KVM). Through it:
  1. Check if your IP is banned: sudo fail2ban-client status sshd.
  2. If yes — unban it: sudo fail2ban-client set sshd unbanip your_IP.
  3. Check the /etc/ssh/sshd_config config for errors (sudo sshd -t).
  4. Restart SSH: sudo systemctl restart sshd.
  5. Check the firewall.

❌ Fail2ban Doesn't Work on CentOS/RHEL or Blocks Everything

• Cause: Incorrect logpath. On CentOS, SSH logs are written to /var/log/secure, not /var/log/auth.log.
• Solution: In /etc/fail2ban/jail.local for [sshd], set logpath = /var/log/secure.

❌ sshd Fails to Start After Port Change

• Cause: The port is already in use by another process or is specified incorrectly (e.g., a letter character).
• Solution: Ensure the port is free: sudo ss -tlnp. Check the sshd_config syntax (sudo sshd -t). Ensure the port is numeric and within the 1-65535 range.

❌ SELinux/AppArmor Blocks SSH on a Non-Standard Port

• Cause: SELinux by default only allows sshd to listen on port 22.
• Solution (for SELinux):

  sudo semanage port -a -t ssh_port_t -p tcp 2222
  

  (The policycoreutils-python-utils package may be required).
• Solution (for AppArmor): Usually not required, but if issues arise, check the profile at /etc/apparmor.d/usr.sbin.sshd.