How to Harden SSH on a Linux Server With Key-Only Login (2026)?

I run a handful of small VPS nodes that I have learned to treat like production servers, and the single change that has cut noise and risk the most is forcing SSH to accept keys only. If you want to harden SSH on a Linux server with key-only login, this guide walks through the exact config I use, the mistakes I have made, and the order to do things so you never lock yourself out.

SSH hardening is the process of tightening the OpenSSH server so that only the people you want can get in, using only the methods you trust. The key idea behind key-only login is simple: you keep a private key file on your laptop, the server holds the matching public key, and authentication happens with asymmetric cryptography. No password ever crosses the network, so brute-force attacks against your SSH port stop working overnight.

Critical safety note: Before disabling password authentication, open a second SSH session and confirm it stays connected. If something breaks, that backup session is your way back in. If you skip this step on a remote box, you can lock yourself out completely.

Quick reference: sshd_config settings for key-only SSH

This table is the cheat sheet I keep open while editing. Every entry is a directive you can paste into /etc/ssh/sshd_config; the value column reflects what I run on production boxes in 2026.

Prerequisites before you start

You need three things: a Linux server you can reach over SSH, a user account with sudo rights, and a local machine (Linux, macOS, or Windows with OpenSSH) where the private key will live.

Verify the OpenSSH server is installed and running.

On Debian or Ubuntu:

sudo apt update
sudo apt install -y openssh-server
sudo systemctl enable --now ssh

On RHEL, CentOS, Rocky, or AlmaLinux:

sudo dnf install -y openssh-server
sudo systemctl enable --now sshd

Make sure your account has sudo access before you go further. Run sudo whoami and check that it prints root.

How to harden SSH on a Linux server with key-only login?

Follow these steps in order. The list is the same one I run on a fresh VPS, and each step builds on the one before it so you always have a way back in.

Step 1: Generate an Ed25519 key pair on your local machine

Ed25519 is the modern default: short keys, fast signatures, and strong resistance to timing attacks. The Reddit r/sysadmin community and the OpenSSH project itself recommend Ed25519 over RSA unless you have a specific compatibility reason.

Run this on your laptop, not on the server:

ssh-keygen -t ed25519 -C "yourname@$(hostname)" -f ~/.ssh/id_ed25519

You will be asked for a passphrase. Set one. A passphrase encrypts the private key at rest, so a stolen laptop file does not become an instant server compromise. Most guides skip this, and I think that is bad advice.

Step 2: Copy your public key to the server

The ssh-copy-id tool appends your public key to the server’s ~/.ssh/authorized_keys file and fixes the right permissions along the way.

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

If you cannot use ssh-copy-id (Windows without it, or a non-standard port), paste the key manually.

cat ~/.ssh/id_ed25519.pub | ssh [email protected] "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Step 3: Test key-based login in a second session

Open a brand new terminal window and SSH in. Do not close your existing session. You should be prompted for your key passphrase, not the server account password.

ssh -i ~/.ssh/id_ed25519 [email protected]

If you want passwordless logins, add the key to ssh-agent on your laptop so you only unlock once per reboot.

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

Confirm the second session works and only then move on. If you skip this test, you are gambling with your access.

Step 4: Back up sshd_config and edit a copy

Always keep a copy of the working config before you change anything. One bad keystroke can lock you out, and the backup is what saves you.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.$(date +%F)

Now edit the live file. I prefer sudo nano for small servers and sudo vi for anything that might be hardened against GUI tools.

sudo nano /etc/ssh/sshd_config

Apply the three changes that turn the server into key-only mode:

PubkeyAuthentication yes
PasswordAuthentication no
PermitRootLogin no

Add the rest of the directives from the quick reference table above. If a line is commented out (starts with #), either uncomment it or append a new line with the value you want.

Step 5: Validate the config and restart sshd

Before the restart, run a syntax check. This is the single most important line in the whole guide because it tells you whether the daemon will even start.

sudo sshd -t && echo "config ok"

If that prints config ok, restart the service.

On Debian or Ubuntu:

sudo systemctl restart ssh

On RHEL, CentOS, Rocky, or AlmaLinux:

sudo systemctl restart sshd

Check that the daemon is listening and active.

sudo systemctl status sshd --no-pager

Step 6: Confirm key-only login works

Open a third terminal and try to log in. The session should succeed with your key passphrase and fail outright if you force a password via ssh -o PubkeyAuthentication=no.

ssh -o PubkeyAuthentication=no [email protected]

A Permission denied (publickey) message confirms the server is now rejecting passwords. Anything else means your config did not take, and you should revisit the syntax check.

Harden SSH further: firewall, port, access, and fail2ban

Key-only login blocks the most common attack, but I treat it as the baseline, not the finish line. The next layers reduce noise, shrink the attack surface, and catch the few probes that manage to get through.

Lock down the SSH port with a firewall

Restrict TCP 22 to your own IP if the server is personal, or to a small CIDR range for a team. UFW (Ubuntu) is the simplest:

sudo ufw allow from 203.0.113.10 to any port 22 proto tcp
sudo ufw reload

For nftables on RHEL-family systems:

sudo nft add rule inet filter input tcp dport 22 ip saddr 203.0.113.10 accept

Should you change the default port?

Moving to a non-standard port like 5022 does not stop a targeted attacker, but it does cut drive-by scanners to almost zero. On the forum threads, most admins say it is worth doing for noise reduction alone. Edit Port in sshd_config and open that port in your firewall before restarting.

Allow only specific users

Whitelisting is the safest way to control who can SSH in. Add one line per user.

AllowUsers deploy alice

For groups, use AllowGroups sshusers and add each login user to the sshusers group.

Set idle and login limits

Idle sessions left on a laptop are a real risk. The two settings below close any connection that has been silent for 10 minutes.

ClientAliveInterval 300
ClientAliveCountMax 2

Pair them with a short LoginGraceTime 30 and MaxAuthTries 3 to cut the time window for any brute-force attempt.

Disable forwarding features you do not use

X11, TCP, and agent forwarding are convenient, but they expand what a compromised account can do. If you do not need them, turn them off.

X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no

Add fail2ban to soak up the leftover probes

Even with key-only auth, the SSH port still gets hit by bots trying older passwords. fail2ban watches the auth log and blocks repeat offenders. On Ubuntu:

sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban

Create a local config file at /etc/fail2ban/jail.local:

[sshd]
enabled = true
port = ssh
maxretry = 3
findtime = 600
bantime = 3600

Reload fail2ban and confirm the jail is active.

sudo systemctl reload fail2ban
sudo fail2ban-client status sshd

Keep the server updated and watch the logs

Enable unattended security updates on Debian/Ubuntu or dnf-automatic on RHEL-family systems. Forward auth logs to a remote syslog server or a SIEM if you can, and skim /var/log/auth.log (or /var/log/secure) once a week.

Optional: add a second factor

For high-value servers, layer Google Authenticator on top of the SSH key. The private key proves “this is the laptop”, and the TOTP code proves “this is the human”. This is the only section most beginner guides skip; I think it is the most valuable for any box that touches customer data.

Troubleshooting: fixing key-only SSH when it breaks

Most lockouts come from one of five issues. Walk through them in order before you panic.

File permissions and ownership

This is the single most common cause of “my key works once then fails” or “SSH still asks for a password”. The server rejects the key if any permission is too open.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_ed25519
chown -R $USER:$USER ~/.ssh

The server-side ~/.ssh directory must be owned by the user, not root, and the home directory should not be writable by anyone else.

Verify the public key actually landed

Log in with your still-working password (or backup session) and confirm the key is in ~/.ssh/authorized_keys on one line, with no extra spaces or line breaks.

cat ~/.ssh/authorized_keys

Run sshd in debug mode

To see exactly why the server is rejecting the key, stop the daemon and run a debug instance on a different port.

sudo systemctl stop sshd
sudo /usr/sbin/sshd -d -p 2222

From your laptop, connect to that port and read the matching error.

ssh -p 2222 -v [email protected]

When you are done, restart the real service.

sudo systemctl start sshd

Check SELinux or AppArmor

On RHEL-family systems, a mislabeled home directory can block sshd from reading authorized_keys. Restore the default context.

restorecon -R ~/.ssh

Recover from a complete lockout

If you have no working SSH session, most cloud providers give you a console (VNC or serial) where you can log in as root and revert /etc/ssh/sshd_config from your backup.

sudo cp /etc/ssh/sshd_config.bak.2026-08-13 /etc/ssh/sshd_config
sudo systemctl restart sshd

That is why the backup file from step 4 matters.

Frequently Asked Questions

Conclusion: your key-only SSH checklist

That is the full workflow to harden SSH on a Linux server with key-only login. The order is the part most guides get wrong: generate the key, copy it, test in a second session, edit the config, run sshd -t, restart, and confirm. Once that is in place, layer the firewall, fail2ban, idle timeouts, and updates on top.

Pick one server you maintain this week and run the steps against it. Keep that backup session open until you have confirmed a fresh key-only login, and you will never have to recover from a lockout.