Manage Multiple Servers with SSH Keys: A Setup Guide

Introduction

As I manage more cloud servers and VPS instances, I also use more AI tools. I wanted a consistent way for my AI agents to help maintain multiple Linux servers without repeatedly typing IP addresses, ports, and passwords.

The approach

SSH key authentication and SSH config aliases offer a practical balance between security and convenience. An alias turns a connection into a simple command such as ssh server-demo.

SSH configuration is stored locally in ~/.ssh. File-transfer tools and AI tools with access to the local terminal can use the same configuration, provided their execution environment has the necessary access.

This guide uses macOS to connect to Linux servers. All addresses, usernames, and names are examples: replace server.example.com with your server's IP address or domain. The steps do not create a remote user; use an existing account permitted to log in over SSH.

Add a new server in five steps

Choose an alias, address, SSH port, and login account before you begin:

The local alias is simply a name that is easy for you to remember.

1. Generate a local key

Run these commands in your local terminal:

mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t ed25519 -f ~/.ssh/server_demo_ed25519 -C "server-demo"

Set a passphrase when prompted. If the file already exists, do not overwrite it; check its purpose or choose another name.

• server_demo_ed25519 is the private key. Keep it on your device; never share it or commit it to Git.
• server_demo_ed25519.pub is the public key you install on the server.

One key can be used on multiple servers. This guide uses a separate key per server so each can be rotated or revoked independently. Generate separate keys for separate local devices rather than copying private keys between them.

2. Install the public key on the server

First confirm that you can sign in to the account using a password or another existing method. Then run locally:

ssh-copy-id -i ~/.ssh/server_demo_ed25519.pub -p 22 [email protected]

On the first connection, verify the host key fingerprint through a trusted channel such as your provider's console before accepting it. When prompted, enter the server account password, not the passphrase you just set for your private key.

If ssh-copy-id is unavailable on macOS, Homebrew users can run brew install ssh-copy-id. Alternatively, use the provider's console to append the entire line from the .pub file to the target account's ~/.ssh/authorized_keys; do not overwrite existing keys. If password login is disabled from the start, install the public key using an existing key, the cloud provider's key-injection feature, or its console.

Install the public key for the account you intend to use. A key installed for root does not automatically authorize ubuntu or another account.

If public-key authentication is disabled on the server

OpenSSH normally allows public-key authentication by default. If the server explicitly disables it, installing a key alone will not enable key login. The following commands run on the server and require administrator privileges and an existing way in, such as a password, another key, or the cloud provider's console.

Keep your current session open and confirm that you can recover access through the console before changing anything. Do not disable password authentication or close the working session until the new login has been verified.

1. Check the effective configuration for the target account. Replace ubuntu with that account and 192.0.2.10 with the client's public IP address from which the SSH connection originates, not the server's address:

sudo sshd -T -C user=ubuntu,addr=192.0.2.10 | grep -Ei '^(pubkeyauthentication|authorizedkeysfile|authorizedkeyscommand|authenticationmethods) '

If pubkeyauthentication yes appears, public-key authentication is already allowed for that scenario; there is no need to change that setting. This alone does not confirm that a key is installed or that login will succeed: also check key locations, permissions, and account restrictions. If matching rules depend on the client hostname or local address, add the applicable host or laddr conditions. If the value is no, continue.

2. Find and change the effective setting. Check /etc/ssh/sshd_config, its Include files, and applicable Match blocks so the setting for the target account becomes:

PubkeyAuthentication yes

Do not blindly append it to the end of a file: an earlier included file or a Match block may affect which value applies. Preserve the existing AuthorizedKeysFile, AuthorizedKeysCommand, and other login policies. If the server uses a key source other than the default ~/.ssh/authorized_keys, install the key according to that configuration.

3. Validate and reload the configuration. On the server, run:

sudo sshd -t

Fix any errors before reloading. On Ubuntu or Debian, the service is commonly reloaded with sudo systemctl reload ssh; on RHEL-family systems, it is commonly sudo systemctl reload sshd. Use the actual service name on your system. Run the sshd -T -C check again to confirm the effective setting.

4. Back on your local machine, confirm that the target account's public key was installed in step 2. After configuring the SSH alias below, open a new terminal window and test ssh server-demo. Only after the new connection succeeds should you consider changing password authentication. If it fails, use the troubleshooting section below. AuthenticationMethods may require an additional authentication factor, so PubkeyAuthentication yes alone does not guarantee login.

Reference: OpenSSH sshd_config manual.

3. Add an SSH alias

Edit your local ~/.ssh/config and append the following while preserving existing entries:

Host server-demo
    HostName server.example.com
    User ubuntu
    Port 22
    IdentityFile ~/.ssh/server_demo_ed25519
    IdentitiesOnly yes
    AddKeysToAgent yes

Host is the alias you type; HostName is the real address; IdentityFile points to the private key. IdentitiesOnly yes limits SSH to configured identities so it does not try many unrelated agent keys.

On macOS, add this line at the end of the same Host block:

    UseKeychain yes

UseKeychain is an extension in Apple's OpenSSH. Do not copy it directly to Linux, Windows, or other OpenSSH builds. If the file has a general Host * block, place the specific host block before it: for most options, SSH uses the first value it reads.

Set local permissions:

chmod 600 ~/.ssh/config ~/.ssh/server_demo_ed25519

4. Save the private-key passphrase on macOS

Use the system tool to store the passphrase in Keychain and load the key into the agent:

/usr/bin/ssh-add --apple-use-keychain ~/.ssh/server_demo_ed25519

The private-key passphrase protects the local key; the server password is a different authentication method. The agent and Keychain reduce repeated prompts to unlock the key.

5. Connect and verify

First connect interactively and confirm the account, host fingerprint, and permissions:

ssh server-demo

Exit the remote session, then test non-interactive public-key authentication from your local terminal:

ssh -o BatchMode=yes -o PreferredAuthentications=publickey server-demo 'whoami; hostname; uptime'

A successful result confirms public-key authentication works in this terminal environment. It does not guarantee that another user, background process, or AI execution environment can access the same SSH agent.

Repeat the five steps for each new server, changing the alias, address, port, account, and key filename.

Everyday connections and file management

Once configured, ssh, scp, and rsync over SSH can all use the same alias:

# Connect
ssh server-demo

# Check load and disk space
ssh server-demo 'uptime; df -h'

# Upload to the remote account's home directory
scp ./example.txt server-demo:~/

# Download to the current directory
scp server-demo:~/example.txt ./

# Preview synchronization; remove --dry-run to execute
rsync -av --dry-run ./project/ server-demo:~/project/

rsync must be installed at both ends. The trailing / on the source means “the contents of this directory.” This example does not delete extra files on the server and does not require the port to be specified again.

As the number of servers grows, keep a private inventory of aliases, purposes, accounts, and public-key fingerprints. Do not store passwords or private keys in it. To see a local public-key fingerprint:

ssh-keygen -lf ~/.ssh/server_demo_ed25519.pub

To rotate a key, install the new public key and verify that it works before removing the old public key's complete line from the server's authorized_keys. Deleting only the local private key does not revoke the server-side authorization.

Troubleshooting

Check which configuration SSH actually uses

ssh -G server-demo | grep -E '^(hostname|user|port|identityfile|identitiesonly) '
ssh -vvv server-demo

Do not disable host-key checking to suppress warnings, and do not regenerate keys as the first response to every failure.

Check the server

Using an existing login or the provider's console, check the following as the target login account:

ls -ld ~ ~/.ssh
ls -l ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Confirm ownership is correct and the home directory is not writable by other users. An administrator can use sudo sshd -T to inspect server configuration, especially PubkeyAuthentication, AuthorizedKeysFile, and authentication restrictions. If there are Match rules, check the real user and connection source too.

Before changing server SSH settings, retain the current session and a console recovery route. Run sudo sshd -t to validate changes, then reload the appropriate service for your distribution. Verify key login from another terminal before considering disabling password authentication.

Let AI tools reuse the connection

An AI tool with access to the local terminal can use the same SSH alias. For example:

Use the local SSH alias server-demo to inspect system load, memory, and disk usage in read-only mode and summarize any problems. Do not change configuration or restart services, and do not read or print private keys.

Its execution environment must be able to read the configuration, use the agent, and reach the server. An ordinary chat window or isolated cloud environment does not automatically gain access to your local SSH setup; you do not need to paste a private key into an AI tool.

Key authentication determines who can sign in; the remote account's permissions determine what they can do afterward. Use a restricted account for long-running automation. For deployments or data changes, define the scope, backup, and rollback plan.

For future reference, the sequence is: generate a local key → install the public key on the server → add a Host alias → unlock the key → test the connection.

References

• OpenSSH: ssh-keygen
• OpenSSH: ssh_config
• Local macOS documentation: man ssh-add and man ssh_config.