Protecting SSH: Keys Off Disk, in a Password Manager

For years my SSH key lived where everyone’s does: ~/.ssh/id_ed25519, mode 600, maybe with a passphrase. That was fine when the only things running as me were programs I’d chosen. It isn’t fine any more. A coding agent runs as you: it can read every file you can, run any command you can, and reach the network. The realistic attack isn’t an intruder, it’s prompt injection. A README, an issue or a web page contains instructions, the agent follows them, and your private key ends up in a curl request.

So I’ve adopted one rule: no secrets at rest. Private keys and tokens live in a password manager, are unlocked into memory, and reach tools through an agent socket. The difference that matters:

• A key in a file can be copied. Once it’s exfiltrated, it’s gone, and you’ll find out when it’s used.
• A key in an agent can only be used, while the agent is unlocked, and each use can require a click. An approval prompt you didn’t expect is the alarm going off.

The setup: Bitwarden as the SSH agent

Bitwarden’s desktop app (Linux, macOS, Windows) includes an SSH agent. 1Password has an equivalent; the shape is the same.

1. Desktop app → Settings → Enable SSH agent. Choose how often it asks for approval: every use, or until the vault locks.

2. New → SSH key. Generate an Ed25519 key there, or import your existing one from the clipboard.

3. Point SSH_AUTH_SOCK at its socket. The path depends on how the app was installed, so my shell config just takes the first one that exists:

   for s in ~/.bitwarden-ssh-agent.sock \
            ~/snap/bitwarden/current/.bitwarden-ssh-agent.sock \
            ~/.var/app/com.bitwarden.desktop/data/.bitwarden-ssh-agent.sock \
            ~/Library/Containers/com.bitwarden.desktop/Data/.bitwarden-ssh-agent.sock; do
     [ -S "$s" ] && export SSH_AUTH_SOCK=$s && break
   done

   These are the .deb/AppImage/direct-download macOS, snap, flatpak and Mac App Store paths, in that order.

4. Check it: ssh-add -l lists the key, and ssh -T git@github.com pops up an approval prompt.

5. Delete the private key from ~/.ssh. Keep the .pub, though. It’s useful, as the next part shows.

Tell ssh which key, by its public half

With several keys in one agent (personal, work), ssh offers them in whatever order the agent gives them, and GitHub happily logs you in as the wrong account. Point -i at the public key and add IdentitiesOnly, and ssh asks the agent for exactly that one:

# ~/.gitconfig
[core]
  sshCommand = "ssh -i ~/.ssh/id_ed25519.pub -o IdentitiesOnly=yes"

The same works in ~/.ssh/config as IdentityFile ~/.ssh/work.pub plus IdentitiesOnly yes per host.

On GNOME, take the agent slot properly

Ubuntu’s GNOME session runs its own agent (gcr-ssh-agent) and exports SSH_AUTH_SOCK to every app it launches. A shell-level override works for shells, but anything started from the desktop, or a terminal multiplexer started before the override, still talks to the empty GNOME agent. Hand the session over instead:

# ~/.config/environment.d/10-ssh-agent.conf
SSH_AUTH_SOCK=${HOME}/snap/bitwarden/current/.bitwarden-ssh-agent.sock

# GNOME's agent is enabled globally, so mask it for your user
systemctl --user mask --now gcr-ssh-agent.socket gcr-ssh-agent.service

Log out and back in, and the whole session uses the password manager.

Errors that confused me

Permission denied (publickey) straight after setting it up. The terminal had been open before the change, so it still pointed at the old agent, which had no keys. exec zsh, or a new tab, fixes it. Run echo $SSH_AUTH_SOCK first whenever SSH is acting strangely.

WARNING: UNPROTECTED PRIVATE KEY FILE! Permissions 0644 for 'id_ed25519.pub' are too open. This one is a red herring. It means the agent you’re connected to doesn’t have this key: ssh couldn’t get it from the agent, so it fell back to reading the .pub file as if it were a private key. Don’t chmod 600 it. That hides the warning and fixes nothing. Check SSH_AUTH_SOCK and ssh-add -l instead. The vault is locked, or you’re on the wrong agent.

Using the key from other devices

My desktop is always on and I reach it from a laptop and a phone (see Home Dev Box over Tailscale). If git push on the desktop uses the desktop’s password manager, the approval prompt appears on a screen I can’t see. So the key should come from the device in front of me.

• Laptop: run the same password-manager agent there, then ssh -A <user>@<machine>. Agent forwarding lends the remote shell your local agent, and approvals appear on the laptop.
• Phone: Bitwarden’s mobile app isn’t an SSH agent, so I generated a separate key for the phone in the SSH client (Termius), registered its public half with GitHub, and turned on agent forwarding for the host. One key per device means a lost phone costs one revocation, not a rotation everywhere.
• mosh doesn’t forward agents. It’s great for typing over a bad connection; use plain ssh when you need the key.

One trap I built for myself: my shell config always overrode SSH_AUTH_SOCK with the local password manager’s socket, which silently threw away the forwarded agent. The fix is to only use the local socket when there isn’t a live one already:

# keep a forwarded agent (ssh -A) or a multiplexer's agent symlink; skip GNOME's empty one
if [ -S "$SSH_AUTH_SOCK" ] && [ "${SSH_AUTH_SOCK:A}" != "$XDG_RUNTIME_DIR/gcr/ssh" ]; then
  : # a live agent is already here: use it
else
  : # ...pick the local password manager's socket (the loop above)
fi

${SSH_AUTH_SOCK:A} is zsh for “resolve symlinks”, because multiplexers like tmux setups and herdr keep a stable symlink to whichever agent was attached last.

Agent forwarding isn’t free: while you’re connected, anyone with root on the remote machine can use your agent. To hop through a box, prefer ssh -J jump target (ProxyJump), which never exposes the agent to the jump host. Only forward to machines you control, and keep approval prompts on.

The server side

Keys are half the story. On the machine you SSH into:

• Don’t expose port 22 to the internet at all. I use Tailscale SSH (tailscale up --ssh): logins are authorised by my identity on a private network and audit-logged, with no port forwarding and no authorized_keys to manage. Check mode adds a browser re-authentication before a session.
• Keys only for the regular OpenSSH server: PasswordAuthentication no and PermitRootLogin no in a file under /etc/ssh/sshd_config.d/.
• Sign commits with the same SSH key (git config gpg.format ssh, user.signingkey ~/.ssh/id_ed25519.pub, and add it to GitHub as a signing key). A leaked push token then can’t produce “Verified” commits in your name.
• Encrypt the disk. Full-disk encryption covers swap, caches, browser profiles and every secret you forgot about. Choose it at install time; adding it later means a reinstall.

Belt and braces for AI agents

Even with nothing left to steal, tell agents not to look. In Claude Code, deny rules in ~/.claude/settings.json:

"permissions": { "deny": [
  "Read(~/.ssh/**)", "Read(**/.env)", "Read(**/.env.*)", "Read(~/.aws/**)", "Read(~/.config/gh/**)"
] }

Then add the Bash sandbox, a devcontainer for repos you don’t trust, and reading every curl, ssh or git push an agent proposes before you approve it. The same thinking extends past SSH: API tokens belong in the password manager and get injected per command (bws run -- npm run dev, op run -- …) instead of sitting in .env files. If a secret has leaked into a repo already, see Removing Secrets from Git History.

The summary is short: a secret on disk is a secret one prompt injection away from leaking. Put it in an agent, make every use ask, and give each device its own key.