One constant source of amazement for me is people not using ssh keys / using passwords with ssh.
Especially at a BigCo, where there are different environments, with different passwords, and password expiry/rotation/complexity rules.
Like, when asking for help, or working together... you say to them "ok, lets ssh to devfoo1234", and they do it, and then type in their password, and maybe get it wrong, then need to reset it, or whatever... and it takes half a minute or more for them to just ssh to some host. Maybe there are several hosts involved, and it all multiplies out.
I mention to them "you know... i never use ssh passwords, i don't actually know my devfoo1234 password... maybe you should google for ssh-keygen, set it up, let me know if you have any problems?" and they're like "oh yeah, thats cool. i should do that sometime later!".... and then they never do, and they are forever messing with passwords.
Since a few years now I only ever use SSH private keys safely hidden behind a HSM with a tinier than tiny attack surface: Yubikeys do it for me (but other vendors would work too). My SSH keys do not have a password but when I log in using SSH, it requires me to physically touch my Yubikey (well one of my Yubikeys).
Windows has great support, surprisingly, for TPM-backed sk keys using Windows Hello and OpenSSH. Protected with physical presence and anti-hammering at the hardware level, and easy to setup by just selecting a sk type key.
I only use password keys for things that need to be scripted.
Keys is great for individual use, or for company use if you have centralized key control (and issue one or more keys per user).
Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines.
Passwords at least are "simple" for people to work with.
> Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines
That hasn't been my experience at all. I've never encountered ssh key sharing in any environment, that would be insane.
We enforced different ssh keys per environment at my previous company: fingerprint of your key would get logged, and if the SIEM detected a reuse of keys across environments (dev, test, prod, etc) you'd get a stern talking to.
Unregulated/decentralized SSH key usage (i.e. allowing ssh-copy-id) is a dream for hackers to move laterally through a network. That's why many orgs disable it, and otherwise haven't invested resources in getting a proper centralized CA/authz server set up.
Every couple of months someone re-discovers SSH certificates, and blogs about them.
I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually.
Things like deploying dev keys to various production environments, instead of generating/registering them within said environment.
One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "secure" read-only account to the other agency's SQL server (which uses the same TLS 1.3 as HTTPS). This is from person in charge of digital security for a government org.
One key technological cause is that PKCS#12 standardizes a format (you've most likely seen it as .PFX files) in which a certificate and its associated private key are bundled. This is in an effort to simplify the software...
So you get a situation where the lay person is given a "certificate" but it's not really just the certificate it's a PFX file and so e.g. no they mustn't show it you, it has their private key inside it and so you will learn that key and if you're honest you've just ruined their day because they need to start over...
I would say in my career I've had at least two occasions where I did that and I felt awful for the person, because I had set out to help them but now things are worse, and I've had a good number of later occasions where I spent a lot more of their time and mine because I knew I need to be very sure whether their "certificate" is actually a certificate (which they can show me, e.g. Teams message me the file) or a PFX file (thus it has their private key) and I must caution them to show nobody the file yet also try to assist them.
Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.
Certs may still be the right approach, but OpenSSH also supports an AuthorizedKeysCommand which could be a secure HTTPS request to a central server to pull down a dynamically generated authorized_keys file content for the particular user and host.
If your endpoints can securely and reliably reach a central server, this gives you maximum control (your authorized_keys HTTPS server can have any custom business logic you want) without having to deal with certs/CAs.
Exactly. This is really useful in larger organizations where you may want more complex rules on access. For example, you can easily build "break glass" or 2nd party approved access on demand. You can put whatever logic you need in a CA front-end.
You can also make all the certs short-lived (and only store them in ram).
And when your or someone else's infra down to such a degree that you need SSH access, you do not want to depend on being able to touch that machine first. The same is true with custom AuthorizedKeysCommands that phone home.
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though.
I assume you gathered a lot of thoughts over these 15 years.
A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
Adding to this: while certs are indeed well-supported by OpenSSH, it's not always the SSH daemon used on alternate or embedded platforms.
For example, OpenWRT used Dropbear [1] instead, which does not support certs. Also, Java programs that implement SSH stuff, like Jenkins, may be doing so using Apache Mina [2] which, though the underlying library supports certs, it is buggy [3] and requires the application to add the UX to also support it.
You can just replace dropbear with openssh on OpenWRT. That was one of the first things I did, since DropBear also doesn't support hardware backed (sk) keys. Just move it to 2222 and disable the service.
I reenabled DB on that alt port when I did the recent major update, just in case, but it wasn't necessary. After the upgrade, OpenSSH was alive and ready.
Might actually be a positive instead of a negative. Gaming use-cases should have not any effect on security policies, these should be as separate as possible, different auth mechanisms for your gaming stuff and your professional stuff ensures nothing gets mixed.
Hah? It being my gaming machine has nothing to do with the problem. It’s also my FPGA development machine, though it gets used less for that. It only happens to be the only Linux workstation in my home (the others are Macs or OpenBSD).
If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two.
In environment where they don't cause frustration they're not worth it.
Not really more to it than that, from my point of view.
It's no different compared to regular SSH private keys. You need to protect it from compromise.
However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.
Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.
It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.
Can't really speak to the point of the guy you're replying to, but the FreeIPA implementation via SSSD does more than just Kerberos tickets. Actually, I think the Kerberos based stuff as it relates to SSH is GSSAPI as part of sshd itself and has little to do with sssd, though I could be wrong.
That said...
If I'm remembering things correctly (and it's been a looong while since I've played with this), FreeIPA's client configures sshd with an AuthorizedKeysCommand that executes a program that queries sssd for the list of authorized keys for a given user. Sssd then uses a plugin to query the LDAP server @ FreeIPA to get the list of keys.
There's also SSHFP (I think) records in DNS if you're using FreeIPA's DNS servers. These provide the host keys for servers for your ssh client to check against. Not sure if that's integrated into ssh itself or something else -- I can't remember how it's implemented offhand -- but it's fairly nifty since you never see the TOFU prompt (or it would be if DNS was actually secure, anyway).
Yes, FreeIPA is Kerberos+LDAP+X.509 CA, and GSSAPI is in OpenSSH (normally with the key exchange patch). SSSD is a local mechanism, not network authentication. I mentioned authorized keys distribution mechanisms elsewhere, but I was thinking authentication (c.f. OIDC), not authorization.
Yes, but its authN components only act locally, and PAM is optional for sshd. It can/does call out to network services like Kerberos/LDAP given a password, of course, but I was thinking of network authN connected directly with OIDC somehow, for which I don't know a mechanism in vanilla OpenSSH. (I don't know what Authentik does for this -- I could imagine it's behind the scenes somehow.) I should probably look it up sometime.
It depends on what you want to do. CA certs are easy to manage, you just put the CA key instead of the SSH public key in authorized_keys.
They also provide a way to get hardware-backed security without messing with SSH agent forwarding and crappy USB security devices. You can use an HSM to issue a temporary certificate for your (possibly temporary) public key and use it as normal. The certificate can be valid for just 1 hour, enough to not worry about it leaking.
Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters).
There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
Despite the drawbacks of its grassroot nature TOFU goes a looooong way.
With my own machines I can just physically check that the server host key matches what the ssh client sees. Once TOFU looks good I'm all set with that host because I don't change any of the keys ever.
In a no-frills corporate unix environment it's enough to have a list of the internal servers' public keys listed on an internal website, accessible via SSL so it's effectively signed by a known corporate identity. You only need to check this list once to validate carrying out TOFU after which you can trust future connections.
In settings with huge fleet of machines or in a very dynamic environment where new machines are rolled out all the time it probably makes things easier to use certificates. Of course, certificates come with some extra work and some extra features so the amount of benefit depend on the case. But at this scale TOFU is breaking down bad on multiple levels so you can't afford a strong opinion against certificates, really.
I wish web browsers could remember server TLS host keys easily too and at least notify me whenever they change even if they'd still accept the new keys via ~trusted CAs.
I've been using SSH since 1996 and I've never once seen anyone actually verify a public key manually like that. If they can MITM your SSH server, why can't they MITM your internal web site? In common practice, TOFU is type "yes" and YOLO.
I work in a corporate setting and the money and time we wasted because of Zscaler and its SSL inspection [1] is beyond your wildest imagination. Whenever I see a "SSL certificate problem: self-signed certificate in certificate chain" error, I know I'm in trouble.
Zscaler was deployed on a Windows 11 machine at a place that a friend worked at. When I assessed the software, its behavior was downright evil like malware. As we all know, it injects its own root certificate into the operating system in order to conduct man-in-the-middle attacks on TLS/HTTPS connections to monitor the user's web activity.
Furthermore, it locks down the web browser's settings so that you cannot use a proxy server to bypass Zscaler's MITM. I saw this behavior in Mozilla Firefox, where the proxy option is set to "No proxy" and all other options are disabled and grayed out; I imagine that it does the same to Google Chrome. If you try to modify the browser's .ini(?) file for proxy settings, Zscaler immediately overwrites it against your will. Zscaler worked very hard to enforce its settings in order to spy on the computer user.
And as you'd expect, if you open up the Zscaler GUI in the system tray, you are presented with the option to disable the software if you have the IT password. Which of course, you don't have. Then again, that might be an epsilon better than the Cybereason antivirus software, which just has a system tray icon with no exit option, and cannot be killed in Task Manager even if you are a local administrator, and imposes a heavy slowdown if you're open hundreds of small text files per second.
I feel the same way about Cisco Umbrella where I work.
The worst breakage by far is protocol breakage; basically anything that uses HTTP as a basis for building some other protocol gets broken all the time. None of the people implementing it seem aware. They buy the vendor's claim that it's "transparent", when in fact even "inspect/trace-only" modes often break all kinds of shit.
and probably some other things I'm forgetting. When this breakage is reported, the first round of replies is typically "I visited that domain in my enterprise-managed browser and it's not blocked". That is, of course, a useless and irrelevant test.
Often it takes hours to even fully diagnose the breakage with enough confidence to point the finger at that tool and not some other endpoint security tool.
I'm not sure if the people buying and deploying tools in this category don't know how much stuff it breaks or just don't care. But the breakage is everywhere and nobody seems prepared for it.
The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious.
Also, I've never had a security issue due to TOFU, have you?
Certificates provide extra features, like revocation.
However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work.
TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax.
The secure way to use SSH is to never use TOFU but to pair the user and the server by copying the public keys between the 2 computers through a secure channel, e.g. either by using a USB memory or by sending the public keys through already existing authenticated encrypted links that pass through other computers. (Such a link may be a HTTPS download link.)
When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
> secure channel, e.g. either by using a USB memory
Aside: We need a "safely dumb" storage plug standard. Right now the flexibility of USB is a double-edged sword, any USB device could potentially be malicious, waiting for the right moment to send keystrokes as a keyboard etc.
Or use the same plug for everything, but support a "passive storage only" that can be enforced by a special adapter or cable.
Just to make it clear: this does not mean that it is fine to blindly accept the message on first use.
The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?
Like I have said the secure way requires the secure copying of both keys before the first connection attempt
The server public key must be copied into "known_hosts" on the client, while the client public key must be copied into "authorized_keys" on the server.
When this is the procedure that is always followed, any message shown by SSH about an unknown host means that the connection must be aborted, because the identity of the server is unknown.
You cannot truly verify the "fingerprint" displayed by SSH, unless you simultaneously have access to another computer, where you have a copy of the fingerprint. What is usually meant by "verifying" is that you remember a few digits of the fingerprint, and those match.
You could have copied the fingerprint from the server, to be able to truly verify it, but that does not make sense, because in that case you should have copied the entire key, not just the fingerprint, and you should have installed it in the client.
When you use only authentication with digital signatures, it does not make sense to use any other procedure, because you must make at least one of the two copies anyway, so when copying the client key to the server you can take the server key, to carry it back to the client.
The TOFU method is meant to be used together with password-based authentication, in less secure applications, where no physical access to the server is required for setting up SSH on the client.
By "less secure" I mean for example applications equivalent to HTTPS, where the client is not really authenticated, e.g. when providing a public password allowing read-only access to an SSH server through Internet.
Sure, I just wanted to make sure that nobody would understand "TOFU is neither necessary nor recommended, just ignore that message and say "yes" when it appears".
Both ends need to be sure of who is on the other end, and there are different ways to achieve that. The way SSH works is that if you haven't copied the server public key locally, it will explicitly ask you to verify it the first time.
I am not sure that "SSH does TOFU". SSH asks you to verify. The human who YOLOs it and approves it without checking is the one doing TOFU, and this is not really secure.
> When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
That is not the case, and is a major advantage of certificates.
What you say is extremely wrong and this misconception is very dangerous.
When an authenticated connection between a client and a server (or any other 2 computers) is established, before the connection can be initiated both ends must already have a piece of information that has been copied to them through a secure channel, otherwise the authentication is impossible.
The 2 pieces of information may allow direct reciprocal authentication, i.e. the client may have the server certificate and the server may have the client certificate.
It is also possible to have indirect authentication, e.g. the server sends the server certificate to the client and the client sends the client certificate to the server during the initiation of the connection.
However, for indirect authentication, both the client and the server must had been previously paired with a server hosting the certification authority.
This means that the root certificate of the CA must have been transferred through secure channels to the client and to the server, and also that the client certificate and the server certificate must have been transferred through secure channels to the CA for signing and then back to the client and the server.
So setting up the computers for indirect authentication through certificates requires significantly more transfers through secure channels than setting them up for direct authentication.
The advantage of indirect authentication appears in an organization with a great number of computers, where indirect authentication reduces the number of secure pairings that must be done from N*(N-1)/2 to N.
However this has nothing to do with certificates. Indirect authentication can be done without certificates, like in Kerberos, or like in SSH if you have a management computer that you pair by copying the public_keys with every computer in the network. Later, whenever you want to enable some client to connect to a SSH server you use the management computer to copy securely the corresponding public_keys between the server and the client.
So no, there is no advantage of certificates related to reducing the number of transfers that must be done through secure side channels, like copying through a portable memory.
Certificates provide other features, like temporary validity and revocation, which can be useful in an enterprise context.
What misconception? You essentially admit that I'm right in this comment but surround it with a bunch of other unrelated words.
If you use SSH certificates, then you have to put the SSH CA cert on each server once, and you have to give signed certs back to each client once each time you sign a cert (which can be once per client or once per connection or anywhere in between).
You do not have to copy anything new to each server when you sign new client certs. Likewise if you sign host keys, you don't have to copy a new host key to your client when you add a new server.
Sure, you could get the same kind of trust relationships with other tools. But SSH certificates provide them and are an advantage of SSH certificates over traditional SSH public key auth.
It honestly feels like you're just dropping this thread into an LLM and asking it to write a disagreeing response.
Not just your ISP. If an attacker slipped a device onto your LAN and also you happened to be sshing to a new box for the first time then TOFU poses a problem. But that's an awfully limited attack surface. It's similar to the difference between leaking a fax while it's sent versus leaking years old emails that are just sitting there on an internet accessible server.
As for your ISP I think you should never rely on TOFU over the public internet. If you really don't want to do ssh certs it's easy enough to make the host key available securely via https.
If you have some form of access to set up the CA config on the box before connecting then you can use the same access channel to avoid needing to rely on TOFU for setting up the key access all the same.
This can be anything from being part of the install script to customized deployment image to physical access to access via a host in virtualized scenarios.
TOFU only really comes into play when the box is already set up and you have no other way to load things onto the box other than connecting via SSH to do so. But, again, that would be the same story if you were intending to go the certificate approach too.
Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates.
If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory.
Using certificates does not add any simplification or any extra security.
For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory.
When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control.
Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation.
In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used.
Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user.
Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.
Touche.. actually a good point, but actually those are two different situations.
With one, I'm accessing a website and trusting that the certificate is signed by someone I trust; so the trust in my browser certificates (which include certificates from hundreds of certificate authorities all over the world, any one of which could be compromised, robbed, or controlled by an adversarial person or even government) is extended to the site that I'm visiting. To say this is weak sauce rather understates how bad this actually is. (To paraphrase Churchill, this is the worst possible design, except for all the rest.)
With the other, I'm logging into a server for the first time (and I could simply deploy the same trusted host key to all my ssh servers via an autoscaling configuration or whatever). I think it's debatable if TOFU is worse or better than your (granted clever) metaphor.
(to those who'd recommend userify, yes - great for the client login issue and definitely increases security, but to parent's point, TOFU is still needed unless you want to distribute host pubkeys)
Pairing is absolutely necessary for bidirectional authentication, where each party must verify the identity of the other end.
To visit this site, there is no pairing, because the site does not know who I am.
In order to verify the identity of the HN site, I must trust that the maintainers of the installation packages of the browsers that I use (Firefox, Vivaldi, Chromium) have ensured that the built-in certificates have reached me through a secure path. This actually requires much more trust than when someone answers "yes" to the SSH unknown host message.
If I use certificates for accessing e.g. the network of my employer, then my work computer must be paired with some corporate server, i.e. a unique certificate has been generated for myself and it has been copied to some certificate authority server for signing and then to my computer, and also a certificate of the local certificate authority has been copied to my personal computer.
While pairing is unavoidable for bidirectional authentication, it is not necessarily direct between the end points. Both end points must have been paired with at least one other computer but they need not have been paired between themselves previously if there exists some path through secure connections that have been originally created by pairing.
When certificates are used, usually the pairings are not done directly between end points, but each computer must be paired with the server hosting the certificate authority.
The term "pairing" is not used frequently, but it should have been preferred, because frequently the users do not understand which are the exact actions on which the security of their communications depend, which leads to various exploits. The critical security actions are those that perform the pairing.
"Pairing" of 2 systems, e.g. A and B, means that some information must be transmitted through a secure channel from A to B and some other information must be transmitted through a secure channel from B to A. An alternative pairing method is to generate both pieces of information on one of the 2 systems and transmit both of them through a secure channel to the other. The information exchange channels must already be secure, because before pairing authentication is impossible.
The pairing between a PC and the server hosting the certificate authority can be done in various ways, depending on where the PC certificate is generated. If the certificate is generated at the certificate authority than both it and the root certificate must be copied through a secure channel to the PC. If the certificate is generated on the PC, it must be sent through a secure channel to the CA for signing, then it must be sent back also through a secure channel.
In practice, administrators are not always careful enough for the channels through which certificates are copied to be really secure. For instance they may be copied through network links that are not yet authenticated, which is equivalent with the TOFU method optionally used by SSH.
In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.
In my view it is more important to stop using software keys so probably use sk (fido) for both host and user.. From there CAs would be a next step.. The level of documentation and example setups is astoundingly poor if you even look at step 2 for any feature. I.e. SK keys are reasonably understood for user keys but the setup as host keys is vague and needs testing to see if it really works.
SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management.
The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow.
Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once.
Common friction points:
1. your signer that has to be up and correct all the time
2. trust roots everywhere (and drifting)
3. TTL tuning nonsense (too short = random lockouts, too long = what was the point)
4. limited on-box state makes debugging harder than it should be
5. failures tend to fan out instead of staying contained
Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough.
What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it.
We went the opposite direction:
1. nodes pull over outbound HTTPS
2. local authorized_keys is the source of truth locally
3. users/roles are visible on the box
4. drift fixes itself quickly
5. no inbound ports, no CA signatures (WELL, not strictly true*!)
You still get central control, but operation and failure modes are local instead of "everyone is locked out right now."
That’s basically what we do at Userify (https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn.
And the part that usually gets hand-waved with SSH CAs:
1. creating the user account
2. managing sudo roles
3. deciding what happens to home directories on removal
4. cleanup vs retention for compliance/forensics
Those don’t go away - they're just not part of the certificate solution.
* (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)
With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes.
I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day.
I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile.
Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?
Instead of using a CA, why not set the key's PIN policy to "once" and use an agent (e.g. https://github.com/FiloSottile/yubikey-agent/) that holds an active session to the yubikey? You start the agent at the beginning of the day, enter the PIN once, and then stop the agent at the end of the day.
Wanted to avoid having the key in USB slot all the time. I have the version that sticks out and i carry it on my key chain. So it's easy to break in a laptop.
The experience might be better right up until you're running it in prod and someone happens to ask about:
Cert revocation (or even expiration)
Sudo roles
User removal and process termination
Is the cert server HA and locked down
How you log in when the cert server is down or under attack (rich target!)
How to easily add Alice to server group A, Bob to B, and Carlos to both A and B, and then to remove them..
(disclaimer we're celebrating our 15th anniversary at https://Userify.com, but those are actually legit concerns and not only a sales pitch. You certainly can build a solid and secure ssh cert infra, but doing it in production is just not an easy set-it-and-forget-it sort of thing.)
On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common.
Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
We added SSH certificates support to pico.sh [1] and it's been great. Utilizing principals gave us the ability to implement a RBAC like system for specific parts of the pico.sh ecosystem. Users get the flexibility they want with limited complexity.
shameless plug: I've been tinkering with a tool to make SSH certificate-based login a bit easier. it's called Sshifu.
basically, you set up a sshifu-server that acts as a certificate authority + SSO server. then on your SSH servers, you configure them to trust this CA (there are helper npx commands / bash scripts to make this easy).
after that, for each user who wants access, they just run:
npx sshifu <sshifu-server-url> <ssh-server-url>
this starts the SSO login flow, sets up the CA public key if needed, and immediately opens an SSH session.
npx is just the easiest way to get started, there are other install options too.
I built this as a smaller alternative to Smallstep / Teleport. it's still very early and mostly vibe-coded, but it's already scratching my own itch.
>then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key.
This sentence is a bit of a red flag, it looks like the author is making a (subtle) mistake in the category of too much security, or at least misjudging the amount of security (objectively measurable entropy) needed. This is of course a less consequential error than too little entropy/security measures, but still if one wants to be a cybersecurity professional, especially one with influence, they must know exactly the right amount needed, because our resources are limited, and each additional bit of entropy and security step not only costs time of the admin that implements it, but of the users that have to follow it, and this can even impact security itself by fatiguing the user and causing them to circumvent measures or ignore alerts.
On to specifically what's wrong:
Either a key file or a password can be used to log in to a server or authenticate to any service in general. Besides the technical implementation, the main difference is whether the secret is stored on the device, or in the user's brain. One is not more correct than the other, there's a lot of tradeoffs, one can ensure more bits and is more ergonomic, the other is not stored on device so it cannot be compromised that way.
That said a 2FA approach, in whatever format, is (generally speaking) safer than any individual method, in that the two secrets are necessary to be granted access. In this scenario one needs both the file and the password to authenticate, even if the password is 4 digits long, that increases the security of the system when compared to no password. An attacker would have to setup a brute force attempt along with a way to verify the decryption was successful. If local decryption confirmation is not possible, then such a brute force attack would require to submit erroneous logins to the server potentially activating tripwires or alerting a monitoring admin.
There's nothing special about the second factor authorization being equal or equivalent in entropy to the first, and there's especially no requirement that a password have more entropy when it's a second authorization, in fact it's the other way around.
tl;dr You can consider each security mechanism in the wider context rather than in isolation and you will see security fatigue go down without compromising security.
Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.
We're in the process of updating the experience to this century! ;)
We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
Life is easier if you can use Kerberos SSO, i.e. GSSAPIAuthentication in OpenSSH. (If we're talking certificates, presumably it is OpenSSH, or does anything else implement them?)
I don't want to have to get a special purpose credential when I have a TGT which can work generally, and is at least required for secure remote filesystem access.
You have to manage extra infrastructure for certificates and, as a user, have the friction of firing up a JavaScript-enabled web browser via an additional tool, assuming "real IdP" means using OIDC. Unfortunately that flow is actually needed for remote systems and something like Edugain federation, since Moonshot/IETF ABFAB failed, but at least Shibboleth can use the TGT, and it's not the Globus horror.
Every serious shop of any real size is already managing an OIDC IdP (you need one for whatever SAAS apps your team is using along with any internal web applications you're using). Why not just link it to something that can issue short-lived SSH certificates? That's also the cleanest way to get strong multifactor auth for SSH (certificates issued only through an OIDC progress minted with MFA requirements).
Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
I'm happy for anyone who doesn't have MS Windows/Active Directory -- so Kerberos -- in their organization, but I'd need (Free)IPA or similar for user/access management anyway. Certificates are an extra layer of SSH-specific complexity, which concerns me for security even if it doesn't involve some third party. MFA is needed once a day, say, for SSO to all Kerberized services. [As I understand it, "managing an OIDC IdP" includes shipping the contents of Active Directory to Entra, heaven help us.]
> Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
Microsoft (if that means anything, but they've done good work) and Red Hat obviously disagree, along with decades' experience. It is malpractice not to secure NFS mounts (and other network filesystems with sensitive data), and that means Kerberos.
If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction.
Sure you need your signing service to be reasonably available, but that’s easily accomplished.
That sounds like a lot of extra steps. How do I validate the authenticity of a signing request? Should my signing machine be able to challenge the requester? (This means that the CA key is on a machine with network access!!)
Replacing the distribution of a revocation list with short-lived certificates just creates other problems that are not easier to solve. (Also, 1h is bonkers, even letsencrypt doesn't do it)
Honestly, we used to replace a lot of pam_ldap and similar sorts of awful solutions. With those, if your LDAP went down even for a heartbeat, you couldn't log in at all.
So I totally agree: if I had to do certificates and didn't have something like Userify, a 1 hour (or even shorter if possible) expiration seems quite worth chasing, especially with suitable highly available configuration. (Of course, TFA doesn't even bother mentioning revocation and expiration, which should give you a clue as to how much fun those are lol)
And for more normal, lower-security requirements or non-HA, 6 or 8 hours or so would probably work and give you plenty of time for even serious system outages before the certs expired.
Not to hard shill or anything (apologies in advance, just skip if you're not interested), but there are two significant security and reliability differences between standard SSH (with or without certificates) and Userify:
1. Userify Cloud updates by default every three minutes, and on-premise Userify Express/Enterprise updates every ten seconds, but it doesn't have to update at all; even if your Userify server goes offline forever, you can still log in because the accounts are standard UNIX accounts (literally created with `useradd`)
2. When accounts are removed, Userify also completely nukes the user account, removes its sudo perms, and totally kill -9 's any tmux/screen/etc sessions (all processes owned by the user are terminated across the entire enterprise within seconds), which is also not something that a certificate expiration would ever do.
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.
One constant source of amazement for me is people not using ssh keys / using passwords with ssh.
Especially at a BigCo, where there are different environments, with different passwords, and password expiry/rotation/complexity rules.
Like, when asking for help, or working together... you say to them "ok, lets ssh to devfoo1234", and they do it, and then type in their password, and maybe get it wrong, then need to reset it, or whatever... and it takes half a minute or more for them to just ssh to some host. Maybe there are several hosts involved, and it all multiplies out.
I mention to them "you know... i never use ssh passwords, i don't actually know my devfoo1234 password... maybe you should google for ssh-keygen, set it up, let me know if you have any problems?" and they're like "oh yeah, thats cool. i should do that sometime later!".... and then they never do, and they are forever messing with passwords.
I just don't get it.
Since a few years now I only ever use SSH private keys safely hidden behind a HSM with a tinier than tiny attack surface: Yubikeys do it for me (but other vendors would work too). My SSH keys do not have a password but when I log in using SSH, it requires me to physically touch my Yubikey (well one of my Yubikeys).
Windows has great support, surprisingly, for TPM-backed sk keys using Windows Hello and OpenSSH. Protected with physical presence and anti-hammering at the hardware level, and easy to setup by just selecting a sk type key.
I only use password keys for things that need to be scripted.
Keys is great for individual use, or for company use if you have centralized key control (and issue one or more keys per user).
Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines.
Passwords at least are "simple" for people to work with.
> Often you either end up with one "dev ssh key" for all machines (which is bad) or you end up with people sharing around keys and unidentified keys on machines
That hasn't been my experience at all. I've never encountered ssh key sharing in any environment, that would be insane.
We enforced different ssh keys per environment at my previous company: fingerprint of your key would get logged, and if the SIEM detected a reuse of keys across environments (dev, test, prod, etc) you'd get a stern talking to.
> Often you either end up with one "dev ssh key" for all machines (which is bad)
Or, conversely: with one "dev password" for all machines.
Unregulated/decentralized SSH key usage (i.e. allowing ssh-copy-id) is a dream for hackers to move laterally through a network. That's why many orgs disable it, and otherwise haven't invested resources in getting a proper centralized CA/authz server set up.
Every couple of months someone re-discovers SSH certificates, and blogs about them.
I'm guilty of it too. My blog post from 15 years ago is nowhere near as good as OP's post, but if I though me of 15 years ago lived up to my standards of today, I'd be really disappointed: https://blog.habets.se/2011/07/OpenSSH-certificates.html
I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.
I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually.
Things like deploying dev keys to various production environments, instead of generating/registering them within said environment.
One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "secure" read-only account to the other agency's SQL server (which uses the same TLS 1.3 as HTTPS). This is from person in charge of digital security for a government org.
> Things like deploying dev keys to various production environments, instead of generating/registering them within said environment.
I can see this happening when a developer is authorized to generate, but not to register. So, they just reuse an already-registered one.
Or when the security team at some other company emails you their private key.
One key technological cause is that PKCS#12 standardizes a format (you've most likely seen it as .PFX files) in which a certificate and its associated private key are bundled. This is in an effort to simplify the software...
So you get a situation where the lay person is given a "certificate" but it's not really just the certificate it's a PFX file and so e.g. no they mustn't show it you, it has their private key inside it and so you will learn that key and if you're honest you've just ruined their day because they need to start over...
I would say in my career I've had at least two occasions where I did that and I felt awful for the person, because I had set out to help them but now things are worse, and I've had a good number of later occasions where I spent a lot more of their time and mine because I knew I need to be very sure whether their "certificate" is actually a certificate (which they can show me, e.g. Teams message me the file) or a PFX file (thus it has their private key) and I must caution them to show nobody the file yet also try to assist them.
Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.
The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.
The main advantage of certificates is that you are able to do that from the CA without touching the target machine.
Certs may still be the right approach, but OpenSSH also supports an AuthorizedKeysCommand which could be a secure HTTPS request to a central server to pull down a dynamically generated authorized_keys file content for the particular user and host.
If your endpoints can securely and reliably reach a central server, this gives you maximum control (your authorized_keys HTTPS server can have any custom business logic you want) without having to deal with certs/CAs.
Exactly. This is really useful in larger organizations where you may want more complex rules on access. For example, you can easily build "break glass" or 2nd party approved access on demand. You can put whatever logic you need in a CA front-end.
You can also make all the certs short-lived (and only store them in ram).
The way I've been doing that is with Shamir Secret Sharing and encrypting keys until glass-breaking is necessary.
generating tons of keys? or just broad keys?
What I've done is generate a cert for the host(s) the user needs, for the time-span they need (subject to authorization logic).
And when your or someone else's infra down to such a degree that you need SSH access, you do not want to depend on being able to touch that machine first. The same is true with custom AuthorizedKeysCommands that phone home.
I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though.
I assume you gathered a lot of thoughts over these 15 years.
Should I invest in making the switch?
A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.
Adding to this: while certs are indeed well-supported by OpenSSH, it's not always the SSH daemon used on alternate or embedded platforms.
For example, OpenWRT used Dropbear [1] instead, which does not support certs. Also, Java programs that implement SSH stuff, like Jenkins, may be doing so using Apache Mina [2] which, though the underlying library supports certs, it is buggy [3] and requires the application to add the UX to also support it.
[1] https://matt.ucc.asn.au/dropbear/dropbear.html
[2] https://mina.apache.org/sshd-project/
[3] I've been dealing for years with NullPointerExceptions causing the connection to crash when presented with certain ed25519 certificates.
You can just replace dropbear with openssh on OpenWRT. That was one of the first things I did, since DropBear also doesn't support hardware backed (sk) keys. Just move it to 2222 and disable the service.
I reenabled DB on that alt port when I did the recent major update, just in case, but it wasn't necessary. After the upgrade, OpenSSH was alive and ready.
Upgrade to a better one in initramfs?
Might actually be a positive instead of a negative. Gaming use-cases should have not any effect on security policies, these should be as separate as possible, different auth mechanisms for your gaming stuff and your professional stuff ensures nothing gets mixed.
Hah? It being my gaming machine has nothing to do with the problem. It’s also my FPGA development machine, though it gets used less for that. It only happens to be the only Linux workstation in my home (the others are Macs or OpenBSD).
If you care about security, I recommend investing into a separate computer for developing hardware and software and another for downloading games on.
You can setup your security any way you like, but nothing beats an air gap in terms of security and simplicity.
remote unlock is also useful when you're not gaming so that feels like the wrong aspect to focus on
If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two.
In environment where they don't cause frustration they're not worth it.
Not really more to it than that, from my point of view.
You will have to manage your SSH CA certificates instead of your keys.
The workflows SSH CA's are extremely janky and insecure.
With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure.
With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
> With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.
Whut? This is literally the opposite.
With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.
And what about the CA?
It's no different compared to regular SSH private keys. You need to protect it from compromise.
However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.
Yes it is different. SSH CA keys are harder to secure and attackers have a much bigger incentive to steal them.
You can also configure multiple CA for client auth, and on the client side multiple ca to verify host keys.
Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.
It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.
[flagged]
I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.
If you mean using OIDC, in that space there's at least https://github.com/EOSC-synergy/ssh-oidc, https://dianagudu.github.io/mccli/ and OpenPubkey-ssh discussed in https://news.ycombinator.com/item?id=43470906 (which might mention more).
How does SSSD support help with SSH authN? I know you can now get Kerberos tickets from FreeIPA using OIDC(?), but I forget if SSSD is involved.
Can't really speak to the point of the guy you're replying to, but the FreeIPA implementation via SSSD does more than just Kerberos tickets. Actually, I think the Kerberos based stuff as it relates to SSH is GSSAPI as part of sshd itself and has little to do with sssd, though I could be wrong.
That said...
If I'm remembering things correctly (and it's been a looong while since I've played with this), FreeIPA's client configures sshd with an AuthorizedKeysCommand that executes a program that queries sssd for the list of authorized keys for a given user. Sssd then uses a plugin to query the LDAP server @ FreeIPA to get the list of keys.
There's also SSHFP (I think) records in DNS if you're using FreeIPA's DNS servers. These provide the host keys for servers for your ssh client to check against. Not sure if that's integrated into ssh itself or something else -- I can't remember how it's implemented offhand -- but it's fairly nifty since you never see the TOFU prompt (or it would be if DNS was actually secure, anyway).
Yes, FreeIPA is Kerberos+LDAP+X.509 CA, and GSSAPI is in OpenSSH (normally with the key exchange patch). SSSD is a local mechanism, not network authentication. I mentioned authorized keys distribution mechanisms elsewhere, but I was thinking authentication (c.f. OIDC), not authorization.
I haven't used SSSD due it not being available for Alpine but doesn't it provide authentication via pam_sss ?
Yes, but its authN components only act locally, and PAM is optional for sshd. It can/does call out to network services like Kerberos/LDAP given a password, of course, but I was thinking of network authN connected directly with OIDC somehow, for which I don't know a mechanism in vanilla OpenSSH. (I don't know what Authentik does for this -- I could imagine it's behind the scenes somehow.) I should probably look it up sometime.
It depends on what you want to do. CA certs are easy to manage, you just put the CA key instead of the SSH public key in authorized_keys.
They also provide a way to get hardware-backed security without messing with SSH agent forwarding and crappy USB security devices. You can use an HSM to issue a temporary certificate for your (possibly temporary) public key and use it as normal. The certificate can be valid for just 1 hour, enough to not worry about it leaking.
Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters).
There are some serious security benefits for larger organizations but it does not sound as if you are part of one.
oh man, I referred back to your blog post when I wrote the ssh certificate authority for $job ... ~10 years ago.
Thank for writing it!
maybe it keeps getting rediscovered, becaues a good tooling around it is still missing?
Despite the drawbacks of its grassroot nature TOFU goes a looooong way.
With my own machines I can just physically check that the server host key matches what the ssh client sees. Once TOFU looks good I'm all set with that host because I don't change any of the keys ever.
In a no-frills corporate unix environment it's enough to have a list of the internal servers' public keys listed on an internal website, accessible via SSL so it's effectively signed by a known corporate identity. You only need to check this list once to validate carrying out TOFU after which you can trust future connections.
In settings with huge fleet of machines or in a very dynamic environment where new machines are rolled out all the time it probably makes things easier to use certificates. Of course, certificates come with some extra work and some extra features so the amount of benefit depend on the case. But at this scale TOFU is breaking down bad on multiple levels so you can't afford a strong opinion against certificates, really.
I wish web browsers could remember server TLS host keys easily too and at least notify me whenever they change even if they'd still accept the new keys via ~trusted CAs.
I've been using SSH since 1996 and I've never once seen anyone actually verify a public key manually like that. If they can MITM your SSH server, why can't they MITM your internal web site? In common practice, TOFU is type "yes" and YOLO.
I work in a corporate setting and the money and time we wasted because of Zscaler and its SSL inspection [1] is beyond your wildest imagination. Whenever I see a "SSL certificate problem: self-signed certificate in certificate chain" error, I know I'm in trouble.
[1] https://www.zscaler.com/resources/security-terms-glossary/wh...
Zscaler was deployed on a Windows 11 machine at a place that a friend worked at. When I assessed the software, its behavior was downright evil like malware. As we all know, it injects its own root certificate into the operating system in order to conduct man-in-the-middle attacks on TLS/HTTPS connections to monitor the user's web activity.
Furthermore, it locks down the web browser's settings so that you cannot use a proxy server to bypass Zscaler's MITM. I saw this behavior in Mozilla Firefox, where the proxy option is set to "No proxy" and all other options are disabled and grayed out; I imagine that it does the same to Google Chrome. If you try to modify the browser's .ini(?) file for proxy settings, Zscaler immediately overwrites it against your will. Zscaler worked very hard to enforce its settings in order to spy on the computer user.
And as you'd expect, if you open up the Zscaler GUI in the system tray, you are presented with the option to disable the software if you have the IT password. Which of course, you don't have. Then again, that might be an epsilon better than the Cybereason antivirus software, which just has a system tray icon with no exit option, and cannot be killed in Task Manager even if you are a local administrator, and imposes a heavy slowdown if you're open hundreds of small text files per second.
I feel the same way about Cisco Umbrella where I work.
The worst breakage by far is protocol breakage; basically anything that uses HTTP as a basis for building some other protocol gets broken all the time. None of the people implementing it seem aware. They buy the vendor's claim that it's "transparent", when in fact even "inspect/trace-only" modes often break all kinds of shit.
I've seen Umbrella break:
and probably some other things I'm forgetting. When this breakage is reported, the first round of replies is typically "I visited that domain in my enterprise-managed browser and it's not blocked". That is, of course, a useless and irrelevant test.Often it takes hours to even fully diagnose the breakage with enough confidence to point the finger at that tool and not some other endpoint security tool.
I'm not sure if the people buying and deploying tools in this category don't know how much stuff it breaks or just don't care. But the breakage is everywhere and nobody seems prepared for it.
SSH certificates aren't X.509 certificates.
The author lists all the advantes of CA certificates, yet doesn't list the disadvantages. OTOH, all the many steps required to set it up make the disadvantages rather obvious.
Also, I've never had a security issue due to TOFU, have you?
> Also, I've never had a security issue due to TOFU, have you?
This is a bit like suggesting you've never been in a car crash, so seat belts must not be worth considering.
Do you feel that beyond the obvious and documented work in setting them up, there are disadvantages to using SSH certificates?
Certificates provide extra features, like revocation.
However, if you do not need the extra features provided by certificates, using SSH-generated keys is strictly equivalent with using certificates and it requires less work.
TOFU is neither necessary nor recommended, it is just a convenience feature, to be used when security may be lax.
The secure way to use SSH is to never use TOFU but to pair the user and the server by copying the public keys between the 2 computers through a secure channel, e.g. either by using a USB memory or by sending the public keys through already existing authenticated encrypted links that pass through other computers. (Such a link may be a HTTPS download link.)
When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
> secure channel, e.g. either by using a USB memory
Aside: We need a "safely dumb" storage plug standard. Right now the flexibility of USB is a double-edged sword, any USB device could potentially be malicious, waiting for the right moment to send keystrokes as a keyboard etc.
Or use the same plug for everything, but support a "passive storage only" that can be enforced by a special adapter or cable.
Assuming you’re not just soliciting random thumb drives from strangers… this doesn’t feel like an actual risk.
When you’re getting things from strangers, use any of the myriad ways to send files over a network, where HTTP / your browser are the safe dumb plug.
The place you buy the thumb drive is also a stranger that can't be trusted.
This kind of paranoia isn't productive.
It was productive enough for the NSA and FBI when they were intercepting equipment deliveries and installing backdoors in them.
A USB stick that can secretly remember any file that looks like a key file or certificate would be very easy.
> TOFU is neither necessary nor recommended
Just to make it clear: this does not mean that it is fine to blindly accept the message on first use.
The "secure way" implies copying the server's public key as well, which people generally don't do, right? Which is equivalent to verifying the fingerprint shown with the TOFU message, correct?
Like I have said the secure way requires the secure copying of both keys before the first connection attempt
The server public key must be copied into "known_hosts" on the client, while the client public key must be copied into "authorized_keys" on the server.
When this is the procedure that is always followed, any message shown by SSH about an unknown host means that the connection must be aborted, because the identity of the server is unknown.
You cannot truly verify the "fingerprint" displayed by SSH, unless you simultaneously have access to another computer, where you have a copy of the fingerprint. What is usually meant by "verifying" is that you remember a few digits of the fingerprint, and those match.
You could have copied the fingerprint from the server, to be able to truly verify it, but that does not make sense, because in that case you should have copied the entire key, not just the fingerprint, and you should have installed it in the client.
When you use only authentication with digital signatures, it does not make sense to use any other procedure, because you must make at least one of the two copies anyway, so when copying the client key to the server you can take the server key, to carry it back to the client.
The TOFU method is meant to be used together with password-based authentication, in less secure applications, where no physical access to the server is required for setting up SSH on the client.
By "less secure" I mean for example applications equivalent to HTTPS, where the client is not really authenticated, e.g. when providing a public password allowing read-only access to an SSH server through Internet.
> Like I have said
Sure, I just wanted to make sure that nobody would understand "TOFU is neither necessary nor recommended, just ignore that message and say "yes" when it appears".
Both ends need to be sure of who is on the other end, and there are different ways to achieve that. The way SSH works is that if you haven't copied the server public key locally, it will explicitly ask you to verify it the first time.
I am not sure that "SSH does TOFU". SSH asks you to verify. The human who YOLOs it and approves it without checking is the one doing TOFU, and this is not really secure.
> When using certificates, a completely identical procedure must be used. After certificates are generated, like also after SSH keys are generated, the certificates must be copied to the client computer and the server computer through secure channels.
That is not the case, and is a major advantage of certificates.
What you say is extremely wrong and this misconception is very dangerous.
When an authenticated connection between a client and a server (or any other 2 computers) is established, before the connection can be initiated both ends must already have a piece of information that has been copied to them through a secure channel, otherwise the authentication is impossible.
The 2 pieces of information may allow direct reciprocal authentication, i.e. the client may have the server certificate and the server may have the client certificate.
It is also possible to have indirect authentication, e.g. the server sends the server certificate to the client and the client sends the client certificate to the server during the initiation of the connection.
However, for indirect authentication, both the client and the server must had been previously paired with a server hosting the certification authority.
This means that the root certificate of the CA must have been transferred through secure channels to the client and to the server, and also that the client certificate and the server certificate must have been transferred through secure channels to the CA for signing and then back to the client and the server.
So setting up the computers for indirect authentication through certificates requires significantly more transfers through secure channels than setting them up for direct authentication.
The advantage of indirect authentication appears in an organization with a great number of computers, where indirect authentication reduces the number of secure pairings that must be done from N*(N-1)/2 to N.
However this has nothing to do with certificates. Indirect authentication can be done without certificates, like in Kerberos, or like in SSH if you have a management computer that you pair by copying the public_keys with every computer in the network. Later, whenever you want to enable some client to connect to a SSH server you use the management computer to copy securely the corresponding public_keys between the server and the client.
So no, there is no advantage of certificates related to reducing the number of transfers that must be done through secure side channels, like copying through a portable memory.
Certificates provide other features, like temporary validity and revocation, which can be useful in an enterprise context.
What misconception? You essentially admit that I'm right in this comment but surround it with a bunch of other unrelated words.
If you use SSH certificates, then you have to put the SSH CA cert on each server once, and you have to give signed certs back to each client once each time you sign a cert (which can be once per client or once per connection or anywhere in between).
You do not have to copy anything new to each server when you sign new client certs. Likewise if you sign host keys, you don't have to copy a new host key to your client when you add a new server.
Sure, you could get the same kind of trust relationships with other tools. But SSH certificates provide them and are an advantage of SSH certificates over traditional SSH public key auth.
It honestly feels like you're just dropping this thread into an LLM and asking it to write a disagreeing response.
Your ISP or telecom has to be compromised for TOFU to be relevant to anything. In practice that never happens.
Not just your ISP. If an attacker slipped a device onto your LAN and also you happened to be sshing to a new box for the first time then TOFU poses a problem. But that's an awfully limited attack surface. It's similar to the difference between leaking a fax while it's sent versus leaking years old emails that are just sitting there on an internet accessible server.
As for your ISP I think you should never rely on TOFU over the public internet. If you really don't want to do ssh certs it's easy enough to make the host key available securely via https.
If you have some form of access to set up the CA config on the box before connecting then you can use the same access channel to avoid needing to rely on TOFU for setting up the key access all the same.
This can be anything from being part of the install script to customized deployment image to physical access to access via a host in virtualized scenarios.
TOFU only really comes into play when the box is already set up and you have no other way to load things onto the box other than connecting via SSH to do so. But, again, that would be the same story if you were intending to go the certificate approach too.
TOFU is convenient, but not necessary.
Choosing to use TOFU is a distinct choice from the choice of using the keys generated by SSH, instead of using certificates.
If you do not want to use TOFU, for extra security, you just have to pair the computers by copying between them the corresponding public keys through a secure channel, e.g. by using a USB memory.
Using certificates does not add any simplification or any extra security.
For real security, you still must pair the communicating computers by copying between them the corresponding certificates, through a secure channel, e.g. a USB memory.
When you use for HTTPS the certificates that have come with your Internet browser, you trust that the installer package for the browser has come to that computer through a secure channel from the authority that has created the certificates. This is usually an assumption much more far fetched than the assumption that you can trust TOFU between computers under your control.
Certificates may be useful in big organizations, if other functionality is needed beyond just establishing secure communication channels, e.g. if you want to use certificate revocation.
In the list of "advantages" enumerated in the parent article, more than half of them are false, because if certificates are implemented correctly, completely equivalent actions must be executed when SSH keys without TOFU are used and when certificates are used.
Perhaps the author meant by writing some of the "advantages" that the actions that supposedly are no longer needed with certificates are done by an administrator, not by the user. However that is also applicable with SSH. An administrator could install the certificates, so that no action is required from the user, but an administrator can also install the SSH public keys, so that no TOFU is ever needed from the user.
Using certificates requires exactly the same steps like using keys generated by SSH (i.e. generating certificates and copying them between computers through secure channels, to pair the servers and the authorized users), but it may need additional steps, caused by the fact that certificates provide additional functionality.
Are you pairing computers by copying certificates to visit this site?
Touche.. actually a good point, but actually those are two different situations. With one, I'm accessing a website and trusting that the certificate is signed by someone I trust; so the trust in my browser certificates (which include certificates from hundreds of certificate authorities all over the world, any one of which could be compromised, robbed, or controlled by an adversarial person or even government) is extended to the site that I'm visiting. To say this is weak sauce rather understates how bad this actually is. (To paraphrase Churchill, this is the worst possible design, except for all the rest.)
With the other, I'm logging into a server for the first time (and I could simply deploy the same trusted host key to all my ssh servers via an autoscaling configuration or whatever). I think it's debatable if TOFU is worse or better than your (granted clever) metaphor.
(to those who'd recommend userify, yes - great for the client login issue and definitely increases security, but to parent's point, TOFU is still needed unless you want to distribute host pubkeys)
Pairing is absolutely necessary for bidirectional authentication, where each party must verify the identity of the other end.
To visit this site, there is no pairing, because the site does not know who I am.
In order to verify the identity of the HN site, I must trust that the maintainers of the installation packages of the browsers that I use (Firefox, Vivaldi, Chromium) have ensured that the built-in certificates have reached me through a secure path. This actually requires much more trust than when someone answers "yes" to the SSH unknown host message.
If I use certificates for accessing e.g. the network of my employer, then my work computer must be paired with some corporate server, i.e. a unique certificate has been generated for myself and it has been copied to some certificate authority server for signing and then to my computer, and also a certificate of the local certificate authority has been copied to my personal computer.
While pairing is unavoidable for bidirectional authentication, it is not necessarily direct between the end points. Both end points must have been paired with at least one other computer but they need not have been paired between themselves previously if there exists some path through secure connections that have been originally created by pairing.
When certificates are used, usually the pairings are not done directly between end points, but each computer must be paired with the server hosting the certificate authority.
The term "pairing" is not used frequently, but it should have been preferred, because frequently the users do not understand which are the exact actions on which the security of their communications depend, which leads to various exploits. The critical security actions are those that perform the pairing.
"Pairing" of 2 systems, e.g. A and B, means that some information must be transmitted through a secure channel from A to B and some other information must be transmitted through a secure channel from B to A. An alternative pairing method is to generate both pieces of information on one of the 2 systems and transmit both of them through a secure channel to the other. The information exchange channels must already be secure, because before pairing authentication is impossible.
The pairing between a PC and the server hosting the certificate authority can be done in various ways, depending on where the PC certificate is generated. If the certificate is generated at the certificate authority than both it and the root certificate must be copied through a secure channel to the PC. If the certificate is generated on the PC, it must be sent through a secure channel to the CA for signing, then it must be sent back also through a secure channel.
In practice, administrators are not always careful enough for the channels through which certificates are copied to be really secure. For instance they may be copied through network links that are not yet authenticated, which is equivalent with the TOFU method optionally used by SSH.
> Also, I've never had a security issue due to TOFU, have you?
You haven't yet had a security issue due to TOFU - FTFY.
Who knows - maybe I'll never have one until I stop using TOFU.
[flagged]
In our dev/stg environment we reinstall half our machines every morning (largely to test our machine setup automation), and SSH host certificates make that so much nicer than having to persist host keys or remove/replace them in known_hosts. Highly recommended.
[flagged]
This post inspired me to add SSH host key signing to WeEncrypt (formerly LessEncrypt): https://github.com/linsomniac/weencrypt
This discussion is full of schizo solutions to "secure" SSH, most of which make no practical sense or have no technical basis.
There really needs to be a definitive best practices guide published by a trusted authority.
In my view it is more important to stop using software keys so probably use sk (fido) for both host and user.. From there CAs would be a next step.. The level of documentation and example setups is astoundingly poor if you even look at step 2 for any feature. I.e. SK keys are reasonably understood for user keys but the setup as host keys is vague and needs testing to see if it really works.
Can you give some examples of which suggestions you think are schizo?
SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management.
The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow.
Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once.
Common friction points:
Revocation is also kind of a lie. Just waiting for expiry and hoping that’s good enough.What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it.
We went the opposite direction:
You still get central control, but operation and failure modes are local instead of "everyone is locked out right now."That’s basically what we do at Userify (https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn.
And the part that usually gets hand-waved with SSH CAs:
Those don’t go away - they're just not part of the certificate solution.* (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)
How do you solve TOFU?
Well, TOFU is really just the model for how the chain of trust is established.
In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs).
If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust.
From an automation/autoscaling angle, the same thing shows up again:
1. either keys are pre-baked / distributed
2. or, something signs them at boot
Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it.
Certificates just add extra steps around the same underlying task.
I agree. I was just wondering if Userify had a solution for distribution the server signatures to the users.
Great question. Not yet ;)
Fair enough :)
With the recent wave of npm hacks stealing private keys, I wanted to limit key's lifetimes.
I've set up a couple of yubikeys as SSH CAs on hosts I manage. I use them to create short lived certs (say 24h) at the start of the day. This way i only have to enter the yubikey pin once a day.
I could not find an easy way to limit maximum certificate lifetime in openssh, except for using the AuthorizedPrincipalCommand, which feels very fragile.
Does anyone else have any experience with a similar setup? How do you limit cert max lifetime?
Instead of using a CA, why not set the key's PIN policy to "once" and use an agent (e.g. https://github.com/FiloSottile/yubikey-agent/) that holds an active session to the yubikey? You start the agent at the beginning of the day, enter the PIN once, and then stop the agent at the end of the day.
Wanted to avoid having the key in USB slot all the time. I have the version that sticks out and i carry it on my key chain. So it's easy to break in a laptop.
The experience might be better right up until you're running it in prod and someone happens to ask about:
(disclaimer we're celebrating our 15th anniversary at https://Userify.com, but those are actually legit concerns and not only a sales pitch. You certainly can build a solid and secure ssh cert infra, but doing it in production is just not an easy set-it-and-forget-it sort of thing.)Sorry to pop your bubble, but a SaaS is the worst possible option.
Then install your own:
You can also address TOFU to some extent using SSHFP DNS records.
Openssh supports checking the DNSSEC signature in the client, in theory, but it's a configure option and I'm not sure if distros build with it.
On top of that you would need something to secure DNS. Like DNSSEC or at the very least use DNS with TLS or DNS over HTTP. None of these are typically enabled by default.
Anything that uses system-resolved is probably doing DNSSEC validation by default. It's becoming much more common.
Additionally, as I mentioned, openssh itself has support for validating the DNSSEC signature even if your local resolver doesn't. I actually don't think it can use the standard resolver for SSHFP records at all, but I'm not sure.
Any idea if there's a standardized location, something like /.well-known/ssh?
This exists, but not very well supported: https://github.com/C2SP/C2SP/blob/main/well-known-ssh-hosts....
But DNSSEC + SSHFP is a better supported solution at the moment
We added SSH certificates support to pico.sh [1] and it's been great. Utilizing principals gave us the ability to implement a RBAC like system for specific parts of the pico.sh ecosystem. Users get the flexibility they want with limited complexity.
[1] https://pico.sh/access-control#ssh-certificates
shameless plug: I've been tinkering with a tool to make SSH certificate-based login a bit easier. it's called Sshifu.
basically, you set up a sshifu-server that acts as a certificate authority + SSO server. then on your SSH servers, you configure them to trust this CA (there are helper npx commands / bash scripts to make this easy).
after that, for each user who wants access, they just run:
npx sshifu <sshifu-server-url> <ssh-server-url>
this starts the SSO login flow, sets up the CA public key if needed, and immediately opens an SSH session.
npx is just the easiest way to get started, there are other install options too.
I built this as a smaller alternative to Smallstep / Teleport. it's still very early and mostly vibe-coded, but it's already scratching my own itch.
would love to hear what you guys think
repo: github.com/azophy/sshifu
>then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key.
This sentence is a bit of a red flag, it looks like the author is making a (subtle) mistake in the category of too much security, or at least misjudging the amount of security (objectively measurable entropy) needed. This is of course a less consequential error than too little entropy/security measures, but still if one wants to be a cybersecurity professional, especially one with influence, they must know exactly the right amount needed, because our resources are limited, and each additional bit of entropy and security step not only costs time of the admin that implements it, but of the users that have to follow it, and this can even impact security itself by fatiguing the user and causing them to circumvent measures or ignore alerts.
On to specifically what's wrong:
Either a key file or a password can be used to log in to a server or authenticate to any service in general. Besides the technical implementation, the main difference is whether the secret is stored on the device, or in the user's brain. One is not more correct than the other, there's a lot of tradeoffs, one can ensure more bits and is more ergonomic, the other is not stored on device so it cannot be compromised that way.
That said a 2FA approach, in whatever format, is (generally speaking) safer than any individual method, in that the two secrets are necessary to be granted access. In this scenario one needs both the file and the password to authenticate, even if the password is 4 digits long, that increases the security of the system when compared to no password. An attacker would have to setup a brute force attempt along with a way to verify the decryption was successful. If local decryption confirmation is not possible, then such a brute force attack would require to submit erroneous logins to the server potentially activating tripwires or alerting a monitoring admin.
There's nothing special about the second factor authorization being equal or equivalent in entropy to the first, and there's especially no requirement that a password have more entropy when it's a second authorization, in fact it's the other way around.
tl;dr You can consider each security mechanism in the wider context rather than in isolation and you will see security fatigue go down without compromising security.
Sadly services such as Github don't support these so it's mostly good for internal infrastructure.
They do, for Enterprise customers only: https://docs.github.com/en/enterprise-cloud@latest/organizat...
They've rolled their host key one time, so there's little reason for them to use it on the host side.
I’d love if they supported an SSH certificate CA for commit signing verification though.
It's very interesting that you can force the execution of a specific command.
I wonder if that can be somehow employed to somehow get kubectl to invoke ssh in order to get an authentication tokens.
I would be cool to authenticate to kubernetes via ssh.
Anyone tried out Userify? It creates/removes ssh pubkeys locally so (like a CA) no authn server needs to be online. But unlike certs, active sessions and processes are terminated when the user access is revoked.
We're in the process of updating the experience to this century! ;)
We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)
Life is easier if you can use Kerberos SSO, i.e. GSSAPIAuthentication in OpenSSH. (If we're talking certificates, presumably it is OpenSSH, or does anything else implement them?)
Why would you do that rather than just hooking SSH up to a real IdP with certificates?
I don't want to have to get a special purpose credential when I have a TGT which can work generally, and is at least required for secure remote filesystem access.
You have to manage extra infrastructure for certificates and, as a user, have the friction of firing up a JavaScript-enabled web browser via an additional tool, assuming "real IdP" means using OIDC. Unfortunately that flow is actually needed for remote systems and something like Edugain federation, since Moonshot/IETF ABFAB failed, but at least Shibboleth can use the TGT, and it's not the Globus horror.
Every serious shop of any real size is already managing an OIDC IdP (you need one for whatever SAAS apps your team is using along with any internal web applications you're using). Why not just link it to something that can issue short-lived SSH certificates? That's also the cleanest way to get strong multifactor auth for SSH (certificates issued only through an OIDC progress minted with MFA requirements).
Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
I'm happy for anyone who doesn't have MS Windows/Active Directory -- so Kerberos -- in their organization, but I'd need (Free)IPA or similar for user/access management anyway. Certificates are an extra layer of SSH-specific complexity, which concerns me for security even if it doesn't involve some third party. MFA is needed once a day, say, for SSO to all Kerberized services. [As I understand it, "managing an OIDC IdP" includes shipping the contents of Active Directory to Entra, heaven help us.]
> Setting up Kerberos in 2026 feels somewhat close to malpractice to me.
Microsoft (if that means anything, but they've done good work) and Red Hat obviously disagree, along with decades' experience. It is malpractice not to secure NFS mounts (and other network filesystems with sensitive data), and that means Kerberos.
All those articles about SSH certificates fall short of explaining how the revocation list can/should be published.
Is that yet another problem that I need to solve with syncthing?
https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS
If you generate short lived certificates via an automated process/service then you don’t really need to manage a revocation list as they will have expired in short order.
But then you can't log in if your box goes offline for any reason.
Hmm. For user certs you can have the service sign them for, say an hour, so long as you can ssh to your server in that time then there’s no need for any other interaction.
Sure you need your signing service to be reasonably available, but that’s easily accomplished.
Maybe I misunderstand?
That works for authn in the happy path: short-lived cert, grab it, connect, done.
Except for everything around that:
* user lifecycle (create/remove/rename accounts)
* authz (who gets sudo, what groups, per-host differences)
* cleanup (what happens when someone leaves)
* visibility (what state is this box actually in right now?)
SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine.
So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct.
On the availability point: "reasonably available" is doing a lot of work ;)
Even with 1-hour certs:
* new sessions depend on the signer
* fleet-wide issues hit everything at once
* incident response gets awkward if the signer is part of the blast radius
The failure mode shifts from a few boxes don't work to nobody can get in anywhere
The pull model just leans the other way:
* nodes converge to desired state
* access continues even if control plane hiccups
* authn and authz live together on the box
Both models can work - it’s more about which failure mode is tolerable to you.
Well, yes, pick your poison.
But for just getting access to role accounts then I find it a lot nicer than distributing public keys around.
And for everything else, a periodic Ansible :-)
Public keys (for OpenSSH) can be in DNS (VerifyHostKeyDNS) or in, say, LDAP via KnownHostsCommand and AuthorizedKeysCommand.
That sounds like a lot of extra steps. How do I validate the authenticity of a signing request? Should my signing machine be able to challenge the requester? (This means that the CA key is on a machine with network access!!)
Replacing the distribution of a revocation list with short-lived certificates just creates other problems that are not easier to solve. (Also, 1h is bonkers, even letsencrypt doesn't do it)
1h is bonkers for certs in https, but it's not unreasonable for authorized user certs, if your issuance path is available enough.
IMHO, if you're pushing revocation lists at low latency, you could also push authorized keys updates at low latency.
Honestly, we used to replace a lot of pam_ldap and similar sorts of awful solutions. With those, if your LDAP went down even for a heartbeat, you couldn't log in at all.
So I totally agree: if I had to do certificates and didn't have something like Userify, a 1 hour (or even shorter if possible) expiration seems quite worth chasing, especially with suitable highly available configuration. (Of course, TFA doesn't even bother mentioning revocation and expiration, which should give you a clue as to how much fun those are lol)
And for more normal, lower-security requirements or non-HA, 6 or 8 hours or so would probably work and give you plenty of time for even serious system outages before the certs expired.
Not to hard shill or anything (apologies in advance, just skip if you're not interested), but there are two significant security and reliability differences between standard SSH (with or without certificates) and Userify:
1. Userify Cloud updates by default every three minutes, and on-premise Userify Express/Enterprise updates every ten seconds, but it doesn't have to update at all; even if your Userify server goes offline forever, you can still log in because the accounts are standard UNIX accounts (literally created with `useradd`)
2. When accounts are removed, Userify also completely nukes the user account, removes its sudo perms, and totally kill -9 's any tmux/screen/etc sessions (all processes owned by the user are terminated across the entire enterprise within seconds), which is also not something that a certificate expiration would ever do.
I've had very good experiences with SSH Communication Security company's (the guys who invented SSH) PrivX product to manage secure remote access, including SSH certificates and also cert based Windows authentication. It supports other kinds of remote targets too, via webui or with native clients. Great product.
[dead]