While it is certainly an interesting bug, I kinda feel that the title is click bait? Because this `cryptsetup luksSuspend` from what I understood is not really officially supported but an extension done in Debian, so if anything this regression only affected Debian? I am not sure if you can blame the kernel for something that is not supported or even widely tested.
I still find this impressive, and it is nice that we now have a test (NixOSTests BTW are awesome, I agree with OP) to avoid this regression from coming back. But from the title it seems to be a widespread issue, not something that affects only one Distro.
Sorry, aimed for a technically precise title and didn't want to bait clicks.
Yes, this does not affect people on stock configurations for the plain reason that they wouldn't expect the volume key to be safe during suspend anyway.
Debian's solution was ported to several (most?) other distributions and I guess quite a few people maintained private ports.
The thread-keyring(7) manpage promises: "A thread keyring is destroyed when the thread that refers to it terminates." For their key upload (from userspace to kernelspace) mechanism, the cryptsetup project relied on this property; but kernel 6.9 introduced a regression invalidating this property.
I don't see any other way? When you sleep (suspend to RAM), everything is stored in RAM and is encrypted but the master key is present in kernel memory (if I recall correctly).
However, if you hibernate (suspend to disk) the entire contents of RAM (including the master key) is written/encrypted to disk and the RAM is cleared.
When you wake the machine up you have to re-enter the passphrase to decrypt the master key to re-load disk contents back to memory.
Yes, if you simply suspend your laptop on most stock Linux distributions, then everything including the master key is still kept in memory. But Debian pioneered the (optional) cryptsetup-suspend addon. This issues a luksSuspend command which is supposed to wipe the key from memory, and on resume asks you to resupply your passphrase.
Up to kernel 6.8, this worked as described; starting with kernel 6.9, it silently didn't.
Exactly. Cryptsetup wouldn't know about the extra copy of the volume key in kernel memory. Which is why, dramatically, it appeared secure ("surely I wouldn't be asked to resupply the passphrase if the volume key is still in memory, right?").
It was still more secure than the default if I understand this correctly. On resume from suspend the laptop would still be locked by the encryption key and without access to the disk even if you can somehow circumvent the lock. The only insecurity was that somewhere in the kernel memory the key still exists so if you can somehow extract that from the live system you can unlock it.
Yes, you are right: LUKS encryption protests your data at rest. An attacker which steals your disk can only gain little, like the information that you have used LUKS (unless you put your LUKS headers elsewhere, separated from the disk) and perhaps disk and disk sector usage statistics.
You need to get quite specific on actual attacks to call this insecure to be clear.
Having access to the raw RAM of a machine suspended but demanding the key to resume is certainly possible but the number of attacks where you don't need this bug is "almost all of them" given at that point if the machine ever unlocks you won in this hypothetical attack even with a bug fix.
If the key has been purged but you can read RAM, then you can do two things:
1. You can extract whatever user data happens to be in RAM.
2. If you can either write RAM or reboot into your own OS, and then return the device to the unsuspecting user who will put in their password, then you can run a fake password dialog and get everything.
1 is bad, since there may be quite a lot of user data in RAM. But it’s not quite as bad as having the disk key, which gives the attacker all the data plus the future ability to decrypt or modify user data given only the physical disk. (Still, a better solution would be encrypting the hibernation image, preventing this attack entirely.)
2 is fully bad, but in many plausible scenarios (e.g. seized device) the attacker cannot just return the device to the user without them knowing something happened. Or even if they can, the method of RAM access may be one where reads are much more practical than writes, such as cold boot attacks involving physically swapping out the RAM.
makes me wonder if there is potential for a more "main stream"/by default friendly version of this, where the key during suspend is encrypted using the TPM even if the TPM isn't a possible unlock from cold boot (i.e. no TMP encrypted volume key in the LECS headers/meta only temporary in memory during suspend)
or the alternative (for more convenient usage) for single user systems auto login on boot + use disc password for doas/sudo?
I've been wondering why hibernate didn't work with encryption, because this seems like the extremely obvious way to handle it, but I have struggled to find anything about it for years - glad to hear it does exist!
But yeah, also rather obviously it's inherently a bit leak-prone. Though it seems probably pretty simple to test, just hibernate and scan all stored data. They could probably even do it on shutdown, as a hash of the key data would be sufficient to detect the key.
If you’re at all serious about security and not user convenience, you deploy BitLocker with a PIN instead of TPM only. And then a whole class of vulnerabilities goes away.
Just a PIN? For most people that's a 4-digit number, which has a worst-case scenario of 10,000 attempts and a median of only a few hundred. Why not use a full 8-digit password?
> For example, when BitLocker is used with a TPM + PIN configuration, the number of PIN guesses is limited over time. A TPM 2.0 in this example could be configured to allow only 32 PIN guesses immediately, and then only one more guess every two hours. This totals a maximum of about 4,415 guesses per year. If the PIN is four digits, all 9999 possible PIN combinations could be attempted in a little over two years.
In that case, the median would still be just over a month, if the PINs were entered in order of how commonly they are used. Even the worst case of two years is still soon enough for a lot of data still be useful.
Also, how is the time limit enforced? With hardware access, it would be easy to change time or increase the clock rate, as well as many other side-channel attacks that could eliminate the wait altogether.
And then depending on how "serious" you are you also don't consider Linux.
But also, threat models and the best way to mitigate them aren't really a linear scale of being <unserious> to <serious>, but a complex consideration of a particular situation.
People just plain suck at opsec. Like Che Guevara might have had a longer career as a revolutionary if he'd used his one time pads only once.
Back in the late 1980s it was clear that it would be no problem at all to hook up a hard drive to a digital phone exchange and record all the calls! I had a strict policy of "don't talk about anything illegal using electronic communication" even when it was rather banal stuff like selling weed.
The carelessness of people at Facebook documenting policies that nobody in their right mind would document boggles my mind: you might as well leave it mysterious why you didn't crack down on scam ads, for instance. When I've been involved in minor conspiracies, say when we had an HR problem with another employee, I've always made the point to meet furtively in person and avoid leaving a paper trail so that I'd never need to explain an email I wrote in front of an unfriendly audience.
The issues you linked with BitLocker are obvious properties of BitLocker-with-SecureBoot-only architecture. If you configure Linux that way, you get similar issues (for example, it's pretty easy to mis-configure TPM sealed disk encryption on Linux to still allow a recovery shell, which will run with the disk unsealed).
BitLocker with a password (the equivalent of the LUKS configuration in question) does not share these issues.
Bitlocker with a password has always felt like a second class citizen to me. You have to dig into a bunch of group policies to use it. Maybe most people don't even realize it exists.
Yah, it seems blatantly hostile how much they hide it.
I can understand the default being TPM-only + online key backup, huge amounts of the population forget their login passwords (which can be involuntary, e.g. head injury) and huge amounts of them still want some backup way to access their data rather than losing it forever.
But for anyone who cares just a little more, or would prefer to lose data in those situations, it's such an abnormal and hidden path that it's clearly blocking tons of people from choosing it.
For the system drive they seem to really strongly prefer PIN, which also doesn't have the problems linked above. I was going to use PIN as my example but didn't want to explain another set of recent BitLocker conspiracy theories yet again; maybe I should have.
It is annoying that they hate password for system drive _so_ much; the reason is actually pretty obvious when you think about how their "happy path" AD-driven enterprise deployment with stupid password rotation requirements works (and FileVault is a nightmare in this scenario), but I wish they'd make it easier for individual power users.
veracrypt lost their drivers license so afaik you should avoid it since it cannot update its drivers any longer. didnt see any news about them reacquiring that license
If you think for one single second that businesses and governments who rely on a lost disk being secure don’t trust bitlocker, I have oceanfront property in Missouri to sell you.
Bitlocker + PIN is as secure as anything.
A vulnerability can’t leak your key if the TPM doesn’t know the entire key and relies on the user to supply the missing parts of the key in the form of a PIN.
> A vulnerability can’t leak your key if the TPM doesn’t know the entire key and relies on the user to supply the missing parts of the key in the form of a PIN.
First off: I agree with your thesis, BitLocker with PIN is Just Fine, equivalent in all practical senses to most disk encryption strategies, and an enterprise standard.
I post this to reinforce what you're saying, because there are a ton of weird theories about how this works that make people think it's weaker than it is.
BitLocker with PIN works like this:
* BitLocker seals an encrypted key IK into the TPM using a policy on the TPM which requires the SHA-256 of the PIN to be sent to the TPM to unlock the record (and has anti-hammering at the TPM level).
* Encryption using another key called the SK. Once the OS acquires the e(IK) from the TPM, it needs to derive SK to decrypt the IK. The SK is generated by applying a salt and 0x100000 rounds SHA256 to the PIN.
* The IK can then decrypt the VMK, which can decrypt the FVEK which is the actual data encryption key used on the on-disk data.
Fully offline (no TPM access) decryption is hopeless; the IK is random and not attached to the PIN at all.
Fully online decryption relies on TPM anti-hammering; if you can manage to hammer the TPM and find a PIN where h(PIN) matches, you can then use the PIN to generate both h(PIN) (unseal TPM) and reverse e(IK).
Partially offline decryption (imagining TPM content is somehow dumped, for example by bus sniff while a user uses their PIN) relies on bruteforcing the key for e(IK), which is possible but difficult (IMO Microsoft should ratchet this up or use a more expensive algorithm than SHA-256 in order to deal with modern hardware capabilities, but they have issues like FIPS to deal with that make new algorithm selection difficult).
Only if you store your key with Microsoft, which is not required or the default if you're using a local account which I assume most privacy sensitive people are.
Unfortunately Microsoft keeps working to destroy that option and force consumers to make a remote account. [0][1] Their consistent moves towards wanting to co-own my computer were one of the many last-straws that made me migrate everything to Linux this year.
> Local-only commands removal: We are removing known mechanisms for creating a local account in the Windows Setup experience (OOBE). While these mechanisms were often used to bypass Microsoft account setup, they also inadvertently skip critical setup screens, potentially causing users to exit OOBE with a device that is not fully configured for use. Users will need to complete OOBE with internet and a Microsoft account, to ensure device is setup correctly.
Not to mention that unless the bitlocker activation flow changed recently, it specifically asks you how to store your backup keys, with a choice given been local options (eg. usb drive, printing it off, etc.) and saving it to your microsoft account.
dell opts you in without telling you. one day you'll just reboot to an unexpected bitlocker screen and have to figure out whether you're getting ransomwared before eventually digging a key out of your microsoft account you weren't aware was there.
Agreed it's optional (I've seen and used that option), but are local accounts even a thing any more? Or are you just referring to "not MDM controlled" accounts?
Tangentially: Microsoft telemetry collects the serial# of your devices and reports it (with your IP and MS account) back to the mothership, and some printers embed their serial# in printed pages.
So take countermeasures if you print something out criticizing any groups that abuse political or law-enforcement powers.
It's still brittle, awkward and puzzlingly awful UX despite being the literal standard for the platform.
Compare it to any of the actively maintained alternatives, Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else) or LUKS on Linux.. heck, even Veracrypt is actually easier to understand and more robust.
FileVault absolutely has an optional iCloud Keychain escrow. That’s how the “unlock with Apple Account” feature works. Apple doesn’t have the keys for iCloud Keychain, but it is still stored in iCloud.
managing a fleet of 100+ laptops with bitlocker is a breeze. its so seemless that the users don't even realize its enabled (i.e. no UX issues, at all).
on the other hand, i am not managing 100+ laptops that use veracrypt. sounds absolutely awful. i've never managed an apple fleet, so i can't speak to that, and will take your word on it.
for personal use, i do not recommend bitlocker (or windows, really), but for already-windows enterprises? absolutely
Flicking a button to turn something on is not what I'm talking about, that's normally the easy part of any setup, and I judge people harshly who only take that aspect of something into consideration when discussing systems.
Brittle is what happens when you haven't logged on to the machine in 60 days, trust with AD is broken, TPM has a glitch and wipes the in device key and forces you into recovery... or god forbid you service the laptop and now you have to enter recovery mode.
Then you're in a nightmare, trying to give someone a super long passphrase over the phone is a not-too-uncommon occurance.
That's assuming you have a good policy for storing the recovery keys. Too loose and they're handed out to everyone, sort of defeating the purpose: too strict and you need the IT department (or specific members), and its still predicated on the notion that you have a policy for it... Given that Admins are a dying breed... I don't think this is workable.
If you compare with Filevault on MacOS: which tracks the credentials of the logged in user; there's no "issue" if the device loses trust because ultimately you always use the real unlock key: not something cached in a "secure storage".
Having dealt with FileVault in this context, it's also frustrating; it's really common to have it fail to follow the logged-in user's credentials, and if you use any kind of federated login, you will frequently get users with FileVault passwords that are either ahead of or behind their system login password.
I think both approaches are valid trade-offs and I think that the default Secure Boot BitLocker configuration, for all its architectural tradeoffs, can probably be credited for an enormous amount of data loss mitigation originating from used hard drives alone.
maybe i am missing something, but how did veracrypt solve all of the admin and policy issues you’re bringing up? (specifically for large enterprise fleets)
We have more issues with FileVault than we do with BitLocker, the latter being a fleet 5 times larger than the former. I find both “fine” for enterprise.
Veracrypt is more difficult to set up - whether on one machine or a fleet. Bitlocker is a few buttons in the UI, configurable via Group Policy, and so much more.
D) Written down on a scrap of paper when I set up the laptop
the fact that they never ask for the passphrase is a weakness of the system. Because now you have an extremely difficult situation as soon as you're off the happy path.
It's also like 64 characters alphanumeric with no capability to copy/paste.
Compare it to Vera/Filevault where the access key is the users passphrase. In MacOS it's literally your account password, which follows along with your in-OS account credentials.
That happens with Veracrypt as well. I have plenty of friends and family who can't remember their WiFi password without remembering where it was written down, and they use that far more often than an encryption recovery code.
In fleets users wouldn't even be setting up their own code.
I've installed Windows thousands of times on dozens, probably hundreds of systems - long ago I even worked on the Windows team and was installing it every day - and in the last 20 years (yes, I ran Vista Ultimate in 2006) I've had to deal with Bitlocker recovery prompts perhaps 20 times - not 20 times per machine, 20 times across all of them.
Or nowhere you know at all if you're a non-technical user who was on a local account Win 11 setup who was tricked by microsoft dark pattern pop-ups getting you to go to "online accounts" which automatically and silently encrypts your drives in the background and then tells you to go to to some shady domain called aka.ms (with another computer, since yours is now locked on a bluescreen and unusable). Basically a typical ransomware message. In truth in this case it's #1 (uploaded to Microsoft) but the non-technical user doesn't know that. Even I thought aka.ms screen was ransomware when my parent called me saying their computer had a "virus".
"iCloud account: Click “Allow my iCloud account to unlock my disk” if you already use iCloud. Click “Set up my iCloud account to reset my password” if you don’t already use iCloud."
"FileVault Full Disk Encryption (FDE) recovery keys are, by default, sent to Apple if the user requests them. Only one payload of this type is allowed per system."
Both Intel/AMD CPUs produced in the last 5 years or so support full transparent (to the OS) memory encryption. So cold boot attacks are a thing of the past if you enable this feature (it's typically disabled because it reduces RAM speed by about 0.5%).
The impact on performance is more along the lines of 1-2% on AMD (though it likely varies by generation (I did extensive benchmarking on Renoir wrt throughput/latency/gpu). But yes small enough to be insignificant unless you run LLMs or game on the iGPU. I imagine that it also uses marginally more power.
AMD also has a second encryption mode where the OS decides what gets transparently encrypted, it doesn't have to be everything. But that mode is poorly documented (or at least the documentation isn't accessible to peasants like me)
Recent news is that this isn't shipping on some consumer-grade CPUs from AMD. There, made it explicit enough there's no room for conversation. Here's the link:
Future AGESA updates will still include memory encryption for all processor models. It's one of the rare instances of consumer backlash creating a clear and immediate response.
The only reason that I do the disk encryption is so that I don't have to worry about people going through my laptop to steal tax documents and/or credit card stuff when I sell the laptop. I of course also wipe the laptop too, but I figure that if the data is encrypted at the drive level then there's very little risk of anyone being able to use some kind of forensics tool and recover data.
The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?
edit: I see now that the prompt was being given and the keys still resided in memory.
The reason this bug is unexpected is that the user is expecting to have to enter their password (because they expect the key to be wiped on suspend), and then _they are_ asked for their password. But there was a copy of the key elsewhere in kernel memory that was never cleared.
> The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?
If that was the case for the people using the debian extra secure extension that should have wiped the memory clean then someone would have found this bug much earlier than two years. Their password was required to be re-entered even though the key was still in memory somewhere.
Did the Feds desperately need a way of getting the key? is this a bugdoor? Has the commits been traced?
Recently I’ve been seeing this pattern a lot and I’m starting to be a little bit suspicious. Maybe it’s because people are more sensible to this and post more on it?
it is a regression. the user space application also would silently fail, it is a chain of oversights.
also having the encryption keys in memory does not mean you can extract them, it is more of unnecessarily letting it there indefinitely, not having it where it shouldn't be.
The whole point of luksSuspend is to not have the encryption keys in memory, as that actually does mean they could be extracted by an attacker who has taken the hardware.
I am far from a security expert, but from the number of "we missed a single line C check across files during refactoring" critical security bugs discovered on a regular basis these days, the whole premise of a "giant secure open source C codebase" seems questionable. It is not specific to C of course, but invariants are arguably even harder to enforce and track consistently (esp under changes to code) in C. Unsure if FP with invariants encoded in types is a practically feasible scalable solution either. Model checking? [LLM] fuzzing? Fewer primitives with clear boundaries? Is that how seLinux was "checked"?
While I can see the shortcomings of C and generally don't recommend it for new projects I don't see this particular bug as a good example of something Rust's borrow checker or some other language's type system will catch. I don't think even static analyzers can catch this.
In my experience a substantial portion of gnarly bugs come down to a violation of a high-level system invariant and those do not strike me as something that can be automated. Even with something like Lean you can prove your program satisfies certain properties but you need to have thought about those properties in the first place. The proof doesn't discover the invariant for you.
If you'd had thought about the relevant security property you could have written a regression test for it which is not hard. IMO the really hard part isn't expressing the implementation safely, but it's the realization that this was a property the implementation needed to preserve.
I agree re Rust vs C - this is not (only) a language issue. What would (roughly) the invariant be here?
In another thread comment below i argue that maybe the system (OS) itself is so complex that it lacks clear contract / the contract evolves too quickly over time (as other parts of the code need to change the given piece of code to extend it to their use case) and that defies clear encoding?
Or we lack easy enough means to describe specs? I tried reading jepsen spec earlier today and despite it being an "integration test" of sorts, it is far from "simple".
Can an entire OS or a system of comparable complexity be decomposed into objects simple enough that their entire intended behavior (with all edge cases) can be explained in a paragraph of human text + half a screen of dense behavioral "spec" - if i do X and do Y, Z should come out / hold _no matter what happens in-between_. Or that's what asserts + fuzzing is effectively supposed to do? Is there a clear distinction between invalid input and failed invariant in typical C code? I guess error code vs seg fault?
This is in effect a state machine, and when you have a type system more complex than C's you can encode state transitions in the type system (either by having state transitions explicitly return a new return type or by using sum types). You still need to architect the system to encode the invariants in types. No language will fix all logic bugs for free. But you can leverage language features to reduce their number.
> You still need to architect the system to encode the invariants in types.
That's the problem though, right? If it's pointed out we all agree the "do not keep credentials alive" is a property that should hold and we can leverage whatever the environment offers to help preserve it. I fully agree modern languages have amazing support for this, but in C you can still run tests. Let's just say I don't think the language's inability to express logic of this kind held all those involved back from testing for it. I personally find "we just didn't think of it" much more likely.
That said, I am not a fan of C and recommend leveraging whatever fantastic modern tooling is available to you.
Like set an generic marker struct IsEncrypted<T> where T is yes or now and only allow its state to change when proven and then write the shutdown function to only take the yes variant?
If you think open source is bad, wait till you see enterprise code. I'm talking full auth bypass due to the stupidest crap. You can do that in any language if you have fools working on the code base.
I explicitly make sure services I lead have Integration tests in CI pipeline to validate the "negative paths" against all APIs with missing, invalid, un-authorised identities, expired, un-authenticated tokens. Of course that still doesn't cover every surface, but even that gets sideways glances from some folks who think we should just test happy paths and why we're testing for access controls in Integration tests.
Even security code. Fortinet, a vendor whose entire thing is security for your network, is consistently getting caught out with default passwords, backdoors, etc.
"It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt" is a reason why people don't open-source more often.
To translate to Rust, it would have been "we missed a single line Rust check"...
This is a bug involving intersecting concerns and a deficit of cross-domain knowledge. It probably would have been the same in Lisp or assembly language.
Yes, I agree. I find the addition of the regression test the true long-term fix. The code is just an opaque incantation that may or may not preserve some property we find worth preserving and we have no way of knowing it keeps preserving it over time as other parts of the system change.
The test actually proves it and while it too can change it has more staying power because it's expressed at a higher level of abstraction ("random arcane weird C shit" in the case of code versus "does this property hold" in the case of a regression test).
I have not looked into this specific issue, but are we sure that a regression here could have been avoided via a localized test? Maybe issues seem to arise from A implementing a feature with tests. B seeing that A lacks some functionality and adding it (potentially with tests), C seeing this (extra) functionality in A, and using in unintended ways not covered by tests (or in an unintended environment) + multiply by many layers of this A-B-C story up and down the stack.
The whole premise of a "giant secure open source C codebase" seems questionable
Because code review is sometimes not much different from an idealized version of the halting problem, where you would have access to a formalized version of a specification.
In other words, there is no strict definition of what is a security issue.
On the other hand, it is (both halting and spec adherence) are checkable under compute and space constraints though? :) I'd say the biggest hurdle are means to describe the spec in way that is easy enough for a human to produce to make it feasible.
Not a DB person either, but things like TLA+ seem very hard to write even with LLMs. Behavioral tests with an enumerable number of random paths to take (aka model checking - eg jepsen) seem more feasible. Although you can't check internal properties of the system (string `pass` or any of it's copies or parts are not held anywhere in memory at any point between lines A and B) unless we can check that two memory dumps are indistinguishable with different pass strings (assuming we abstracted away storage devices in a test environment).. Also not sure if it's "easy enough" to write such tests either.
Maybe the reason is that OS domain objects / primitives are too complex and not "isolatable" enough / lack a clear contract at all? (Hence multi file refactorings that break invariants.)
Not sure why you're getting downvoted, this is the entire point of open source.
Does such a bug exist in Windows? OSX? Who checks? If someone finds the key in memory, can they tell what conditions might be causing it and where?
Their only recourse under those situations is to hand it off to the OS Vendor and trust that what they implement does solve the problem, and trust that it wasn't a deliberate back-door that is now being replaced by another back-door.
Oh, and large companies quite often fix these horrific issues silently, especially in online services where the customer can't diff bins. We're talking auth bypasses and RCE's that you'll never know about.
Maybe but still a little better than closed source like Windows. Everytime whem someone asked me if I could hack my way into his Windows PC I always told "After all it's Windows, how bad can it be?" Doing that since 25 years still waiting for a Windows machine that doesn't open... On the other hand I failed to open about 50% of Apple Devices I was asked to open and about 10% of Linux machines. (Not because Linux is insecure by itself but because most Linux distros install with insecure defaults and users don't care.)
On my laptop with Fedora I just configured Linux to hibernate to disk after 15 minutes of suspend. Powering memory off ensures that bugs like this Debian-specific would not matter.
Plus what Debian extension to Linux tooling does although nice in theory, but in practice if one really worries about cold-boot attacks, then all keys and important documents has to be wiped out from memory, not only LUKS keys.
So hibernating is really the only proper way to protect against cold boot.
Interesting idea. On the other hand on the latest SSD with hardware encryption the raw disk speed under Linux can be over 5 GB/s so on my laptop with 64 GB of RAM the full restoration from disk takes like 45 seconds. With LUKS it is like 2 times slower. Which is not a problem at all. So I do not see much value in memory encryption in suspend.
Do you mean with with current software? How to avoid it in general is straightforward.
If you're the only one with the disk password then the simple answer is make both passwords the same and make the different parts of the system communicate better.
If you want multiple users, give them each a different boot password and encrypt a separate copy of the disk key with each one. That password can be their login password too, or it can encrypt their login.
> Except that, for more than two years, the encryption key remained resident in memory across suspend, leaving it there for the taking by anyone who seized the still-powered laptop.
I don't get it. Obviously, the laptop is locked when it resumes, how is that key "for the taking by anyone"? I'm not saying it is impossible to read out RAM from a locked laptop, but surely not by "anyone".
There are attacks that allow dumping RAM if the device is powered on though and you have physical access. Depending on config it may be very easy (just plug in a dumper over Thunderbolt on USB C and do direct memory access) or hard (freeze and swap physical RAM to an unlocked machine).. but the idea was defense-in-depth here; a well configured device should both be hard to dump RAM on and it should not give encryption keys if an attacker succeeds.
Anyone with physical access. I think it is understandable from the phrase.
There is a common misconception about how lock-screens in general work - they usually just prevents using the current hardware and software as it is to access the current OS. But the disk encryption is the main thing that prevents modification and other kind of access to actual data. And if the disk encryption key is lying in the memory, then effectively, the disk encryption is bypassed if someone can access the machine physically and assuming that there are no sufficient tampering protections in place for that machine.
Anyone with physical access, significant tools, and experience. The FBI has people who can pull data out of memory after freezing the RAM but the average laptop thief doesn’t so how serious this is depends significantly on your threat model. If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.
While that is true, the fact is that encryption is technically useless for anyone who is not constantly powering off and on, which is surely a bunch of people. That this is not widely understood is IMO a problem. And excellent reason for Debian's feature.
> Anyone with physical access. I think it is understandable from the phrase.
Sorry, I'm probably dense, I still don't get it. You steal a laptop, you open it, the screen is locked with a password/fingerprint whatever. How do you read out the RAM from that laptop?
Several options. One is you restart and boot from a live system where you are root, and then dump all memory. This is described in the paper with the witty title "Lest We Remember: Cold Boot Attacks on Encryption Keys":
Other options: DMA attacks. Also you never know what the Intel Management Engine hidden in your computer is doing. It's running a version of Minix you don't have any control over, and it has full access to memory.
on the subject of encryption keys and memory there is something you can do:
- if your CPU supports it, enable memory encryption.
- if your TPM module supports this look for MemoryOverwriteRequestControl & MemoryOverwriteRequestControlLock (/sys/firmware/efi/efivars/) and toggle them. make sure that your computer always reboots and never powers off. memory will always be wiped on boot.
It's because of vulnerabilities like this that I enable Intel's "total memory encryption" feature. No plaintext leaves the CPU package. DIMM swap attacks become useless. Moreover, it's basically free: the cryptography happens directly in the memory controller, in hardware, inline with the bus transactions the CPU is doing anyway.
I don't see how that solves this problem. there is a string in memory that gets saved on suspend. that string when read by the CPU has the same properties it had before. if the CPU is using rot-13, the string is still rot-13 and the attacker doesn't need to spend the compute needed to crack rot-13, the CPU will simply do that as normal.
Definitely not a symptom of Linux being a hodgepodge of code thrown together from a thousand different sources and no one person could tell you how it all fits.
Bugs happen in all code. The difference is, anyone can fix stuff in open source. Closed source bugs are out of control and must be worked around. Usually by switching to OSS
I wonder if you think other OSes are any different?
TempleOS is the only thing that comes to mind that doesn't fit your description and it's not practically useful.
Any sufficiently large codebase is a mix of ideas and concepts implemented by different people with different priorities over a large timespan and if you can fit the entire thing in your head it's not very interesting or complex.
Qubes OS, the Linux distribution aspiring to offer a reasonably secure operating system, pioneering a "every app runs in a virtual machine" approach in the Linux laptop/desktop space, tracks this at the following issue:
The *BSDs, Mac, and Windows all keep critical code in the same tree as the OS.
Something like disk encryption would be immediately visible.
So you don't have this mess of 80 different distros with 60 different versions of systemd, 20 that don't use it, a million kernel versions and it's all thrown together in a Costco-sized trash bag and we call the output "Linux".
In my experience any software system (not just operating system) after crossing a certain limit on complexity and age looks exactly as hodgepodge of code pieces thrown together, sometimes from different sources even if developed by one org. All major OSs have long crossed those limits, I believe.
Windows for ages did not really keep all the code in one repo. There were like a dozen parallel repos for e.g. the shell, kernel, IE, etc. Also every feature was developed on team-level branches; integrating all those branches often caused unexpected bugs.
While it is certainly an interesting bug, I kinda feel that the title is click bait? Because this `cryptsetup luksSuspend` from what I understood is not really officially supported but an extension done in Debian, so if anything this regression only affected Debian? I am not sure if you can blame the kernel for something that is not supported or even widely tested.
I still find this impressive, and it is nice that we now have a test (NixOSTests BTW are awesome, I agree with OP) to avoid this regression from coming back. But from the title it seems to be a widespread issue, not something that affects only one Distro.
Sorry, aimed for a technically precise title and didn't want to bait clicks.
Yes, this does not affect people on stock configurations for the plain reason that they wouldn't expect the volume key to be safe during suspend anyway.
Debian's solution was ported to several (most?) other distributions and I guess quite a few people maintained private ports.
The thread-keyring(7) manpage promises: "A thread keyring is destroyed when the thread that refers to it terminates." For their key upload (from userspace to kernelspace) mechanism, the cryptsetup project relied on this property; but kernel 6.9 introduced a regression invalidating this property.
what debian version first shipped 6.9?
I don't see any other way? When you sleep (suspend to RAM), everything is stored in RAM and is encrypted but the master key is present in kernel memory (if I recall correctly).
However, if you hibernate (suspend to disk) the entire contents of RAM (including the master key) is written/encrypted to disk and the RAM is cleared.
When you wake the machine up you have to re-enter the passphrase to decrypt the master key to re-load disk contents back to memory.
Yes, if you simply suspend your laptop on most stock Linux distributions, then everything including the master key is still kept in memory. But Debian pioneered the (optional) cryptsetup-suspend addon. This issues a luksSuspend command which is supposed to wipe the key from memory, and on resume asks you to resupply your passphrase.
Up to kernel 6.8, this worked as described; starting with kernel 6.9, it silently didn't.
So you would still be asked for a passphrase, even though it's already available?
Exactly. Cryptsetup wouldn't know about the extra copy of the volume key in kernel memory. Which is why, dramatically, it appeared secure ("surely I wouldn't be asked to resupply the passphrase if the volume key is still in memory, right?").
It was still more secure than the default if I understand this correctly. On resume from suspend the laptop would still be locked by the encryption key and without access to the disk even if you can somehow circumvent the lock. The only insecurity was that somewhere in the kernel memory the key still exists so if you can somehow extract that from the live system you can unlock it.
Yes, you are right: LUKS encryption protests your data at rest. An attacker which steals your disk can only gain little, like the information that you have used LUKS (unless you put your LUKS headers elsewhere, separated from the disk) and perhaps disk and disk sector usage statistics.
You need to get quite specific on actual attacks to call this insecure to be clear.
Having access to the raw RAM of a machine suspended but demanding the key to resume is certainly possible but the number of attacks where you don't need this bug is "almost all of them" given at that point if the machine ever unlocks you won in this hypothetical attack even with a bug fix.
I don’t know about “almost all”.
If the key has been purged but you can read RAM, then you can do two things:
1. You can extract whatever user data happens to be in RAM.
2. If you can either write RAM or reboot into your own OS, and then return the device to the unsuspecting user who will put in their password, then you can run a fake password dialog and get everything.
1 is bad, since there may be quite a lot of user data in RAM. But it’s not quite as bad as having the disk key, which gives the attacker all the data plus the future ability to decrypt or modify user data given only the physical disk. (Still, a better solution would be encrypting the hibernation image, preventing this attack entirely.)
2 is fully bad, but in many plausible scenarios (e.g. seized device) the attacker cannot just return the device to the user without them knowing something happened. Or even if they can, the method of RAM access may be one where reads are much more practical than writes, such as cold boot attacks involving physically swapping out the RAM.
makes me wonder if there is potential for a more "main stream"/by default friendly version of this, where the key during suspend is encrypted using the TPM even if the TPM isn't a possible unlock from cold boot (i.e. no TMP encrypted volume key in the LECS headers/meta only temporary in memory during suspend)
or the alternative (for more convenient usage) for single user systems auto login on boot + use disc password for doas/sudo?
I've been wondering why hibernate didn't work with encryption, because this seems like the extremely obvious way to handle it, but I have struggled to find anything about it for years - glad to hear it does exist!
But yeah, also rather obviously it's inherently a bit leak-prone. Though it seems probably pretty simple to test, just hibernate and scan all stored data. They could probably even do it on shutdown, as a hash of the key data would be sufficient to detect the key.
FYI: VeraCrypt is not the defacto encryption software for Windows.
Oh, which one is it?
(You don't mean BitLocker, right?)
It absolutely is and they have most the enterprise market.
Okay, yes, sure. It definitely is the most-used encryption software for Windows.
But I would never trust it a second, being proprietary and known for issues. You likely know that, but for the benefit of others:
38C3 - Windows BitLocker: Screwed without a Screwdriver https://media.ccc.de/v/38c3-windows-bitlocker-screwed-withou... https://www.youtube.com/watch?v=5eNtT2p12cM
If you’re at all serious about security and not user convenience, you deploy BitLocker with a PIN instead of TPM only. And then a whole class of vulnerabilities goes away.
It's probably all security theater. There's only so much trust you can put into some shitty vendor's TPM implementation
"Disk must be in expected hardware environment" versus "Same environment plus PIN" makes a huge difference if a thief simply steals a whole computer.
Just a PIN? For most people that's a 4-digit number, which has a worst-case scenario of 10,000 attempts and a median of only a few hundred. Why not use a full 8-digit password?
Because the TPM effectively rate limits brute forcing of the key.
https://learn.microsoft.com/en-us/windows/security/hardware-...
> For example, when BitLocker is used with a TPM + PIN configuration, the number of PIN guesses is limited over time. A TPM 2.0 in this example could be configured to allow only 32 PIN guesses immediately, and then only one more guess every two hours. This totals a maximum of about 4,415 guesses per year. If the PIN is four digits, all 9999 possible PIN combinations could be attempted in a little over two years.
In that case, the median would still be just over a month, if the PINs were entered in order of how commonly they are used. Even the worst case of two years is still soon enough for a lot of data still be useful.
Also, how is the time limit enforced? With hardware access, it would be easy to change time or increase the clock rate, as well as many other side-channel attacks that could eliminate the wait altogether.
The time limit is enforced by the TPM itself which defends against tampering.
> the TPM effectively rate limits
I had a friend working at trusted compute at Microsoft, and he had so many stories.
These TPM firmwares are often written by shitty companies that have no fxcking clue what they are doing.
Most TPM implementations are a clown show, companies just want to check a box on paper so they say "look! We have a TPM!" and move on.
No one uses a 4-digit pin for BitLocker. No one who knows what they are doing, anyway.
My employer requires at least an 18-digit PIN, and not just numbers, either.
If you're really serious, you use a strong password, not a PIN.
If you are at all serious about security you don't consider Windows.
Depending on how serious you are you also don't consider MacOS.
And then you kinda have a couple of things to chose from but ultimately you need to build your own security depending on your attack/threat model
And then depending on how "serious" you are you also don't consider Linux.
But also, threat models and the best way to mitigate them aren't really a linear scale of being <unserious> to <serious>, but a complex consideration of a particular situation.
People just plain suck at opsec. Like Che Guevara might have had a longer career as a revolutionary if he'd used his one time pads only once.
Back in the late 1980s it was clear that it would be no problem at all to hook up a hard drive to a digital phone exchange and record all the calls! I had a strict policy of "don't talk about anything illegal using electronic communication" even when it was rather banal stuff like selling weed.
The carelessness of people at Facebook documenting policies that nobody in their right mind would document boggles my mind: you might as well leave it mysterious why you didn't crack down on scam ads, for instance. When I've been involved in minor conspiracies, say when we had an HR problem with another employee, I've always made the point to meet furtively in person and avoid leaving a paper trail so that I'd never need to explain an email I wrote in front of an unfriendly audience.
The issues you linked with BitLocker are obvious properties of BitLocker-with-SecureBoot-only architecture. If you configure Linux that way, you get similar issues (for example, it's pretty easy to mis-configure TPM sealed disk encryption on Linux to still allow a recovery shell, which will run with the disk unsealed).
BitLocker with a password (the equivalent of the LUKS configuration in question) does not share these issues.
Bitlocker with a password has always felt like a second class citizen to me. You have to dig into a bunch of group policies to use it. Maybe most people don't even realize it exists.
Yah, it seems blatantly hostile how much they hide it.
I can understand the default being TPM-only + online key backup, huge amounts of the population forget their login passwords (which can be involuntary, e.g. head injury) and huge amounts of them still want some backup way to access their data rather than losing it forever.
But for anyone who cares just a little more, or would prefer to lose data in those situations, it's such an abnormal and hidden path that it's clearly blocking tons of people from choosing it.
For the system drive they seem to really strongly prefer PIN, which also doesn't have the problems linked above. I was going to use PIN as my example but didn't want to explain another set of recent BitLocker conspiracy theories yet again; maybe I should have.
It is annoying that they hate password for system drive _so_ much; the reason is actually pretty obvious when you think about how their "happy path" AD-driven enterprise deployment with stupid password rotation requirements works (and FileVault is a nightmare in this scenario), but I wish they'd make it easier for individual power users.
veracrypt lost their drivers license so afaik you should avoid it since it cannot update its drivers any longer. didnt see any news about them reacquiring that license
Assuming this is what you are referring to, it was resolved within a few days. The incident being resolved just didn't make headlines. https://sourceforge.net/p/veracrypt/discussion/general/threa...
If you think for one single second that businesses and governments who rely on a lost disk being secure don’t trust bitlocker, I have oceanfront property in Missouri to sell you.
Bitlocker + PIN is as secure as anything.
A vulnerability can’t leak your key if the TPM doesn’t know the entire key and relies on the user to supply the missing parts of the key in the form of a PIN.
> A vulnerability can’t leak your key if the TPM doesn’t know the entire key and relies on the user to supply the missing parts of the key in the form of a PIN.
First off: I agree with your thesis, BitLocker with PIN is Just Fine, equivalent in all practical senses to most disk encryption strategies, and an enterprise standard.
I post this to reinforce what you're saying, because there are a ton of weird theories about how this works that make people think it's weaker than it is.
BitLocker with PIN works like this:
* BitLocker seals an encrypted key IK into the TPM using a policy on the TPM which requires the SHA-256 of the PIN to be sent to the TPM to unlock the record (and has anti-hammering at the TPM level).
* Encryption using another key called the SK. Once the OS acquires the e(IK) from the TPM, it needs to derive SK to decrypt the IK. The SK is generated by applying a salt and 0x100000 rounds SHA256 to the PIN.
* The IK can then decrypt the VMK, which can decrypt the FVEK which is the actual data encryption key used on the on-disk data.
Fully offline (no TPM access) decryption is hopeless; the IK is random and not attached to the PIN at all.
Fully online decryption relies on TPM anti-hammering; if you can manage to hammer the TPM and find a PIN where h(PIN) matches, you can then use the PIN to generate both h(PIN) (unseal TPM) and reverse e(IK).
Partially offline decryption (imagining TPM content is somehow dumped, for example by bus sniff while a user uses their PIN) relies on bruteforcing the key for e(IK), which is possible but difficult (IMO Microsoft should ratchet this up or use a more expensive algorithm than SHA-256 in order to deal with modern hardware capabilities, but they have issues like FIPS to deal with that make new algorithm selection difficult).
Reminder that by using Bitlocker, you're using a closed source encryption for which Microsoft will happily hand out your recovery key on request.
https://www.forbes.com/sites/thomasbrewster/2026/01/22/micro...
Only if you store your key with Microsoft, which is not required or the default if you're using a local account which I assume most privacy sensitive people are.
> if you're using a local account
Unfortunately Microsoft keeps working to destroy that option and force consumers to make a remote account. [0][1] Their consistent moves towards wanting to co-own my computer were one of the many last-straws that made me migrate everything to Linux this year.
> Local-only commands removal: We are removing known mechanisms for creating a local account in the Windows Setup experience (OOBE). While these mechanisms were often used to bypass Microsoft account setup, they also inadvertently skip critical setup screens, potentially causing users to exit OOBE with a device that is not fully configured for use. Users will need to complete OOBE with internet and a Microsoft account, to ensure device is setup correctly.
[0] https://blogs.windows.com/windows-insider/2025/10/06/announc...
[1] https://www.windowslatest.com/2025/10/07/microsoft-confirms-...
Not to mention that unless the bitlocker activation flow changed recently, it specifically asks you how to store your backup keys, with a choice given been local options (eg. usb drive, printing it off, etc.) and saving it to your microsoft account.
dell opts you in without telling you. one day you'll just reboot to an unexpected bitlocker screen and have to figure out whether you're getting ransomwared before eventually digging a key out of your microsoft account you weren't aware was there.
Agreed it's optional (I've seen and used that option), but are local accounts even a thing any more? Or are you just referring to "not MDM controlled" accounts?
Tangentially: Microsoft telemetry collects the serial# of your devices and reports it (with your IP and MS account) back to the mothership, and some printers embed their serial# in printed pages.
So take countermeasures if you print something out criticizing any groups that abuse political or law-enforcement powers.
Bitlocker can use keys that are local only, but the default for home editions of Windows was to use the online account to back it up.
'Happily' is also a stretch, as they really don't have a choice if served a valid court order.
If you want encryption that is safe from the US government, keys need to be stored in your head. Anything physical is subject to court orders.
for enterprises, where this doesn't really matter, bitlocker is great.
if by "great" you really mean "fine".
It's still brittle, awkward and puzzlingly awful UX despite being the literal standard for the platform.
Compare it to any of the actively maintained alternatives, Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else) or LUKS on Linux.. heck, even Veracrypt is actually easier to understand and more robust.
FileVault absolutely has an optional iCloud Keychain escrow. That’s how the “unlock with Apple Account” feature works. Apple doesn’t have the keys for iCloud Keychain, but it is still stored in iCloud.
>if by "great" you really mean "fine".
no, i mean great.
managing a fleet of 100+ laptops with bitlocker is a breeze. its so seemless that the users don't even realize its enabled (i.e. no UX issues, at all).
on the other hand, i am not managing 100+ laptops that use veracrypt. sounds absolutely awful. i've never managed an apple fleet, so i can't speak to that, and will take your word on it.
for personal use, i do not recommend bitlocker (or windows, really), but for already-windows enterprises? absolutely
Flicking a button to turn something on is not what I'm talking about, that's normally the easy part of any setup, and I judge people harshly who only take that aspect of something into consideration when discussing systems.
Brittle is what happens when you haven't logged on to the machine in 60 days, trust with AD is broken, TPM has a glitch and wipes the in device key and forces you into recovery... or god forbid you service the laptop and now you have to enter recovery mode.
Then you're in a nightmare, trying to give someone a super long passphrase over the phone is a not-too-uncommon occurance.
That's assuming you have a good policy for storing the recovery keys. Too loose and they're handed out to everyone, sort of defeating the purpose: too strict and you need the IT department (or specific members), and its still predicated on the notion that you have a policy for it... Given that Admins are a dying breed... I don't think this is workable.
If you compare with Filevault on MacOS: which tracks the credentials of the logged in user; there's no "issue" if the device loses trust because ultimately you always use the real unlock key: not something cached in a "secure storage".
Having dealt with FileVault in this context, it's also frustrating; it's really common to have it fail to follow the logged-in user's credentials, and if you use any kind of federated login, you will frequently get users with FileVault passwords that are either ahead of or behind their system login password.
I think both approaches are valid trade-offs and I think that the default Secure Boot BitLocker configuration, for all its architectural tradeoffs, can probably be credited for an enormous amount of data loss mitigation originating from used hard drives alone.
maybe i am missing something, but how did veracrypt solve all of the admin and policy issues you’re bringing up? (specifically for large enterprise fleets)
If you use your key every day you tend not to forget it.
If I as an admin give you your key: it is “leaked” effectively.
>If you use your key every day you tend not to forget it.
hoping users don’t forget their password is a very weak policy.
specifically, the policy and admin points you brought up above, how does veracrypt solve them?
Have you never gone on vacation and forgotten your daily-use password upon return?
Managing an Apple fleet is similarly fine, and that includes using any of the MDM tooling that also does key escrow on enterprise Filevault devices.
We have more issues with FileVault than we do with BitLocker, the latter being a fleet 5 times larger than the former. I find both “fine” for enterprise.
Veracrypt is more difficult to set up - whether on one machine or a fleet. Bitlocker is a few buttons in the UI, configurable via Group Policy, and so much more.
What is brittle or awkward?
"PLEASE ENTER YOUR BITLOCKER RECOVERY KEY"
Where is it?
A) Uploaded to microsoft
B) Somewhere in EntraID?
C) Somewhere in our onprem AD?
D) Written down on a scrap of paper when I set up the laptop
the fact that they never ask for the passphrase is a weakness of the system. Because now you have an extremely difficult situation as soon as you're off the happy path.
It's also like 64 characters alphanumeric with no capability to copy/paste.
Compare it to Vera/Filevault where the access key is the users passphrase. In MacOS it's literally your account password, which follows along with your in-OS account credentials.
That happens with Veracrypt as well. I have plenty of friends and family who can't remember their WiFi password without remembering where it was written down, and they use that far more often than an encryption recovery code.
In fleets users wouldn't even be setting up their own code.
I've installed Windows thousands of times on dozens, probably hundreds of systems - long ago I even worked on the Windows team and was installing it every day - and in the last 20 years (yes, I ran Vista Ultimate in 2006) I've had to deal with Bitlocker recovery prompts perhaps 20 times - not 20 times per machine, 20 times across all of them.
Or nowhere you know at all if you're a non-technical user who was on a local account Win 11 setup who was tricked by microsoft dark pattern pop-ups getting you to go to "online accounts" which automatically and silently encrypts your drives in the background and then tells you to go to to some shady domain called aka.ms (with another computer, since yours is now locked on a bluescreen and unusable). Basically a typical ransomware message. In truth in this case it's #1 (uploaded to Microsoft) but the non-technical user doesn't know that. Even I thought aka.ms screen was ransomware when my parent called me saying their computer had a "virus".
> Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else)
Did you read the documentation?
https://support.apple.com/guide/mac-help/protect-data-on-you...
"iCloud account: Click “Allow my iCloud account to unlock my disk” if you already use iCloud. Click “Set up my iCloud account to reset my password” if you don’t already use iCloud."
https://developer.apple.com/documentation/devicemanagement/f...
"FileVault Full Disk Encryption (FDE) recovery keys are, by default, sent to Apple if the user requests them. Only one payload of this type is allowed per system."
Can.
If you click "Allow my iCloud account to unlock my disk", your recovery key is escrowed to Apple, tied to your Apple Account.
If you don't select that option it never does.
I should have said "without your explicit permission", but I assumed we were all adults and understood that.
The main point is that it's using your account password to unlock, the recovery key is for if you forget your account password.
No, you were just plain wrong. You said “never”, when in reality BitLocker and FileVault both have optional escrow.
Does that mean it's not the de facto standard on Windows?
So exactly like FileVault?
Both Intel/AMD CPUs produced in the last 5 years or so support full transparent (to the OS) memory encryption. So cold boot attacks are a thing of the past if you enable this feature (it's typically disabled because it reduces RAM speed by about 0.5%).
The impact on performance is more along the lines of 1-2% on AMD (though it likely varies by generation (I did extensive benchmarking on Renoir wrt throughput/latency/gpu). But yes small enough to be insignificant unless you run LLMs or game on the iGPU. I imagine that it also uses marginally more power.
AMD also has a second encryption mode where the OS decides what gets transparently encrypted, it doesn't have to be everything. But that mode is poorly documented (or at least the documentation isn't accessible to peasants like me)
Recent news is that this isn't shipping on some consumer-grade CPUs from AMD. There, made it explicit enough there's no room for conversation. Here's the link:
https://arstechnica.com/security/2026/06/users-cry-foul-afte...
AMD backtracked on that strategy: https://www.techpowerup.com/350142/amd-to-restore-tsme-memor...
Future AGESA updates will still include memory encryption for all processor models. It's one of the rare instances of consumer backlash creating a clear and immediate response.
This is one of those regressions that's easy to miss because everything still "works." Security bugs often don't announce themselves.
Right! Which is why integration tests for these kinds of features are all the more important.
It was also fun to write, and enabled git-bisecting to isolate the specific kernel refactoring which introduced this bug: https://github.com/NixOS/nixpkgs/pull/532499
I don't think this bothers me.
The only reason that I do the disk encryption is so that I don't have to worry about people going through my laptop to steal tax documents and/or credit card stuff when I sell the laptop. I of course also wipe the laptop too, but I figure that if the data is encrypted at the drive level then there's very little risk of anyone being able to use some kind of forensics tool and recover data.
Assuming the encryption key is strong, the wiping is theoretically redundant.
And assuming the crypto algorithm has no fundamental flaws, it's applied correctly, and the software implementation has no bugs.
All of which are things people have on occasion believed to be true and found out later they were wrong about.
Agreed. It's also very low effort and as such I'm ok with the redundancy.
Indeed. Truth be told, I do it too.
I don't have to re-enter my boot password after Sleep, so obviously the encryption key is still in memory.
Obviously your distro isn’t using cryptsetup-luksSuspend.
Correct.
The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?
edit: I see now that the prompt was being given and the keys still resided in memory.
Because debian users with luks-suspend have to re-enter their boot password after sleep.
The reason this bug is unexpected is that the user is expecting to have to enter their password (because they expect the key to be wiped on suspend), and then _they are_ asked for their password. But there was a copy of the key elsewhere in kernel memory that was never cleared.
Ah, my bad. Yes, if the user was being presented with the prompt on wake, I see the problem.
I have never had that setup so I was confused.
> The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?
If that was the case for the people using the debian extra secure extension that should have wiped the memory clean then someone would have found this bug much earlier than two years. Their password was required to be re-entered even though the key was still in memory somewhere.
Well, potentially a key might be stored in TPM. But I don't think that's better
I would hope that it is harder to get into the TPM than into the RAM.
Did the Feds desperately need a way of getting the key? is this a bugdoor? Has the commits been traced? Recently I’ve been seeing this pattern a lot and I’m starting to be a little bit suspicious. Maybe it’s because people are more sensible to this and post more on it?
it is a regression. the user space application also would silently fail, it is a chain of oversights. also having the encryption keys in memory does not mean you can extract them, it is more of unnecessarily letting it there indefinitely, not having it where it shouldn't be.
The whole point of luksSuspend is to not have the encryption keys in memory, as that actually does mean they could be extracted by an attacker who has taken the hardware.
I am far from a security expert, but from the number of "we missed a single line C check across files during refactoring" critical security bugs discovered on a regular basis these days, the whole premise of a "giant secure open source C codebase" seems questionable. It is not specific to C of course, but invariants are arguably even harder to enforce and track consistently (esp under changes to code) in C. Unsure if FP with invariants encoded in types is a practically feasible scalable solution either. Model checking? [LLM] fuzzing? Fewer primitives with clear boundaries? Is that how seLinux was "checked"?
While I can see the shortcomings of C and generally don't recommend it for new projects I don't see this particular bug as a good example of something Rust's borrow checker or some other language's type system will catch. I don't think even static analyzers can catch this.
It's basically something like this:
original: DoTheThing()
new: DoTheThingSlightlyDifferentButKeepMyCredentialsAlive()
fix: DoTheThingSlightlyDifferentButDoInFactNOTKeepMyCredentialsAlive()
In my experience a substantial portion of gnarly bugs come down to a violation of a high-level system invariant and those do not strike me as something that can be automated. Even with something like Lean you can prove your program satisfies certain properties but you need to have thought about those properties in the first place. The proof doesn't discover the invariant for you.
If you'd had thought about the relevant security property you could have written a regression test for it which is not hard. IMO the really hard part isn't expressing the implementation safely, but it's the realization that this was a property the implementation needed to preserve.
I agree re Rust vs C - this is not (only) a language issue. What would (roughly) the invariant be here?
In another thread comment below i argue that maybe the system (OS) itself is so complex that it lacks clear contract / the contract evolves too quickly over time (as other parts of the code need to change the given piece of code to extend it to their use case) and that defies clear encoding?
Or we lack easy enough means to describe specs? I tried reading jepsen spec earlier today and despite it being an "integration test" of sorts, it is far from "simple".
Can an entire OS or a system of comparable complexity be decomposed into objects simple enough that their entire intended behavior (with all edge cases) can be explained in a paragraph of human text + half a screen of dense behavioral "spec" - if i do X and do Y, Z should come out / hold _no matter what happens in-between_. Or that's what asserts + fuzzing is effectively supposed to do? Is there a clear distinction between invalid input and failed invariant in typical C code? I guess error code vs seg fault?
This is in effect a state machine, and when you have a type system more complex than C's you can encode state transitions in the type system (either by having state transitions explicitly return a new return type or by using sum types). You still need to architect the system to encode the invariants in types. No language will fix all logic bugs for free. But you can leverage language features to reduce their number.
> You still need to architect the system to encode the invariants in types.
That's the problem though, right? If it's pointed out we all agree the "do not keep credentials alive" is a property that should hold and we can leverage whatever the environment offers to help preserve it. I fully agree modern languages have amazing support for this, but in C you can still run tests. Let's just say I don't think the language's inability to express logic of this kind held all those involved back from testing for it. I personally find "we just didn't think of it" much more likely.
That said, I am not a fan of C and recommend leveraging whatever fantastic modern tooling is available to you.
Like set an generic marker struct IsEncrypted<T> where T is yes or now and only allow its state to change when proven and then write the shutdown function to only take the yes variant?
The premise of a secure open codebase is fine.
The problem is being more auditable does not automatically make it more audited.
There have to be enough people with skill taking enough time to work on it.
If you think open source is bad, wait till you see enterprise code. I'm talking full auth bypass due to the stupidest crap. You can do that in any language if you have fools working on the code base.
I explicitly make sure services I lead have Integration tests in CI pipeline to validate the "negative paths" against all APIs with missing, invalid, un-authorised identities, expired, un-authenticated tokens. Of course that still doesn't cover every surface, but even that gets sideways glances from some folks who think we should just test happy paths and why we're testing for access controls in Integration tests.
Even security code. Fortinet, a vendor whose entire thing is security for your network, is consistently getting caught out with default passwords, backdoors, etc.
https://community.spiceworks.com/t/hard-coded-password-backd...
This sort of thing leads to every kind of exploit, like
https://www.linkedin.com/pulse/half-worlds-fortinet-firewall...
"It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt" is a reason why people don't open-source more often.
To translate to Rust, it would have been "we missed a single line Rust check"...
This is a bug involving intersecting concerns and a deficit of cross-domain knowledge. It probably would have been the same in Lisp or assembly language.
The lesson here is that if a feature (at a minimum) does not have a associated test case, it is not actually a feature.
Yes, I agree. I find the addition of the regression test the true long-term fix. The code is just an opaque incantation that may or may not preserve some property we find worth preserving and we have no way of knowing it keeps preserving it over time as other parts of the system change.
The test actually proves it and while it too can change it has more staying power because it's expressed at a higher level of abstraction ("random arcane weird C shit" in the case of code versus "does this property hold" in the case of a regression test).
I have not looked into this specific issue, but are we sure that a regression here could have been avoided via a localized test? Maybe issues seem to arise from A implementing a feature with tests. B seeing that A lacks some functionality and adding it (potentially with tests), C seeing this (extra) functionality in A, and using in unintended ways not covered by tests (or in an unintended environment) + multiply by many layers of this A-B-C story up and down the stack.
The whole premise of a "giant secure open source C codebase" seems questionable
Because code review is sometimes not much different from an idealized version of the halting problem, where you would have access to a formalized version of a specification.
In other words, there is no strict definition of what is a security issue.
On the other hand, it is (both halting and spec adherence) are checkable under compute and space constraints though? :) I'd say the biggest hurdle are means to describe the spec in way that is easy enough for a human to produce to make it feasible.
Not a DB person either, but things like TLA+ seem very hard to write even with LLMs. Behavioral tests with an enumerable number of random paths to take (aka model checking - eg jepsen) seem more feasible. Although you can't check internal properties of the system (string `pass` or any of it's copies or parts are not held anywhere in memory at any point between lines A and B) unless we can check that two memory dumps are indistinguishable with different pass strings (assuming we abstracted away storage devices in a test environment).. Also not sure if it's "easy enough" to write such tests either.
Maybe the reason is that OS domain objects / primitives are too complex and not "isolatable" enough / lack a clear contract at all? (Hence multi file refactorings that break invariants.)
In open source, someone (many, many) someone’s can at least check.
Closed source…..
Not sure why you're getting downvoted, this is the entire point of open source.
Does such a bug exist in Windows? OSX? Who checks? If someone finds the key in memory, can they tell what conditions might be causing it and where?
Their only recourse under those situations is to hand it off to the OS Vendor and trust that what they implement does solve the problem, and trust that it wasn't a deliberate back-door that is now being replaced by another back-door.
Security researchers find security bugs in closed source operating systems all of the time.
Yup, it’s just harder to know for sure.
Oh, and large companies quite often fix these horrific issues silently, especially in online services where the customer can't diff bins. We're talking auth bypasses and RCE's that you'll never know about.
"Million eyeballs" argument was always kinda meh.
Maybe but still a little better than closed source like Windows. Everytime whem someone asked me if I could hack my way into his Windows PC I always told "After all it's Windows, how bad can it be?" Doing that since 25 years still waiting for a Windows machine that doesn't open... On the other hand I failed to open about 50% of Apple Devices I was asked to open and about 10% of Linux machines. (Not because Linux is insecure by itself but because most Linux distros install with insecure defaults and users don't care.)
Of course, I'm not saying closed source is better. I'm saying that just because it's open it doesn't mean people really care to read it thoroughly.
On my laptop with Fedora I just configured Linux to hibernate to disk after 15 minutes of suspend. Powering memory off ensures that bugs like this Debian-specific would not matter.
Plus what Debian extension to Linux tooling does although nice in theory, but in practice if one really worries about cold-boot attacks, then all keys and important documents has to be wiped out from memory, not only LUKS keys.
So hibernating is really the only proper way to protect against cold boot.
> So hibernating is really the only proper way to protect against cold boot.
I agree; or resurrecting FridgeLock: https://www.sec.in.tum.de/i20/publications/fridgelock-preven...
Interesting idea. On the other hand on the latest SSD with hardware encryption the raw disk speed under Linux can be over 5 GB/s so on my laptop with 64 GB of RAM the full restoration from disk takes like 45 seconds. With LUKS it is like 2 times slower. Which is not a problem at all. So I do not see much value in memory encryption in suspend.
Hmm, where does it get a key to decrypt memory on resume?
AFAIK it's practical only if you make use of TPM. And if you do, you're basically at mercy of TPM.
But if you do this, don't you have to enter two passwords each time you wake? One for LUKS, one for the system login?
Well yes and I don't see how this can be avoided.
Do you mean with with current software? How to avoid it in general is straightforward.
If you're the only one with the disk password then the simple answer is make both passwords the same and make the different parts of the system communicate better.
If you want multiple users, give them each a different boot password and encrypt a separate copy of the disk key with each one. That password can be their login password too, or it can encrypt their login.
https://xkcd.com/538/
(No, no, I take this stuff seriously too, but it had to be said)
> Except that, for more than two years, the encryption key remained resident in memory across suspend, leaving it there for the taking by anyone who seized the still-powered laptop.
I don't get it. Obviously, the laptop is locked when it resumes, how is that key "for the taking by anyone"? I'm not saying it is impossible to read out RAM from a locked laptop, but surely not by "anyone".
There are attacks that allow dumping RAM if the device is powered on though and you have physical access. Depending on config it may be very easy (just plug in a dumper over Thunderbolt on USB C and do direct memory access) or hard (freeze and swap physical RAM to an unlocked machine).. but the idea was defense-in-depth here; a well configured device should both be hard to dump RAM on and it should not give encryption keys if an attacker succeeds.
Anyone with physical access. I think it is understandable from the phrase.
There is a common misconception about how lock-screens in general work - they usually just prevents using the current hardware and software as it is to access the current OS. But the disk encryption is the main thing that prevents modification and other kind of access to actual data. And if the disk encryption key is lying in the memory, then effectively, the disk encryption is bypassed if someone can access the machine physically and assuming that there are no sufficient tampering protections in place for that machine.
Anyone with physical access, significant tools, and experience. The FBI has people who can pull data out of memory after freezing the RAM but the average laptop thief doesn’t so how serious this is depends significantly on your threat model. If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.
> If you’re not a major criminal, bitcoin whale, or intelligence target this is almost certainly academic.
Thanks, that's what I thought.
While that is true, the fact is that encryption is technically useless for anyone who is not constantly powering off and on, which is surely a bunch of people. That this is not widely understood is IMO a problem. And excellent reason for Debian's feature.
> Anyone with physical access. I think it is understandable from the phrase.
Sorry, I'm probably dense, I still don't get it. You steal a laptop, you open it, the screen is locked with a password/fingerprint whatever. How do you read out the RAM from that laptop?
Several options. One is you restart and boot from a live system where you are root, and then dump all memory. This is described in the paper with the witty title "Lest We Remember: Cold Boot Attacks on Encryption Keys":
https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...
Other options: DMA attacks. Also you never know what the Intel Management Engine hidden in your computer is doing. It's running a version of Minix you don't have any control over, and it has full access to memory.
>How do you read out the RAM from that laptop?
the term to look up is "cold boot attack" (https://en.wikipedia.org/wiki/Cold_boot_attack).
tons of cool live demonstrations of how it works on youtube if you've got the 20-40 minutes to spare
Still, this is a pretty crazy definition of "anyone".
you dump the physical memory, then decrypt the disk offline
on the subject of encryption keys and memory there is something you can do:
- if your CPU supports it, enable memory encryption.
- if your TPM module supports this look for MemoryOverwriteRequestControl & MemoryOverwriteRequestControlLock (/sys/firmware/efi/efivars/) and toggle them. make sure that your computer always reboots and never powers off. memory will always be wiped on boot.
https://trustedcomputinggroup.org/wp-content/uploads/TCG-PC-...
Proper capitalization makes English easier to read.
It's because of vulnerabilities like this that I enable Intel's "total memory encryption" feature. No plaintext leaves the CPU package. DIMM swap attacks become useless. Moreover, it's basically free: the cryptography happens directly in the memory controller, in hardware, inline with the bus transactions the CPU is doing anyway.
I don't see how that solves this problem. there is a string in memory that gets saved on suspend. that string when read by the CPU has the same properties it had before. if the CPU is using rot-13, the string is still rot-13 and the attacker doesn't need to spend the compute needed to crack rot-13, the CPU will simply do that as normal.
Definitely not a symptom of Linux being a hodgepodge of code thrown together from a thousand different sources and no one person could tell you how it all fits.
Bugs happen in all code. The difference is, anyone can fix stuff in open source. Closed source bugs are out of control and must be worked around. Usually by switching to OSS
Of course it's (indirectly) a symptom of that.
What's the alternative? Proprietary closed-source operating systems owned by corps who can be compelled to insert covert backdoors?
If BSD was as popular as Linux it would have the exact same problems.
I wonder if you think other OSes are any different?
TempleOS is the only thing that comes to mind that doesn't fit your description and it's not practically useful.
Any sufficiently large codebase is a mix of ideas and concepts implemented by different people with different priorities over a large timespan and if you can fit the entire thing in your head it's not very interesting or complex.
Qubes OS, the Linux distribution aspiring to offer a reasonably secure operating system, pioneering a "every app runs in a virtual machine" approach in the Linux laptop/desktop space, tracks this at the following issue:
https://github.com/QubesOS/qubes-issues/issues/2890
QubesOS is Xen based. Not Linux.
The *BSDs, Mac, and Windows all keep critical code in the same tree as the OS.
Something like disk encryption would be immediately visible.
So you don't have this mess of 80 different distros with 60 different versions of systemd, 20 that don't use it, a million kernel versions and it's all thrown together in a Costco-sized trash bag and we call the output "Linux".
In my experience any software system (not just operating system) after crossing a certain limit on complexity and age looks exactly as hodgepodge of code pieces thrown together, sometimes from different sources even if developed by one org. All major OSs have long crossed those limits, I believe.
Windows for ages did not really keep all the code in one repo. There were like a dozen parallel repos for e.g. the shell, kernel, IE, etc. Also every feature was developed on team-level branches; integrating all those branches often caused unexpected bugs.
"Mythos, find me a bug in LUKS. I know there is one in there".