> Memory safety bugs, which are errors in handling memory in native programming languages, are common code issues. They lead to security vulnerabilities as well as stability problems.
>Armv9 introduced the Arm Memory Tagging Extension (MTE), a hardware extension that allows you to catch use-after-free and buffer-overflow bugs in your native code.
No MTE and AddressSanitizer are implemented completely differently under the hood and catch different kinds of memory bugs.
MTE tracks provenance of pointers which means it catches bugs where a valid pointer derived from one allocation is used to access another allocation. Provenance is indicated by a fixed number of tags available. So there’s a 7% chance of not detecting an occurrence of a memory bug.
ASan is implemented differently: it adds red zones next to allocations. In theory it could have a false negative if a pointer jumps over the poisoned region. But it works well for stack memory in addition to heap memory. MTE doesn’t protect your stack allocated objects.
This is all true but it also is true that MTE was in part built to accelerate address sanitizer.
Kostya/et al who pushed for and designed the extension, was trying to accelerate address sanitizer so it could be on all the time. Among other things.
In fact, most presentations presented it literally as a way to do hardware accelerated ASAN (again, among other things), so the post you responded to is correct in that sense.
(I was there at the time, helping him figure out how to push for it)
Heh, you can kinda think of MTE as ASan except instead of a small range of a guaranteed redzone around heap pointers, it's a massive `2^56 * (random number, ≥0, on average 15)`-byte "redzone" (and some padding up to a multiple of 16 bytes which can predictably hide a bug, though at least such a bug won't corrupt unrelated heap). Stack handling is a significant difference though.
To expand a bit: 16-byte chunks of memory can be associated with a four bit tag. Then you steal four unused high bits from your pointers to store a tag value. When memory has a tag, a pointer used to access it must have the matching tag in its high bits. Your malloc implementation can then assign a different tag to adjacent allocations and any overflow into an adjacent allocation will have a mismatched tag and will trap. Likewise, change the tag on free and an attempt to use the pointer after the allocation has been freed will trap.
Of course, this doesn't come for free. Four bits per 16 bytes means a 3% increase in memory needed for tagged memory, hardware overhead for checking tags on memory accesses, and software overhead of setting/changing/clearing tags as necessary. This overhead is low (Apple shipped this in flagship hardware a year ago and nobody's complaining about performance there) but not zero, and an implementation with poor performance could be a real problem.
Implementing MTE is mostly another step in Apple’s efforts towards memory safety. There are bunch of things they’ve done to try to make this work a little better, but for obvious reasons, this is never going to be turned on by default for user apps and is only enabled for specific pathways.
> Because EMTE tag checking imposes a performance cost, we designed Memory Integrity Enforcement to take advantage of our secure allocators first and use EMTE to protect only smaller individual allocations within a type bucket, which software allocators can’t defend on their own.
Which seems to imply that outside of Apple’s targeted use in the kernel, the developer use of MTE on Apple platforms requires that you migrate your code to use their typed allocators…which I don’t know how many people are going to do that (and it is clear their answer is going to be to point people to Swift since it makes these types explicit for the compiler and even then, I am sure there are bugs in Swift in certain cases). Memory tagging is neat tech, but I don’t know if it’s going to gain all that much traction. It requires a burden that I suspect most app developers are not willing to take on and given that the user interaction is a crashy app. This is a path to nowhere except those of us who understand the trade-off.
Ive had MTE enabled on GrapheneOS for the past 3 years for pretty much all apps I use without any issues.
Theres a couple of apps that use C++ libraries for which I disable MTE.
Most android apps using Java or Kotalin which hugely helps with avoiding buggy apps crashing with MTE enabled.
Android can and does regularly force developers to make changes to their apps to improve privacy and security on the platform. Like theyve done with other changes they could gradually force developers to fix bugs in their apps. Maybe offer a MTE opt out for apps that use memory unsafe languages.
Think its likely Apple and Google will increasingly push forward the use of MTE in Android and iOS
What does the malloc or hardware do if for some reason your program needs to access memory addresses that overlap your tag bytes ?
I know that it's a really improbable scenario and the OS would also just refuse you allocations at some point, but what would the malloc implementation and the MTE do in such a case ? Fail the allocation ? trap when reading the pointer since it would point to the "wrong place" ?
If you mean the memory that stores the actual tags associated with each 16-byte chunk of memory, that's a separate region that should be inaccessible except for using the special instructions for reading/writing tags.
4 bits per 16 bytes is ~ 1/4 the cost of ECC, which is possible to implement with just an extra cycle or two of latency in the memory controller. MTE seems similarly lightweight. Costs some transistors and a percent of a percent of power budget, but much like ECC it seems a fair bargain.
It also explains that the fact ASan works well with, e.g., std::vector in C++ doesn't come for free and if you want ASan to detect bugs when your custom collections or allocators are used you have to use special ASan API to mark [in]accessible memory.
Unfortunately, the headline is somewhat optimistic compared to the reality, I think.
The thread makes it sound like it could have been disabled due to errata or performance issues. Basically, it looks like the software Google is shipping intentionally doesn't use MTE on the Pixel 11 hardware. That raises the question of...what does Google know is wrong with MTE on the Pixel 11?
> it looks like the software Google is shipping intentionally doesn't use MTE on the Pixel 11 hardware. That raises the question of...what does Google know is wrong with MTE on the Pixel 11?
Apple replaced MTE with an upgraded version that can run in synchronous mode all the time without the performance hit.
> Consider that MTE can be configured to report memory corruption either synchronously or asynchronously. In the latter mode, memory corruption doesn’t immediately raise an exception, leaving a race window open for attackers... We believe memory safety protections need to be strictly synchronous, on by default, and working continuously.
> Google may just be getting ready to follow suit.
That may well be, but it's a regression for the time being, since Advanced Protection mode presumably no longer uses MTE (or a replacement for MTE), whereas it did on the Pixel 8, 9, and 10.
It seems like they should've continued to offer MTE until the replacement was ready.
There's only a significant cost to synchronous MTE.
MTE ships two modes. synchronous mode and asynchronous mode. SYNC is slower but gives you far better traces and throws an SEGV_MTESERR as soon as violations happen. ASYNC however is async so there's a bit of a delay between a violation and the "catch" that throws SEGV_MTEAERR.
Strictly speaking async is worse for security because there's a brief window of time where the process "gets away with it" but the main differentiator is that because things don't stop the moment the violation occurs, SEGV_MTEAERR traces tend to be some degree of "out of date" vs SEGV_MTESERR which capture the exact state of the world the moment the error occurs.
The main tradeoff here is that async MTE has basically negligible cost. Something like 1-5% in practice but in microbenchmarks you can see up to 50%. Vs synchronous MTE where the penalty is on average closer to like 10-15% but in microbenchmarks it can be like 5-6x slower.
So yeah the perf cost is there but it's really not a major issue.
------------
The main issue is that apps and services crash when an MTE segfault occurs. So this means that to the uninformed end user apps appear spuriously unstable with no meaningful context. And it's not just apps. On graphene I see MTE segfaults semi regularly from various system daemons (mainly related to GPS/nav) and occasionally in Google Play Services itself.
From time to time I get them in Youtube and I constantly get them in the Twitch app. It's very annoying and the apps just crash when you happen to do some particular action leaving you walking on your toes to avoid accidentally tapping whatever magic pattern happens to invoke a MTE SIGSEGV until the next update. Doubly so since most apps provide no meaningful interface for uploading log traces to report these issues.
-------------
If Google wanted to roll this out without the spurious crashes they'd need to deploy it and eat the minor perf hit but they'd need to register a global signal handler to capture these MTE SIGSEGVs and report them back to the services in question without crashing the app.
And most app devs don't care so they'd be taking a minor (or major) perf hit in exchange for logging errors that developers always ignore. Doubly so in the modern day of "just have AI fix/do XYZ and who cares about the consequences as long as it runs".
I'm not surprised they dropped it but I do honestly wish they'd forced the issue and just deployed it and forced apps to fix their shit.
Given the current use of AI tools I would think that SEGV_MTESERR crashes would be low hanging fruit to fix. AI tools do ok with identifying memory corruption issues, given a smoking gun, in my experience.
SEGV_MTESERR is reasonably low hanging fruit until it's in a binary blob and the IC manufacturer said service interfaces with refuses to do anything to fix it.
Likewise if it's because a language is being lazy with how it handles memory (see: wireguard-go SEGV_MTESERR because golang's string implementation was being overly permissive with pointers).
And of course this only applies to SEGV_MTESERR. SEGV_MTESERR has a much higher runtime overhead that Google and co would not tolerate (but Graphene does) so they'd likely only be willing to use SEGV_MTEAERR which provides traces that point at a completely different place than where the fault occurred (but still provide a little bit of useful info).
Nothing is wrong. They just don't use MTE yet, they probably have other priorities. They wanted to save some costs until they will use MTE, so they removed some hardware acceleration and downgraded the GPU. Simple as that.
AFAIK the other Pixel devices, going back to the 8, do use MTE in the stock OS. Some MTE features are only enabled when "Advanced Protection" is turned on in the OS settings, but the stock OS on those devices does make use of MTE. In the Android documentation, MTE is explicitly called out as one of the protection features enabled in the Advanced Protection mode.
So, I don't think it's accurate to say they don't use it yet. They were using it, and then they stopped. I think that's one of the reasons the linked GOS thread says they're concerned that "MTE may actually be broken due to CPU errata".
It’s absurd that a small project like Graphene is able to run rings around a giant like Google in the security sphere. Almost makes you wonder if some of those vulnerabilities are intentionally allowed to exist.
Vulnerabilities in the world’s most popular (by volume) mobile OS could provide a plausibly deniable global espionage backdoor.
The perception that they are able to run rings around Google is not necessarily an accurate one. There are tradeoffs in any engineering situation and GrapheneOS devs make different ones based on different requirements. Google is one of the reasons MTE even exists. I don't think GrapheneOS devs would be capable of helping push forward that technology in the same capacity.
Fair, and also Google is not a monolith. I’m sure that their own security experts who helped push things like MTE are also confused and frustrated by the lack of internal adoption.
A lot of heavy lifting is involved at the lower levels of security and Graphene doesn’t have to deal with this work, generally. I guess my surprise comes at the “fit and finish” stage where Google regularly seems to be lacking.
Do we know if Google is operating on the frontier of the trade-off curve? What do we lose from Pixel Android when GrapheneOS security changes are made?
Very little if anything at all. Depends a bit how you draw the line between an app and the base OS.
Contactless payments dont work with Google Pay - but that is due to Google using attestatiom to check if the OS is GMS lincesed (GrapheneOS is not).
There are alternative contactless payment options that work fine.
There's quite a list you can get even from GrapheneOS's posts - among other things, there's a rather large set of apps that will outright crash when MTE is enabled.
Elsewhere in the thread, someone reports that with MTE enforcement enabled, there are lots of crashes. And app and system service developers don't seem responsive to them.
That's not something that's really acceptable on a $500+ phone... so if you're Google, you're not going to turn that on by default and you're not really going to be interested in keeping it as a feature that users can turn on.
Graphene has a different focus, so they can push the tradeoff towards enforcement and crashes rather than non enforcement and fewer crashes.
Google could perhaps do more with their crash logs and pushing app developers to fix MTE crashes, but they don't often do a lot of that; and they get called out for being developer unfriendly when they do enforcement.
Because it (anecdotally) crashes too much, but only on $$$ devices with cool CPUs that can use it. People buy expensive phones because they're supposed to work better.
They should really do some sort of sampling thing to generate crash dumps and find big offenders and increase the coverage over time.
For Google employee devices, 0.1% of background execution starts while charging in an idle period (overnight bedtime charging) will use MTE. When any specific device hits an MTE crash, back-off sampling for 1 week on that device. Modulate the sampling rate so crashes are manageable.
When Google employee devices are not crashing overnight at a high rate, then start sampling background execution during the day for employees and overnight background execution on general user devices. Finally, sample on foreground execution, again for Google employee devices first.
If there's significant variation in crashes by app, you probably need to setup a way to set sampling rates to zero or very low for application versions that have been identified as a known problem and don't need additional traces.
I wonder what causes the crashes. Is it a buggy MTE implementation (or buggy hardware), or are app devs doing something that would only cause crashes under MTE? Maybe it’s something they are using like a library or framework?
There's a lot of speculation that US model censorship around "cyber capabilities" is about protecting an inventory of non-public vulnerabilities used for intelligence purposes. That same explanation could cover Google's MTE actions as a result of pressure from intelligence agencies. Nobody should be surprised if they prioritize what they see as national over personal security.
Since GrapheneOS is building on something that Google is giving away for free, I do hope that they release something better. Everything else would be.. kind of weird, no?
99.9% od the code GrapheneOS uses and 100% of hardware (including one like Titan security chip handling storage encryption and key store) is from Google.
So what are you on about man - it's easy to take someone else's work and put tweaks on it that wouldn't work at scale.
I don't interact with Graphene or any of their folks at all; I'm just a distant observer like most folks here.
Still, I can't help but think that Graphene seems to want to complain about everything and anything that doesn't fit their niche use case. (As much as it seems beloved here, people that flash custom Android OSs are the very definition of niche users.)
My personal attitude to Graphene seems to get a bit more negative with each one of these "rants" and I doubt I'd go far out of my way to help them, even if I had exposure to them.
> that Graphene seems to want to complain about everything and anything that doesn't fit their niche use case
What would you want them to complain about instead? Of course they'll complain about that, just like Googlers will complain about things affecting their stock price, no one is surprised that people care about stuff they're personally involved in, it makes a lot of sense.
Now if these complaints weren't accurate, then I'd walk with you and feel a bit more negative with each piece. But the ones I've looked into, have been spot on, so who cares if it's for their specific niche? I expect them to care about their niche, that's why those people all work together in that organization in the first place.
> As much as it seems beloved here, people that flash custom Android OSs are the very definition of niche users.
Hmm... Let's try reframing this: "as much as it seems beloved here, people that install their own operating systems on PCs are the very definition of niche users"
I'm absolutely certain that's how IBM felt before the clones. But the ability to install what they wanted on a defacto standard platform is what launched the computing revolution. I think we'd still be living in a sterile monopolistic environment with $10k compilers otherwise.
Folks installing their own ROMs on phones are only niche because they've been pushed out at every opportunity using locked bootloaders, embedded security processors, factory installed secret keys, etc.
Despite all that, there's still thriving communities developing and using custom ROMs on their phones. That demonstrates more than niche demand.
> Hmm... Let's try reframing this: "as much as it seems beloved here, people that install their own operating systems on PCs are the very definition of niche users"
I mean yeah, 99% of people never installed an OS and never will, what's your point here?
> 99% of people never installed an OS and never will, what's your point here?
That the 1% who do build visicalc, Linux, the internet, Google, and every application and innovation that happens outside the corporate wall. The entire ecosystem everyone else ends up using.
And that calling that niche is ridiculous, shortsighted, and shooting oneself as a platform owner in the foot.
What are you talking about? You're making zero sense. I've been in the niche of modifying and installing my own ROMs, then to kernels, and now doing the same things commercially.
It is a niche, it always was. There was never a mass community of users, the most installs you used to see on any given AOSP based project is maybe 20-30K for the most popular devices like Redmis, Pocos or Google Nexus phones.
It's still a niche, and things that users do and used to do to their phones are fundamentally incompatible with Android's security model. Root access, magisk, xposed, overall zero or near zero security validation from the security perspective on all ROMs but Graphene (maybe some others, I haven't followed the sphere for a while).
Again, only because the platforms are locked down and there is no single standardized target for universally bootable ROMs in the way a Linux ISO can boot on any PC.
Kinda sounds like you might be younger and didn't live through the 8bit micro -> PC transition. Standardization of the platform led to a cambrian explosion.
> Again, only because the platforms are locked down and there is no single standardized target for universally bootable ROMs in the way a Linux ISO can boot on any PC.
Couple of things here. Android was never that open to begin with. There were always (and still going) issues with the kernel sources being withheld indefinitely, AOSP side changes are almost never released at all, iirc only Google released anything. Bootloaders were always an issue, not all manufacturers had unlockable (or easily unlockable) bootloaders.
Ironically, Google were always the spearhead of the standartization in the Android world. Project Treble and stuff resulted in the fact that almost any Android device released these days can run a generic Android image, it's not gonna be as functional as the original Android, but it's something.
Yes, I am on the younger side, that's why I know how it was in the Android world. Nobody is interested in reinstalling OS-es on their phones because the amount of effort to create an Android ISO that's as usable as what's preinstalled is truly gigantic. Android flavours are heavily customized for the needs of the OEM, and for the hardware that they're gonna run on.
It's a niche, always was and always will be. A niche powered by a few large players like LOS or GOS, that actually get paid somehow or heavily work for the idea, and an army of teenage unpaid maintainers, who quit the maintainership as soon as they get a properly paying job that takes most of their time (me).
> Nobody is interested in reinstalling OS-es on their phones because the amount of effort to create an Android ISO that's as usable as what's preinstalled is truly gigantic.
It's pretty clear that you have an extremely limited exposure to the custom Android development space, since you never rebutted any of the points I brought up. Custom android was never easy and never will be. And it's fundamentally incompatible with the concept of having a truly secure env on your device (same as in the PC space, ironically enough).
Son, I've been building and installing custom roms since the HTC Dream, was involved in OpenMoko, and have owned nearly every open or Linux-running phone of the last 30 years back to the Sharp Zaurus and iPaqs.
Your custom rom experience sits wholly within a larger category of computer systems experience about which I've been speaking.
The sort of fragmentation you describe is a symptom, not a foregone conclusion.
Perhaps they complain because that's literally the only way to get Google to take notice?
Let's be real, AOSP doesn't exist any more. Google have closed down nearly everything. All the development happens in private, you've stopped addressing bugs raised by the public, the source of patches are only infrequently released, device trees are gone.
All the development happens in private, you've stopped addressing bugs raised by the public, the source of patches are only infrequently released, device trees are gone.
To emphasize this point a bit more: only "QPR0" (major release) and QPR3 are released as part of AOSP. QPR1 and QPR3 are not released at all anymore, but contain fixes for vulnerabilities that are not marked high/critical (so don't end up in ASB). It is not clear to me whether OEMs get access to QPR1 and QPR3, but Google are not only witholding features, but also a set of security fixes.
Besides that, they are torpedoing other systems through Play Integrity.
IMO it would be best if AOSP was spun off from Google into its own org that actually cares about developing an open source system for others (both open source systems like GrapheneOS/Lineage and commercial vendors like Samsung) and that would have an attestation system that is open to vendors that have good device security.
If they're just some "niche use case" then why would Motorola partner with them? The way they see it,
> By combining GrapheneOS’s pioneering engineering with Motorola’s decades of security expertise, real‑world user insights, and Lenovo’s ThinkShield solutions, the collaboration will advance a new generation of privacy and security technologies. In the coming months, Motorola and the GrapheneOS Foundation will continue to collaborate on joint research, software enhancements, and new security capabilities, with more details and solutions to roll out as the partnership evolves.
They’re the second largest manufacturer of Android smartphones in the US, and 10% of the global market. Seems a bit unreasonable to dismiss them out of hand on that basis.
I think that's a worthwhile point to consider but it's only relevant if we move the goalposts from "GrapheneOS is only used by Android ROM enthusiasts" to "GrapheneOS is only supported by one small Android phone manufacturer".
To be frank, though, I don't see any of this line of reasoning as relevant; it's just appeals to greater authorities on either end. If AOSP is only for manufacturers there's really no reason for it to be open source in the first place. And then folks who care about actually improving security end-to-end outside of whatever's convenient to implement by those beholden to the quarterly profit metrics are up a creek.
Personally, if this whole GrapheneOS/Motorola thing doesn't improve the state of the ecosystem I'm going back to Apple or whatever other manufacturer makes it clear they take security seriously.
You might not like their style of speech, at least they care about their users. Maybe Google uses nice flowery language that makes the reader feel nice—IDC—actions speak louder than words.
Stock Pixel is an awful experience. So many useless notifications, popups, ads, privacy not by default.
Company: "We care about your privacy" meanwhile 1400 corporations they share data with
GrapheneOS: "There's zero telemetry in GrapheneOS"
The more you read the more you realize they are nearly always correct.
No, users are secondary to the mission. If they happen to overlap, that's a happy coincidence. And caring for a user stops if Daniel has the slightest interpretation of their comments being an attack (i.e. any mild criticism or disagreement about direction)
They care about their mission and Daniel's stance first and foremost
I wouldn't be surprised if this comment is branded an attack
> Stock Pixel is an awful experience. So many useless notifications, popups, ads, privacy not by default.
Huh? What pixel are you on? You only get notifications from stuff you install after the initial setup is done. And even then you can outright mute applications, completely.
I was on stock recent pixel A17 for a few hours before flashing GrapheneOS. They took over the power button for Gemini, there's gemini in Messages, there's 50 apps you don't need. Yeah you can disable notifications but it was constantly giving tips and tricks and other bullshit. The OS feels super bloated compared to GrapheneOS.
I hate the power button change myself, but 1. It was turned into the Assistant button by default for many years now. 2. Can be switched back easily in settings.
> there's 50 apps you don't need
Where are you getting your pixels? Mine came with all the essentials, I don't remember disabling anything in particular. No game demos, no preloaded meta cancer, just the normal google suite.
> The OS feels super bloated compared to GrapheneOS.
When was that ever not the case? When was a stock OS lighter than a custom flavour on any phone? You can't ship Android in the state that custom OS-es do, you won't pass the certification to get Google's goodies like OTA service and preinstalled play store. Don't quite get the Gemini argument either. You're buying a Gemini phone that touts AI everywhere on its promotional page. What do you expect?
Even in the EU spyware use is prevalent (and i 'd guess everywhere else in the world). There have been many scandals of government authorized commercial spyware been deployed against journalists. Is it really that niche a mobile OS that tries to not be exploitable by them?
For example, a user being able to inspect and edit the files written by an app, no matter where or how those files were written, would be an anti-spyware feature: you could better observe the behavior of a closed-source application.
Grapene opposes this feature because the app security model protects the app AGAINST the device user editing or reading protected files.
Graphene's philosophy is enforcing the Android security model. The Android security model gives guarantees to the app developer about how their app can behave, even where the device's owner wishes otherwise. See: Play Integrity.
GrapheneOS have mentioned wanting to expand the logging/intrusion detection capabilities of their Auditor app but contend with the need to include it as a system app which is against their philosophy (PoLP). It is not accurate to say they don't want to do anything about spyware.
They are also completely against Play Integrity as implemented on principle.
Also your argument about a user inspecting and editing application files feels like a strawman argument. For example many spyware use malicious links to infect the devices, not malicious apps.
Let's say I want to secure my system against an app developer deciding to delete my data stored in their app.
To do that, I wish to store a copy of all the files the app has written to my filesystem, and put that copy outside the app's control. This is explicitly to contain data the app's developer does not WANT me to be able to keep.
A. Is my being unable to do this "more secure"? If so, why is the specific threat I described to my data integrity - an app developer deleting my data - invalid?
B. Does GrapheneOS support this protection, ensuring the device owner is secure against the app developer, or do they instead secure the app developer against the user?
I want to install an app now, and protect against its developer doing a rugpull on my local files later. I do not want to need to review each new app version in advance as I install it.
This is clearly a legitimate case where what the Android security model says I should be permitted to do makes me less secure against an attack by the app developer.
So other projects are not supposed to critize Google? Graphene's focus is on security and they complain about lack of security in your products. Seems valid to me.
Also security and using non-Google OS are not niche usecases. I'm not sure how you are working on Android, the most popular OS while claiming security is a niche usecase. In fact, I have less confidence in security of your work.
The way I read the comment (admittedly putting words in their mouth), is that there are different ways of "complaining". I am someone who is actually quite sympathetic to Graphene's project and have considered getting one of their Motorola supported phones when they come out. But every time I encounter their writing, they come off as unnecessarily abrasive. They could be making the same points with a lot less potential bridge burning.
They are not neutrally pointing out places where their preferred priorities aren't being met (which would be reasonable), but they are speaking (forgive me the hyperbole) like religious zealots. I don't blame someone who doesn't share their particular viewpoint with being hesitant to work with someone like that.
As a consumer, I don't particularly care. I'm not going to make my decision based on their comms style. But a developer who actually has to work with them? Yeah I could imagine it mattering.
Agreed. They clearly know their shit and they are stunningly great engineers, but Graphene OS would be much more loved and respected if they had a PR person to take care of online communications.
The "religious zealots" have their own motherfucking Cellebrite column. That means they are right.
They can be abrasive all they want. People who are wrong still need to sit down and listen instead of spitefully "burning bridges".
And that's assuming they're actually guilty of this "abrasiveness" that people accuse them of. I don't actually think that's true. I think their comments are extremely informative and filled to the brim with valuable technical information. I've spent quite some time reading strcat's comments here and it's actually changed my viewpoints on some security matters. For example, I'm no longer as obsessed with knowing what's inside firmware as I used to be, it's better to minimize and isolate such black boxes so they can do no harm. I started valuing IOMMU support and grouping in PC motherboards.
If there's any "abrasiveness" in their posts, it's probably just an undercurrent of frustration with a world that simply refuses to understand or cooperate. A feeling I can't help but have enormous sympathy for.
> They can be abrasive all they want. People who are wrong still need to sit down and listen instead of spitefully "burning bridges".
While I wish that were how humans work, it definitely is not. You can make the strongest argument ever and still lose if you deliver it in a way that the listener can't process it because you're triggering emotions. The best arguments IME are factual and respectful.
The best thing about being right is it doesn't matter if the listeners can process it, you're still right.
There are consequences to ignoring GrapheneOS developers. Insecure phones can get people killed.
The best arguments are respectful and measured, but few things in this world are ideal. It's in your best interests to tolerate any perceived "abrasiveness". Ignore the the guys who make Cellebrite sweat at your own peril.
The argument about the seriousness of the topic seems to be to be arguing for taking into account human psychology and toning down rhetoric (note: not failing to make the claims, or dropping points, just changing tone) in order to make it more likely that you get listened to. To the earlier commenters point: it would be great if nothing other than correctness + importance determined likelihood of being listened to. But that's not the way the world works.
Nobody at google will get demoted if vulns in grapheneOS can’t get fixed due to their oss strategy and result in deaths. Yelling at them angrily will not change this. When was the last time someone abrasive called you a moron and corrected you - did you come away feeling like you will take the lesson to heart and adjust your behaviour?
I am sympathetic to GrapheneOS, but I have also worked with them and they are not easy to work with. Google is in no way required to take on criticism from OSS community- GPL and MIT do not actually require the rights holder to not be an asshole. In such a social dynamic, one will find greater success using language that appeases instead of repels the asshole.
I personally would wish that the AOSP communities would work together. I would love to see some oft their work to also land on other devices even if that's not 100% secure without bootloader relocking. It sometimes seems to me that the perfect seems to be the enemy of the good sometimes. I still understand their sentiment and their goals. I hope that the Motorola thing takes off. Still think a crossover with true aftermarket roms would not hurt.
Thankfully the project doesn't care about your personal attitude. That "niche use case" literally saves lives in countries where saying the wrong thing can put you to death. Since when is calling something out a rant?
In a world where economics makes security hardly a secondary concern, a situation exploited by both the intelligence and surveillance broker sectors, dogmatic sanctimony for high security is a feature and not a bug.
This doesn't read as a rant to me, but as calm and factual, regarding a genuine security regression that merits public attention. What is your interest in mischaracterizing it?
Could you please explain why supporting MTE/potentially EMTE in production as a goal represents a niche use case? Isn't mitigating memory corruption issues a mainstream ideal? How else would you propose to do it?
I'm not an AOSP engineer, but that was my thought reading GOS's comments: Why be negative toward the people who you want help from? They don't need more stress in their day and wouldn't want to engage, with the promise of enjoying more of the same treatment. If it's consolation, much of their communication seems the same - it has nothing to do with AOSP or Google. :)
Communication is a challenging skill and not all good engineers are good at it. It's stressful to be in a role that demands lots communication if you feel you aren't good at it (I'm not speaking for GOS leaders - I don't know what they think).
Going out on a limb, hoping it helps - the most powerful single solution IME is compassion: Compassion toward yourself and toward others. They are engineers like you, trying to get through a stressful day like you, and they could use some pleasant interaction from someone who understands what it's like. You could use some of that too. Running GOS isn't easy, I'm sure.
I'm not an AOSP engineer, but that was my thought reading GOS's comments: Why be negative toward the people who you want help from?
What help though? Google has closed off AOSP and only does source code drops twice a year. Google has embargoed security patches for three months and only provides them to OEMs of Google-certified Android phones, not other AOSP-based projects. Google stopped providing git trees of kernel sources and instead requires projects to submit a request for a Google drive link for each kernel version that takes up to weeks to process. Google is shutting out open Android systems through Play Integrity.
Google is not helping anymore, over the last 1-2 years they have tried everything to sabotage AOSP-based projects. The only reason that they are not fully closing AOSP is probably because 1.) they would get in hot water with regulators; and 2.) AOSP will probably get forked.
> people that flash custom Android OSs are the very definition of niche users.
This is a mischaracterization. The niche isn't Android hobbyists, it's people with what should be a basic expectation for privacy. I wouldn't flash GOS or any other OS if I could safely avoid it.
Daniel Micay who's (most likely) running the project again has a history of being paranoid and coming up with perceived attacks from people on the internet. The loudest one by far was the story with Louis Rossmann. I've been following Micay for close to a decade. This is the exact pattern that he operates on. Tons and tons of drama for no reason.
I truly, sincerely hope that I'm wrong, and that he isn't actually at the helm, because then Motorola are under serious thread of getting an essay saying how they're a terrible company sending assasins to his house, after they disagree on some little thing.
Tbf with all the information that has come to light since then Rossmann and FUTO were also not behaving correctly and it was fair from him to deny coming onto their show (where they have also invited fascists before)
Not sure what you're referring to. I'm talking about that time that Micay alleged that Rossmann launched a stalking and harassment campaign against him. I don't know of any shows he's been invited to.
Maybe if Google did not shove their spyware down people's throats and actually allowed users control of their phones, there wouldn't be a need for projects like Graphene and Lineage. Until then, complaints are more than justified.
Security shouldn’t be a niche use case. There’s a constant trickle of CVEs, and spyware vendors are known to abuse these exploits in their software. All this on devices that are reachable in the US through a text or MMS, sent to an easily located 10 digit number that isn’t easily changed. These are devices that people now use for all kinds of sensitive tasks!
Security should be the number one priority, frankly. Graphene has shown that this is possible, and they have tried multiple times to get Google to integrate their work.
> My personal attitude to Graphene seems to get a bit more negative with each one of these "rants" and I doubt I'd go far out of my way to help them, even if I had exposure to them.
My thoughts entirely, I prefer to use CalyxOS, which for some reason in the eyes of GrapheneOS is pure evil.
I've never seen them call it "evil". I've seen them debunk CalyxOS security claims as well as criticism of GrapheneOS, and they usually provide some serious reasoning and technical information when they do it. They know what they're talking about.
Don't take it personally. I'm a huge fan of Linux, and GrapheneOS routinely comes here and calls it a huge security liability. And they are right.
... and my opinion becomes more positive with each rant. We'll have to agree to disagree. The only reason I buy Pixels for myself and my family members is because of GrapheneOS, otherwise it would be used out of date hardware for LineageOS or some kind of Linux phone. I am thankful that they are attempting to diversify with Motorola, being entirely Pixel dependent has been a project vulnerability; anytime Google decided to lock down the boot loader, it would have been curtains for the project.
So not only a price increase: less RAM, performance scraping backwards, crippled/lost features, a camera system that captures stuttering audio and video imperfectly often, and Pixels dropping out of AOSP.
Yet, you can pick up a Samsung S26 for 600 Euro that will absolutely demolish a Pixel 11 in practically every aspect except cameras. Years ahead in CPU/GPU performance, etc.
I am almost certainly going to live with whatever drawbacks in terms of camera quality, battery life, etc. Come with their Motorola phone when it's time to upgrade. MTE is such a non-negotiable for modern digital security on phones it's crazy Google would be so okay with this regression.
What's especially stuck in my mind lately is how insecure basically all desktop OS' feel. In at the point of buying a second and third GPU for my desktop to run my email and browser in dedicated VMs because everything feels as watertight as a sieve. Qubes seems more and more appealing in a world where every open source software supply chain is under seige, corporate software underprioritizes security, and most sites will stop at almost nothing to surveil you.
I truly lament this new reality where MY computers I PURCHASED feel to use like I'm reaching blind into a paper bag filled with razor blades.
>What's especially stuck in my mind lately is how insecure basically all desktop OS' feel. In at the point of buying a second and third GPU for my desktop to run my email and browser in dedicated VMs because everything feels as watertight as a sieve. Qubes seems more and more appealing in a world where every open source software supply chain is under seige, corporate software underprioritizes security, and most sites will stop at almost nothing to surveil you.
I made the same observations and conclusions as you, but a few years ago. I have been daily driving Qubes OS exclusively since on my Thinkpads. If you haven't started to yet, I implore you to consider the following when buying a system for Qubes: mid-line CPU with plenty of PCs (turn off the ECs), As much RAM as you can afford (seriously, budgetmaxxx on RAM), dedicated secondary GPU with the minimum specs you need is a must for inference (may have to widen the budget a little bit after budgetmaxxing RAM).
Expect to not game on the platform. I have not tried this but if you can't cut the digital crack-cocaine habit then it may be possible to set up a gaming qube which takes the dGPU passed through: get a monitor + second USB keyboard & mouse, plug the monitor into the dGPU's HDMI ports, and pass through the secondary peripherals. Digital crack-cocai... err games... don't like having their frames pushed over X11 forwarding even if the dGPU is passed through, so you will need the monitor; the secondary input devices are so the mouse and keyboard inputs don't leave the gaming qube.
> What's especially stuck in my mind lately is how insecure basically all desktop OS' feel.
I ended up switching away from using my Mac to using an iPad M5 + Magic Keyboard as my sole computing device for the security benefits, and it’s been working pretty well and in retrospect I don’t miss anything from macOS aside from a chromium-based browser. However I’ve said it before that switching back to using a Mac after using iPadOS for so long feels like a serious regression, for all it’s faults the iPad + iPadOS feels like a futuristic technology preview of macOS.
For development work I use GitHub Codespaces (and self-host Coder in the future hopefully). I was moving towards cloud/container-based development already and it works nicely. Plus NPM vulnerabilities are no longer a serious issue. Again the only thing I miss is Chromium, but I’ve altered front-end development planning to target Safari/Firefox since websites shouldn’t be implementing non-standard Chrome-exclusive features anyway imo (I’m saying this as someone who highly prefers chromium lol).
GrapheneOS: "Multiple Google engineers we've contacted have said they aren't able to give us any information about this so we're left doing reverse engineering and relying on leaks. The leaks do not seem reliable and do not match what we see. Our concern is that MTE may actually be broken due to CPU errata."
> It's possible to enable reserving the tag memory for MTE via `fastboot oem mte on`, boot a non-stock kernel ignoring arm64.nomte and use MTE.
What I'd like to know is, is it not sufficient to just check the feature bits in ID_AA64PFR1_EL1? (Isn't this the first thing you'd try when trying to determine if the hardware supports some feature?)
If they were making the determination solely based on the fact that `arm64.nomte` is being passed to the kernel (or based on what features are exposed by kernel interfaces), it may have been better to say "MTE is seemingly disabled in the current Android release" rather than claiming that it's simply not present in hardware. If you're asking the Linux kernel about hardware features, maybe it takes cmdline arguments into account when presenting that info to userspace.
The TRM[^1] mentions that some of the feature bits depend on BROADCASTMTE (presumably some CPU input pin), but maybe that signal isn't constant and is allowed to change based on what happens in firmware/the bootloader?
Also, why the claim about the lack of hardware acceleration for MTE in the caches, is there evidence for that, or is this also a misunderstanding?
I think it's reasonable to assume that the perf impact of MTE is non-negligible (on cores in older Pixel devices[^2], MTE apparently suffers from the fact that checked stores are serializing!), but it's entirely possible that this does not follow from some physical design concession when implementing the SoC. The characterization of all this as some kind of cost-cutting measure is not necessarily accurate.
Fascinating, thank you. It seems strange that the C1-Pro/C1-Ultra TRM mention that ID_AA64PFR1_EL1[11:8] should still be non-zero even when BROADCASTMTE is low.
edit: Oh, I guess Linux does emulate the feature registers, doh. Since that user is booting with arm64.nomte, the kernel is changing those feature bits to zero! This probably explains the confusion here about hardware support: in Linux userspace, you still trap for system register reads, and this is just abstracted away from you...
I think there’s a very high chance it’s net positive. It’s growing quickly and approaching 10% marketshare for premium phones (the most profitable segment) in the US.
Phones typically have 2 operating systems: one to handle telephonic functions like managing tower connections and separate operating system for user applications. The security implication here is that the telephonic operating system has access to the same system resources as the application operating system at the same time, right? The problem is your data is always available in plaintext to the telephonic OS because it has to be stored somewhere before it gets encrypted, so assuming graphene OS doesn't handle both functions its not a perfectly closed loop system.
> The security implication here is that the telephonic operating system has access to the same system resources as the application operating system at the same time, right?
> The baseband is isolated on all of the officially supported devices. Memory access is partitioned by the IOMMU and limited to internal memory and memory shared by the driver implementations. The baseband on the officially supported devices with a Qualcomm SoC implements Wi-Fi and Bluetooth as internal sandboxed processes rather than having a separate baseband for those like earlier devices.
A separate OS runs the security chip (like Titan). Another could be running in EL3 (Trusted Execution Environment), yet many could be running in Realms (mutually untrusted VMs). The later stage bootloaders and some firmware (ex: GPUs) themselves could be considered OSes in their own right.
No, shared peripherals need to go through a central arbiter for access. In the case of things like storage the AP's OS, eg android, provides that roll. This is also true for the OS that runs under the trustzone. Be careful about spreading rumors without validating them.
Most devices have a large number of operating systems. The camera also has one, and the storage has one if it's eMMC. The WiFi/BT chip has one. The AP has at least two, because of trustzone.
People bring up the baseband (ie. cell modem) having its own OS because someone gave a talk on it at defcon or something, but it's just one of many.
Boo. I was delighted by its introduction in pixel 8. I thought this was a resurgence of the old Google, actually giving a f about software, security and the end user.
iPhone 17 has it now. maybe Google's work here is done
So, where does the restriction for side loading apps live, on the device or the OS? Curious about getting the Pixel 11 Pro if/when I can install GrapheneOS.
GrapheneOS themselves said it wasn't worth getting the 11 over the 10 on Twitter. It has a worse GPU and less RAM, and the increase in CPU speed is so negligible that often you end up with better performance using the previous generation. In addition, GrapheneOS isn't even available on 11 series devices yet.
The 11 series has a new MediaTek modem compared to previous Pixels(since 7 I think?). The old modem had power efficiency issues during poor cellular signal as well as a relatively lengthy GPS lock acquisition.
I don't think there is enough evidence yet about either but I can attest that GPS feels improved over my 7. It's only been 1 week though.
The non-Pro 11 has a much larger photo sensor and has more base storage. Everyone has different priorities. I'm not saying which is better but trying to give a more complete picture.
Yeah the camera seems to better which could be deciding to some people. But while the base storage has increased the price did too, so a base 265gb might cost the same as an upgraded one from 128gb.
Sideloading has always been allowed, and that new "advanced flow" is something entirely decided by the OS so I don't think it has or will have any effect with GrapheneOS use.
It's actually recommended to get least of your apps as possible via the play store and use 3rd party stores if you are going for maximum privacy or de-googling.
I've done some research and am considering as well, though I might wait for the motorola phones to release. Have you encountered apps that didn't work? Like banking and stuff seems to break a lot of the time.
Next 6% might need some fancy setups, like setting up storage scopes or other privacy settings so they can function as expected. I won't pretend they are always intuitive, but if you really can't figure it out, someone on the internet has already had the issue and posted their solution online
Another 3% are your banking and similar apps. Some work, but some are needy, and I have these in another profile which has the sandboxed Google play mirror and other Google services that GrapheneOS mocks
Then is your last 1% that are so stubborn that not only require Google play, but they have the audacity to require you be logged in to Google play or they won't start. Personally, I just refuse to use these apps when I get one, as they are random dumb ones, like some incredibly privacy invading videogame app. However I'm sure there will be some app you find that is like this. I guess you can log in, or heck make a third profile and hopefully don't need that app often. I guarantee there is a friendlier alternative that is worth looking into if you do feel you really need that service.
My strong instinct is that GrapheneOS's plan could be problematic:
Generally, do not to invest in features that the upstream vendor isn't fully committed to. It might work today, but tomorrow or next year, etc., it might not. They might even deploy other things that are incompatible or do other things that assume the feature is disabled. In fact, plans to do those things might be the reason they disabled MTE.
Also, never support products or projects that won't work well. When someone's Pixel 11 is slow, they will think and post that 'GrapheneOS is slow'. Blaming Google will not be very credible and will look like you're just trying to duck responsibility. Your brand's name goes whatever you release; people will associate their experience with GrapheneOS (also because GOS is the new, unconventional thing; Google is the established norm); if their experience is bad, they will associate it with GOS.
Their plan is to seemingly comprehensively test MTE on the Pixel 11 to see if it's usable for them. I'm not sure how that's "problematic", in fact it seems like it's exactly what they should be doing.
Their real focus is on their Motorola partnership, anyway.
GrapheneOS is a non-profit, publicly naming the actual issues is the only thing they can do to make Google fix those issues. And it is clear that security/privacy is their first priority, not "brand image"
Some on HN defend everything GOS does (maybe not you, but there are some). Those people are hurting GOS.
Good security engineering isn't sufficient: GOS could accomplish the security engineering by making an OS for themselves on a private git server and never distributing it.
But if we want to help the public with security and privacy, then the outcome depends on GOS's brand image - perception drives public adoption, who will never read long posts on arcane engineering matters. It depends on GOS's relationship with Google and its individual engineers, a relationship in which Google and the engineers hold all the cards - they could choose to cut off GOS completely tomorrow at no cost to themselves.
It's an engineer's fantasy that you can ignore those things and just do engineering; life just isn't like that. Doing that would imperil or greatly limit GOS, which hurts everyone.
The thread made it sound like it was more- that the actual release of pixel 11 did not have the firmware to support MTE, not that it was just disabled.
If you don't know, making things up isn't a solution. I don't know you; would it be useful to fabricate a life story and what you had for breakfast today? What would be the point?
All you can really do is deal with uncertainty, which is the norm for most things in life.
The performance isn't really that bad but it's definitely a hit. The bigger issue is that it causes apps and services and occasionally also the kernel to throw MTE SIGSEGV which becomes a substantial problem if the developers for said apps or services don't care enough to fix them.
So the TLDR is that the main issue is that it causes a whole bunch of stuff to segfault when bad memory accesses occur but where they wouldn't segfault without MTE. It makes the phone feel unstable to average users since stuff just crashes with no real end-user-facing explanations.
For anyone else not knowing what MTE stands for:
> Memory safety bugs, which are errors in handling memory in native programming languages, are common code issues. They lead to security vulnerabilities as well as stability problems.
>Armv9 introduced the Arm Memory Tagging Extension (MTE), a hardware extension that allows you to catch use-after-free and buffer-overflow bugs in your native code.
https://developer.android.com/ndk/guides/arm-mte
MTE is essentially hardware-accelerated AddressSanitiser[1].
[1]: https://clang.llvm.org/docs/AddressSanitizer.html
No MTE and AddressSanitizer are implemented completely differently under the hood and catch different kinds of memory bugs.
MTE tracks provenance of pointers which means it catches bugs where a valid pointer derived from one allocation is used to access another allocation. Provenance is indicated by a fixed number of tags available. So there’s a 7% chance of not detecting an occurrence of a memory bug.
ASan is implemented differently: it adds red zones next to allocations. In theory it could have a false negative if a pointer jumps over the poisoned region. But it works well for stack memory in addition to heap memory. MTE doesn’t protect your stack allocated objects.
This is all true but it also is true that MTE was in part built to accelerate address sanitizer.
Kostya/et al who pushed for and designed the extension, was trying to accelerate address sanitizer so it could be on all the time. Among other things.
In fact, most presentations presented it literally as a way to do hardware accelerated ASAN (again, among other things), so the post you responded to is correct in that sense.
(I was there at the time, helping him figure out how to push for it)
Heh, you can kinda think of MTE as ASan except instead of a small range of a guaranteed redzone around heap pointers, it's a massive `2^56 * (random number, ≥0, on average 15)`-byte "redzone" (and some padding up to a multiple of 16 bytes which can predictably hide a bug, though at least such a bug won't corrupt unrelated heap). Stack handling is a significant difference though.
To expand a bit: 16-byte chunks of memory can be associated with a four bit tag. Then you steal four unused high bits from your pointers to store a tag value. When memory has a tag, a pointer used to access it must have the matching tag in its high bits. Your malloc implementation can then assign a different tag to adjacent allocations and any overflow into an adjacent allocation will have a mismatched tag and will trap. Likewise, change the tag on free and an attempt to use the pointer after the allocation has been freed will trap.
Of course, this doesn't come for free. Four bits per 16 bytes means a 3% increase in memory needed for tagged memory, hardware overhead for checking tags on memory accesses, and software overhead of setting/changing/clearing tags as necessary. This overhead is low (Apple shipped this in flagship hardware a year ago and nobody's complaining about performance there) but not zero, and an implementation with poor performance could be a real problem.
Implementing MTE is mostly another step in Apple’s efforts towards memory safety. There are bunch of things they’ve done to try to make this work a little better, but for obvious reasons, this is never going to be turned on by default for user apps and is only enabled for specific pathways.
> Because EMTE tag checking imposes a performance cost, we designed Memory Integrity Enforcement to take advantage of our secure allocators first and use EMTE to protect only smaller individual allocations within a type bucket, which software allocators can’t defend on their own.
Which seems to imply that outside of Apple’s targeted use in the kernel, the developer use of MTE on Apple platforms requires that you migrate your code to use their typed allocators…which I don’t know how many people are going to do that (and it is clear their answer is going to be to point people to Swift since it makes these types explicit for the compiler and even then, I am sure there are bugs in Swift in certain cases). Memory tagging is neat tech, but I don’t know if it’s going to gain all that much traction. It requires a burden that I suspect most app developers are not willing to take on and given that the user interaction is a crashy app. This is a path to nowhere except those of us who understand the trade-off.
Ive had MTE enabled on GrapheneOS for the past 3 years for pretty much all apps I use without any issues.
Theres a couple of apps that use C++ libraries for which I disable MTE.
Most android apps using Java or Kotalin which hugely helps with avoiding buggy apps crashing with MTE enabled.
Android can and does regularly force developers to make changes to their apps to improve privacy and security on the platform. Like theyve done with other changes they could gradually force developers to fix bugs in their apps. Maybe offer a MTE opt out for apps that use memory unsafe languages.
Think its likely Apple and Google will increasingly push forward the use of MTE in Android and iOS
Typed allocator migration just requires a recompile, it’s not a big job.
What does the malloc or hardware do if for some reason your program needs to access memory addresses that overlap your tag bytes ?
I know that it's a really improbable scenario and the OS would also just refuse you allocations at some point, but what would the malloc implementation and the MTE do in such a case ? Fail the allocation ? trap when reading the pointer since it would point to the "wrong place" ?
If you mean the memory that stores the actual tags associated with each 16-byte chunk of memory, that's a separate region that should be inaccessible except for using the special instructions for reading/writing tags.
4 bits per 16 bytes is ~ 1/4 the cost of ECC, which is possible to implement with just an extra cycle or two of latency in the memory controller. MTE seems similarly lightweight. Costs some transistors and a percent of a percent of power budget, but much like ECC it seems a fair bargain.
In case anyone wants to learn more how ASan works, I wrote a blog post on it at https://blog.trailofbits.com/2024/05/16/understanding-addres...
It also explains that the fact ASan works well with, e.g., std::vector in C++ doesn't come for free and if you want ASan to detect bugs when your custom collections or allocators are used you have to use special ASan API to mark [in]accessible memory.
Unfortunately, the headline is somewhat optimistic compared to the reality, I think.
The thread makes it sound like it could have been disabled due to errata or performance issues. Basically, it looks like the software Google is shipping intentionally doesn't use MTE on the Pixel 11 hardware. That raises the question of...what does Google know is wrong with MTE on the Pixel 11?
> it looks like the software Google is shipping intentionally doesn't use MTE on the Pixel 11 hardware. That raises the question of...what does Google know is wrong with MTE on the Pixel 11?
Apple replaced MTE with an upgraded version that can run in synchronous mode all the time without the performance hit.
> Consider that MTE can be configured to report memory corruption either synchronously or asynchronously. In the latter mode, memory corruption doesn’t immediately raise an exception, leaving a race window open for attackers... We believe memory safety protections need to be strictly synchronous, on by default, and working continuously.
https://security.apple.com/blog/memory-integrity-enforcement...
Google may just be getting ready to follow suit.
> Google may just be getting ready to follow suit.
That may well be, but it's a regression for the time being, since Advanced Protection mode presumably no longer uses MTE (or a replacement for MTE), whereas it did on the Pixel 8, 9, and 10.
It seems like they should've continued to offer MTE until the replacement was ready.
I've read that there's a significant performance cost to MTE on Android (or on Tensor). It might just be that.
There's only a significant cost to synchronous MTE.
MTE ships two modes. synchronous mode and asynchronous mode. SYNC is slower but gives you far better traces and throws an SEGV_MTESERR as soon as violations happen. ASYNC however is async so there's a bit of a delay between a violation and the "catch" that throws SEGV_MTEAERR.
Strictly speaking async is worse for security because there's a brief window of time where the process "gets away with it" but the main differentiator is that because things don't stop the moment the violation occurs, SEGV_MTEAERR traces tend to be some degree of "out of date" vs SEGV_MTESERR which capture the exact state of the world the moment the error occurs.
The main tradeoff here is that async MTE has basically negligible cost. Something like 1-5% in practice but in microbenchmarks you can see up to 50%. Vs synchronous MTE where the penalty is on average closer to like 10-15% but in microbenchmarks it can be like 5-6x slower.
So yeah the perf cost is there but it's really not a major issue.
------------
The main issue is that apps and services crash when an MTE segfault occurs. So this means that to the uninformed end user apps appear spuriously unstable with no meaningful context. And it's not just apps. On graphene I see MTE segfaults semi regularly from various system daemons (mainly related to GPS/nav) and occasionally in Google Play Services itself.
From time to time I get them in Youtube and I constantly get them in the Twitch app. It's very annoying and the apps just crash when you happen to do some particular action leaving you walking on your toes to avoid accidentally tapping whatever magic pattern happens to invoke a MTE SIGSEGV until the next update. Doubly so since most apps provide no meaningful interface for uploading log traces to report these issues.
-------------
If Google wanted to roll this out without the spurious crashes they'd need to deploy it and eat the minor perf hit but they'd need to register a global signal handler to capture these MTE SIGSEGVs and report them back to the services in question without crashing the app.
And most app devs don't care so they'd be taking a minor (or major) perf hit in exchange for logging errors that developers always ignore. Doubly so in the modern day of "just have AI fix/do XYZ and who cares about the consequences as long as it runs".
I'm not surprised they dropped it but I do honestly wish they'd forced the issue and just deployed it and forced apps to fix their shit.
Given the current use of AI tools I would think that SEGV_MTESERR crashes would be low hanging fruit to fix. AI tools do ok with identifying memory corruption issues, given a smoking gun, in my experience.
SEGV_MTESERR is reasonably low hanging fruit until it's in a binary blob and the IC manufacturer said service interfaces with refuses to do anything to fix it.
Likewise if it's because a language is being lazy with how it handles memory (see: wireguard-go SEGV_MTESERR because golang's string implementation was being overly permissive with pointers).
And of course this only applies to SEGV_MTESERR. SEGV_MTESERR has a much higher runtime overhead that Google and co would not tolerate (but Graphene does) so they'd likely only be willing to use SEGV_MTEAERR which provides traces that point at a completely different place than where the fault occurred (but still provide a little bit of useful info).
Are there any alternatives for a similar runtime security improvement?
Nothing is wrong. They just don't use MTE yet, they probably have other priorities. They wanted to save some costs until they will use MTE, so they removed some hardware acceleration and downgraded the GPU. Simple as that.
AFAIK the other Pixel devices, going back to the 8, do use MTE in the stock OS. Some MTE features are only enabled when "Advanced Protection" is turned on in the OS settings, but the stock OS on those devices does make use of MTE. In the Android documentation, MTE is explicitly called out as one of the protection features enabled in the Advanced Protection mode.
So, I don't think it's accurate to say they don't use it yet. They were using it, and then they stopped. I think that's one of the reasons the linked GOS thread says they're concerned that "MTE may actually be broken due to CPU errata".
And axed 4GB of RAM. In many ways, the Pixel 10 Pro is the better device for running GrapheneOS (and maybe even Pixel UI). Well done Google!
The thermometer was nice too if you have kids.
It’s absurd that a small project like Graphene is able to run rings around a giant like Google in the security sphere. Almost makes you wonder if some of those vulnerabilities are intentionally allowed to exist.
Vulnerabilities in the world’s most popular (by volume) mobile OS could provide a plausibly deniable global espionage backdoor.
The perception that they are able to run rings around Google is not necessarily an accurate one. There are tradeoffs in any engineering situation and GrapheneOS devs make different ones based on different requirements. Google is one of the reasons MTE even exists. I don't think GrapheneOS devs would be capable of helping push forward that technology in the same capacity.
Fair, and also Google is not a monolith. I’m sure that their own security experts who helped push things like MTE are also confused and frustrated by the lack of internal adoption.
A lot of heavy lifting is involved at the lower levels of security and Graphene doesn’t have to deal with this work, generally. I guess my surprise comes at the “fit and finish” stage where Google regularly seems to be lacking.
Do we know if Google is operating on the frontier of the trade-off curve? What do we lose from Pixel Android when GrapheneOS security changes are made?
Google's spyware (Play Services) ceases to work normally and Graphene had to patch it to run as an unprivileged app. So that's one "loss."
I personally don't bother with Google dependencies at all. If an app requires them to work, I will do without it.
Actually functionality, stability of apps and features people use.
What functionality and features are lost in the base Pixel OS (not apps) as a direct result of GrapheneOS's changes to AOSP?
Very little if anything at all. Depends a bit how you draw the line between an app and the base OS. Contactless payments dont work with Google Pay - but that is due to Google using attestatiom to check if the OS is GMS lincesed (GrapheneOS is not). There are alternative contactless payment options that work fine.
Moving the goalposts to "it's not their fault that it doesn't work" doesn't really change it doesn't work.
There's quite a list you can get even from GrapheneOS's posts - among other things, there's a rather large set of apps that will outright crash when MTE is enabled.
Could you list a couple of the biggest functionality losses from AOSP due to GrapheneOS changes? I might be misunderstanding what you mean.
Elsewhere in the thread, someone reports that with MTE enforcement enabled, there are lots of crashes. And app and system service developers don't seem responsive to them.
That's not something that's really acceptable on a $500+ phone... so if you're Google, you're not going to turn that on by default and you're not really going to be interested in keeping it as a feature that users can turn on.
Graphene has a different focus, so they can push the tradeoff towards enforcement and crashes rather than non enforcement and fewer crashes.
Google could perhaps do more with their crash logs and pushing app developers to fix MTE crashes, but they don't often do a lot of that; and they get called out for being developer unfriendly when they do enforcement.
That was partially true a couple years ago that there were some crashes related to it. It is absolutely not the case anymore.
Why not force it on an API change? It wouldn’t be the first time there was a breaking change.
Because it (anecdotally) crashes too much, but only on $$$ devices with cool CPUs that can use it. People buy expensive phones because they're supposed to work better.
They should really do some sort of sampling thing to generate crash dumps and find big offenders and increase the coverage over time.
For Google employee devices, 0.1% of background execution starts while charging in an idle period (overnight bedtime charging) will use MTE. When any specific device hits an MTE crash, back-off sampling for 1 week on that device. Modulate the sampling rate so crashes are manageable.
When Google employee devices are not crashing overnight at a high rate, then start sampling background execution during the day for employees and overnight background execution on general user devices. Finally, sample on foreground execution, again for Google employee devices first.
If there's significant variation in crashes by app, you probably need to setup a way to set sampling rates to zero or very low for application versions that have been identified as a known problem and don't need additional traces.
I wonder what causes the crashes. Is it a buggy MTE implementation (or buggy hardware), or are app devs doing something that would only cause crashes under MTE? Maybe it’s something they are using like a library or framework?
Its buggy apps. Ive been using MTE for pretty much all the apps on my GrapheneOS phone for 3 years without any problems.
2 apps have C++ libraries and MTE makes these crash due to memeory bugs, so I disable MTE for these.
There's a lot of speculation that US model censorship around "cyber capabilities" is about protecting an inventory of non-public vulnerabilities used for intelligence purposes. That same explanation could cover Google's MTE actions as a result of pressure from intelligence agencies. Nobody should be surprised if they prioritize what they see as national over personal security.
>Almost makes you wonder if some of those vulnerabilities are intentionally allowed to exist.
the fact that someone can say that post-Dual_EC_DRBG/project-rubicon/eternalblue/eternalromance exposure boggles the mind.
Since GrapheneOS is building on something that Google is giving away for free, I do hope that they release something better. Everything else would be.. kind of weird, no?
What do you mean? Graphene is also free software and everyone including Google can use it and learn from it.
Just to nitpick, the facilitation by AOSP & ARM is available so it is not about the OS but about Google as a Hardware manufacturer.
They're running rings around google... by enabling features that google develops/support?
99.9% od the code GrapheneOS uses and 100% of hardware (including one like Titan security chip handling storage encryption and key store) is from Google.
So what are you on about man - it's easy to take someone else's work and put tweaks on it that wouldn't work at scale.
AOSP eng here;
I don't interact with Graphene or any of their folks at all; I'm just a distant observer like most folks here.
Still, I can't help but think that Graphene seems to want to complain about everything and anything that doesn't fit their niche use case. (As much as it seems beloved here, people that flash custom Android OSs are the very definition of niche users.)
My personal attitude to Graphene seems to get a bit more negative with each one of these "rants" and I doubt I'd go far out of my way to help them, even if I had exposure to them.
> that Graphene seems to want to complain about everything and anything that doesn't fit their niche use case
What would you want them to complain about instead? Of course they'll complain about that, just like Googlers will complain about things affecting their stock price, no one is surprised that people care about stuff they're personally involved in, it makes a lot of sense.
Now if these complaints weren't accurate, then I'd walk with you and feel a bit more negative with each piece. But the ones I've looked into, have been spot on, so who cares if it's for their specific niche? I expect them to care about their niche, that's why those people all work together in that organization in the first place.
> As much as it seems beloved here, people that flash custom Android OSs are the very definition of niche users.
Hmm... Let's try reframing this: "as much as it seems beloved here, people that install their own operating systems on PCs are the very definition of niche users"
I'm absolutely certain that's how IBM felt before the clones. But the ability to install what they wanted on a defacto standard platform is what launched the computing revolution. I think we'd still be living in a sterile monopolistic environment with $10k compilers otherwise.
Folks installing their own ROMs on phones are only niche because they've been pushed out at every opportunity using locked bootloaders, embedded security processors, factory installed secret keys, etc.
Despite all that, there's still thriving communities developing and using custom ROMs on their phones. That demonstrates more than niche demand.
> Hmm... Let's try reframing this: "as much as it seems beloved here, people that install their own operating systems on PCs are the very definition of niche users"
I mean yeah, 99% of people never installed an OS and never will, what's your point here?
> 99% of people never installed an OS and never will, what's your point here?
That the 1% who do build visicalc, Linux, the internet, Google, and every application and innovation that happens outside the corporate wall. The entire ecosystem everyone else ends up using.
And that calling that niche is ridiculous, shortsighted, and shooting oneself as a platform owner in the foot.
What are you talking about? You're making zero sense. I've been in the niche of modifying and installing my own ROMs, then to kernels, and now doing the same things commercially.
It is a niche, it always was. There was never a mass community of users, the most installs you used to see on any given AOSP based project is maybe 20-30K for the most popular devices like Redmis, Pocos or Google Nexus phones.
It's still a niche, and things that users do and used to do to their phones are fundamentally incompatible with Android's security model. Root access, magisk, xposed, overall zero or near zero security validation from the security perspective on all ROMs but Graphene (maybe some others, I haven't followed the sphere for a while).
> It is a niche, it always was.
Again, only because the platforms are locked down and there is no single standardized target for universally bootable ROMs in the way a Linux ISO can boot on any PC.
Kinda sounds like you might be younger and didn't live through the 8bit micro -> PC transition. Standardization of the platform led to a cambrian explosion.
https://en.wikipedia.org/wiki/Influence_of_the_IBM_PC_on_the...
> Again, only because the platforms are locked down and there is no single standardized target for universally bootable ROMs in the way a Linux ISO can boot on any PC.
Couple of things here. Android was never that open to begin with. There were always (and still going) issues with the kernel sources being withheld indefinitely, AOSP side changes are almost never released at all, iirc only Google released anything. Bootloaders were always an issue, not all manufacturers had unlockable (or easily unlockable) bootloaders.
Ironically, Google were always the spearhead of the standartization in the Android world. Project Treble and stuff resulted in the fact that almost any Android device released these days can run a generic Android image, it's not gonna be as functional as the original Android, but it's something.
Yes, I am on the younger side, that's why I know how it was in the Android world. Nobody is interested in reinstalling OS-es on their phones because the amount of effort to create an Android ISO that's as usable as what's preinstalled is truly gigantic. Android flavours are heavily customized for the needs of the OEM, and for the hardware that they're gonna run on.
It's a niche, always was and always will be. A niche powered by a few large players like LOS or GOS, that actually get paid somehow or heavily work for the idea, and an army of teenage unpaid maintainers, who quit the maintainership as soon as they get a properly paying job that takes most of their time (me).
> Nobody is interested in reinstalling OS-es on their phones because the amount of effort to create an Android ISO that's as usable as what's preinstalled is truly gigantic.
You've restated my thesis.
> Yes, I am on the younger side
I could tell.
It's pretty clear that you have an extremely limited exposure to the custom Android development space, since you never rebutted any of the points I brought up. Custom android was never easy and never will be. And it's fundamentally incompatible with the concept of having a truly secure env on your device (same as in the PC space, ironically enough).
Son, I've been building and installing custom roms since the HTC Dream, was involved in OpenMoko, and have owned nearly every open or Linux-running phone of the last 30 years back to the Sharp Zaurus and iPaqs.
Your custom rom experience sits wholly within a larger category of computer systems experience about which I've been speaking.
The sort of fragmentation you describe is a symptom, not a foregone conclusion.
Perhaps they complain because that's literally the only way to get Google to take notice?
Let's be real, AOSP doesn't exist any more. Google have closed down nearly everything. All the development happens in private, you've stopped addressing bugs raised by the public, the source of patches are only infrequently released, device trees are gone.
Wouldn't you complain?
All the development happens in private, you've stopped addressing bugs raised by the public, the source of patches are only infrequently released, device trees are gone.
To emphasize this point a bit more: only "QPR0" (major release) and QPR3 are released as part of AOSP. QPR1 and QPR3 are not released at all anymore, but contain fixes for vulnerabilities that are not marked high/critical (so don't end up in ASB). It is not clear to me whether OEMs get access to QPR1 and QPR3, but Google are not only witholding features, but also a set of security fixes.
Besides that, they are torpedoing other systems through Play Integrity.
IMO it would be best if AOSP was spun off from Google into its own org that actually cares about developing an open source system for others (both open source systems like GrapheneOS/Lineage and commercial vendors like Samsung) and that would have an attestation system that is open to vendors that have good device security.
If they're just some "niche use case" then why would Motorola partner with them? The way they see it,
> By combining GrapheneOS’s pioneering engineering with Motorola’s decades of security expertise, real‑world user insights, and Lenovo’s ThinkShield solutions, the collaboration will advance a new generation of privacy and security technologies. In the coming months, Motorola and the GrapheneOS Foundation will continue to collaborate on joint research, software enhancements, and new security capabilities, with more details and solutions to roll out as the partnership evolves.
https://motorolanews.com/motorola-three-new-b2b-solutions-at...
Motorola is not a big Android phone maker.
They’re the second largest manufacturer of Android smartphones in the US, and 10% of the global market. Seems a bit unreasonable to dismiss them out of hand on that basis.
Neither is google.
I think that's a worthwhile point to consider but it's only relevant if we move the goalposts from "GrapheneOS is only used by Android ROM enthusiasts" to "GrapheneOS is only supported by one small Android phone manufacturer".
To be frank, though, I don't see any of this line of reasoning as relevant; it's just appeals to greater authorities on either end. If AOSP is only for manufacturers there's really no reason for it to be open source in the first place. And then folks who care about actually improving security end-to-end outside of whatever's convenient to implement by those beholden to the quarterly profit metrics are up a creek.
Personally, if this whole GrapheneOS/Motorola thing doesn't improve the state of the ecosystem I'm going back to Apple or whatever other manufacturer makes it clear they take security seriously.
You might not like their style of speech, at least they care about their users. Maybe Google uses nice flowery language that makes the reader feel nice—IDC—actions speak louder than words.
Stock Pixel is an awful experience. So many useless notifications, popups, ads, privacy not by default.
Company: "We care about your privacy" meanwhile 1400 corporations they share data with
GrapheneOS: "There's zero telemetry in GrapheneOS"
The more you read the more you realize they are nearly always correct.
> at least they care about their users
No, users are secondary to the mission. If they happen to overlap, that's a happy coincidence. And caring for a user stops if Daniel has the slightest interpretation of their comments being an attack (i.e. any mild criticism or disagreement about direction)
They care about their mission and Daniel's stance first and foremost
I wouldn't be surprised if this comment is branded an attack
> If they happen to overlap, that's a happy coincidence.
I can't know what the developers of any OS are actually thinking, but based their actions, GrapheneOS does more for their users than any other OS.
Therefore I assume that doing good things for users equals care for users. It's probably stupid to try to guess about care.
Care: https://xprivate.lol/GrapheneOS/status/2094974844639068534#m
https://xprivate.lol/cryptofyre/status/2093880623005880440#m
> Stock Pixel is an awful experience. So many useless notifications, popups, ads, privacy not by default.
Huh? What pixel are you on? You only get notifications from stuff you install after the initial setup is done. And even then you can outright mute applications, completely.
I was on stock recent pixel A17 for a few hours before flashing GrapheneOS. They took over the power button for Gemini, there's gemini in Messages, there's 50 apps you don't need. Yeah you can disable notifications but it was constantly giving tips and tricks and other bullshit. The OS feels super bloated compared to GrapheneOS.
I hate the power button change myself, but 1. It was turned into the Assistant button by default for many years now. 2. Can be switched back easily in settings.
> there's 50 apps you don't need Where are you getting your pixels? Mine came with all the essentials, I don't remember disabling anything in particular. No game demos, no preloaded meta cancer, just the normal google suite.
> The OS feels super bloated compared to GrapheneOS.
When was that ever not the case? When was a stock OS lighter than a custom flavour on any phone? You can't ship Android in the state that custom OS-es do, you won't pass the certification to get Google's goodies like OTA service and preinstalled play store. Don't quite get the Gemini argument either. You're buying a Gemini phone that touts AI everywhere on its promotional page. What do you expect?
Even in the EU spyware use is prevalent (and i 'd guess everywhere else in the world). There have been many scandals of government authorized commercial spyware been deployed against journalists. Is it really that niche a mobile OS that tries to not be exploitable by them?
Graphene doesn't position itself against spyware.
For example, a user being able to inspect and edit the files written by an app, no matter where or how those files were written, would be an anti-spyware feature: you could better observe the behavior of a closed-source application.
Grapene opposes this feature because the app security model protects the app AGAINST the device user editing or reading protected files.
Graphene's philosophy is enforcing the Android security model. The Android security model gives guarantees to the app developer about how their app can behave, even where the device's owner wishes otherwise. See: Play Integrity.
GrapheneOS have mentioned wanting to expand the logging/intrusion detection capabilities of their Auditor app but contend with the need to include it as a system app which is against their philosophy (PoLP). It is not accurate to say they don't want to do anything about spyware.
They are also completely against Play Integrity as implemented on principle.
Graphene puts a HEAVY emphasis on security.
Also your argument about a user inspecting and editing application files feels like a strawman argument. For example many spyware use malicious links to infect the devices, not malicious apps.
Let's say I want to secure my system against an app developer deciding to delete my data stored in their app.
To do that, I wish to store a copy of all the files the app has written to my filesystem, and put that copy outside the app's control. This is explicitly to contain data the app's developer does not WANT me to be able to keep.
A. Is my being unable to do this "more secure"? If so, why is the specific threat I described to my data integrity - an app developer deleting my data - invalid?
B. Does GrapheneOS support this protection, ensuring the device owner is secure against the app developer, or do they instead secure the app developer against the user?
Let's say you are a Graphene OS user. Why are you even using such an app?
Also you are misrepresenting the threat model. The problem is not an app deleting it's own legitimate data.
P.S. On GrapheneOS you can block apps from getting Internet access to limit their bad ideas.
I do not feel you addressed my point at all.
I want to install an app now, and protect against its developer doing a rugpull on my local files later. I do not want to need to review each new app version in advance as I install it.
This is clearly a legitimate case where what the Android security model says I should be permitted to do makes me less secure against an attack by the app developer.
So other projects are not supposed to critize Google? Graphene's focus is on security and they complain about lack of security in your products. Seems valid to me.
Also security and using non-Google OS are not niche usecases. I'm not sure how you are working on Android, the most popular OS while claiming security is a niche usecase. In fact, I have less confidence in security of your work.
The way I read the comment (admittedly putting words in their mouth), is that there are different ways of "complaining". I am someone who is actually quite sympathetic to Graphene's project and have considered getting one of their Motorola supported phones when they come out. But every time I encounter their writing, they come off as unnecessarily abrasive. They could be making the same points with a lot less potential bridge burning.
They are not neutrally pointing out places where their preferred priorities aren't being met (which would be reasonable), but they are speaking (forgive me the hyperbole) like religious zealots. I don't blame someone who doesn't share their particular viewpoint with being hesitant to work with someone like that.
As a consumer, I don't particularly care. I'm not going to make my decision based on their comms style. But a developer who actually has to work with them? Yeah I could imagine it mattering.
Agreed. They clearly know their shit and they are stunningly great engineers, but Graphene OS would be much more loved and respected if they had a PR person to take care of online communications.
The "religious zealots" have their own motherfucking Cellebrite column. That means they are right.
They can be abrasive all they want. People who are wrong still need to sit down and listen instead of spitefully "burning bridges".
And that's assuming they're actually guilty of this "abrasiveness" that people accuse them of. I don't actually think that's true. I think their comments are extremely informative and filled to the brim with valuable technical information. I've spent quite some time reading strcat's comments here and it's actually changed my viewpoints on some security matters. For example, I'm no longer as obsessed with knowing what's inside firmware as I used to be, it's better to minimize and isolate such black boxes so they can do no harm. I started valuing IOMMU support and grouping in PC motherboards.
If there's any "abrasiveness" in their posts, it's probably just an undercurrent of frustration with a world that simply refuses to understand or cooperate. A feeling I can't help but have enormous sympathy for.
> They can be abrasive all they want. People who are wrong still need to sit down and listen instead of spitefully "burning bridges".
While I wish that were how humans work, it definitely is not. You can make the strongest argument ever and still lose if you deliver it in a way that the listener can't process it because you're triggering emotions. The best arguments IME are factual and respectful.
The best thing about being right is it doesn't matter if the listeners can process it, you're still right.
There are consequences to ignoring GrapheneOS developers. Insecure phones can get people killed.
The best arguments are respectful and measured, but few things in this world are ideal. It's in your best interests to tolerate any perceived "abrasiveness". Ignore the the guys who make Cellebrite sweat at your own peril.
The argument about the seriousness of the topic seems to be to be arguing for taking into account human psychology and toning down rhetoric (note: not failing to make the claims, or dropping points, just changing tone) in order to make it more likely that you get listened to. To the earlier commenters point: it would be great if nothing other than correctness + importance determined likelihood of being listened to. But that's not the way the world works.
Nobody at google will get demoted if vulns in grapheneOS can’t get fixed due to their oss strategy and result in deaths. Yelling at them angrily will not change this. When was the last time someone abrasive called you a moron and corrected you - did you come away feeling like you will take the lesson to heart and adjust your behaviour?
I am sympathetic to GrapheneOS, but I have also worked with them and they are not easy to work with. Google is in no way required to take on criticism from OSS community- GPL and MIT do not actually require the rights holder to not be an asshole. In such a social dynamic, one will find greater success using language that appeases instead of repels the asshole.
When you are a minority you have to be incredibly loud for any chance to sway things your way.
Not saying whether it's a good thing or a bad thing, but just the nature of reality.
I personally would wish that the AOSP communities would work together. I would love to see some oft their work to also land on other devices even if that's not 100% secure without bootloader relocking. It sometimes seems to me that the perfect seems to be the enemy of the good sometimes. I still understand their sentiment and their goals. I hope that the Motorola thing takes off. Still think a crossover with true aftermarket roms would not hurt.
Thankfully the project doesn't care about your personal attitude. That "niche use case" literally saves lives in countries where saying the wrong thing can put you to death. Since when is calling something out a rant?
Graphene relies on donations, and will be dependent on more than just the users where their lives are at risk.
If everyone in North America and Europe stopped using Graphene, I doubt they would last very long.
High standards are great, but they have to interact with the community and the sanctimony doesn't help.
In a world where economics makes security hardly a secondary concern, a situation exploited by both the intelligence and surveillance broker sectors, dogmatic sanctimony for high security is a feature and not a bug.
This doesn't read as a rant to me, but as calm and factual, regarding a genuine security regression that merits public attention. What is your interest in mischaracterizing it?
Could you please explain why supporting MTE/potentially EMTE in production as a goal represents a niche use case? Isn't mitigating memory corruption issues a mainstream ideal? How else would you propose to do it?
I'm not an AOSP engineer, but that was my thought reading GOS's comments: Why be negative toward the people who you want help from? They don't need more stress in their day and wouldn't want to engage, with the promise of enjoying more of the same treatment. If it's consolation, much of their communication seems the same - it has nothing to do with AOSP or Google. :)
Communication is a challenging skill and not all good engineers are good at it. It's stressful to be in a role that demands lots communication if you feel you aren't good at it (I'm not speaking for GOS leaders - I don't know what they think).
Going out on a limb, hoping it helps - the most powerful single solution IME is compassion: Compassion toward yourself and toward others. They are engineers like you, trying to get through a stressful day like you, and they could use some pleasant interaction from someone who understands what it's like. You could use some of that too. Running GOS isn't easy, I'm sure.
I'm not an AOSP engineer, but that was my thought reading GOS's comments: Why be negative toward the people who you want help from?
What help though? Google has closed off AOSP and only does source code drops twice a year. Google has embargoed security patches for three months and only provides them to OEMs of Google-certified Android phones, not other AOSP-based projects. Google stopped providing git trees of kernel sources and instead requires projects to submit a request for a Google drive link for each kernel version that takes up to weeks to process. Google is shutting out open Android systems through Play Integrity.
Google is not helping anymore, over the last 1-2 years they have tried everything to sabotage AOSP-based projects. The only reason that they are not fully closing AOSP is probably because 1.) they would get in hot water with regulators; and 2.) AOSP will probably get forked.
> people that flash custom Android OSs are the very definition of niche users.
This is a mischaracterization. The niche isn't Android hobbyists, it's people with what should be a basic expectation for privacy. I wouldn't flash GOS or any other OS if I could safely avoid it.
Daniel Micay who's (most likely) running the project again has a history of being paranoid and coming up with perceived attacks from people on the internet. The loudest one by far was the story with Louis Rossmann. I've been following Micay for close to a decade. This is the exact pattern that he operates on. Tons and tons of drama for no reason.
I truly, sincerely hope that I'm wrong, and that he isn't actually at the helm, because then Motorola are under serious thread of getting an essay saying how they're a terrible company sending assasins to his house, after they disagree on some little thing.
Tbf with all the information that has come to light since then Rossmann and FUTO were also not behaving correctly and it was fair from him to deny coming onto their show (where they have also invited fascists before)
Not sure what you're referring to. I'm talking about that time that Micay alleged that Rossmann launched a stalking and harassment campaign against him. I don't know of any shows he's been invited to.
Maybe if Google did not shove their spyware down people's throats and actually allowed users control of their phones, there wouldn't be a need for projects like Graphene and Lineage. Until then, complaints are more than justified.
"I don't know anything but I don't like Graphene"
Well we know Google isn't enabling MTE, while LLM-enabled exploits are multiplying rapidly. Maybe you need to get back to work?
Security shouldn’t be a niche use case. There’s a constant trickle of CVEs, and spyware vendors are known to abuse these exploits in their software. All this on devices that are reachable in the US through a text or MMS, sent to an easily located 10 digit number that isn’t easily changed. These are devices that people now use for all kinds of sensitive tasks!
Security should be the number one priority, frankly. Graphene has shown that this is possible, and they have tried multiple times to get Google to integrate their work.
They are good enough to have their own Cellebrite column. If they complain about something, you should probably listen.
> I doubt I'd go far out of my way to help them, even if I had exposure to them.
Too busy crippling sideloading I guess
> My personal attitude to Graphene seems to get a bit more negative with each one of these "rants" and I doubt I'd go far out of my way to help them, even if I had exposure to them.
My thoughts entirely, I prefer to use CalyxOS, which for some reason in the eyes of GrapheneOS is pure evil.
I've never seen them call it "evil". I've seen them debunk CalyxOS security claims as well as criticism of GrapheneOS, and they usually provide some serious reasoning and technical information when they do it. They know what they're talking about.
Don't take it personally. I'm a huge fan of Linux, and GrapheneOS routinely comes here and calls it a huge security liability. And they are right.
... and my opinion becomes more positive with each rant. We'll have to agree to disagree. The only reason I buy Pixels for myself and my family members is because of GrapheneOS, otherwise it would be used out of date hardware for LineageOS or some kind of Linux phone. I am thankful that they are attempting to diversify with Motorola, being entirely Pixel dependent has been a project vulnerability; anytime Google decided to lock down the boot loader, it would have been curtains for the project.
So not only a price increase: less RAM, performance scraping backwards, crippled/lost features, a camera system that captures stuttering audio and video imperfectly often, and Pixels dropping out of AOSP.
Google Pixel has the marketshare it deserves.
There's no phone that isn't like that this year
Fairphone gen6+ (vs gen6) got an increase in ram from 8 to 12 and from Snapdragon 7s gen3 to gen4. MSRP only went from 599 to 649.
Yet, you can pick up a Samsung S26 for 600 Euro that will absolutely demolish a Pixel 11 in practically every aspect except cameras. Years ahead in CPU/GPU performance, etc.
Yet no GrapheneOS.
I am almost certainly going to live with whatever drawbacks in terms of camera quality, battery life, etc. Come with their Motorola phone when it's time to upgrade. MTE is such a non-negotiable for modern digital security on phones it's crazy Google would be so okay with this regression.
What's especially stuck in my mind lately is how insecure basically all desktop OS' feel. In at the point of buying a second and third GPU for my desktop to run my email and browser in dedicated VMs because everything feels as watertight as a sieve. Qubes seems more and more appealing in a world where every open source software supply chain is under seige, corporate software underprioritizes security, and most sites will stop at almost nothing to surveil you.
I truly lament this new reality where MY computers I PURCHASED feel to use like I'm reaching blind into a paper bag filled with razor blades.
>What's especially stuck in my mind lately is how insecure basically all desktop OS' feel. In at the point of buying a second and third GPU for my desktop to run my email and browser in dedicated VMs because everything feels as watertight as a sieve. Qubes seems more and more appealing in a world where every open source software supply chain is under seige, corporate software underprioritizes security, and most sites will stop at almost nothing to surveil you.
I made the same observations and conclusions as you, but a few years ago. I have been daily driving Qubes OS exclusively since on my Thinkpads. If you haven't started to yet, I implore you to consider the following when buying a system for Qubes: mid-line CPU with plenty of PCs (turn off the ECs), As much RAM as you can afford (seriously, budgetmaxxx on RAM), dedicated secondary GPU with the minimum specs you need is a must for inference (may have to widen the budget a little bit after budgetmaxxing RAM).
Expect to not game on the platform. I have not tried this but if you can't cut the digital crack-cocaine habit then it may be possible to set up a gaming qube which takes the dGPU passed through: get a monitor + second USB keyboard & mouse, plug the monitor into the dGPU's HDMI ports, and pass through the secondary peripherals. Digital crack-cocai... err games... don't like having their frames pushed over X11 forwarding even if the dGPU is passed through, so you will need the monitor; the secondary input devices are so the mouse and keyboard inputs don't leave the gaming qube.
> What's especially stuck in my mind lately is how insecure basically all desktop OS' feel.
I ended up switching away from using my Mac to using an iPad M5 + Magic Keyboard as my sole computing device for the security benefits, and it’s been working pretty well and in retrospect I don’t miss anything from macOS aside from a chromium-based browser. However I’ve said it before that switching back to using a Mac after using iPadOS for so long feels like a serious regression, for all it’s faults the iPad + iPadOS feels like a futuristic technology preview of macOS.
For development work I use GitHub Codespaces (and self-host Coder in the future hopefully). I was moving towards cloud/container-based development already and it works nicely. Plus NPM vulnerabilities are no longer a serious issue. Again the only thing I miss is Chromium, but I’ve altered front-end development planning to target Safari/Firefox since websites shouldn’t be implementing non-standard Chrome-exclusive features anyway imo (I’m saying this as someone who highly prefers chromium lol).
12 hours before this was posted to HN: https://grapheneos.social/@GrapheneOS/117198980596943121
GrapheneOS: "Multiple Google engineers we've contacted have said they aren't able to give us any information about this so we're left doing reverse engineering and relying on leaks. The leaks do not seem reliable and do not match what we see. Our concern is that MTE may actually be broken due to CPU errata."
> It's possible to enable reserving the tag memory for MTE via `fastboot oem mte on`, boot a non-stock kernel ignoring arm64.nomte and use MTE.
What I'd like to know is, is it not sufficient to just check the feature bits in ID_AA64PFR1_EL1? (Isn't this the first thing you'd try when trying to determine if the hardware supports some feature?)
If they were making the determination solely based on the fact that `arm64.nomte` is being passed to the kernel (or based on what features are exposed by kernel interfaces), it may have been better to say "MTE is seemingly disabled in the current Android release" rather than claiming that it's simply not present in hardware. If you're asking the Linux kernel about hardware features, maybe it takes cmdline arguments into account when presenting that info to userspace.
The TRM[^1] mentions that some of the feature bits depend on BROADCASTMTE (presumably some CPU input pin), but maybe that signal isn't constant and is allowed to change based on what happens in firmware/the bootloader?
Also, why the claim about the lack of hardware acceleration for MTE in the caches, is there evidence for that, or is this also a misunderstanding?
I think it's reasonable to assume that the perf impact of MTE is non-negligible (on cores in older Pixel devices[^2], MTE apparently suffers from the fact that checked stores are serializing!), but it's entirely possible that this does not follow from some physical design concession when implementing the SoC. The characterization of all this as some kind of cost-cutting measure is not necessarily accurate.
[^1]: https://support.arm.com/documentation/108014/0101/?lang=en
[^2]: https://arxiv.org/pdf/2601.11786
ID_AA64PFR1_EL1 was discussed five days ago
https://discuss.grapheneos.org/d/41564-pixel-11-doesnt-meet-...
Fascinating, thank you. It seems strange that the C1-Pro/C1-Ultra TRM mention that ID_AA64PFR1_EL1[11:8] should still be non-zero even when BROADCASTMTE is low.
edit: Oh, I guess Linux does emulate the feature registers, doh. Since that user is booting with arm64.nomte, the kernel is changing those feature bits to zero! This probably explains the confusion here about hardware support: in Linux userspace, you still trap for system register reads, and this is just abstracted away from you...
https://github.com/torvalds/linux/blob/841e384b841a3d89c50b4...
A "Google bad" essay gets a lot more clicks.
"Aside from MTE, Google downgraded the Pixel 11 series GPU compared to the Pixel 10 series in addition to reducing RAM for base Pro models."
I will make a note not to buy this model.
Is this the worst quality control lapse since the Huawei Nexus core failures?
This isn't a quality control issue, but a beancounter-chooses-cheaper-parts issue.
One must believe that MTE was intended for use on this device, as it is present on everything down to the Pixel 6.
Everything else is in the realm of bean counters, I would think.
The GPU claim is contradicted by pretty much every benchmark though.
A company making multi billions of dollars in profit still looking to save money selling their flagship phone?
Google makes money, I'm not sure Pixel does.
Zero chance the Pixel division is net positive
I think there’s a very high chance it’s net positive. It’s growing quickly and approaching 10% marketshare for premium phones (the most profitable segment) in the US.
https://axis-intelligence.com/google-pixel-statistics/
They’re also strongly tied in with Google Fi which is almost certainly very profitable.
It is net positive for keeping folks in the Google Ads funnel. That is Androids purpose after all.
agreed h/w negative, but how far removed is pixel from android itself and then the ad tech platform, playstore, etc in general?
Phones typically have 2 operating systems: one to handle telephonic functions like managing tower connections and separate operating system for user applications. The security implication here is that the telephonic operating system has access to the same system resources as the application operating system at the same time, right? The problem is your data is always available in plaintext to the telephonic OS because it has to be stored somewhere before it gets encrypted, so assuming graphene OS doesn't handle both functions its not a perfectly closed loop system.
IOMMU
The way it should have been all along.
> Phones typically have 2 operating systems
A separate OS runs the security chip (like Titan). Another could be running in EL3 (Trusted Execution Environment), yet many could be running in Realms (mutually untrusted VMs). The later stage bootloaders and some firmware (ex: GPUs) themselves could be considered OSes in their own right.
Also, the SIM card has its own operating system.
Also, most smartphones today have at least one browser installed, they're basically their own tiny little OSes at this point.
No, shared peripherals need to go through a central arbiter for access. In the case of things like storage the AP's OS, eg android, provides that roll. This is also true for the OS that runs under the trustzone. Be careful about spreading rumors without validating them.
Most devices have a large number of operating systems. The camera also has one, and the storage has one if it's eMMC. The WiFi/BT chip has one. The AP has at least two, because of trustzone.
People bring up the baseband (ie. cell modem) having its own OS because someone gave a talk on it at defcon or something, but it's just one of many.
Shouldn't the headline read "GrapheneOS: Pixel 11 has bare minimum hardware level MTE support"?
As currently written, one would take away an entirely different message from the headline.
Boo. I was delighted by its introduction in pixel 8. I thought this was a resurgence of the old Google, actually giving a f about software, security and the end user.
iPhone 17 has it now. maybe Google's work here is done
https://www.reddit.com/r/apple/comments/1ndcrug/iphone_17_in...
So, where does the restriction for side loading apps live, on the device or the OS? Curious about getting the Pixel 11 Pro if/when I can install GrapheneOS.
GrapheneOS themselves said it wasn't worth getting the 11 over the 10 on Twitter. It has a worse GPU and less RAM, and the increase in CPU speed is so negligible that often you end up with better performance using the previous generation. In addition, GrapheneOS isn't even available on 11 series devices yet.
The 11 series has a new MediaTek modem compared to previous Pixels(since 7 I think?). The old modem had power efficiency issues during poor cellular signal as well as a relatively lengthy GPS lock acquisition.
I don't think there is enough evidence yet about either but I can attest that GPS feels improved over my 7. It's only been 1 week though.
The non-Pro 11 has a much larger photo sensor and has more base storage. Everyone has different priorities. I'm not saying which is better but trying to give a more complete picture.
Yeah the camera seems to better which could be deciding to some people. But while the base storage has increased the price did too, so a base 265gb might cost the same as an upgraded one from 128gb.
interesting. what about the side loading apps? I assume that's an OS and Graphene would likely allow it?
Sideloading has always been allowed, and that new "advanced flow" is something entirely decided by the OS so I don't think it has or will have any effect with GrapheneOS use.
It's actually recommended to get least of your apps as possible via the play store and use 3rd party stores if you are going for maximum privacy or de-googling.
I’m in the process of switching from an iPhone to Graphene on a pixel 10 Pro.
I can load apps from the web (disabled by default iirc), the local file system, a sandboxed instance of the Play store, Obtainium, f-droid, wherever.
Took some getting used to, but it’s nice feeling like I actually control and can choose how my phone works.
I've done some research and am considering as well, though I might wait for the motorola phones to release. Have you encountered apps that didn't work? Like banking and stuff seems to break a lot of the time.
All percentages are made up but feel right to me.
90% of apps work, no finagling required
Next 6% might need some fancy setups, like setting up storage scopes or other privacy settings so they can function as expected. I won't pretend they are always intuitive, but if you really can't figure it out, someone on the internet has already had the issue and posted their solution online
Another 3% are your banking and similar apps. Some work, but some are needy, and I have these in another profile which has the sandboxed Google play mirror and other Google services that GrapheneOS mocks
Then is your last 1% that are so stubborn that not only require Google play, but they have the audacity to require you be logged in to Google play or they won't start. Personally, I just refuse to use these apps when I get one, as they are random dumb ones, like some incredibly privacy invading videogame app. However I'm sure there will be some app you find that is like this. I guess you can log in, or heck make a third profile and hopefully don't need that app often. I guarantee there is a friendlier alternative that is worth looking into if you do feel you really need that service.
The OS, obviously
My strong instinct is that GrapheneOS's plan could be problematic:
Generally, do not to invest in features that the upstream vendor isn't fully committed to. It might work today, but tomorrow or next year, etc., it might not. They might even deploy other things that are incompatible or do other things that assume the feature is disabled. In fact, plans to do those things might be the reason they disabled MTE.
Also, never support products or projects that won't work well. When someone's Pixel 11 is slow, they will think and post that 'GrapheneOS is slow'. Blaming Google will not be very credible and will look like you're just trying to duck responsibility. Your brand's name goes whatever you release; people will associate their experience with GrapheneOS (also because GOS is the new, unconventional thing; Google is the established norm); if their experience is bad, they will associate it with GOS.
Their plan is to seemingly comprehensively test MTE on the Pixel 11 to see if it's usable for them. I'm not sure how that's "problematic", in fact it seems like it's exactly what they should be doing.
Their real focus is on their Motorola partnership, anyway.
GrapheneOS is a non-profit, publicly naming the actual issues is the only thing they can do to make Google fix those issues. And it is clear that security/privacy is their first priority, not "brand image"
Some on HN defend everything GOS does (maybe not you, but there are some). Those people are hurting GOS.
Good security engineering isn't sufficient: GOS could accomplish the security engineering by making an OS for themselves on a private git server and never distributing it.
But if we want to help the public with security and privacy, then the outcome depends on GOS's brand image - perception drives public adoption, who will never read long posts on arcane engineering matters. It depends on GOS's relationship with Google and its individual engineers, a relationship in which Google and the engineers hold all the cards - they could choose to cut off GOS completely tomorrow at no cost to themselves.
It's an engineer's fantasy that you can ignore those things and just do engineering; life just isn't like that. Doing that would imperil or greatly limit GOS, which hurts everyone.
A shady Android distribution with shady finances that is everyday on HN.
Weird how reports about GrapheneOS always come with some overtly generalized FUD that add nothing of value to the topic at hand.
To clarify, regarding the confusion: it was disabled, leading them to think it didn't outright exist.
Why disabled? Well, its performance is apparently quite poor. But why?
...Apparently to save money.
The thread made it sound like it was more- that the actual release of pixel 11 did not have the firmware to support MTE, not that it was just disabled.
Is the thread wrong or did I misread it?
Nah sorry my bad. I mixed up "it's currently disabled" and "the hardware is there" into "it was earlier disabled".
> Apparently to save money.
No evidence of this is presented.
Its not like Google is transparent about this, so speculation is all we can do
If you don't know, making things up isn't a solution. I don't know you; would it be useful to fabricate a life story and what you had for breakfast today? What would be the point?
All you can really do is deal with uncertainty, which is the norm for most things in life.
The performance isn't really that bad but it's definitely a hit. The bigger issue is that it causes apps and services and occasionally also the kernel to throw MTE SIGSEGV which becomes a substantial problem if the developers for said apps or services don't care enough to fix them.
So the TLDR is that the main issue is that it causes a whole bunch of stuff to segfault when bad memory accesses occur but where they wouldn't segfault without MTE. It makes the phone feel unstable to average users since stuff just crashes with no real end-user-facing explanations.
[dead]
For those don't know you will either code in Rust or make sure MTE enabled.