Ex-Apple engineer here. I really like your implementation. A few years ago I built a similar tool to help me automate the testing of some of my native macOS apps. Being able to run multiple UI automation tests simultaneously was the big win in my case.
My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.
The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on
Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfooding helps.
A subset of power users want to their usage to be profiled (me, if I trust the company. Brave, Mozilla, Mullvad, 1Password, Bitwarden, Valve, companies like that). But most power users will not want that because of privacy worries.
From that you get two situations.
Opt-in:
- Regular users: click all 'ok' through setup at lightning speed, no telemetry enabled.
- Most power users: consciously don't check the box to opt-in because of privacy worries.
- Big picture power users: consciously check the opt-in box given they trust you (because they want their usage patterns to be profiled and optimized for).
Opt-out:
- Regular users: click all 'ok' through setup at lightning speed, telemetry enabled.
- Most power users: consciously check the box to opt-out because of privacy worries.
- Big picture power users: consciously don't check the opt-out box given they trust you (because they want their usage patterns to be profiled and optimized for).
If they really were they would turn it off. And stop using Gmail and Android.
The overwhelming majority of people don't care about digital privacy because the cost is opaque to them.
Also, telemetry when done right isn't "spying". Again, it is anonymized and used to see, for example, where the hot paths and paper cuts in applications are.
i think that in a free society, you should be able to sell the product you want to sell. but, you should give information of what you are selling to the customer.
if it has telemetry, then it is a tool the customer buys, that also has the function of listening and reporting to others, how it is being used.
you want to sell it - no problem. but tell the customer, "look, this is bugged, and it's going to tell me what you are doing. but it's a great product." anything with opt-out telemetry needs a big version of that warning on the top of the page.
personally i am not a buyer. but that's my preference.
Again: telemetry isn't "spying" and it isn't "bugging" the application. It collects usage patterns: how often is which button being pressed by which type of user.
It is not collecting data on you personally nor is it collecting the actual data you enter.
Of course the question remains if a company has "maskAllImages: true, maskAllTextInputs: true" (and I also wonder if they hide UI elements like message titles / contents), but that's why I mentioned I only turn on telemetry for companies that seem to explicitly, consistently and robustly care about privacy and security.
Telemetry (if it’s truly telemetry) is nowhere close to “tracking”. People conflate the two all the time. One can provide useful, anonymous metrics (e.g. “user enabled feature X”) without doing anything but incrementing the counter for “feature X”.
The “Firefox Problem” is that all the power users disable telemetry, so all the “cool” features that power users like (but never get used by “regular people”) get ignored or removed instead of improved because, according to the metrics, “nobody uses them”.
The user doesn't conflate the two, the developers do, and that's why we turn off telemetry, because its damn close to tracking.
Knowing what (vulnerable) version of software a user is using transmitted in the clear was absolutely a part of the NSA monitoring error information from windows crash logs https://www.schneier.com/blog/archives/2017/08/nsa_collects_... - so forgive me if I do not trust the developer to know what makes me unsafe or not.
If you enable telemetry by default I will do my best to never use your product.
If Charmin put sensors in toilet paper rolls to optimize the wiping experience, it would be dystopian. Why do we give software a pass? Privacy is a right not a telemetry problem and opt-out by default is non-consensual surveillance.
In fairness Charmin is probably backed by millions of dollars of market research on simple user questions like softness, tendency to crumble, size, etc., while free software faces more criticism for issues that are exponentially more difficult to express.
Ok, replace Charmin with a toilet paper startup disrupting the industry. They wouldn’t be given a pass either. Still disgusting.
It should probably be noted that if there’s no agreement, collecting telemetry without opt-in probably violates several state and federal laws. Not that these are enforced, but it would be nice if they were.
Fair criticism. We took a similar approach to established dev tools like Homebrew, with an anonymous, opt-out telemetry to understand install issues, crashes, and high-level usage. For cua-driver specifically, telemetry is limited to command/tool-level events and basic environment metadata. We don’t send screenshots, recordings, app contents, prompts, typed text, file paths, or tool arguments. That said, we should make the opt-out path clearer
We don't have a specific testing framework yet. cua-driver is closer to an automation interface than a test runner. that said, you could definitely build one on top of it. For reference these are some of our integration tests:
https://github.com/trycua/cua/tree/main/libs/cua-driver/Test...
One useful trick is to cua-driver 'launch_app' instead of the default 'open' or other osascript, since it can start the app without raising/focusing it, and the tests don't disturb your active desktop while they run
This is one of the coolest hacks I've seen recently. Having done some much less involved MacOS hacking, I can't help but wonder if we may finally see momentum behind some flavor of agent-friendly Linux/Android if Apple doesn't give us more ways to let agents interact with our machines.
really appreciate it. macOS has powerful primitives already, but they weren’t designed as one coherent agent API so you end up stitching together and hitting roadblocks. If Apple doesn't make this more first-class, Linux/Android-style environments may move faster because they’re easier to instrument. I think the OpenAI/Jony Ive AI hardware rumors are yet another signal that people may start building agent-native CUA devices instead of retrofitting agents onto existing desktops
Thanks for starting that thread, I definitely drew some inspiration from it. But ultimately the secret sauce for the background click came from discovering yabai's window_manager_focus_window_without_raise https://github.com/asmvik/yabai/blob/f17ef88116b0d988b834bb2...
Incredible! I’m interested in doing something similar on windows, have you looked into that at all? Apparently codex computer use plans to support this on windows in the future. Were you able to see how codex was doing it, or the inspiration was just “they’ve shown it’s possible”?
I did something similar on Windows by creating a "virtual desktop," where I can give the app focus without stealing it from another one. The idea was to basically reimplement RemoteApp without needing a dedicated Windows server.
However, in that case, the app is not visible to the user unless you use "connect" to the virtual desktop; to do it, I implemented (WIP) a simple VNC server in C#.
Thanks! We haven't gone deep on Windows yet because we're still focused on polishing the macOS release. We want to go deeper on the Mac experience before going broader across platforms, and there are still a lot of features we want to ship and use cases we want to share.
And to think that ARexx was released nearly 40 years ago! Amazing that modern computing still hasn't caught on to some of the capabilities AmigaOS introduced.
I tried out their Loom vm software a couple of months back. Worked well, fwiw. I'm not using it anymore because I decided to just give agents direct (supervised) access to my devices.
Thanks for trying out Lume! We definitely haven't given up on the idea of sandboxing GUI agents in local macOS VMs. Cua Driver is aimed at a different use case though, letting coding agents and general agents use the Mac you're already on, asynchronously and in the background. That also makes the economics better since multiple agents can share the same machine instead of each needing its own VM
Same here. I give agents supervised direct access on my Mac for a side project. Session stealing is annoying. VM feels overkill for solo dev, but hate that the cursor jumps around while I try to do other things. Background driver sounds like the missing middle ground.
The audit trail question is interesting and I haven't seen it come up much. When an agent clicks through an ERP or edits a file, you've got logs, but how do you explain the "why" behind each decision to, say, a compliance team?
Curious if that's something you're thinking about or if it's too early.
Ex-Apple engineer here. I really like your implementation. A few years ago I built a similar tool to help me automate the testing of some of my native macOS apps. Being able to run multiple UI automation tests simultaneously was the big win in my case.
My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.
The problem with opt-in telemetry is that 95% of users don't change defaults, and the 5% who do are your power users. They're not representative of the average user. And only a subset of them will turn it on
Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfooding helps.
I'm confused.
You claim power users opt in to telemetry, and then immediately say power users opt out.
A subset of power users want to their usage to be profiled (me, if I trust the company. Brave, Mozilla, Mullvad, 1Password, Bitwarden, Valve, companies like that). But most power users will not want that because of privacy worries.
From that you get two situations.
Opt-in:
- Regular users: click all 'ok' through setup at lightning speed, no telemetry enabled.
- Most power users: consciously don't check the box to opt-in because of privacy worries.
- Big picture power users: consciously check the opt-in box given they trust you (because they want their usage patterns to be profiled and optimized for).
Opt-out:
- Regular users: click all 'ok' through setup at lightning speed, telemetry enabled.
- Most power users: consciously check the box to opt-out because of privacy worries.
- Big picture power users: consciously don't check the opt-out box given they trust you (because they want their usage patterns to be profiled and optimized for).
power users opt in to opt in telemetry, and power users opt out of opt out telemetry. Power users click all the buttons.
The problem with opt-in telemetry is that 95% of users are sick and tired of being spied on with every little thing they do.
If they really were they would turn it off. And stop using Gmail and Android.
The overwhelming majority of people don't care about digital privacy because the cost is opaque to them.
Also, telemetry when done right isn't "spying". Again, it is anonymized and used to see, for example, where the hot paths and paper cuts in applications are.
i think that in a free society, you should be able to sell the product you want to sell. but, you should give information of what you are selling to the customer.
if it has telemetry, then it is a tool the customer buys, that also has the function of listening and reporting to others, how it is being used.
you want to sell it - no problem. but tell the customer, "look, this is bugged, and it's going to tell me what you are doing. but it's a great product." anything with opt-out telemetry needs a big version of that warning on the top of the page.
personally i am not a buyer. but that's my preference.
Again: telemetry isn't "spying" and it isn't "bugging" the application. It collects usage patterns: how often is which button being pressed by which type of user.
It is not collecting data on you personally nor is it collecting the actual data you enter.
go on posthog: https://posthog.com/session-replay
"Watch people use your product" - that's their own words!
it literally sends you screen capture. and yes, you can id the user and capture inputs.
YC 2020. we love to see it.
https://posthog.com/questions/how-to-hide-text-inputs
Of course the question remains if a company has "maskAllImages: true, maskAllTextInputs: true" (and I also wonder if they hide UI elements like message titles / contents), but that's why I mentioned I only turn on telemetry for companies that seem to explicitly, consistently and robustly care about privacy and security.
And how would I know if you did it right or not?
This is called trust.
Telemetry (if it’s truly telemetry) is nowhere close to “tracking”. People conflate the two all the time. One can provide useful, anonymous metrics (e.g. “user enabled feature X”) without doing anything but incrementing the counter for “feature X”.
The “Firefox Problem” is that all the power users disable telemetry, so all the “cool” features that power users like (but never get used by “regular people”) get ignored or removed instead of improved because, according to the metrics, “nobody uses them”.
The user doesn't conflate the two, the developers do, and that's why we turn off telemetry, because its damn close to tracking.
Knowing what (vulnerable) version of software a user is using transmitted in the clear was absolutely a part of the NSA monitoring error information from windows crash logs https://www.schneier.com/blog/archives/2017/08/nsa_collects_... - so forgive me if I do not trust the developer to know what makes me unsafe or not.
If you enable telemetry by default I will do my best to never use your product.
As you can see with TikTok / Instagram usage…regular people who are not on HN could not care less about that.
If Charmin put sensors in toilet paper rolls to optimize the wiping experience, it would be dystopian. Why do we give software a pass? Privacy is a right not a telemetry problem and opt-out by default is non-consensual surveillance.
In fairness Charmin is probably backed by millions of dollars of market research on simple user questions like softness, tendency to crumble, size, etc., while free software faces more criticism for issues that are exponentially more difficult to express.
Ok, replace Charmin with a toilet paper startup disrupting the industry. They wouldn’t be given a pass either. Still disgusting.
It should probably be noted that if there’s no agreement, collecting telemetry without opt-in probably violates several state and federal laws. Not that these are enforced, but it would be nice if they were.
>Ok, replace Charmin with a toilet paper startup disrupting the industry.
"Move Fast and Break Things" is an ominous motto for a disruptive toilet paper startup.
So is "Fake It Till You Make It".
"Crowd Sourced Enshittification" is more like it.
i think it's not so much non-consensual, it's misrepresentation.
it's bugged. the same as a mole in your company. or a sculpture with a listening device in it.
tell the user that your thing is bugged!
Fair criticism. We took a similar approach to established dev tools like Homebrew, with an anonymous, opt-out telemetry to understand install issues, crashes, and high-level usage. For cua-driver specifically, telemetry is limited to command/tool-level events and basic environment metadata. We don’t send screenshots, recordings, app contents, prompts, typed text, file paths, or tool arguments. That said, we should make the opt-out path clearer
Would you be open to sharing what you built for running the automation tests? I could really use this right now.
We don't have a specific testing framework yet. cua-driver is closer to an automation interface than a test runner. that said, you could definitely build one on top of it. For reference these are some of our integration tests: https://github.com/trycua/cua/tree/main/libs/cua-driver/Test...
One useful trick is to cua-driver 'launch_app' instead of the default 'open' or other osascript, since it can start the app without raising/focusing it, and the tests don't disturb your active desktop while they run
Nice. thanks for sharing!
This is one of the coolest hacks I've seen recently. Having done some much less involved MacOS hacking, I can't help but wonder if we may finally see momentum behind some flavor of agent-friendly Linux/Android if Apple doesn't give us more ways to let agents interact with our machines.
really appreciate it. macOS has powerful primitives already, but they weren’t designed as one coherent agent API so you end up stitching together and hitting roadblocks. If Apple doesn't make this more first-class, Linux/Android-style environments may move faster because they’re easier to instrument. I think the OpenAI/Jony Ive AI hardware rumors are yet another signal that people may start building agent-native CUA devices instead of retrofitting agents onto existing desktops
Nice! Thanks for the technical writeup, ~2 weeks from me wondering how it's implemented [1] to being able to play with a replicated version!
[1] https://news.ycombinator.com/item?id=47799128
Thanks for starting that thread, I definitely drew some inspiration from it. But ultimately the secret sauce for the background click came from discovering yabai's window_manager_focus_window_without_raise https://github.com/asmvik/yabai/blob/f17ef88116b0d988b834bb2...
What is specific about this for using with agents? As opposed to offering it as a general automation library for any use?
Nothing prevents using it as a general automation library.
If you want to use it directly as an automation framework, you can take a Swift dependency on 'CuaDriverCore': https://cua.ai/docs/cua-driver/guide/getting-started/swift-i...
Being new to the idea of using agents to run programs on one’s computer, could someone provide several use cases?
A few examples i'm excited about:
- Closing the coding feedback loop by having agents verify their own changes in a real app
- Automating repetitive workflows across apps that don't have good APIs
- Agents recording product demos of them using software. One compelling use case here: https://x.com/trycua/status/2047383207612645426
- Creating CLI and APIs for apps by reverse implementing their GUI, e.g. see: https://github.com/HKUDS/CLI-Anything
Incredible! I’m interested in doing something similar on windows, have you looked into that at all? Apparently codex computer use plans to support this on windows in the future. Were you able to see how codex was doing it, or the inspiration was just “they’ve shown it’s possible”?
I did something similar on Windows by creating a "virtual desktop," where I can give the app focus without stealing it from another one. The idea was to basically reimplement RemoteApp without needing a dedicated Windows server. However, in that case, the app is not visible to the user unless you use "connect" to the virtual desktop; to do it, I implemented (WIP) a simple VNC server in C#.
Thanks! We haven't gone deep on Windows yet because we're still focused on polishing the macOS release. We want to go deeper on the Mac experience before going broader across platforms, and there are still a lot of features we want to ship and use cases we want to share.
And to think that ARexx was released nearly 40 years ago! Amazing that modern computing still hasn't caught on to some of the capabilities AmigaOS introduced.
I tried out their Loom vm software a couple of months back. Worked well, fwiw. I'm not using it anymore because I decided to just give agents direct (supervised) access to my devices.
Thanks for trying out Lume! We definitely haven't given up on the idea of sandboxing GUI agents in local macOS VMs. Cua Driver is aimed at a different use case though, letting coding agents and general agents use the Mac you're already on, asynchronously and in the background. That also makes the economics better since multiple agents can share the same machine instead of each needing its own VM
Same here. I give agents supervised direct access on my Mac for a side project. Session stealing is annoying. VM feels overkill for solo dev, but hate that the cursor jumps around while I try to do other things. Background driver sounds like the missing middle ground.
http://tart.run makes the VM part easy. So what if it's overkill?
Its looking great.
The audit trail question is interesting and I haven't seen it come up much. When an agent clicks through an ERP or edits a file, you've got logs, but how do you explain the "why" behind each decision to, say, a compliance team?
Curious if that's something you're thinking about or if it's too early.