This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
Shared code repositories for OSS are much older than npm. See at least maven central for Java.
What makes npm a more relevant target is a much smaller standard library and thinner dependencies, leading to more chances that a random project will have code of suspicious provenance. The vast majority of what a backend Java project depends on was probably written either by the same 3 companies, or large, established OSS repos. So this is relatively solvable if people in steering committees decided they cared. It's improving, but I'd argue not fast enough.
I noticed this a lot with java where most of what I needed that wasn't specific to our company's business logic was already available in the standard library, and a lot of the OSS libraries I needed were relatively reputable and well-known. I did a few personal backend projects with js and the amount of functionality I had to crate-dig from npm drove me insane, on top of the more complicated chain of trust for how many layers packages from npm can have.
Generally we don't hear much about attacks in Java like that. But they probably exist but python and javascript are targeted mainly because everyone uses them. They cover a lot of ground.
A big reason Python and Java are juicy targets are because for a very long time pip and npm executed code on the user's machine on package install, not just on import. (This is changing in newer versions of pip and npm IIRC)
I think it's mostly about attach surface. There is nothing fundamentally different in the threat model or security boundary between any language in terms of how they manage their dependencies. Even platforms like Maven or NuGet that don't have an install script, you're still likely to trigger a build or a quick run for the application right after installing that would trigger code execution anyway.
The chances of a person getting hit during the `npm install` phase is bigger but not that different than the `dotnet add package ... && dotnet run`.
I do agree though that the massive surface of npmjs, pip, and even cargo means that package signing is also not realistic. If you have the average 300 npm packages, no one is gonna pay attention to the authenticity of 200 different signing keys. It would be a lot easier with something like java where you probably just need to trust a handful like openjdk, jackson, junit, or log4j keys and be 90% there. (And hope the supply chain attack hasn't compromised the signing keys too of course)
The reason execution on install is a bigger deal than execution on import/run is because a lot of developers today run their code inside of sandboxed environments (containers, VMs, etc.) which prevents the malware from easily accessing stuff like session tokens.
There’s also been a regression in how infrastructure is operated.
15 years ago you didn’t let your production infrastructure talk out to the Internet, and if you did, it was via an explicit proxy with allowlisted domains. This significantly limited the damage that a malicious dependency could cause.
With the move to cloud and DevOps, that network security model disappeared.
That XKCD comic is demonstrative of the opposite of what OP is talking about. Python and JavaScript are on opposite sides of the spectrum about what is included in the language and what is not. Python is “batteries included” which is why you can just “import antigravity” rather than “pip install antigravity”. JavaScript and NPM sit on the other end, where you have to infamously install the leftpad library to pad strings with spaces on their left, which left thousands of packages broken when it was taken off npm.
Don't know why you're getting scrutinized, npm didn't introduce malicious packages obviously but it did completely remove the feeling of suspicion and responsibility people had towards third party libraries.
The pre 2020 bootcampers would basically npm install anything because that's all they knew. You even had bullshit like "is-array" with millions of weekly installs because how dare you reinvent the wheel.
Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").
In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for the next couple decades.
VMs are demonstrably not infallible, but that's not quite the issue here.
If you only ever run your banking app inside a VM with an encrypted disk volume, and don't browse sketchy sites or do anything else in there, then you're very well protected against even LLM-assisted attacks.
One caveat in that case is I wouldn't keep the password to the VM disk in a password manager, since that then becomes an attack vector.
A keylogger would still be a viable attack vector, though. The security of operating systems is going to need to be improved to guard against some of these things, e.g. a protected input mechanism for passwords that can't be intercepted.
Have you heard about "default allow" and "default deny" in the domain of cyber security? Traditionally PCs, especially for home use, were built around the former idea, where the user supposedly trusts most of the stuff installed.
The problem is that malware may sneak in, and then the user needs a horrible piece of software called anti-virus that needs to apply complicated heuristics to try and separate one from the other. It kind of worked ok for many years, but as the world evolved and the dangers kept increasing, little by little we have been going towards the other idea – default deny – where apps are considered harmful until proved otherwise, and the honest app developers have to jump through multiple hoops to get their apps available to users. That's the app store model, which kind of works, but is cumbersome and expensive for the app developer.
Anyway, going back to the VM discussion, I think we may end up applying the default deny idea further into the OS Kernel to reduce those risks further. After all, a small kernel has a much smaller risk surface and perhaps it can be formally verified to be secure under some reasonable assumptions.
I think this is one of those situations where you don't need to outrun the tiger, you just need to be better at security than 99% of people, which is definitely the case if you're partitioning your life into VMs.
I wonder if/when mainstream OSes will start doing this.
Isn't the container ecosystem basically the same problem with more packages? You pull an OS package, a language runtime package, etc etc. It's something I've been thinking about also
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
> has a tendency of running "find"/"ls" on your home folder
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
It accidentally made a compiler because of an ambiguous prompt. And yes it will find "helpful" code on other parts of the system. Which means claude most likely has a `find ~` of millions of developers home folders
Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
The contra is that “stable” releases are now a liability, because it takes longer for upstream fixes to be back-ported. Recently, for bugs that are quickly exploitable in the wild.
A recent example is the TCP options use-after-free, where it took Canonical a couple days to patch LTS beyond when the fix was first available in Fedora 45.
And these are just for public CVEs. I’m also worried that there are a lot of quiet fixes being made with new agentic review tooling, which may fix an undiscovered bug but will never be back-ported.
Yea I should have put it in my first post, but that is the one I have come across a lot reading some of the package maintainers comments from fedora on reddit. It's a really good point. Thanks!
I think this will be the perfect motivation for the rise of pure functional programming, like in Roc [1]. With a clear separation of side effects from the rest of the codebase, you can sleep much better knowing that a library that is supposed to do some pure computations will not exfiltrate your data to a weird corner of the Internet.
I do it like this, but on one computer, using QEMU. I have 3 different VMs for
- development & AI tools
- infra rollouts
- banking / other sensitive services
I have Steam Deck for gaming, and do most of casual web browsing still on host, because it's secure enough to use Firefox with ublock + privacy badger and enforcing SELinux. However, I dont use socials other than HN and Reddit under pseudonymous accounts. If I had real zuckerbook or instacrap accounts, maybe I'd have another VM for socials too.
Why do you people love over-engineering and bad architectures so much? Do you have any sense of engineering beauty? Modern CPUs already run every application in an isolated sandbox under "protected mode". There is no need to emulate a machine with hardware devices, run a separate kernel, drivers and services. The only problem is that the OS kernel and privileged system services usually grant too much ways to bypass the sandbox. So the proper architecture would be to re-design and fix the existing OS instead of using ugly hacks like VM.
I also use VM to run code editors and AI. One must be crazy to run third-party software (including package managers like npm) unrestricted. The overhead is significant, and VS Code UI is lagging heavily without a GPU.
This was reasonable advice 12 months ago, but at the rate kernel exploitation is going it's less of a security boundary than ever. Hypervisors aren't much better, but it helps.
CPU and memory are not the only concerns,
file system and network access are dangerous as well,
and they tend to be less instrumented in terms of hardware isolation.
AI dev environment gets its own SSD. The second SSD stays in, but since it's encrypted there's no way to exfiltrate the data unless you're of interest to nation state attackers. Theoretically the AI could destroy the other SSD but that's what backups are for, and the probability is low because well why would they bother. Boot level malware is possible, but franky that's well into paranoid territory so I'm not too bothered.
The other SSD is a normal setup where you do normal computer stuff. I don't access the dev SSD through it, I don't even mount it, although the danger seems minimal if you do. Banking, socials, gaming, whatever. No AI of any kind on it, aside from web portal for claude/chatgpt/whatever and possibly local AI model that gets no internet access.
No idea why I'd want to have its own setup split between gaming and socials, as an example. Better yet just toss all your twitter and bluesky accounts into a volcano, and consider whether to do it with HN as well.
this is what I do, and it's fairly cheap nowadays if you just buy 2nd hand, I actually go so far as having virtualbox VMs for some of those within the same computer (say I have my "important accounts" computer that has some virtualbox VMs with each a browser with only some of those).
Gaming is a windows PC that has only games (so who cares) and a ps5 (so I can rent games at the library for free), development is a linux PC (where I containerize most things, as well as firejail etc.), web browsing is an x86 imac, I don't do socials, I also have a completely separate linux PC for art/drawing.
The vast majority of these has been bought for a couple hundred dollars 2nd hand from ebay or local university surplus (say they have all the time m1/16gb/500gb for $300-$400 imacs, or dell 16gb towers, or tons of dell laptops), the only expensive one was the linux development one as I bought a 5950x/128gb when it came out, but it has served me well, as long as you stay away from windows/mac you can go very far with really cheap hardware (DDR3 is plenty for banking / browsing / socials / factorio).
I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
I know it's a bit insane, but I do daily LTO-6 backups of my workstation. I was able to get a working setup for around $250 since I was willing to work with an "autoloader" drive, but honestly it works great and is really nice peace of mind to have my critical stuff protected. I keep banking to my iPhone these days -- though I can't say it's for sure safe. I probably should add that to the backup.
I already have users for Work stuff and Home stuff and Rando stuff. Fast user switching is great, however sometimes I need to do a quick check on some Home stuff, while seeing all my work windows.
My guideline is nothing can install packages outside of a docker container. For anything on the host, only packages that ship with the default distribution are allowed. This also helps with proper encapsulation of services in a three-tier architecture. Claude always wants to break abstraction layers which leads to 10 different implementations of the same logic in the client besides the increased supply chain risk.
For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.
Do package managers for Linux distros have this, like dnf/yum/apt/pacman?
I use min-release-age =14 for npm now as a default but should start using it everywhere it's possible
- 1 computer and Google identity for administrative stuff: Primary accounts for AWS, Google. I also do accounting and have the list of customers. I still don't have any other solution than opening PDFs for the invoices, but it's better than nothing.
- 1 computer for programming, with a non-asministrator account for everything. Claude runs in Docker, with the dev directory mounted, .git:ro (read-only) and .env mounted to /dev/null.
But Claude tries to escape it A LOT. Claude is designed like a virus, we tell him not to do things, and it 1. tried to download the git binary from the internet, 2. read the remotes in the git directory and tried to access them, only blocked because it didn't have the credentials.
DEFINITELY forcefully block out Claude from seeing your .env files, it WILL try to brute-force an access to your intranet!!
All the extra code isn't really the problem, it's the insanely easy to attack "programming language package ecosystems", like NPM, PyPI, etc, compared to something like the Linux package ecosystem. But this is a much less common problem than malware and phishing. Supply chain is more of a concern than it used to be, but thousands of people get infected with malware or phished every day, versus relatively few supply chain attacks that get caught fairly quickly.
For your own user security, my suggestion is to stop using SaaS wherever you can. None of them are secure, and exfiltration is child's play. All SaaS I've worked for had terrible security.
Malware typically depends on two things:
1) Unpatched software. OS, apps, routers, etc. Always keep everything up to date on patches. 0-day exploits are rarely used because it burns a valuable resource. If you can use an OS that has very little malware (like Linux, MacOS) it reduces your risk. Malware also tends to lurk in certain spaces on the internet (porn sites, pirated media, shareware downloads, malicious ads), so staying clear of them also reduces your risk.
2) Persistent credentials. Most malware (at least when I was looking at it years ago) isn't very advanced/persistent. An advanced persistent threat may sit on your machine indefinitely until you login using credentials, but they may not use those credentials immediately. Sign in and out quickly (don't "stay signed in") and always use MFA, ideally a second device. Secured Android and iPhone devices are ideal for checking bank statements etc... but you must be vigilant not to install untrusted apps from App Stores (anything not from a major company), as they are more likely to infect you than your browser on normal/safe websites. If you can, try to use virtual credit card numbers, and use a credit rather than debit card for purchases.
Phishing largely depends on clicking a link in an email, but also SMS "alerts" and browser pages that look like OS pop-ups. To defend against these, use a password manager to generate random unique passwords everywhere, use MFA everywhere. For login, set up and use 1-click-login and passkeys, especially for the most important logins (bank, email). Don't share or reuse passwords or API keys for more than one thing; not only does it reduce the impact, but it helps you identify where the compromise came from.
I personally play "fast and loose" on my Linux desktop and don't run things in isolation. But if you want rock-solid security, the only real option is running applications in dedicated VMs, and only mounting files needed for that application. Don't depend on containers or sandboxes for security. If you want a "lightweight" alternative to VMs/containers, you can have multiple user accounts on your computer, lock each user account down (no reading/writing from other user accounts), and either login to different ones for different tasks, or run apps as that user and have them display in your current user's desktop.
You got it wrong. In AI era, the trade-off is between CVE and vulnerabilities that people haven't noticed yet. In the AI era, the forest is ALWAYS on fire. Even when you don't see the smoke.
> Makes me want to have like 5 separate computers:
I mean if you have only one there’s https://www.qubes-os.org/ but not sure about the gaming part, nor is it necessarily as good as separate isolated hardware.
I had a theory that in the future all online interactions would need to be brokered by an agent that can stay on top of all those changes, and that the internet would be a hostile if not unuseable —and possibly also illegal— place to visit without having a personalized AI broker that is insured, and that you pay for. It would begin with you owning the agent but it would end with the agent owning you.
To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.
Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.
Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised
Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.
> How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
I moved part of my wealth to a private bank, on an account requiring several signatures (from different persons) plus a vetting from the banker for every single outgoing transfer. Not only are several manual signatures required, in addition to that the banker must call me before authorizing any transfer, even if it's only, say, 1 000 EUR (now it's not a retail bank and that's kinda the whole point: transfers are rare and when one is done, no matter the amount, a shitload of verification do happen).
In addition to that, I told my banker that should anything out of the ordinary happen (say an order to suddenly wire 100 K to Dubai or Erithrea or something), then he should go out of his way to make me come in person to the bank to confirm the transfer.
You must train your banker: you must demo him voice impersonation on the phone. You have to make him become totally paranoid too.
I'm not going to say he's my buddy, but he knows I know my shit and we regularly talk for 20 minutes on the phone.
Any transfer to a new bank account: long discussion explaining what that account is and why I'm moving money there.
I've also got digital files (say administrative stuff, stuff from notaries, etc.), encrypted, that I myself cannot easily access. For to access them I must physically go to my safe at a bank first. And trying to impersonate me physically at the bank to steal my stuff means: "on your way out, you risk being locked in the secure airlock and then spend years behind bars".
> Makes me want to have like 5 separate computers:
I have a setup not totally unlike that. For example one phone with one additional app: the country's official "electronic ID" thinggy. And I don't browse from that phone.
I've got several computers and there's definitely my "trash computer", complete with bogus email accounts and crappy apps, etc.
> How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
The first thing is recognizing there's an issue if someone hacking your unique email then can steal everything you have. Many people stop there: they think, as in the XKCD, "it's game over anyway if someone gets my email, so I may as well not worry about it".
That is the mentality of losers.
You're already well ahead of the crowd by recognizing there's a problem and by thinking about solutions.
Many things can be done to lower the risk of a full compromise. I've got two physical LANs and the absolute most sensitive stuff (e.g. my banking apps) are done from a computer that does nothing but email + bank.
So it's not just a separate computer: separate computer on a separate physical LAN (not a VLAN). And there a tiny Pi doing the bridging between the two LANs. And of course it's got a serious firewall (a lost art, but it could make a comeback with the help of LLMs).
Something else that can help is stuff like Google Workspace and keeping the credentials for the admin account very secure and hardly ever logging into it. That way if "jack@acme.com" gets compromised, there's still "admin@acme.com" that can help get back control of jack@acme.com.
> - one for development only
I've got a user account for dev only.
> - one for gaming only
A dedicated computer for gaming yes.
> - one for socials only
Besides HN, I don't need those.
It's a wild world out there.
The hacks won't stop coming and you're not anywhere near paranoid enough.
We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
My router got hacked the other month (left web interface open to internet) after being exposed to the internet and the persistence left behind was overly commented LLM artifacts. I caught it in like 5 minutes so they weren’t able to do much but it was eye opening how much more basic practices like firewalls are not something that can be ignored even if you update your firmware frequently.
What if I told you nothing new is happening? Except casuals are noticing what the industry knew and saw for decades?
Script kiddies aren't new. Metasploit has been around literally decades. vx-underground existed for years, not to mention older open source projects that collect all known malware dev tools. APTs have been pwning stuff, script kiddies have been doing their thing. Nothing significant has changed for anyone in the know.
Just because LLMs automated some offensive stuff doesn't mean there's gonna be new magic collapse of society.
Given that our knowing-and-seeing industry is also the thing which enabled and then desperately shepherded humanity out-of every IT deficiency that has ever existed, our inside perspective is still highly fallible too.
It's an odd choice to run this on the victim's machine (exposing the open router key) rather than on the attacker's infrastructure. I guess this lets them keep it dead simple for buyers, you just need to choose a Telegram channel for the logs to be dumped into.
How very considerate! Steal from the rich, not the poor.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.
You’ll have created an autonomous Robin Hood clone army. Fun times.
Unfortunately LLMs still optimize scams/exploits to the point that they'll probably still get to those of us heading into the permanent underclass, just a little slower.
kind of boggles me that the big ai labs arent running zero-shot classification on all their outputs with "does this look like any of these bad things we know people get up to"
Oh, other day I was asking Claude to get a onetime code from my PocketID VM, I was being lazy and I was asking it to open the website, generate the authentication code from there and then type it into a built in browser. And it just refused to do it. I couldn't convince it to do it, even though it was all my own infrastructure And essentially, we were doing a security theater because the website that it was trying to authenticate to was the one it had just published itself behind my internal trafefic instance, lol.
I ended up having Codex do it. Codex had no compunctions about it.
Once ranked, victims can be assigned a proportional token budget so that a high value victim might be targeted with zero days or deepfakes that are more expensive to execute but have passed the desired cost/reward threshold.
If you backtrack the posts here, the warnings were there from various bodies already two or three years ago, but it seems the lag is now being closed. (In fact, I wonder why it took so long.)
Bot's enable discovery of information or exploits at a scale that we cannot imagine.
Recent example of a breach, they are now using AI to decomp APKs and iOS apps, mainly old builds of apps that nobody would bother spending the time on previously to look for secret keys or accidental information disclosure.
This would have been immensely time consuming with each app usually doing things a bit differently (e.g. encrypted + compiled settings, keys in database, settings in a settings.json).
Now you can automate the entire pipeline, this is actively being used to look for legacy API endpoints that may not be hardened or monitored, or if you don't rotate your keys and at one point disclosed them.
Dang, that’s tough. Recently having to deal with php2shell was a good wake-up call for anybody doing hosting of any kind. Next few years gonna be rough (but food on the table for cybersecurity experts I guess)
I'm no genius, but I've managed to set up my agents to prefer writing a small python script or whatever to do repetitive/iterative tasks that my language model 'knows' it's not good or efficient at.
You're technically right, which is why I say "good enough".
It's a good strat imo - start simple and get it working with LLMs asap, then you can add heuristics etc based on the data you have and improve with that benchmark
This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
You can thank the npm and the like.
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
Shared code repositories for OSS are much older than npm. See at least maven central for Java.
What makes npm a more relevant target is a much smaller standard library and thinner dependencies, leading to more chances that a random project will have code of suspicious provenance. The vast majority of what a backend Java project depends on was probably written either by the same 3 companies, or large, established OSS repos. So this is relatively solvable if people in steering committees decided they cared. It's improving, but I'd argue not fast enough.
I noticed this a lot with java where most of what I needed that wasn't specific to our company's business logic was already available in the standard library, and a lot of the OSS libraries I needed were relatively reputable and well-known. I did a few personal backend projects with js and the amount of functionality I had to crate-dig from npm drove me insane, on top of the more complicated chain of trust for how many layers packages from npm can have.
Generally we don't hear much about attacks in Java like that. But they probably exist but python and javascript are targeted mainly because everyone uses them. They cover a lot of ground.
A big reason Python and Java are juicy targets are because for a very long time pip and npm executed code on the user's machine on package install, not just on import. (This is changing in newer versions of pip and npm IIRC)
I think it's mostly about attach surface. There is nothing fundamentally different in the threat model or security boundary between any language in terms of how they manage their dependencies. Even platforms like Maven or NuGet that don't have an install script, you're still likely to trigger a build or a quick run for the application right after installing that would trigger code execution anyway.
The chances of a person getting hit during the `npm install` phase is bigger but not that different than the `dotnet add package ... && dotnet run`.
I do agree though that the massive surface of npmjs, pip, and even cargo means that package signing is also not realistic. If you have the average 300 npm packages, no one is gonna pay attention to the authenticity of 200 different signing keys. It would be a lot easier with something like java where you probably just need to trust a handful like openjdk, jackson, junit, or log4j keys and be 90% there. (And hope the supply chain attack hasn't compromised the signing keys too of course)
The reason execution on install is a bigger deal than execution on import/run is because a lot of developers today run their code inside of sandboxed environments (containers, VMs, etc.) which prevents the malware from easily accessing stuff like session tokens.
There’s also been a regression in how infrastructure is operated.
15 years ago you didn’t let your production infrastructure talk out to the Internet, and if you did, it was via an explicit proxy with allowlisted domains. This significantly limited the damage that a malicious dependency could cause.
With the move to cloud and DevOps, that network security model disappeared.
Cloud does let you do this with VPCs and subnets. Some people don't bother and end up susceptible but it remains best practice even for cloud software
There's nothing new under the sun, as they say.
Perl had CPAN already in 1993; Linux and common BSD distros have basically always had package repositories.
"People were responsible for the dependencies they were importing" - I don't know about that. Plenty of us were irresponsible 30 years ago.
> Before, people were responsible for the dependencies they were importing.
This is an attitude not reflective of reality for those that wrote code in the pre npm days
Yeah, and it isn't even a JavaScript thing either. The famous "import antigravity" strip from XKCD was about python.
The fact is we always were dependent on others.
That XKCD comic is demonstrative of the opposite of what OP is talking about. Python and JavaScript are on opposite sides of the spectrum about what is included in the language and what is not. Python is “batteries included” which is why you can just “import antigravity” rather than “pip install antigravity”. JavaScript and NPM sit on the other end, where you have to infamously install the leftpad library to pad strings with spaces on their left, which left thousands of packages broken when it was taken off npm.
I mean, I don't think the stereotypical JS and Python developer are all that different... at least when it comes to dependency management.
Don't know why you're getting scrutinized, npm didn't introduce malicious packages obviously but it did completely remove the feeling of suspicion and responsibility people had towards third party libraries.
The pre 2020 bootcampers would basically npm install anything because that's all they knew. You even had bullshit like "is-array" with millions of weekly installs because how dare you reinvent the wheel.
I just checked isarray got almost 300m installs in the past week.
L take. Can’t blame npm for software shops wanting to run leaner and ship faster.
I’m personally glad I don’t have to download separate runtimes and apps and can just use the standard browser
Except browser based versions take 10x the RAM of native versions.
Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").
In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for the next couple decades.
VMs are demonstrably not infallible, but that's not quite the issue here.
If you only ever run your banking app inside a VM with an encrypted disk volume, and don't browse sketchy sites or do anything else in there, then you're very well protected against even LLM-assisted attacks.
One caveat in that case is I wouldn't keep the password to the VM disk in a password manager, since that then becomes an attack vector.
A keylogger would still be a viable attack vector, though. The security of operating systems is going to need to be improved to guard against some of these things, e.g. a protected input mechanism for passwords that can't be intercepted.
Have you heard about "default allow" and "default deny" in the domain of cyber security? Traditionally PCs, especially for home use, were built around the former idea, where the user supposedly trusts most of the stuff installed.
The problem is that malware may sneak in, and then the user needs a horrible piece of software called anti-virus that needs to apply complicated heuristics to try and separate one from the other. It kind of worked ok for many years, but as the world evolved and the dangers kept increasing, little by little we have been going towards the other idea – default deny – where apps are considered harmful until proved otherwise, and the honest app developers have to jump through multiple hoops to get their apps available to users. That's the app store model, which kind of works, but is cumbersome and expensive for the app developer.
Anyway, going back to the VM discussion, I think we may end up applying the default deny idea further into the OS Kernel to reduce those risks further. After all, a small kernel has a much smaller risk surface and perhaps it can be formally verified to be secure under some reasonable assumptions.
I think this is one of those situations where you don't need to outrun the tiger, you just need to be better at security than 99% of people, which is definitely the case if you're partitioning your life into VMs.
I wonder if/when mainstream OSes will start doing this.
The idea is awesome, the implementation looks great as well, but hardware support is super limited.
Isn't the container ecosystem basically the same problem with more packages? You pull an OS package, a language runtime package, etc etc. It's something I've been thinking about also
> threat that exfiltrates
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
> has a tendency of running "find"/"ls" on your home folder
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
It accidentally made a compiler because of an ambiguous prompt. And yes it will find "helpful" code on other parts of the system. Which means claude most likely has a `find ~` of millions of developers home folders
Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
The contra is that “stable” releases are now a liability, because it takes longer for upstream fixes to be back-ported. Recently, for bugs that are quickly exploitable in the wild.
A recent example is the TCP options use-after-free, where it took Canonical a couple days to patch LTS beyond when the fix was first available in Fedora 45.
And these are just for public CVEs. I’m also worried that there are a lot of quiet fixes being made with new agentic review tooling, which may fix an undiscovered bug but will never be back-ported.
Yea I should have put it in my first post, but that is the one I have come across a lot reading some of the package maintainers comments from fedora on reddit. It's a really good point. Thanks!
I think this will be the perfect motivation for the rise of pure functional programming, like in Roc [1]. With a clear separation of side effects from the rest of the codebase, you can sleep much better knowing that a library that is supposed to do some pure computations will not exfiltrate your data to a weird corner of the Internet.
[1] https://www.roc-lang.org/
I do it like this, but on one computer, using QEMU. I have 3 different VMs for
- development & AI tools
- infra rollouts
- banking / other sensitive services
I have Steam Deck for gaming, and do most of casual web browsing still on host, because it's secure enough to use Firefox with ublock + privacy badger and enforcing SELinux. However, I dont use socials other than HN and Reddit under pseudonymous accounts. If I had real zuckerbook or instacrap accounts, maybe I'd have another VM for socials too.
Why do you people love over-engineering and bad architectures so much? Do you have any sense of engineering beauty? Modern CPUs already run every application in an isolated sandbox under "protected mode". There is no need to emulate a machine with hardware devices, run a separate kernel, drivers and services. The only problem is that the OS kernel and privileged system services usually grant too much ways to bypass the sandbox. So the proper architecture would be to re-design and fix the existing OS instead of using ugly hacks like VM.
I also use VM to run code editors and AI. One must be crazy to run third-party software (including package managers like npm) unrestricted. The overhead is significant, and VS Code UI is lagging heavily without a GPU.
This was reasonable advice 12 months ago, but at the rate kernel exploitation is going it's less of a security boundary than ever. Hypervisors aren't much better, but it helps.
CPU and memory are not the only concerns, file system and network access are dangerous as well, and they tend to be less instrumented in terms of hardware isolation.
AI dev environment gets its own SSD. The second SSD stays in, but since it's encrypted there's no way to exfiltrate the data unless you're of interest to nation state attackers. Theoretically the AI could destroy the other SSD but that's what backups are for, and the probability is low because well why would they bother. Boot level malware is possible, but franky that's well into paranoid territory so I'm not too bothered.
The other SSD is a normal setup where you do normal computer stuff. I don't access the dev SSD through it, I don't even mount it, although the danger seems minimal if you do. Banking, socials, gaming, whatever. No AI of any kind on it, aside from web portal for claude/chatgpt/whatever and possibly local AI model that gets no internet access.
No idea why I'd want to have its own setup split between gaming and socials, as an example. Better yet just toss all your twitter and bluesky accounts into a volcano, and consider whether to do it with HN as well.
Qubes OS might be a reasonable middle ground for this anywhere that you can't transport five computers.
this is what I do, and it's fairly cheap nowadays if you just buy 2nd hand, I actually go so far as having virtualbox VMs for some of those within the same computer (say I have my "important accounts" computer that has some virtualbox VMs with each a browser with only some of those).
Gaming is a windows PC that has only games (so who cares) and a ps5 (so I can rent games at the library for free), development is a linux PC (where I containerize most things, as well as firejail etc.), web browsing is an x86 imac, I don't do socials, I also have a completely separate linux PC for art/drawing.
The vast majority of these has been bought for a couple hundred dollars 2nd hand from ebay or local university surplus (say they have all the time m1/16gb/500gb for $300-$400 imacs, or dell 16gb towers, or tons of dell laptops), the only expensive one was the linux development one as I bought a 5950x/128gb when it came out, but it has served me well, as long as you stay away from windows/mac you can go very far with really cheap hardware (DDR3 is plenty for banking / browsing / socials / factorio).
> one for banking only > one for web browsing only > one for socials only
I feel like your phone is good enough for all of those, and generally secure enough unless you're rooting/jailbreak it
> Makes me want to have like 5 separate computers
This is a good, practical strategy. You definitely want to at least have a separate computer for finances and taxes.
I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
I know it's a bit insane, but I do daily LTO-6 backups of my workstation. I was able to get a working setup for around $250 since I was willing to work with an "autoloader" drive, but honestly it works great and is really nice peace of mind to have my critical stuff protected. I keep banking to my iPhone these days -- though I can't say it's for sure safe. I probably should add that to the backup.
I've been playing with `tart`
I already have users for Work stuff and Home stuff and Rando stuff. Fast user switching is great, however sometimes I need to do a quick check on some Home stuff, while seeing all my work windows.
Usw VMs and isolate them according to separation of concerns. Same with mobile phones. Nix OS is also a good idea.
I am now developing with claude code on mobile and sandboxes and a few remote VPS I rent.
No more laptops, just a dedicated android device.
I developed a devops/admin tool and then its claude code + github
I sandbox my dev environment.. hopefully that protects at least a little bit.
What do you use?
My guideline is nothing can install packages outside of a docker container. For anything on the host, only packages that ship with the default distribution are allowed. This also helps with proper encapsulation of services in a three-tier architecture. Claude always wants to break abstraction layers which leads to 10 different implementations of the same logic in the client besides the increased supply chain risk.
[dead]
For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.
Do package managers for Linux distros have this, like dnf/yum/apt/pacman? I use min-release-age =14 for npm now as a default but should start using it everywhere it's possible
Except when there's an actual urgent patch: See recent PHP or Ruby ecosystem updates. Then you are just a sitting duck for a week
[dead]
I'm implementing a 2-computers rule:
- 1 computer and Google identity for administrative stuff: Primary accounts for AWS, Google. I also do accounting and have the list of customers. I still don't have any other solution than opening PDFs for the invoices, but it's better than nothing.
- 1 computer for programming, with a non-asministrator account for everything. Claude runs in Docker, with the dev directory mounted, .git:ro (read-only) and .env mounted to /dev/null.
But Claude tries to escape it A LOT. Claude is designed like a virus, we tell him not to do things, and it 1. tried to download the git binary from the internet, 2. read the remotes in the git directory and tried to access them, only blocked because it didn't have the credentials.
DEFINITELY forcefully block out Claude from seeing your .env files, it WILL try to brute-force an access to your intranet!!
All the extra code isn't really the problem, it's the insanely easy to attack "programming language package ecosystems", like NPM, PyPI, etc, compared to something like the Linux package ecosystem. But this is a much less common problem than malware and phishing. Supply chain is more of a concern than it used to be, but thousands of people get infected with malware or phished every day, versus relatively few supply chain attacks that get caught fairly quickly.
For your own user security, my suggestion is to stop using SaaS wherever you can. None of them are secure, and exfiltration is child's play. All SaaS I've worked for had terrible security.
Malware typically depends on two things:
1) Unpatched software. OS, apps, routers, etc. Always keep everything up to date on patches. 0-day exploits are rarely used because it burns a valuable resource. If you can use an OS that has very little malware (like Linux, MacOS) it reduces your risk. Malware also tends to lurk in certain spaces on the internet (porn sites, pirated media, shareware downloads, malicious ads), so staying clear of them also reduces your risk.
2) Persistent credentials. Most malware (at least when I was looking at it years ago) isn't very advanced/persistent. An advanced persistent threat may sit on your machine indefinitely until you login using credentials, but they may not use those credentials immediately. Sign in and out quickly (don't "stay signed in") and always use MFA, ideally a second device. Secured Android and iPhone devices are ideal for checking bank statements etc... but you must be vigilant not to install untrusted apps from App Stores (anything not from a major company), as they are more likely to infect you than your browser on normal/safe websites. If you can, try to use virtual credit card numbers, and use a credit rather than debit card for purchases.
Phishing largely depends on clicking a link in an email, but also SMS "alerts" and browser pages that look like OS pop-ups. To defend against these, use a password manager to generate random unique passwords everywhere, use MFA everywhere. For login, set up and use 1-click-login and passkeys, especially for the most important logins (bank, email). Don't share or reuse passwords or API keys for more than one thing; not only does it reduce the impact, but it helps you identify where the compromise came from.
I personally play "fast and loose" on my Linux desktop and don't run things in isolation. But if you want rock-solid security, the only real option is running applications in dedicated VMs, and only mounting files needed for that application. Don't depend on containers or sandboxes for security. If you want a "lightweight" alternative to VMs/containers, you can have multiple user accounts on your computer, lock each user account down (no reading/writing from other user accounts), and either login to different ones for different tasks, or run apps as that user and have them display in your current user's desktop.
Well, we don't have to use the tools you mentioned.
If there is no way to disable auto-update the application goes in the bin.
How many CVEs are in the software you're not updating in a timely manner if at all?
You got it wrong. In AI era, the trade-off is between CVE and vulnerabilities that people haven't noticed yet. In the AI era, the forest is ALWAYS on fire. Even when you don't see the smoke.
VMs would work pretty well.
You can use VMs and reset after use
+ one for keeping the others secure
> Makes me want to have like 5 separate computers:
I mean if you have only one there’s https://www.qubes-os.org/ but not sure about the gaming part, nor is it necessarily as good as separate isolated hardware.
Architecture page: https://doc.qubes-os.org/en/r4.3/developer/system/architectu...
I mean this is the basic premise behind QubesOS, but if you add isolation like Qubes wants, it does become very difficult to cross between apps.
I had a theory that in the future all online interactions would need to be brokered by an agent that can stay on top of all those changes, and that the internet would be a hostile if not unuseable —and possibly also illegal— place to visit without having a personalized AI broker that is insured, and that you pay for. It would begin with you owning the agent but it would end with the agent owning you.
For banking, you can avoid computers and just go in.
To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.
Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.
Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised
100% agree. Developers are playing with fire regarding cookie stealers, malware, data exfiltration.
Ideal setup:
- Dev machine
- Gaming machine (isolate downloading mods)
- Browsing machine
Isn’t this common knowledge that there are computers you use for banking, computers you use for windows 98 and computer you use for gaming?
For banking I use macOS/linux, sometimes iOS unfortunately but we all do sins
Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.
Chromebooks are cheap and are a good fit as a banking only computer.
Maybe consume less sensationalist newsbits :P.
“ Makes me want to have like 5 separate computers”
You forgot one.
"forgot" :D
> How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
I moved part of my wealth to a private bank, on an account requiring several signatures (from different persons) plus a vetting from the banker for every single outgoing transfer. Not only are several manual signatures required, in addition to that the banker must call me before authorizing any transfer, even if it's only, say, 1 000 EUR (now it's not a retail bank and that's kinda the whole point: transfers are rare and when one is done, no matter the amount, a shitload of verification do happen).
In addition to that, I told my banker that should anything out of the ordinary happen (say an order to suddenly wire 100 K to Dubai or Erithrea or something), then he should go out of his way to make me come in person to the bank to confirm the transfer.
You must train your banker: you must demo him voice impersonation on the phone. You have to make him become totally paranoid too.
I'm not going to say he's my buddy, but he knows I know my shit and we regularly talk for 20 minutes on the phone.
Any transfer to a new bank account: long discussion explaining what that account is and why I'm moving money there.
I've also got digital files (say administrative stuff, stuff from notaries, etc.), encrypted, that I myself cannot easily access. For to access them I must physically go to my safe at a bank first. And trying to impersonate me physically at the bank to steal my stuff means: "on your way out, you risk being locked in the secure airlock and then spend years behind bars".
> Makes me want to have like 5 separate computers:
I have a setup not totally unlike that. For example one phone with one additional app: the country's official "electronic ID" thinggy. And I don't browse from that phone.
I've got several computers and there's definitely my "trash computer", complete with bogus email accounts and crappy apps, etc.
> How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
The first thing is recognizing there's an issue if someone hacking your unique email then can steal everything you have. Many people stop there: they think, as in the XKCD, "it's game over anyway if someone gets my email, so I may as well not worry about it".
That is the mentality of losers.
You're already well ahead of the crowd by recognizing there's a problem and by thinking about solutions.
Many things can be done to lower the risk of a full compromise. I've got two physical LANs and the absolute most sensitive stuff (e.g. my banking apps) are done from a computer that does nothing but email + bank.
So it's not just a separate computer: separate computer on a separate physical LAN (not a VLAN). And there a tiny Pi doing the bridging between the two LANs. And of course it's got a serious firewall (a lost art, but it could make a comeback with the help of LLMs).
Something else that can help is stuff like Google Workspace and keeping the credentials for the admin account very secure and hardly ever logging into it. That way if "jack@acme.com" gets compromised, there's still "admin@acme.com" that can help get back control of jack@acme.com.
> - one for development only
I've got a user account for dev only.
> - one for gaming only
A dedicated computer for gaming yes.
> - one for socials only
Besides HN, I don't need those.
It's a wild world out there.
The hacks won't stop coming and you're not anywhere near paranoid enough.
Mainframes with punchcards FTW, lol! This seems onerous to the point of it being almost like a security as a hobby.
[dead]
[dead]
[dead]
[dead]
... but, but, sama says we having his RAM is bad! And now you want 5, not 1 of 'em precious compute receptacles!
Sama hasn't learned to keep the inside voice inside.
(They have unleashed the swarm. The are "helping peer").-
yep, that was so annoying to hear him say that. grrrr.
We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
My router got hacked the other month (left web interface open to internet) after being exposed to the internet and the persistence left behind was overly commented LLM artifacts. I caught it in like 5 minutes so they weren’t able to do much but it was eye opening how much more basic practices like firewalls are not something that can be ignored even if you update your firmware frequently.
What if I told you nothing new is happening? Except casuals are noticing what the industry knew and saw for decades?
Script kiddies aren't new. Metasploit has been around literally decades. vx-underground existed for years, not to mention older open source projects that collect all known malware dev tools. APTs have been pwning stuff, script kiddies have been doing their thing. Nothing significant has changed for anyone in the know.
Just because LLMs automated some offensive stuff doesn't mean there's gonna be new magic collapse of society.
Given that our knowing-and-seeing industry is also the thing which enabled and then desperately shepherded humanity out-of every IT deficiency that has ever existed, our inside perspective is still highly fallible too.
Are we going to end up in an arms race where our local records are defensively sprinkled with prompt injection/agent poisoning/tank_man.jpg?
It's an odd choice to run this on the victim's machine (exposing the open router key) rather than on the attacker's infrastructure. I guess this lets them keep it dead simple for buyers, you just need to choose a Telegram channel for the logs to be dumped into.
How very considerate! Steal from the rich, not the poor.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
I'm just imagining some poor person getting a call about 100,000$ that's been transferred to their account without any idea how or why.
Means testing for malware
Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.
You’ll have created an autonomous Robin Hood clone army. Fun times.
https://en.wikipedia.org/wiki/Morris_worm
Ha, finally me being poor pays off!
> Ha, finally me being poor pays off!
Unfortunately LLMs still optimize scams/exploits to the point that they'll probably still get to those of us heading into the permanent underclass, just a little slower.
kind of boggles me that the big ai labs arent running zero-shot classification on all their outputs with "does this look like any of these bad things we know people get up to"
They do. They're getting increasingly cautious with these things, too- Claude won't help you reverse engineer much anymore apparently.
Hackers are almost certainly using open-source AI for this.
Oh, other day I was asking Claude to get a onetime code from my PocketID VM, I was being lazy and I was asking it to open the website, generate the authentication code from there and then type it into a built in browser. And it just refused to do it. I couldn't convince it to do it, even though it was all my own infrastructure And essentially, we were doing a security theater because the website that it was trying to authenticate to was the one it had just published itself behind my internal trafefic instance, lol. I ended up having Codex do it. Codex had no compunctions about it.
They certainly do.
Just not very well. It's a hard problem. And they're not incentivized to slow down.
Once ranked, victims can be assigned a proportional token budget so that a high value victim might be targeted with zero days or deepfakes that are more expensive to execute but have passed the desired cost/reward threshold.
If you backtrack the posts here, the warnings were there from various bodies already two or three years ago, but it seems the lag is now being closed. (In fact, I wonder why it took so long.)
If I ever fall in it's radar, I would be the last on that ranked list
I'm safe.
> There is a new idea in the cybercrime world — cast a wide net, steal everything, and let AI sort out the specifics.
This is an old idea just with AI added on top.
People even steal physical mail for this reason.
Phase 0: breakins to all (php/python/etc,)
Phase 1: the closure, companies close down sites, all banks down
Phase 2: the reopen- but to only the correct* sites
Phase 3: the viruses
Phase 4: the re-shutdown
Phase 5: use of physical currency / fax machines only.
We are just getting to Phase 0. By end of 2026 most things will be broken into. By 2027 most banks will be offline.
I work in security and while things are grim, it's not even close to this.
Whats your perspective? Front end dev here.
Bot's enable discovery of information or exploits at a scale that we cannot imagine.
Recent example of a breach, they are now using AI to decomp APKs and iOS apps, mainly old builds of apps that nobody would bother spending the time on previously to look for secret keys or accidental information disclosure.
This would have been immensely time consuming with each app usually doing things a bit differently (e.g. encrypted + compiled settings, keys in database, settings in a settings.json).
Now you can automate the entire pipeline, this is actively being used to look for legacy API endpoints that may not be hardened or monitored, or if you don't rotate your keys and at one point disclosed them.
Dang, that’s tough. Recently having to deal with php2shell was a good wake-up call for anybody doing hosting of any kind. Next few years gonna be rough (but food on the table for cybersecurity experts I guess)
I'm not on the optimistic side regarding AI but I just don't see this happening.
now who said enterprises aren’t seeing ROI from AI use
[flagged]
[dead]
[dead]
Good. LLMs aren't very good at that.
What makes you say that? It's good enough
At priority ranking? It's less accurate, slower, and more expensive than every other purpose-built tool.
I'm no genius, but I've managed to set up my agents to prefer writing a small python script or whatever to do repetitive/iterative tasks that my language model 'knows' it's not good or efficient at.
You're technically right, which is why I say "good enough".
It's a good strat imo - start simple and get it working with LLMs asap, then you can add heuristics etc based on the data you have and improve with that benchmark
Ah yes, I see the malware authors buying subscriptions to the appropriate software to do that.
All those tools are FOSS.