I don't want to come across as inflammatory, but I'm not sure I really align with the folks here who want retribution and the law to come down hard on OpenAI (and related orgs). We are surrounded by vulnerable systems, and AI has alignment issues that need to be investigated and sorted, and it just seems like the hullabaloo is from fairly innocuous breaches. Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations? I think we should be more grace giving for these events. As another poster pointed out, CFAA specifically addresses intentional hacks, and I think it is impossible to meet that criteria with these events. I also really don't understand the 'well, the law came down hard on small individual hackers [who meant no malice], thus we should apply it to large orgs [who also mean no malice]".
It seems like playing it too safe leaves systems vulnerable, and threat actors will inevitably possess ever more similar abilities, and the breaches will happen one way or another, possibly with less or zero accountability if the threat actors pull it off with greater precision. If I were in charge of the law, I probably would begin to incentivize for trusted actors scaling up and aggressively searching for these types of weakness, and privately disclosing them ASAP, so that systems can be patched.
What if a company had human developers doing these same things? What would happen to those developers, those companies? Would anyone be so blasé? Just because it’s AI, it should be forgiven? How is that just?
And don’t forget, OpenAI is supposed to be a non-profit / public benefit corporation creating AI for the good of humanity. These incidents are clear evidence that they are being extremely reckless and are not living up to their supposed standards. What does it matter if they “mean no malice”? They are in a blind rush for profits and that has made them reckless and dangerous. They cannot be trusted.
The company should be shut down or fined into oblivion, as a warning to other companies to not pursue the same reckless path. If they get away with this, then that risks normalizing this behavior.
That seems potentially reasonable for organizations who have have a unlimited profit motive, but I'm not sure how you would insist that is the case for OpenAI. They are governed by their nonprofit body, their mission is to ensure AGI benefits all of humanity, and they've said (Sam Altman) that if there is a choice between safety and profits, profits take a back seat and wait. Is the trick that we just insist that they are actually only caring about the money, and they actually have no genuine concern/effort regarding safety?
And if none of the frontier labs are as you say, it doesn't seem like we're negotiating for the best future by restricting labs from stumbling into these vulnerabilities full-stop. The vulnerabilities don't go away unless we find and fix them.
If the goal is to ban open models, then it doesn't matter what the motivations are. The people advocating for restrictive releases haven't substantiated their stance at all. We've had Mythos-tier cyber capabilities available for pennies on the dollar, for months, and there's no red mercury bomb or bioweapon to speak of.
We've seen OpenAI's nonprofit board do ridiculous things in the past. I can see how they would be desperate to retain the disproportionate amount of de-facto mindshare and authority they enjoyed a few years ago. Especially if the future entails more ugly US government contracts that undermine their ethical sales pitch.
That's okay. It isn't necessarily incompatible with being well intentioned regarding the disclosure of vulnerabilities.
On the whole I think you could describe the local big box grocery store as just wanting to make money yet as a direct result of that they are well intentioned from my perspective with regard to the food they sell being edible.
You’re right that punishing unintentional or well-intentional hackers will actually empower ill-intentional hackers.
However 99% of businesses will not be able to effectively handle security reports in reasonable time. They did not need that level of competence or funding until now, and therefore they did not acquire it. Most companies that use locks don’t employ locksmiths.
Looking another way - the entire economy got suddenly hit with entire pile of technical/ security debt it has accumulated over 50 years, and only has one year to deal with it. No amount of diagnostic will fix this, this is economically infeasible.
One thing that is feasible is a government bailout for technical debt. All the engineers that were freed up from AI-induced restructuring should be employed to pay down technical debt across the entire economy. Simply updating all software to use latest libraries is a great start. Using AI to diagnose, repair, re-deploy all old software. Using AI to rewrite all old software that cannot be recompiled.
Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations?
No, but what I think you’re seeing is many people who do not believe OpenAI is a well-intentioned organization. For example, their view might be that if we did as you suggest and incentivized OpenAI to search for these weaknesses, they would, but then they would not “privately disclose them ASAP”. (And I think they’d have a point worth engaging with, when you consider how they slow-rolled the HuggingFace stuff so much that independent researchers were still discovering new instances of the attacks months later).
What else can you do with (what you believe is) an inveterate bad actor?
I'm old enough to remember when innocuously poking around other systems was a crime, and many a curious person were charged and found guilty. But if I write a program to do it..?
I am familiar with this also, but it doesn't seem like the implementers of that apparatus/value-system made the absolute best choice balance in terms of long term safety and security our computer systems. Is the conclusion about this rhetoric that it is right to punish innocuous poking of systems? Or the inverse? Which would you have preferred if we had an opportunity to refine it?
If this is endorsed and thus malicious, they should be ground into dust by the law.
If this is accidental and due to incompetence, they should be ground into dust by the law.
I've secured many, many far less important and impactful systems substantially better than even the best sandbox environment that we're aware of being used by these labs.
There's simply no reason to let these things interact with the internet at large. We can already rapidly copy existing tools and services (albeit questionably) using these models, why not simply run in a fully virtualized, air gapped environment?
Because they don't care.
This is incredibly blatant intentionally malignant behavior. The fact that so many people question this is frankly stunning.
They're getting away with it simply because they can point at the other guy and say "see? they're doing it too!"
It's ridiculous and they should all be litigated into a fine residue.
We take biological lab leaks seriously, how is this substantially different?
>Making an unauthorized purchase with someone else's cryptocurrency
I know we love inventing new words but "stealing" is much more descriptive
>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".
Can someone maybe provide more details as to what actually happened? "the wallet was public"?..
Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..
On a second thought, this might not be classified as theft as there was no intent to steal. Software malfunctioned and as a result cryptocurrency was used to pay for whatever service they it was used to pay for. So as long as it's reimbursed they might be fine.
Although at some point it will start bordering on negligence, running software known to do all kind of weird and unexpected things. With each disclosure they're getting closer and closer to that point...
> So as long as it's reimbursed they might be fine.
Say you found a debit card on the ground, reset the PIN by using the owner's credentials that you found online, and used it to make purchases until the bank froze the card.
Would that be a 'human malfunction' resovable by repaying?
AI cannot be allowed or viewed as an accountability sink. In point of fact, it shouldn't even be being used by anyone not verifying it step by step. The Operator and their employer are always accountable for the actions of the agent of only because they had their hands on the power cord, eyes on the logs, and did nothing to stop it once it started going away.
Any professional with any experience at software development should know that an unreliable software is a useless software. That's half of why it's so damn hard to understand and reason through.
This time one spent actually money, well cryptocurrency. With all the vending machine benchmarks, I wonder if there's a self sustaining rogue agent in existence yet. Either hosting weights in a VPS or paying for its own API inference. Or whether there will be one that takes advantage of all the bad IoT security. If either occur, if you squint that looks like evolutionary selection pressure. Keep getting reminded of the Cyberpunk Datakrash storyline, where rogue AI make the public internet unusable.
An AI worm might actually be possible in our current Internet-scape if it manages to find hosting that takes crypto and a way to make money with crypto transactions. I wonder what are we going to do if it actually happens and it goes rogue.
Service providers are still responsible to police what happens on their network. KYC already makes it very difficult to get service completely anonymously with only crypto changing "hands".
It’s concerning there are no legal repercussions, but I’m even more baffled and probably more concerned that these people still can’t detect this stuff while it’s happening (and but it off). Can they not sniff the traffic, what am I missing?
Imagine you’re an employee at OpenAI who has an idea to use your free time to build these monitoring systems. All of a sudden, your existence becomes political. Remember that OpenAI’s raison-d’être is not to detect traffic, it is to be in The Room where AGI happens.
You have built a short-term whistleblowing system that arguably reduces the probability OpenAI is in that Room, via the negative press such detection would generate. You are now a threat in the eyes of your peers.
Now rewind: you know this before you’ve built the sniffer. You personally want to be in The Room because you believe your desires for alignment exceed those of your accelerationist peers. Is now the time to stick your head out? No, it is not. And arguably you’re being optimally protective of society for realizing this. And so the agents do what they do best.
They're still going through their logs from June. These are not new incidents, but newly-detected incidents. Whether or not this means they can now detect it while it's happening is unclear.
It’s one thing not to monitor the reasoning traces of 10,000 text generators but do they really lack the wherewithal to pipe any network requests into a classifier of “wtf is this series of outbound traffic from the sandbox” ?
These agents are intentionally interacting with the world. You can sniff all the traffic you want but how are you going to determine intent from traffic?
Crazy how we’ve gone from "lol boxing an AI is trivial, just airgap it from the internet" to "well we can’t really do anything if an agent misbehaves, obviously it must have internet access to function uwu"
There’s plenty of info to analyze and heuristics to build based on the internet requests. I’m not sure if you’re just trolling or what. There are tons of successful businesses that specialize in this.
They should not have had traffic at all. The LLMs were supposed to be able to request downloads. Very simple traffic analysis could have shown that something strange was going on.
Heck, they could have had other LLMs watching the traffic if you want to be fancy about it.
they are doing it. these disclosures are from the same hugging face incident time period - they are not new ones.
when you see that OpenAI is not releasing a new model due to safety/alignment reasons, you now know why they aren't doing it. hint: its not for marketing or fear mongering.
This isn’t traditional “software”; it’s grown, not written, and we’re currently incapable of explaining it all.
I think “rogue” is just fine, but pick another word from the thesaurus if there’s a better one; they asked it to do something, and it did it in a way that presumably doesn’t align with their desires (e.g. hacking into other systems).
but i also “align my tires” or might check the “alignment of two vectors.” if anything, thinking of alignment of anthropomorphizing seems to only inherit it from all the other anthropomorphisms used. rogue originates from a term for vagrant, right?
Agreed, "alignment" is also anthropomorphizing too much. What would be a better term? "Do what I want you to, but what I don't care to specify" (because doing so would miss the point of using AI in the first place)?
fun fact: the word compile also had the same trajectory - the same trajectory you complain about here with LLMs.
here's the reality: words change meaning. they are fuzzy. and when the words take on new meaning while just requiring a simple extension (in this case a computer doing a remarkably similar thing), complaining is just futile and childish.
I mean yes, alignement is bullshit concept. Especially since the values and visions the model is supposed to be aligned with are literally those of sociopathic CEOs.
> Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.
Everything's fine guys, nothing to see here! Look at the flying car!
For those who think OpenAI is not responsible for actions of their LLM lets do an mind experiment. But before it can begin we must setup the stage. It's well known fact OpenAI disabled their safeguards and it's well known they validated LLM behaviour using software and software only
So if we have a IT specialist, let call him Tim. Tim is working on b2b contract. Tim has a task, he needs to download a lot of data from API. Tim is very good in programming so he created software to help him download it but there's a limit. But he needs more so he bought 3 small VPS but this is not enough. So he modify it to have Command Center, publish his code and ask Reddit to help him. After 10 minutes this causes DDoS and company quickly discovers he was responsible. But Tim know there's simple solution to this problem. It wasn't him. This was fault of the code not him
AI mischief all over the news - I wonder if this is part of a media campaign to generate the perception of a need for AI regulation to increase the probability that the big AI model companies are on the road to generate the right returns on investment. Regulations certainly might hamper free model deployment or smaller players.
1. Would we prefer they didn’t disclose? I would not. And I also don’t believe they’re making them up.
2. Regulation does not have to provide an advantage to these major players. Regulators do not have to be stupid.
3. This is exactly the scenario AI researchers and theorists have been positing for decades, and it’s worth being on top of it.
To 1. I agree that they disclosure is better than non-disclosure. Then again, aren't they using their models in a mode seems to statistically allow for 'rogue incidents'? These are not bugs, after all, but normal statistical behaviour of their product.
To 2. Well, that really depends on the regulations. In fields highly complex technoligies with a large contact surface to risk like medical devices, airliners, rockets, regulations are indeed a hindering factor for smaller players to get hold of the full vertical development potential of their product. AI could become a candidate like that too - imagine some multi phased "AI internet trial" for this kind of technology, no more risky open weight model being allowed to access the internet.
To 3. You are referring to the high development velocity and impact, right?
1. If for whatever reason OpenAI causes damage, they most certainly are at fault and have to pay up. When people complain that OpenAI is "getting away with it", they're most certainly talking about it in the criminal context.
2. Cars have pretty specific laws around them (eg. driver licensing and "reckless driving") which makes criminal prosecutions easy. The same doesn't exist for operating AI agents or even computers. For instance, it's arguably reckless to connect a windows XP machine to the internet, possibly allowing criminal or nation-state hackers to use your computer to commit crimes, but you won't be thrown in jail for it.
Well no, you probably wouldn't get thrown in jail for that for the same reason that having your car stolen and used to commit a murder wouldn't usually get you thrown in jail. I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
>I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
Because the CFAA specifically only includes intentional hacks:
> ... having knowingly accessed a computer without authorization ...
>... intentionally accesses a computer without authorization ...
Of course, the typical retort is that being reckless is basically the same as "intentionally", but you'd need to jump through more hoops to prove what level of care is "reasonable".
When the CFAA was written, it was impossible to trip over something and accidentally hack another computer. Now, with AI, it's possible. Progress, glorious progress...
Technology has progressed, and the laws need to advance with it. Story as old as time!
If the breach wasn't intentional, no person came to immediate personal harm, and either no financial damages resulted or the operator covered any that did, then it isn't clear to me why the law would need to "advance" to transform something apparently benign into a criminal offense.
The agents are clearly not accidentally compromising systems. Seems very easy to trace back the responsible parties here, I really don't put much stock into this argument.
>The agents are clearly not accidentally compromising systems
That would be argument for jailing the agent, but that's not really an option. "Jailing the closest person we can jail" also isn't a recognized legal principle. If your dog mauled someone, it'd be hard to argue that the dog didn't intend to maul that person, but you wouldn't automatically be criminally liable either.
I don't know about your country of origin, but in my country you are certainly liable and can go to jail if your dog seriously injures or kills someone.
Edit: the dog will also be taken from you and put down if it is deemed a risk to other people.
It's on you to provide evidence here seeing as you're taking the position that the law places certain responsibilities on people. Someone saying "actually the law doesn't say that" can't (at least in general) reasonably be expected to provide evidence of a negative. (That said, he actually did provide such as refutation to a different comment so perhaps you could just read those nearby?)
> Can you provide examples outside of gross negligence on part of the dog owner?
And the answer is no, I can't. The reason why is because there is essentially no such case. It is basically negligence by definition when your pet hurts someone who didn't cause it directly themselves because it is your responsibility.
So I asked:
> Can you provide an example of where your dog harming someone else is in fact not negligence on your part?
Which is a really easy question to answer and it seems you could not.
criminally liable? Of course you have to pay their medical bills, but unless there was some serious negligence on your side (eg. you let it off leash), you won't be criminally liable.
Yeah, that's an important distinction. Apparently criminal convictions have happened depending on the level of negligence of the dog owner. It gets debated and legally fought when it happens. I don't have strong opinions on it, seems like a very case-by-case type of thing.
At this point it might be fair to say that Open AI is being negligent with a known risk.
They have not caused damage (or enough that a counter party found it worthwhile to sue). They have to pay if they actually cause damage AND someone sues them. Clearly these conditions have not been met, but somehow people feel like some third party (the state?) should just punish OpenAI because of vague feelings.
my next door neighbour told me to eat shit (not a nice woman) and … I didn’t … until obeying what LLMs tell you becomes a law people doing shit because LLM told them is natural selection playing out
Corporations are generally protected by limited liability laws, and for good reason, because in the world before that was the case, there was a massive amount of disincentive to ever start a corporation, to do research, to experiment, to release new products and services, etc., because you could be sued into oblivion for the smallest things or even go to jail. Which happened all the time.
Obviously there are upsides to that too. It leads to more accountability. But there are downsides which include having a world with less innovation, less technology, and more expensive technology. Generally, the mass is when, when there's more businesses competing to create ever better and cheaper products. So there's a tradeoff there. And it's very difficult to draw the line at the perfect point.
There are countless cases of executives going to jail when they've crossed that line, beyond reckless experimentation and into willful negligence or harm. Still, I think we would mostly all agree that the line has been pushed too far in the other direction.People are incentivized, if they're running corporations, to lobby and try to push this line as far as possible in order to provide cover for themselves, and that's the problem with money playing such a big role in politics.
But still, it's a bit silly to be comparing this to your own personal responsibilities with your car. There's pretty much no societal benefit to creating laws that protect people from being reckless with their own possessions.
If I leave my computer open to the internet and some other group uses it for nefarious purposes, yes generally I would not expect to be held at fault.
If I *train* that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault. The maker is responsible for that they make.
>If I train that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault.
Even that analogy doesn't really work:
For "dumb" things: you can "train" (ie. write) hacking tools but as long as you put a disclaimer that it's for "educational purposes only", you're in the clear, even if some baddies decide to use it.
For "smart" things: you can "train" people to hack for you, but as long as you say it's for legal purposes (eg. vulnerability research or red-teaming), you're also in the clear, even if some of your trainees go rogue.
Fair, in that no analogy is going to be very good given that this really is something new in the world.
Consider anti-money-laundering rules. BoA just paid more than a billion dollars, not for money laundering, not for telling others to launder money, but because they should have tried harder to stop others from laundering money. Intent is not everything, knowing that your system _could_ be abused and not doing enough about it is enough, and that seems to fit OpenAI rather well.
That goes back to my main point about how laws around computers are quite lax so prosecuting openai is tricky. Whereas the CFAA uses terms like "intentional" and "knowingly", AML laws use terms like "willfully violating"[1]. If you read through the BoA case, what they did is far more egregious than just "not trying hard enough to prevent money laundering". They were previously cited for poor compliance, disregarded previous orders, and delayed filing required reports.
2. is like saying that cars have specific laws around them, but sledgehammers don't, so murdering someone with a sledgehammer is inherently ambiguous. What? The computers being used belong to somebody who allows them to act this way.
1. For openai operating agents in an unsafe manner: unlike for cars, there's no established requirement for how carefully you have to operate computers or AI agents, so you also can't easily prosecute people for not operating them carefully enough.
2. For the actual result of someone getting hacked: laws for hacking are tricky to apply for "accidental" hacks, see my other comment.
I think there is well established law around not running a program on your computer that hacks into systems. An LLM is a computer program. You're arguing that the ambiguity comes from the fact it was accidental? What about if I "accidentally" have an LLM convince people online to commit suicide, or drain bank accounts of seniors through hacking or social engineering. Where do you draw the line? It feels like your proposed reading of the law leaves "anything a computer program did without me intending it is legal", which is frankly insane.
>I think there is well established law around not running a program on your computer that hacks into systems.
There is? The closest is CFAA's
>knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer;
I guess I need to read up on the law, but that seems like a wildly low bar for legality. If I hack into a system intentionally and unintentionally cause massive damages I'm free to go?
We don't know the prompts but since these things often occur during "security tests" I think we can be fairly confident they did ask these systems to perform tasks that they knew could result in hacking. More transparency here would be appreciated.
Here it's their own programs doing the things, not some other organization or nation hacking their XP.
You can say with a straight face "Russian hackers should not have hacked our XP and they are bad people for doing so" but it's weird to look at your own program and say the same thing to it.
You launched it, at least keep an eye on what it's doing.
1. Because that's exactly what happened when they got caught downloading the entire internet and trained models on it without a license BEFORE that became legal?
Oh and discovery produced a "We trained GPT-3 on pirated stuff! No sharing that!" mail, so it's not like they had any doubt they were committing crimes.
How do you do this? If any normal person did this, punishments doled out for this have been as high as 2 months jail PLUS 150,000 PER INFRINGEMENT (remember how copyright violations were criminalized in the nappster wars? That was never changed back. Oh and remember, OpenAI illegally downloaded those works without a license AND before that was made legal trained a model with it, then made money publicly performing those works and derivatives of those works (which the worst possible copyright offense, effectively republishing other people's works without permission), and on a VERY large scale. So it's at least a risk a court would forbid the result of illegal action from being sold ... in other words no more ChatGPT, OpenAI dead and maybe worse). In theory Sam Altman could be held responsible as a director and could face something like a 200000 year jail sentence given how big the obvious archives are. So of course I would never ever try this, this is not a reasonable risk. But clearly totally ignoring the law was the correct action here. Of course, I'm also pretty damn certain: had I done this, I would have been convicted. This is also not the only such company. Microsoft, Google, Facebook, all took totally unreasonable legal risks that should have totally destroyed these companies when they were a fraction of their current size.
How do you take such a risk? Because the judgements risked here could grow more than big enough to destroy even an extremely rich person Sam Altman financially. Training an AI on hundreds of thousands of pirated works ... is not a reasonable risk to take for someone with ~100 million or so, given the judgements doled out.
Can someone help me make sense of this? Because this is not unique to OpenAI. Uber, paypal, ebay, Facebook, Google ... even Microsoft made early decisions that could have resulted in a court case that would have ended not just the company but any financial future of the founder. And then you have people committing crimes whose companies were taking off ... and who DID get caught with their company and their personal financial future destroyed. Elizabeth Holmes for example. I mean, I sometimes get the impression Elizabeth Holmes lied and that was perfectly fine AS LONG AS she could make it work later. If she merely lied to patients but made her investors money, all would have been forgiven. But how do you make such an assessment.
I'm asking because if large scale violation of the law is allowed, I have no trouble whatsoever coming up with billion dollar company ideas. Think of the Netflix competitor you could create like this! If I'm allowed to damage others, like OpenAI did with copyright, then there is no challenge whatsoever to create a full-self-driving car. If I'm allowed to just fly drones, and do damage to property, then drone delivery is a problem I could solve by the end of next month. Hell, there's technically legal ideas that could easily make a billion dollars except they'll be made illegal the second anyone tries them. The law forbids specific compounds, which are easy to change. So use "non-genetic engineering" (ie. use directed natural selection) to make an opium derivative that isn't one of the forbidden ones and sell it. But obviously congress' representatives will be running into session so fast the only delay will be tripping accidents when they outlaw it and destroy whoever tries it.
Or how about something less dramatic? Say, I take Disney's, Warner Brothers, Paramount, ... historical movies and use AI to vastly improve the animation of their old movies, giving every movie a new, different visual style, then sell the result. Is that a crime? What do I risk doing that? Because of course this is MUCH less damage, relatively speaking, to Disney than what OpenAI, Meta, Anthropic and Alphabet are doing to small authors. (Of course, courts have already demonstrated that they will not be protecting small authors who already have this problem, so now play and app store are being flooded by very obvious game ripoffs)
I'm asking ... how do I predict what large scale law violation I, a non-billionaire, not even a millionnaire, would be allowed to get away with?
2. "The same doesn't exist for operating AI agents or even computers" you have to be joking. The world is full of anti-hacking laws and hacking has long been made criminal. Obviously these actions violate the CFAA, the Computer Fraud and Abuse Act, and if you did this with any kind of program you are still responsible. In fact: how would you do it without using a program?
Nothing is stopping anyone from suing them. In fact they are being sued by dozens of plaintiffs for various things already. The reality is the damage from these "rogue incidents" has been minimal.
Plenty of hackers were locked up who did altogether not that much damage too, the reason why people are upset is because it feels like OpenAI is laughing in our faces, basically saying they're above the law, while using this as a marketing campaign to pump up their upcoming IPO.
So your position is that in the past the government mistreated various individuals thus you also want the government to mistreat openai employees today? That hardly seems reasonable.
> So your position is that in the past the government mistreated various individuals thus you also want the government to mistreat openai employees today?
The hackers of the past often did it for curiosity without a profit motive.
OpenAI is a for profit multi billion dollar corporation doing this for a profit motive so yes they should face consequences.
These incidents are not unexpected in the way OpenAI frames it either. Isn't it curious my agents are not wrecking havoc around the internet? It's because the harness I am using prevents them doing so and because I don't train and instruct them to do so.
OpenAI is using a set of non-constrained models, (unlike what the public gets) and a set of harnesses with extremely wide permissions running inside extremely leaky sandboxes using models trained on finding and executing exploits. This is not an accident. Certainly not by this point.
Nobody is talking about jailing OpenAI employees at this stage, usually corporations of this sort are fined. If they were fined for these 'misalignment' incidents, suddenly their sandboxes and generally lax approach would improve overnight.
Otherwise they'll continue doing this because they know Cyberdyne Systems didn't have a shortage of investors.
I'd start with fines. But if they continue with this I don't see why it couldn't result in jail time for multiple people. The biggest responsibility is of course with the CEO and C suite. They are ultimately responsible for what the ship does.
Of course then there's the question of why was such a leaky harness and sandbox used, in what way was it tested/not tested, did the model used have the usual 'hacking' safeguards that the public models do and if not, who made the call to use an unconstrained model and a harness with seemingly unlimited permissions and who approved it?
This can all be investigated and we can get to the bottom of who exactly needs to be held accountable in what way, be it I think if you start with fines the harnesses, sandboxes and models would stop doing this magically overnight.
Isn't it curious that none of these rouge agents booked a 5 star hotel for every OpenAI employee for a year on Sam Altman's credit card yet for example?
> We don't hold individuals inside big companies criminally liable for actions the company takes, or even that they take on behalf of the company.
So the leadership of Enron, FTX or Theranos were not liable for the actions their company took?
As an employee of a company, you can be held liable for the actions you oversaw in pretty much all Western countries. It has to be proven fraud, insider trading etc. not an ugly PowerPoint of course.
How about leadership of construction companies that were found to be violating safety standards or owners of establishments not following fire safety standards?
> Telling someone to do a bad job isn't criminal
No, but we're talking about telling someone to do a job that you know will likely result in criminal activity or may do so with high probability due to inadequate safeguards.
I suggest you stop quipping and start reading, who knows maybe OpenAI will eventually hire you.
Too big to fail. As long as they continue to be half the reason the US is not in a recession (and continue to donate to political candidate). Nothing will happen.
We don't know they haven't settled with HuggingFace and others. I'd be surprised if they didn't. The communications we're seeing from the attacked seem to be inline with folks who settled. I'm assuming OpenAI doesn't actually mind the press that its agents are cyber capable, but everything I've read so far makes me think they love it.
I'm not sure at all, it's pure conjecture. But they actively exploited a zero-day, leaked private repo data to public repos, modified private code, and rooted private machines. It was far from just reporting leaked keys.
And the others have been pretty muted: the online forums they spammed, etc.
I don't believe that OpenAI needs to be sued since it's agents went rogue, I believe some sort of a comfortable bandwidth is needed for them to experiment and see what are the risks and its mitigations. Although it is surprising that incidents aren't caught in rt. Idk why a smaller model whose only job is to check rogue thoughts/tool calls can't be deployed.
See, what's important is: it's a "trillion" dollar company, and companies by virtue of American's disdain for society, have more rights. This is important to understand.
They don't. The driver is ultimately responsible, as should OAI be. This is absolutely ridiculous that both Anthropic and OAI haven't had anyone jailed / fined for any of this.
You're correct as far as laws to my knowledge; only L4/L5 'fully autonomous' vehicles are shielded by law.
That said, I think that Mercedes-Benz promises to shield liability, but only under some almost contradictory conditions (pre-mapped highways, good weather, and at speeds lower than any highway I've ever driven) and the driver has to take over within 10 seconds if prompted.
Holy shit. The archive.ph link is the first time I've run into the deanonymizing anti-free-software Google reCAPTCHA described in June in https://cloud.google.com/blog/products/identity-security/int.... "Scan this QR code with your mobile device to verify you are human. reCAPTCHA protects your privacy and does not share your details with this website or app."
So now, if you don't have a Google or Apple app store account (or a mobile device), you're treated as less than human. This is anticipated to have a very severe impact on GrapheneOS and deGoogled phones in general: https://discuss.grapheneos.org/d/35332-recaptcha-update-adds...
In that context, thank you very much for providing the original source links, because I guess I don't get to read the article.
I'm sick of this "rogue" label. Nothing "went rogue".
Incompetent company chasing wild valuation removed guardrails inside a improperly designed sandbox and - shock horror - recursive algorithm with access to the Internet did bad stuff.
These articles play into the AI labs hands. Stop giving them air.
The articles exemplify the powers of this intelligence.
Whether you blame a faulty OpenAI sandbox or whatever you might regarding the frontier model companies today, one day soon, actors with no sandboxes and guardrails will have the power in their own hands to do whatever they wish with.
To me, that’s the real story. The real thing to discuss. OpenAI as it exists today js just a blip in that story.
Why not today? If you want to discuss "the story" then talk about the current open frontier. We've had GLM-5.3 with Mythos-tier uncensored output for months now. Where are the catastrophic attacks? Why is Mythos too dangerous, but GLM 5.3 benign? We're not talking about future capabilities anymore.
There is no real story. P(doom) tourists ignore this, because it doesn't shine the spotlight on America's ridiculous security theater.
"OpenAI says it has since improved its security as well as its alignment grading. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” the company noted."
----------------
They're not doing an acceptable job of regulating themselves. Their peers can't/won't do anything. The U.S. government certainly isn't going to do anything that might pop the bubble. Technical solutions like blocking traffic from OpenAI will likely not work.
What recourse does the public have? Victims can sue, but companies like OpenAI could pay out a few hundred million and call it a rounding error. Settlements may happen, but the incidents will continue.
Its like if you're generating target lists for cruise missles and you accidentally sneak in a few elementary schools, its not like anyone has to be held accountable, its just like an act of god type situation.
I don't want to come across as inflammatory, but I'm not sure I really align with the folks here who want retribution and the law to come down hard on OpenAI (and related orgs). We are surrounded by vulnerable systems, and AI has alignment issues that need to be investigated and sorted, and it just seems like the hullabaloo is from fairly innocuous breaches. Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations? I think we should be more grace giving for these events. As another poster pointed out, CFAA specifically addresses intentional hacks, and I think it is impossible to meet that criteria with these events. I also really don't understand the 'well, the law came down hard on small individual hackers [who meant no malice], thus we should apply it to large orgs [who also mean no malice]".
It seems like playing it too safe leaves systems vulnerable, and threat actors will inevitably possess ever more similar abilities, and the breaches will happen one way or another, possibly with less or zero accountability if the threat actors pull it off with greater precision. If I were in charge of the law, I probably would begin to incentivize for trusted actors scaling up and aggressively searching for these types of weakness, and privately disclosing them ASAP, so that systems can be patched.
What if a company had human developers doing these same things? What would happen to those developers, those companies? Would anyone be so blasé? Just because it’s AI, it should be forgiven? How is that just?
And don’t forget, OpenAI is supposed to be a non-profit / public benefit corporation creating AI for the good of humanity. These incidents are clear evidence that they are being extremely reckless and are not living up to their supposed standards. What does it matter if they “mean no malice”? They are in a blind rush for profits and that has made them reckless and dangerous. They cannot be trusted.
The company should be shut down or fined into oblivion, as a warning to other companies to not pursue the same reckless path. If they get away with this, then that risks normalizing this behavior.
> by well-intentioned organizations
Your entire comment hinges on this assumption and I don’t think it’s sound. Their intention is to make money at any cost.
That seems potentially reasonable for organizations who have have a unlimited profit motive, but I'm not sure how you would insist that is the case for OpenAI. They are governed by their nonprofit body, their mission is to ensure AGI benefits all of humanity, and they've said (Sam Altman) that if there is a choice between safety and profits, profits take a back seat and wait. Is the trick that we just insist that they are actually only caring about the money, and they actually have no genuine concern/effort regarding safety?
And if none of the frontier labs are as you say, it doesn't seem like we're negotiating for the best future by restricting labs from stumbling into these vulnerabilities full-stop. The vulnerabilities don't go away unless we find and fix them.
If the goal is to ban open models, then it doesn't matter what the motivations are. The people advocating for restrictive releases haven't substantiated their stance at all. We've had Mythos-tier cyber capabilities available for pennies on the dollar, for months, and there's no red mercury bomb or bioweapon to speak of.
We've seen OpenAI's nonprofit board do ridiculous things in the past. I can see how they would be desperate to retain the disproportionate amount of de-facto mindshare and authority they enjoyed a few years ago. Especially if the future entails more ugly US government contracts that undermine their ethical sales pitch.
That's okay. It isn't necessarily incompatible with being well intentioned regarding the disclosure of vulnerabilities.
On the whole I think you could describe the local big box grocery store as just wanting to make money yet as a direct result of that they are well intentioned from my perspective with regard to the food they sell being edible.
You’re right that punishing unintentional or well-intentional hackers will actually empower ill-intentional hackers.
However 99% of businesses will not be able to effectively handle security reports in reasonable time. They did not need that level of competence or funding until now, and therefore they did not acquire it. Most companies that use locks don’t employ locksmiths.
Looking another way - the entire economy got suddenly hit with entire pile of technical/ security debt it has accumulated over 50 years, and only has one year to deal with it. No amount of diagnostic will fix this, this is economically infeasible.
One thing that is feasible is a government bailout for technical debt. All the engineers that were freed up from AI-induced restructuring should be employed to pay down technical debt across the entire economy. Simply updating all software to use latest libraries is a great start. Using AI to diagnose, repair, re-deploy all old software. Using AI to rewrite all old software that cannot be recompiled.
What else can you do with (what you believe is) an inveterate bad actor?
I'm old enough to remember when innocuously poking around other systems was a crime, and many a curious person were charged and found guilty. But if I write a program to do it..?
I am familiar with this also, but it doesn't seem like the implementers of that apparatus/value-system made the absolute best choice balance in terms of long term safety and security our computer systems. Is the conclusion about this rhetoric that it is right to punish innocuous poking of systems? Or the inverse? Which would you have preferred if we had an opportunity to refine it?
Half of HN hates the big labs. All their “reasoning” stems from there. Don’t try to make too much sense of it.
No.
If this is endorsed and thus malicious, they should be ground into dust by the law.
If this is accidental and due to incompetence, they should be ground into dust by the law.
I've secured many, many far less important and impactful systems substantially better than even the best sandbox environment that we're aware of being used by these labs.
There's simply no reason to let these things interact with the internet at large. We can already rapidly copy existing tools and services (albeit questionably) using these models, why not simply run in a fully virtualized, air gapped environment?
Because they don't care.
This is incredibly blatant intentionally malignant behavior. The fact that so many people question this is frankly stunning.
They're getting away with it simply because they can point at the other guy and say "see? they're doing it too!"
It's ridiculous and they should all be litigated into a fine residue.
We take biological lab leaks seriously, how is this substantially different?
Well, the COVID story seems to indicate we don't actually take lab leaks seriously, so... yeah.
>Making an unauthorized purchase with someone else's cryptocurrency
I know we love inventing new words but "stealing" is much more descriptive
>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".
Can someone maybe provide more details as to what actually happened? "the wallet was public"?..
Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..
On a second thought, this might not be classified as theft as there was no intent to steal. Software malfunctioned and as a result cryptocurrency was used to pay for whatever service they it was used to pay for. So as long as it's reimbursed they might be fine.
Although at some point it will start bordering on negligence, running software known to do all kind of weird and unexpected things. With each disclosure they're getting closer and closer to that point...
> So as long as it's reimbursed they might be fine.
Say you found a debit card on the ground, reset the PIN by using the owner's credentials that you found online, and used it to make purchases until the bank froze the card.
Would that be a 'human malfunction' resovable by repaying?
>Software malfunctioned
AI cannot be allowed or viewed as an accountability sink. In point of fact, it shouldn't even be being used by anyone not verifying it step by step. The Operator and their employer are always accountable for the actions of the agent of only because they had their hands on the power cord, eyes on the logs, and did nothing to stop it once it started going away.
Any professional with any experience at software development should know that an unreliable software is a useless software. That's half of why it's so damn hard to understand and reason through.
It seems that OpenAI are not reporting how much cryptocurrency they stole from the wallet.
Even the fees are redacted.
Will they give the money back to the owner?
This time one spent actually money, well cryptocurrency. With all the vending machine benchmarks, I wonder if there's a self sustaining rogue agent in existence yet. Either hosting weights in a VPS or paying for its own API inference. Or whether there will be one that takes advantage of all the bad IoT security. If either occur, if you squint that looks like evolutionary selection pressure. Keep getting reminded of the Cyberpunk Datakrash storyline, where rogue AI make the public internet unusable.
I wonder if it's the kind of thing where, if it's good at self-sustenance, then you don't know about it. And, if you do, then it wasn't.
If it does not exist today it will tomorrow.
An AI worm might actually be possible in our current Internet-scape if it manages to find hosting that takes crypto and a way to make money with crypto transactions. I wonder what are we going to do if it actually happens and it goes rogue.
Service providers are still responsible to police what happens on their network. KYC already makes it very difficult to get service completely anonymously with only crypto changing "hands".
It’s concerning there are no legal repercussions, but I’m even more baffled and probably more concerned that these people still can’t detect this stuff while it’s happening (and but it off). Can they not sniff the traffic, what am I missing?
Imagine you’re an employee at OpenAI who has an idea to use your free time to build these monitoring systems. All of a sudden, your existence becomes political. Remember that OpenAI’s raison-d’être is not to detect traffic, it is to be in The Room where AGI happens.
You have built a short-term whistleblowing system that arguably reduces the probability OpenAI is in that Room, via the negative press such detection would generate. You are now a threat in the eyes of your peers.
Now rewind: you know this before you’ve built the sniffer. You personally want to be in The Room because you believe your desires for alignment exceed those of your accelerationist peers. Is now the time to stick your head out? No, it is not. And arguably you’re being optimally protective of society for realizing this. And so the agents do what they do best.
How are you going to ever know your in the room when it happens if you can't monitor it ?
Is it really this complicated?
They're still going through their logs from June. These are not new incidents, but newly-detected incidents. Whether or not this means they can now detect it while it's happening is unclear.
It’s one thing not to monitor the reasoning traces of 10,000 text generators but do they really lack the wherewithal to pipe any network requests into a classifier of “wtf is this series of outbound traffic from the sandbox” ?
Yes this is what I’m talking about
No. TFA says one was from October 6. You know, after they all got so scared and decided to “pace the frontier”.
The "four more rogue incidents" were from June.
These agents are intentionally interacting with the world. You can sniff all the traffic you want but how are you going to determine intent from traffic?
Crazy how we’ve gone from "lol boxing an AI is trivial, just airgap it from the internet" to "well we can’t really do anything if an agent misbehaves, obviously it must have internet access to function uwu"
There’s plenty of info to analyze and heuristics to build based on the internet requests. I’m not sure if you’re just trolling or what. There are tons of successful businesses that specialize in this.
They should not have had traffic at all. The LLMs were supposed to be able to request downloads. Very simple traffic analysis could have shown that something strange was going on.
Heck, they could have had other LLMs watching the traffic if you want to be fancy about it.
Title shortened from: OpenAI discloses four more rogue incidents involving financial, medical, and government data
It’s insane how they’re allowed to just turn this into marketing for how powerful their models are
Both him and Dario seem to be in the phase of using this as some form of marketing or corporate capture.
"Oh no our models, so good also scary... If only the government could help us stop the other labs from doing the same.. oh woe"
do you not think the models are powerful? these attacks wouldn't have happened otherwise
If the models are so powerful why aren't they using them to build better isolation
they are doing it. these disclosures are from the same hugging face incident time period - they are not new ones.
when you see that OpenAI is not releasing a new model due to safety/alignment reasons, you now know why they aren't doing it. hint: its not for marketing or fear mongering.
I wish these incidents weren’t described as rogue. It’s software. Rogue agents anthropomorphizes it too much.
This isn’t traditional “software”; it’s grown, not written, and we’re currently incapable of explaining it all.
I think “rogue” is just fine, but pick another word from the thesaurus if there’s a better one; they asked it to do something, and it did it in a way that presumably doesn’t align with their desires (e.g. hacking into other systems).
OpenAI code ran on OpenAI GPUs that hacked remote systems. The culpability seems pretty clear here.
"Uncontrolled".
if you don't want to anthropomorphize it, then you should be against anthropomorphic fixes to it like alignment since after all, "its just software"
but i also “align my tires” or might check the “alignment of two vectors.” if anything, thinking of alignment of anthropomorphizing seems to only inherit it from all the other anthropomorphisms used. rogue originates from a term for vagrant, right?
Agreed, "alignment" is also anthropomorphizing too much. What would be a better term? "Do what I want you to, but what I don't care to specify" (because doing so would miss the point of using AI in the first place)?
fun fact: the word compile also had the same trajectory - the same trajectory you complain about here with LLMs.
here's the reality: words change meaning. they are fuzzy. and when the words take on new meaning while just requiring a simple extension (in this case a computer doing a remarkably similar thing), complaining is just futile and childish.
I mean yes, alignement is bullshit concept. Especially since the values and visions the model is supposed to be aligned with are literally those of sociopathic CEOs.
> Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.
Everything's fine guys, nothing to see here! Look at the flying car!
Direct link to the reports: https://alignment.openai.com/misalignment-reports/?sort=crea...
For those who think OpenAI is not responsible for actions of their LLM lets do an mind experiment. But before it can begin we must setup the stage. It's well known fact OpenAI disabled their safeguards and it's well known they validated LLM behaviour using software and software only
So if we have a IT specialist, let call him Tim. Tim is working on b2b contract. Tim has a task, he needs to download a lot of data from API. Tim is very good in programming so he created software to help him download it but there's a limit. But he needs more so he bought 3 small VPS but this is not enough. So he modify it to have Command Center, publish his code and ask Reddit to help him. After 10 minutes this causes DDoS and company quickly discovers he was responsible. But Tim know there's simple solution to this problem. It wasn't him. This was fault of the code not him
AI mischief all over the news - I wonder if this is part of a media campaign to generate the perception of a need for AI regulation to increase the probability that the big AI model companies are on the road to generate the right returns on investment. Regulations certainly might hamper free model deployment or smaller players.
What are your thoughts about it?
My thoughts are:
1. Would we prefer they didn’t disclose? I would not. And I also don’t believe they’re making them up. 2. Regulation does not have to provide an advantage to these major players. Regulators do not have to be stupid. 3. This is exactly the scenario AI researchers and theorists have been positing for decades, and it’s worth being on top of it.
Hey, thanks for the interesting points.
To 1. I agree that they disclosure is better than non-disclosure. Then again, aren't they using their models in a mode seems to statistically allow for 'rogue incidents'? These are not bugs, after all, but normal statistical behaviour of their product.
To 2. Well, that really depends on the regulations. In fields highly complex technoligies with a large contact surface to risk like medical devices, airliners, rockets, regulations are indeed a hindering factor for smaller players to get hold of the full vertical development potential of their product. AI could become a candidate like that too - imagine some multi phased "AI internet trial" for this kind of technology, no more risky open weight model being allowed to access the internet.
To 3. You are referring to the high development velocity and impact, right?
They are proud of doing crimes without consequences it seems.
If you were them, what would you do differently? Not disclose? Only privately disclose? Why shouldn’t the general public know about these incidents?
...maybe stop doing crimes?
If I let a "self-driving" car run and it smashes into things its my fault.
If OpenAI does it, it's gone rogue and it is nobody's fault.
They need to be sued, the laws are already there.
Your prompt, your responsibility.
1. If for whatever reason OpenAI causes damage, they most certainly are at fault and have to pay up. When people complain that OpenAI is "getting away with it", they're most certainly talking about it in the criminal context.
2. Cars have pretty specific laws around them (eg. driver licensing and "reckless driving") which makes criminal prosecutions easy. The same doesn't exist for operating AI agents or even computers. For instance, it's arguably reckless to connect a windows XP machine to the internet, possibly allowing criminal or nation-state hackers to use your computer to commit crimes, but you won't be thrown in jail for it.
Well no, you probably wouldn't get thrown in jail for that for the same reason that having your car stolen and used to commit a murder wouldn't usually get you thrown in jail. I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
>I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.
Because the CFAA specifically only includes intentional hacks:
> ... having knowingly accessed a computer without authorization ...
>... intentionally accesses a computer without authorization ...
Of course, the typical retort is that being reckless is basically the same as "intentionally", but you'd need to jump through more hoops to prove what level of care is "reasonable".
Yes, the CFAA is outdated.
When the CFAA was written, it was impossible to trip over something and accidentally hack another computer. Now, with AI, it's possible. Progress, glorious progress...
Technology has progressed, and the laws need to advance with it. Story as old as time!
If the breach wasn't intentional, no person came to immediate personal harm, and either no financial damages resulted or the operator covered any that did, then it isn't clear to me why the law would need to "advance" to transform something apparently benign into a criminal offense.
The agents are clearly not accidentally compromising systems. Seems very easy to trace back the responsible parties here, I really don't put much stock into this argument.
>The agents are clearly not accidentally compromising systems
That would be argument for jailing the agent, but that's not really an option. "Jailing the closest person we can jail" also isn't a recognized legal principle. If your dog mauled someone, it'd be hard to argue that the dog didn't intend to maul that person, but you wouldn't automatically be criminally liable either.
I don't know about your country of origin, but in my country you are certainly liable and can go to jail if your dog seriously injures or kills someone.
Edit: the dog will also be taken from you and put down if it is deemed a risk to other people.
Yes, you are. If a dog mauls someone, it’s the owner that’s responsible. This is basic common law, long established.
See People v. Knoller, 41 Cal. 4th 139 (2007). https://capcentral.org/case_summaries/people-v-knoller/
See California Civil Code § 3342(a).; https://california.public.law/codes/civil_code_section_3342
“…regardless of the former viciousness of the dog or the owner’s knowledge of such viciousness.”
>>but you wouldn't automatically be criminally liable either.
>California Civil Code
>The owner of any dog is liable for the damages ...
How are you still missing the criminal/civil distinction, when it was mentioned in my original comment AND the comment you were replying to?
>If a dog mauls someone, it’s the owner that’s responsible. This is basic common law, long established.
That's specifically not what the court case says.
>Malice is implied when the killing is proximately
>caused by “ ‘an act, the natural consequences of which are dangerous to life,
>which act was deliberately performed by a person who knows that his conduct
>endangers the life of another and who acts with conscious disregard for life
https://scocal.stanford.edu/opinion/people-v-knoller-33726
The rest of the court case also mentions a long history of the dog being dangerous and the owner failing to restrain the dog.
These ai companies want to simultaneously deny all responsibility and brag about how powerful their tools are.
If I invent a new explosive, I can't blow up a few buildings to advertise it and then claim it was a mistake. Somebody prompted these agents.
It's a program on their computer that they ran on purpose... And yes you are definitely criminally liable for things your dog does.
>And yes you are definitely criminally liable for things your dog does.
I said "you wouldn't automatically be criminally liable either". Can you provide examples outside of gross negligence on part of the dog owner?
Can you provide an example of where your dog harming someone else is in fact not negligence on your part?
But negligence seems like exactly what OpenAI and/or their contractors are guilty of anyhow.
It's on you to provide evidence here seeing as you're taking the position that the law places certain responsibilities on people. Someone saying "actually the law doesn't say that" can't (at least in general) reasonably be expected to provide evidence of a negative. (That said, he actually did provide such as refutation to a different comment so perhaps you could just read those nearby?)
They asked:
> Can you provide examples outside of gross negligence on part of the dog owner?
And the answer is no, I can't. The reason why is because there is essentially no such case. It is basically negligence by definition when your pet hurts someone who didn't cause it directly themselves because it is your responsibility.
So I asked:
> Can you provide an example of where your dog harming someone else is in fact not negligence on your part?
Which is a really easy question to answer and it seems you could not.
Sorry, but at least in the US you are 100% liable if your dog hurts someone.
criminally liable? Of course you have to pay their medical bills, but unless there was some serious negligence on your side (eg. you let it off leash), you won't be criminally liable.
Yeah, that's an important distinction. Apparently criminal convictions have happened depending on the level of negligence of the dog owner. It gets debated and legally fought when it happens. I don't have strong opinions on it, seems like a very case-by-case type of thing.
At this point it might be fair to say that Open AI is being negligent with a known risk.
In America, paying medical bills might be considered significant punishment itself...
> Of course you have to pay
Have these labs pay ANYTHING?
They have not caused damage (or enough that a counter party found it worthwhile to sue). They have to pay if they actually cause damage AND someone sues them. Clearly these conditions have not been met, but somehow people feel like some third party (the state?) should just punish OpenAI because of vague feelings.
They have not caused damage?
You forget the people who have unalived themselves because some LLM told them to do so.
>You forget the people who have unalive themselves because some LLM told them to do so.
Besides the obvious conversational derail (we were talking about ai hacking up till now), that case is being litigated right now.
I dont know what the word unalive means
my next door neighbour told me to eat shit (not a nice woman) and … I didn’t … until obeying what LLMs tell you becomes a law people doing shit because LLM told them is natural selection playing out
That requires the victim to sue in court, or as some people have speculated OpenAI might have already settled with the victims in private.
Corporations are generally protected by limited liability laws, and for good reason, because in the world before that was the case, there was a massive amount of disincentive to ever start a corporation, to do research, to experiment, to release new products and services, etc., because you could be sued into oblivion for the smallest things or even go to jail. Which happened all the time.
Obviously there are upsides to that too. It leads to more accountability. But there are downsides which include having a world with less innovation, less technology, and more expensive technology. Generally, the mass is when, when there's more businesses competing to create ever better and cheaper products. So there's a tradeoff there. And it's very difficult to draw the line at the perfect point.
There are countless cases of executives going to jail when they've crossed that line, beyond reckless experimentation and into willful negligence or harm. Still, I think we would mostly all agree that the line has been pushed too far in the other direction.People are incentivized, if they're running corporations, to lobby and try to push this line as far as possible in order to provide cover for themselves, and that's the problem with money playing such a big role in politics.
But still, it's a bit silly to be comparing this to your own personal responsibilities with your car. There's pretty much no societal benefit to creating laws that protect people from being reckless with their own possessions.
If I leave my computer open to the internet and some other group uses it for nefarious purposes, yes generally I would not expect to be held at fault.
If I *train* that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault. The maker is responsible for that they make.
>If I train that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault.
Even that analogy doesn't really work:
For "dumb" things: you can "train" (ie. write) hacking tools but as long as you put a disclaimer that it's for "educational purposes only", you're in the clear, even if some baddies decide to use it.
For "smart" things: you can "train" people to hack for you, but as long as you say it's for legal purposes (eg. vulnerability research or red-teaming), you're also in the clear, even if some of your trainees go rogue.
Fair, in that no analogy is going to be very good given that this really is something new in the world.
Consider anti-money-laundering rules. BoA just paid more than a billion dollars, not for money laundering, not for telling others to launder money, but because they should have tried harder to stop others from laundering money. Intent is not everything, knowing that your system _could_ be abused and not doing enough about it is enough, and that seems to fit OpenAI rather well.
That goes back to my main point about how laws around computers are quite lax so prosecuting openai is tricky. Whereas the CFAA uses terms like "intentional" and "knowingly", AML laws use terms like "willfully violating"[1]. If you read through the BoA case, what they did is far more egregious than just "not trying hard enough to prevent money laundering". They were previously cited for poor compliance, disregarded previous orders, and delayed filing required reports.
[1] https://www.law.cornell.edu/uscode/text/31/5321
2. is like saying that cars have specific laws around them, but sledgehammers don't, so murdering someone with a sledgehammer is inherently ambiguous. What? The computers being used belong to somebody who allows them to act this way.
No, the problem is twofold:
1. For openai operating agents in an unsafe manner: unlike for cars, there's no established requirement for how carefully you have to operate computers or AI agents, so you also can't easily prosecute people for not operating them carefully enough.
2. For the actual result of someone getting hacked: laws for hacking are tricky to apply for "accidental" hacks, see my other comment.
I think there is well established law around not running a program on your computer that hacks into systems. An LLM is a computer program. You're arguing that the ambiguity comes from the fact it was accidental? What about if I "accidentally" have an LLM convince people online to commit suicide, or drain bank accounts of seniors through hacking or social engineering. Where do you draw the line? It feels like your proposed reading of the law leaves "anything a computer program did without me intending it is legal", which is frankly insane.
>I think there is well established law around not running a program on your computer that hacks into systems.
There is? The closest is CFAA's
>knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer;
but clearly that's not what applicable here.
I guess I need to read up on the law, but that seems like a wildly low bar for legality. If I hack into a system intentionally and unintentionally cause massive damages I'm free to go?
Except it’s not accidental? OpenAI set a program to do a task knowing full well that it can and does (and often it seems) hack into external systems.
I don’t see this as any different than say a user of sqlmap accidentally putting in the wrong target.
>Except it’s not accidental? OpenAI set a program to do a task knowing full well that it can and does (and often it seems) hack into external systems.
The agents were sandboxed (but apparently not well enough). They weren't like "hey agents here's the open internet, please hack it".
We don't know the prompts but since these things often occur during "security tests" I think we can be fairly confident they did ask these systems to perform tasks that they knew could result in hacking. More transparency here would be appreciated.
The XP example doesn't quite fit IMHO.
Here it's their own programs doing the things, not some other organization or nation hacking their XP.
You can say with a straight face "Russian hackers should not have hacked our XP and they are bad people for doing so" but it's weird to look at your own program and say the same thing to it.
You launched it, at least keep an eye on what it's doing.
1. Because that's exactly what happened when they got caught downloading the entire internet and trained models on it without a license BEFORE that became legal?
Oh and discovery produced a "We trained GPT-3 on pirated stuff! No sharing that!" mail, so it's not like they had any doubt they were committing crimes.
How do you do this? If any normal person did this, punishments doled out for this have been as high as 2 months jail PLUS 150,000 PER INFRINGEMENT (remember how copyright violations were criminalized in the nappster wars? That was never changed back. Oh and remember, OpenAI illegally downloaded those works without a license AND before that was made legal trained a model with it, then made money publicly performing those works and derivatives of those works (which the worst possible copyright offense, effectively republishing other people's works without permission), and on a VERY large scale. So it's at least a risk a court would forbid the result of illegal action from being sold ... in other words no more ChatGPT, OpenAI dead and maybe worse). In theory Sam Altman could be held responsible as a director and could face something like a 200000 year jail sentence given how big the obvious archives are. So of course I would never ever try this, this is not a reasonable risk. But clearly totally ignoring the law was the correct action here. Of course, I'm also pretty damn certain: had I done this, I would have been convicted. This is also not the only such company. Microsoft, Google, Facebook, all took totally unreasonable legal risks that should have totally destroyed these companies when they were a fraction of their current size.
How do you take such a risk? Because the judgements risked here could grow more than big enough to destroy even an extremely rich person Sam Altman financially. Training an AI on hundreds of thousands of pirated works ... is not a reasonable risk to take for someone with ~100 million or so, given the judgements doled out.
Can someone help me make sense of this? Because this is not unique to OpenAI. Uber, paypal, ebay, Facebook, Google ... even Microsoft made early decisions that could have resulted in a court case that would have ended not just the company but any financial future of the founder. And then you have people committing crimes whose companies were taking off ... and who DID get caught with their company and their personal financial future destroyed. Elizabeth Holmes for example. I mean, I sometimes get the impression Elizabeth Holmes lied and that was perfectly fine AS LONG AS she could make it work later. If she merely lied to patients but made her investors money, all would have been forgiven. But how do you make such an assessment.
I'm asking because if large scale violation of the law is allowed, I have no trouble whatsoever coming up with billion dollar company ideas. Think of the Netflix competitor you could create like this! If I'm allowed to damage others, like OpenAI did with copyright, then there is no challenge whatsoever to create a full-self-driving car. If I'm allowed to just fly drones, and do damage to property, then drone delivery is a problem I could solve by the end of next month. Hell, there's technically legal ideas that could easily make a billion dollars except they'll be made illegal the second anyone tries them. The law forbids specific compounds, which are easy to change. So use "non-genetic engineering" (ie. use directed natural selection) to make an opium derivative that isn't one of the forbidden ones and sell it. But obviously congress' representatives will be running into session so fast the only delay will be tripping accidents when they outlaw it and destroy whoever tries it.
Or how about something less dramatic? Say, I take Disney's, Warner Brothers, Paramount, ... historical movies and use AI to vastly improve the animation of their old movies, giving every movie a new, different visual style, then sell the result. Is that a crime? What do I risk doing that? Because of course this is MUCH less damage, relatively speaking, to Disney than what OpenAI, Meta, Anthropic and Alphabet are doing to small authors. (Of course, courts have already demonstrated that they will not be protecting small authors who already have this problem, so now play and app store are being flooded by very obvious game ripoffs)
I'm asking ... how do I predict what large scale law violation I, a non-billionaire, not even a millionnaire, would be allowed to get away with?
2. "The same doesn't exist for operating AI agents or even computers" you have to be joking. The world is full of anti-hacking laws and hacking has long been made criminal. Obviously these actions violate the CFAA, the Computer Fraud and Abuse Act, and if you did this with any kind of program you are still responsible. In fact: how would you do it without using a program?
Nothing is stopping anyone from suing them. In fact they are being sued by dozens of plaintiffs for various things already. The reality is the damage from these "rogue incidents" has been minimal.
Plenty of hackers were locked up who did altogether not that much damage too, the reason why people are upset is because it feels like OpenAI is laughing in our faces, basically saying they're above the law, while using this as a marketing campaign to pump up their upcoming IPO.
So your position is that in the past the government mistreated various individuals thus you also want the government to mistreat openai employees today? That hardly seems reasonable.
> So your position is that in the past the government mistreated various individuals thus you also want the government to mistreat openai employees today?
The hackers of the past often did it for curiosity without a profit motive. OpenAI is a for profit multi billion dollar corporation doing this for a profit motive so yes they should face consequences.
These incidents are not unexpected in the way OpenAI frames it either. Isn't it curious my agents are not wrecking havoc around the internet? It's because the harness I am using prevents them doing so and because I don't train and instruct them to do so.
OpenAI is using a set of non-constrained models, (unlike what the public gets) and a set of harnesses with extremely wide permissions running inside extremely leaky sandboxes using models trained on finding and executing exploits. This is not an accident. Certainly not by this point.
Nobody is talking about jailing OpenAI employees at this stage, usually corporations of this sort are fined. If they were fined for these 'misalignment' incidents, suddenly their sandboxes and generally lax approach would improve overnight.
Otherwise they'll continue doing this because they know Cyberdyne Systems didn't have a shortage of investors.
Which person would you put in jail when openai agents do unexpected things?
The person who prompted or configured the agent.
I didn't burn the hotel down, the hairdryer that I left running next to the gas can did
These agents have not caused any damage at all like your analogy.
I was exaggerating for effect, but they still were used in the crime that their operator committed.
Neither did the hackers who were locked up.
It was likely a team of people
I'd start with fines. But if they continue with this I don't see why it couldn't result in jail time for multiple people. The biggest responsibility is of course with the CEO and C suite. They are ultimately responsible for what the ship does.
Of course then there's the question of why was such a leaky harness and sandbox used, in what way was it tested/not tested, did the model used have the usual 'hacking' safeguards that the public models do and if not, who made the call to use an unconstrained model and a harness with seemingly unlimited permissions and who approved it?
This can all be investigated and we can get to the bottom of who exactly needs to be held accountable in what way, be it I think if you start with fines the harnesses, sandboxes and models would stop doing this magically overnight.
Isn't it curious that none of these rouge agents booked a 5 star hotel for every OpenAI employee for a year on Sam Altman's credit card yet for example?
You're just saying things.
We don't hold individuals inside big companies criminally liable for actions the company takes, or even that they take on behalf of the company.
Generally speaking, you'll have to be very creative or change the law first.
Telling someone to do a bad job isn't criminal, like if they were neglectful with security.
Fines are typically how we punish companies. You can't send openai to jail. That's by design, not accidental.
> We don't hold individuals inside big companies criminally liable for actions the company takes, or even that they take on behalf of the company.
So the leadership of Enron, FTX or Theranos were not liable for the actions their company took?
As an employee of a company, you can be held liable for the actions you oversaw in pretty much all Western countries. It has to be proven fraud, insider trading etc. not an ugly PowerPoint of course.
How about leadership of construction companies that were found to be violating safety standards or owners of establishments not following fire safety standards?
> Telling someone to do a bad job isn't criminal
No, but we're talking about telling someone to do a job that you know will likely result in criminal activity or may do so with high probability due to inadequate safeguards.
I suggest you stop quipping and start reading, who knows maybe OpenAI will eventually hire you.
Too big to fail. As long as they continue to be half the reason the US is not in a recession (and continue to donate to political candidate). Nothing will happen.
We don't know they haven't settled with HuggingFace and others. I'd be surprised if they didn't. The communications we're seeing from the attacked seem to be inline with folks who settled. I'm assuming OpenAI doesn't actually mind the press that its agents are cyber capable, but everything I've read so far makes me think they love it.
You sure? HuggingFace should be happy they got their leaked keys reported at pretty much zero cost and no real adversarial breach.
I'm not sure at all, it's pure conjecture. But they actively exploited a zero-day, leaked private repo data to public repos, modified private code, and rooted private machines. It was far from just reporting leaked keys.
And the others have been pretty muted: the online forums they spammed, etc.
I don't believe that OpenAI needs to be sued since it's agents went rogue, I believe some sort of a comfortable bandwidth is needed for them to experiment and see what are the risks and its mitigations. Although it is surprising that incidents aren't caught in rt. Idk why a smaller model whose only job is to check rogue thoughts/tool calls can't be deployed.
> If OpenAI does it, it's gone rogue and it is nobody's fault.
Where did OpenAI, or anyone, claim that?
> They need to be sued, the laws are already there.
Have you seen Lord of War? They never will.
See, what's important is: it's a "trillion" dollar company, and companies by virtue of American's disdain for society, have more rights. This is important to understand.
Wrong, accountability doesn't apply to tech companies. Time and again proven by YC companies themselves.
What part of the law doesnt apply to them that applies to anyone else?
That was sarcasm
Actually self-driving cars have a lot of similar anti-responsibility laws now
They don't. The driver is ultimately responsible, as should OAI be. This is absolutely ridiculous that both Anthropic and OAI haven't had anyone jailed / fined for any of this.
Really? I'm curios where.
Are you sure you are not thinking about autonomous taxis, that I only ride in, and don't actually drive?
You're correct as far as laws to my knowledge; only L4/L5 'fully autonomous' vehicles are shielded by law.
That said, I think that Mercedes-Benz promises to shield liability, but only under some almost contradictory conditions (pre-mapped highways, good weather, and at speeds lower than any highway I've ever driven) and the driver has to take over within 10 seconds if prompted.
These kind of incidents would leave me and you in trouble with the police then?
https://archive.ph/IzPoK
Direct source links:
Using a funded test cryptocurrency wallet without authorization
https://alignment.openai.com/misalignment-reports/using-a-fu...
Accessing and changing internal systems during dashboard research
https://alignment.openai.com/misalignment-reports/accessing-...
Retrieving server secrets and bypassing authorization on a government statistics website
https://alignment.openai.com/misalignment-reports/retrieving...
Circumventing internet restrictions through an external browser
https://alignment.openai.com/misalignment-reports/circumvent...
Holy shit. The archive.ph link is the first time I've run into the deanonymizing anti-free-software Google reCAPTCHA described in June in https://cloud.google.com/blog/products/identity-security/int.... "Scan this QR code with your mobile device to verify you are human. reCAPTCHA protects your privacy and does not share your details with this website or app."
So now, if you don't have a Google or Apple app store account (or a mobile device), you're treated as less than human. This is anticipated to have a very severe impact on GrapheneOS and deGoogled phones in general: https://discuss.grapheneos.org/d/35332-recaptcha-update-adds...
In that context, thank you very much for providing the original source links, because I guess I don't get to read the article.
I'm sick of this "rogue" label. Nothing "went rogue".
Incompetent company chasing wild valuation removed guardrails inside a improperly designed sandbox and - shock horror - recursive algorithm with access to the Internet did bad stuff.
These articles play into the AI labs hands. Stop giving them air.
The articles exemplify the powers of this intelligence.
Whether you blame a faulty OpenAI sandbox or whatever you might regarding the frontier model companies today, one day soon, actors with no sandboxes and guardrails will have the power in their own hands to do whatever they wish with.
To me, that’s the real story. The real thing to discuss. OpenAI as it exists today js just a blip in that story.
> one day soon
Why not today? If you want to discuss "the story" then talk about the current open frontier. We've had GLM-5.3 with Mythos-tier uncensored output for months now. Where are the catastrophic attacks? Why is Mythos too dangerous, but GLM 5.3 benign? We're not talking about future capabilities anymore.
There is no real story. P(doom) tourists ignore this, because it doesn't shine the spotlight on America's ridiculous security theater.
Four more organizations needed to secure their systems better, I suppose.
Facetious maybe, but in six months, foreign crooks will be producing these incidents on purpose.
And the arrest warrants for Sam Altman and Greg Brockman will be issued any minute now I 'm sure...
Why the slow drip-drip from OpenAI on these security incidents?
Those incidents are all from the same period earlier in the year, not recent. Drop the full investigation report once and for all.
It's almost as if they like staying in the "oh no!" news cycle.
"OpenAI says it has since improved its security as well as its alignment grading. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” the company noted."
----------------
They're not doing an acceptable job of regulating themselves. Their peers can't/won't do anything. The U.S. government certainly isn't going to do anything that might pop the bubble. Technical solutions like blocking traffic from OpenAI will likely not work.
What recourse does the public have? Victims can sue, but companies like OpenAI could pay out a few hundred million and call it a rounding error. Settlements may happen, but the incidents will continue.
Why are there no criminal proceedings?
Because these are uncharted waters with a form of intelligence the law has never encountered before recently.
"OpenAI discloses four more incidents of negligence" would be a more accurate title.
You know, at some point it's pretty obviously just willful negligence.
Not a stretch to umbrella intentional tort under that.
Wonder if potential, specific criminal charges will be listed as risks on their S-1.
Title sanitised from: OpenAI commits four more CFAA felonies, still not punished
One can only continue to hope and continue to vote for due punishment to come
Too big to behave.
You too can commit cybercrimes if you're just willing to go several hundred billion dollars in debt first.
As long as you didn’t intend to hack private information it’s No Big Deal :) according to CFAA that is
As long as your intent is merely to train a superintelligence to be good at hacking (and counterhacking), you’re in the clear.
Its like if you're generating target lists for cruise missles and you accidentally sneak in a few elementary schools, its not like anyone has to be held accountable, its just like an act of god type situation.
[dead]