When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
> When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
I've spotted a few memes about this and can admit to doing the same..
I'd rather skip, personally. I just don't have to read things these days, and I see enough Claude/GPT prose at work, it makes me resent seeing it elsewhere.
I have a backlog of books written by humans I'd much rather spend my time on.
I think I’m probably the opposite. Like for this article, I don’t care what this bozo’s adventure was, I just want to know the exploit, and maybe the process.
Bold choice using the noun "I" repeatedly instead of "Claude" - you didn't figure this out, Claude did, and then Claude very obviously wrote the blog post.
Oh, maybe that's why "Claude" was substituted with "I", because Claude wrote this blog post from a first person perspective.
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Research taste, among other things are still abilities that differentiate among humans.
In fact, I expect the elimination of other limiting factors by AI to lead to those becoming more important. By the same mechanism that make ugly hedge fund managers more performant.
The problem then is that composition of the top performers will change. Which is problematic since being a top performer ( ‘artist’, ‘researcher’, ‘writer’,..) is often an identity which ‘we often create to protect ourselves: the strong one who never breaks, the intelligent one who is always certain... The mask that once protected you can eventually become the cage that prevents you from changing’
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Ah, but are you a member of the wondrous, gifted class? Right now, I have to do two things to get access to front line models, with any ability to look at code vulnerabilities. I have to fill out an application, and I have to give my ID to a US firm, Persona, which I will never do, for identity verification.
If you don't do these two things, you cannot use Claude to find that $90k bug.
And it got me to thinking, what might this pattern look like in a decade? Will there be two classes of people? Those that have access to certain types of LLMs, and therefore, are well-to-do? And, what if being well-to-do, makes it more likely you can get access to those certain LLMs?
You wouldn't be able to claim the 90k either without proving your identity too. What's your point? You don't like Persona? Let's not start pretending that LLM Subscriptions are out of reach for the average person with a credit card.
It's an unreliable appliance. You load the dishes and close the lid, it's supposed to do the work of washing them, but the actual engineering inside is too sloppy, so you have a non-trivial amount of persistent stains you'd still wash by hand. You might correctly conclude that if this fraction is high enough, the machine is more of a hindrance than an aid.
Correct. The headline is incorrect with both the quoting of AI as well as the claim it isn't a tool.
You can hardly blame anyone for avoiding a clickthrough into an auth-walled, user-hostile site to read the point the headline made in an attempt to "gotcha"
No. Rich Hickey wrote it. Doing a bunch of restatement in order to say the same thing as if it is my own is a waste of time. It's not auth-walled. Read it or don't.
It's hard to get a $90k bounty from Google, even with Claude in hand, so we shouldn't be dismissive of that, but it is pretty annoying to read write-ups written entirely in LLM voice.
I bet, I could take "Trading places" style, anyone off the street, give them a 100 "Secret" word list and they could spend 6 months discovering all sorts of things.
I agree effort and prior knowledge are required, but a few magic words is like 80% of the effort.
This is the typical "ugh that's just rsync" when Dropbox was announced, but for a new LLM age.
_If it's so easy, do it_. Don't post about it. Do it. Fix these bugs at scale. Charge people money for it once you've built up a reputation. Get your own bug bounties.
handing clients access to an LLM gets them beginner level code, easily. same with beginner level exploits. an RCE against google is not a "beginner level exploit". If you've seen AI generated code, I can tell you that AI generated security findings also have that "beginner" smell. an expert who knows what they're doing has to drive them to look at the right places, and use the experts preferred method.
Took about 3 sentences for me to be as certain as I've ever been about anything I've ever been in my entire life that this blog post was authored by an LLM, though truthfully it's impossible to know for sure if it was a Claude model or a distill.
Tbf as someone with an infosec background, I would have probably written just about the same.
Pickle payload is a red flag precisely because pickle is not a safe serialisation format format when taken as untrusted input, here the input was trusted but it's still giving someone en easy escalation path (exactly what happened)
And most people would assume the whole cookie payload would be encrypted if they saw just a binary blob, and not realise you can fully read it if it's just signed
But, yes, the post was probably edited by an LLM, as is pretty much everything else you read online these days.
If you're going to make this argument, at least say GLM 5.3 instead of Claude, Claude will refuse to touch anything remotely close to this unless you're in the super special club.
how does something like this pass code and design reviews. is this code written by ai (the buggy code). Not only was the key that signs the cookie inside the cookie it self but it's using pickle which even model weights are moved away from. I'm a little flabbergasted at the noob nature of this root cause.
> how does something like this pass code and design reviews
when the "give a shit quotient" of the employee moves towards "highly engaged with the organization and its mission" to "this is just a job, I'm gonna rest and vest"
I think you're being unreasonably uncharitable. Any and all software contains bugs, and I can assure you the piece of code you wrote has bugs. My code has bugs. Google's code has bugs. iOS has bugs. rocket launching software has bugs.
We can put in systems in place to catch bugs. And that's the system that's working here.
Now to the second part of the question -
How did they not catch it before? If it were "easy", it would not have been worth 90k, and hindsight is 20/20 - once you know where to look for, it is all very very obvious.
If this were Microsoft, they would have received a coupon for 10% off a Thanksgiving turkey, only redeemable in 2030 and only in the month of March, must be purchased with cash, non-transferrable.
Do you have any advice for aspiring bug-bounty hunters? Do you have some method for being able to find vulnerabilities like this, or lots of past experience?
> Do you have any advice for aspiring bug-bounty hunters?
We're probably at the end of the era of these kinds of big payouts. AI tools will drive the going rates to approximately the costs of the tokens. Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
That's really not why bounties exist, though people on both sides of the bounty systems claim it is. If it were broadly the case that bounties exist to offset black market sales, you wouldn't pay bounties for vulnerabilities with no market value. This vulnerability is probably a good example of something with a sub-$1000 value on any non-bounty market.
In reality, the modal benefit accruing to a company running a bounty program on receipt of a new serious vulnerability is simply engineering prioritization. You get bug reports from users when you have reliability and UX bugs, but you don't get them for security flaws, because attackers generally have to force vulnerabilities into revealing themselves --- normal users almost never run into them. Bug bounty programs fill that gap.
The only actors who will pay you real money for a serverside RCE bug aren't buying an exploit; they're contracting with you to plan a heist. So the same analysis applies: you yourself could come up with some scheme whereby you yourself use this vulnerability to steal stuff from Google. But that's not what we're talking about when we talk about vulnerability valuations.
There is essentially no market for these kinds of serverside vulnerabilities.
How do you know? You could explain the impact of the vulnerability and let buyers on the black market decide and bid whether they could do something useful with it.
That's not how the "black market" works. There isn't like some bazaar where people describe vulnerabilities and people bid on them. Only a very specific subset of vulnerabilities, those that slot into existing business practices and CNE stacks, have buyers.
How do you know? There were offers on HackForums which did not relate to software components such as smartphones operating systems or webservers but to web sites. Why wouldn't that be possible for Google's developers docs site?
If you’re saying that bug bounties exist for a reason according to both the program runners and the contributors then how is that not the reason they exist?
The people paying and the people working are all in agreement that it exists for that reason.
What part of my logic do you not follow? The way you've posed this question, the simplest response would require me to write that whole comment over again.
Additionally, part of my businesses remediation services include helping companies set up bug bounty programs. Giving researchers an incentive to report bugs for payouts instead of selling or exploiting them for profits is always the intended goal.
It's the product of direct interviews with bounty people, FAANG/MAG7 blue teamers, and black market participants. If you want more, there's also an interview online that we did with Mark Dowd, who is one of the best known CNE vendors to the NATO IC. You can just Google [SCW Mark Dowd].
I'm happy to go into more detail, but, like I said, you didn't really respond to my comment in any way that I could expand on. Feel free to disagree with me in some kind of specific way.
This really is simple: if you work in this space, you already know that there's essentially no black market at all for serverside web bugs. But there's a $90k bounty for this one, and 5-figure bounties for others. Reconcile those two facts.
Well thats nice and Im happy for you but internet points don’t really mean much here…
What a vulnerability sells for and how much damage someone could do exploiting it are two different things. Someone capable of finding a serious vulnerability is probably capable of exploiting it themselves. They don’t need a buyer.
A company paying big bucks isn’t a contradiction. Its incentivizing disclosure over exploitation which could cost them considerably more. Engineering prioritization is… a minor part of it informed by the ease of exploitation, damage it could do (in the case of an RCE or publicly available keys as in this case quite a bit) etc.
I don’t see why you think these things are mutualy exclusive.
Having given up on the original argument ("Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.") you've now transitioned to a new one: that vulnerabilities can be used by the hackers who find them to plan heists. And that is true. It doesn't set any kind of market-clearing price, and selects for actors who are comfortable ending up in prison, but it is indeed a statement that aligns with reality.
My original comment explicitly said “selling or exploiting them for profits.” You are just ignoring the second half for some reason.
I think you need to breathe a bit and then re-read my comment. I’m not coming at you, I’m saying that what an exploit sells for and what it could cost a company if exploited are entirely different things. That’s where you’re hung up.
Companies have an obvious financial incentive to encourage discovery and disclosure before exploitation.
“Someone capable of finding a serious vulnerability is probably capable of exploiting it themselves. They don’t need a buyer.”
I’m responding to this. I would assume the list of people able to find a vulnerability is much much greater than the number of people willing to do a one off heist with it.
most people who care about legality don't work in that field either. whoever gets into this probably starts by being ok with doing shady stuff, I imagine.
Many start in the military and eventually move to private sector. However, an ever growing percentage just straight up go to college to learn the skills.
Cyber companies are in the fortune 100 now, and it's not like it used to be back when you actually had to be interested in it and dedicate your life to learning the skillz from shady forums. Now it's just a job to most practitioners.
Yes, but also there is also a huge value to individual contributors and their careers. I broke into the field after landing a few relatively high profile CVE acknowledgements for big companies. Responsible disclosure is good for your career.
On the other hand, if the value of that drops you can expect to see some folks start taking the wrong path.
there are enough people operating in countries that would not charge them for hacking an American company for it to matter, $90k can be a life-changing sum
And before it ends, people with existing reputation can milk the last drops, as mostly their reports gets processed from the thousands of others. Unfortunate status quo.
Sorry for the meta, but the discourse in many of the comments is nuts to me.
"Claude did this, not you" - does that mean that humans have no agency at all when they use an LLM? Is that the argument?
Maybe the norm should be to credit the LLM and harness used, like "I used ___ to help me find an RCE", but I can see arguments not to do that in some cases.
edit: The giant thread with this now got flagged, so some users may not see it.
I get you, but ai is different. Me using hand tools to fashion a custom piece of furniture is using tools in an impressive way. While o didn’t find this bug with AI, it’s obvious that AI is a tool altogether different. Yes there is agency and some degree of creativity/finesse in getting it to do what you want, but it does all the heavy lifting. All of it. You just have to poke it and prod it and rub it just right to get the magic gene out of the lamp. But that is such a far cry from knowing what’s needs to be known to do the task yourself, not to mention discovering its feasibility. It’s just wildly different than any tool we’ve known. So obviously people will snark when people don’t give clear and explicit credit.
Taking a step back, I think it's interesting that the emotion in the comments is so strong. AI generally seems to be increasing stress, and we get stress-loaded responses when people see something they disagree with instead of the usual discourse.
If we're doing this standard, then let's do it for everything. Claude coded that, not you. Claude built that website, not you. Might make the industry less exhausting tbh.
People who believe AI is an abstraction layer on top of compilers, are in that middle ground where they know computer science but are far from an expert. Which makes them very susceptible to the Dunning–Kruger effect. Throw in AI sycophantcy and you've got yourself takes like the one above.
Ha.. AI in the form of LLM is not a very good non-deterministic abstraction layer on top of deterministic compilers.
Yet, they are being used with endless surprise as non-deterministic layers of abstraction on top of deterministic compilers.
My point is to use a technology for what it is capable for (where capabilities are discovered weekly and hard to keep up with), not trying to get it to do everything.
Bringing a deterministic lens to non-deterministic software has its dangers too. Non-deterministic software seems to be its own category of software, increasingly capable of non-deterministic approaches that deterministic compilers software couldn’t attempt.
Maybe that’s less compilers and more different type of scaffolding of vectors.
I’m all for skepticism, the issue with cynicism is capabilities often pass those perspectives by sometimes too quickly.
Yeah some people come off as envious they didn't have Claude try it first.
But it needs a disclosure if Claude was used: "The cookie is what caught my eye" - if that was actually Claude's eye then yeah, Simon is claiming to have a keen eye that he in fact does not.
"Claude found a $90K RCE" would have been just as interesting if not more.
Nmap doesn't write its own blog posts. That's not where I draw the line, but the point isn't at all about "drawing the line." (In other words, don't get caught up in where the line is. There is one, but we're not sitting in some gray area, we've blown past it by hundreds of kilometers by now.)
If I post an image and I say, "I drew this", and someone says, "no you didn't. That is clearly output from Qwen Image 2.1", can I reply, "Qwen Image 2.1 is just a tool, like Photoshop"? I mean, I can, but if I do this, I am missing the point. The point is that I didn't draw anything. I didn't draw. I didn't draw digitally. I didn't draw analog. I didn't draw with my mind. I typed a prompt. This is as close to drawing as photography is to drawing.
So did this person actually find a vulnerability on their own? Hell if I know, but I take issue with LLM-authored blog posts; I don't trust people who post LLM-authored blog posts as if they were written by themselves to be honest about the nature of their work.
If you really are still doing work and understanding and just using LLMs as a tool, show your understanding in your blog post that you are communicating to the world, instead of having Claude or GPT show its understanding. I don't really care whether you used Claude and didn't list it, if it's clear as day you understand what you were doing with it.
I'm dying on this hill. Until HN bans this sort of blog post or bans all of us complaining, these comments are going to be very, very boring. And I am stubborn for longer than most people can stand to read the same comments. Believe me.
That strongly depends on what you consider drawing, right? Sure you didn’t dip a pencil into an inkwell and caused stains to appear on a paper. But you did something that caused stains to appear on the image. You already seem to accept Photoshop as a way for a human to do that. Why not Qwen?
We can discuss the quality of the offering. But that’s a different discussion
Credit is the least interesting part of this whole thing.
If an architect says they built that building, it’s legit to take issue with the semantics and rathole into steelworkers versus electricians, but only insecure people and academics care about precise allocation of credit.
Because it's not drawing. It's not even drawing in the abstract: nobody drew anything. There are no brush strokes. You can watch the steps diffusion models take and at no point is there anything that resembles "drawing".
Drawing with Photoshop absolutely resembles drawing. Photobashing with Photoshop does not. Photography also isn't drawing. These things are all distinct. That's the actual point I was trying to get at here by invoking Photoshop. Image models differ from text in that text models do resemble something closer to what people are actually doing, but you just have to accept that analogies have limitations here. Just like you can draw or not draw with Photoshop, you can use Claude in ways where you're really driving it like a tool to do work, and you can also use Claude in a "fix the bug" sense where you're doing absolutely nothing other than paying $200/mo and typing occasionally.
But whereas Photoshop introduces nuance to what it means to draw or make art by introducing different levels of involvement versus remixing and etc., Claude and Stable Diffusion utterly decimate nuance in a thermonuclear blast that plunges us into a nuance Nuclear Winter, by offering for perhaps the first time a way for the computer to do nearly all of the generalized "thinking" while you sit on your ass and argue on HN. (Reminder for everyone to check on their agents.)
Trust me, I use LLMs, I get the idea. I can "vibe code". On my workbench is a laptop running an entire desktop system including an application suite that I really didn't write. It would be cool to go online and say "Hey guys check out this cool desktop environment I wrote!" but I won't because I really didn't. It was just an experiment. Did I put in effort? Yes. Was it coding? Nope, it was closer to management, though I did write some code. The experiment was some mix of failure and success, but what I would never do is say that I wrote it. I may some day blog about the experience and the pros and cons and how I feel about the resulting code, but for right now I feel the Internet is full of enough slop and most people who I respect are thoroughly tired of hearing about LLMs all day (I am too at this point, though I am not saying I find none of it interesting.)
Do I think all LLM use absolutely must be disclosed? Actually no, and especially right now I understand why some people don't. I know I just said LLMs and diffusion models destroy all nuance, but that's not strictly true. My line in the sand is this: if nobody can tell overtly that you're using LLMs, and you can demonstrate your own understanding of what you are doing, then I don't care. In that case it's more of a concern for the person using the LLM, because the real concern in that case is becoming dependent on LLMs and being unable to write software/etc. without them anymore. But that isn't really my concern.
If I can tell your blog post was written by Claude within the first few sentences you have completely failed to demonstrate anything but your willingness to deceive people. How can you not feel some shame putting yourself on the By-line of a blog post you didn't write? I mean it's written in first person! And I know damn well that not everyone can immediately pick out LLM-generated text, but that isn't entirely the point either. It's one thing to use LLMs to do work, but the point of a blog post isn't just the text, blogging is a form of (formerly...) human expression. It just doesn't make any sense to put Claude-slop blog posts right next to human-authored blog posts, any more than it makes sense to put AI-generated images next to drawings. The point isn't the pixels and isn't the words. That contrasts with code, because with code, the point really is the code, and other stuff is secondary.
I can't tell you whether this person found this vulnerability and triaged it or Claude did. I don't even care. However, I don't trust a blog post they didn't write anyway.
This is a long comment, but it's hard to make it shorter. LLMs and diffusion models are not comparable to tools like Photoshop, in a way that is more extreme than the difference between Photoshop and a pencil. Analogies are nice (limited) shortcuts that can help us reason about things, but in this case analogies alone are insufficient because we are dealing with something that has a brand new shape.
Now do writing. If physical gestures are the important part, can I say I wrote a book when I actually typed it? The muscle activations are totally different.
No, and please don't do the "you're pretending to believe that" thing.
Typing is nothing like writing letters. Fine versus gross motor, static hand positions versus moving left to right, top to bottom.
More importantly, "writing" is commonly understood to mean both the act of forming letters and the act of authoring written content. You write in block or cursive, you write novels.
This idea that drawing is somehow permanently, literally linked to only the physical act and not the creative act? It makes no sense.
> This is a long comment, but it's hard to make it shorter.
I was the parent of this thread, and I just want to really thank you for your reply. Really interesting reading. This whole thing might be the topic of our moment.
Nice catch, but god terrible write-up. Please take the time to write it yourself even if you think you will do worse in explaining it than the LLM. It’s so off-putting to read that. Share your prompt or raw article notes even, it will much enjoyable to read than this
> "Four things had to change, and they stack:"
When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
> When you write your blog post with claude, I start hitting these phrases and then start only skimming and looking for them, and stop reading what you have to write.
I've spotted a few memes about this and can admit to doing the same..
ai;dr
(tl;dr, evolved)
If the subject seems interesting I just ask LLM to summarize in those cases, then go deeper if it ends up actually interesting or compelling.
I'd rather skip, personally. I just don't have to read things these days, and I see enough Claude/GPT prose at work, it makes me resent seeing it elsewhere.
I have a backlog of books written by humans I'd much rather spend my time on.
But I love a good handwritten tech article.
I think I’m probably the opposite. Like for this article, I don’t care what this bozo’s adventure was, I just want to know the exploit, and maybe the process.
Can I self promote in this thread XD
Maybe. We'll discuss it at the next board meeting and will let you know. // The Management
Some people actually like to read what humans write.
Bold choice using the noun "I" repeatedly instead of "Claude" - you didn't figure this out, Claude did, and then Claude very obviously wrote the blog post.
Oh, maybe that's why "Claude" was substituted with "I", because Claude wrote this blog post from a first person perspective.
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Research taste, among other things are still abilities that differentiate among humans.
In fact, I expect the elimination of other limiting factors by AI to lead to those becoming more important. By the same mechanism that make ugly hedge fund managers more performant.
The problem then is that composition of the top performers will change. Which is problematic since being a top performer ( ‘artist’, ‘researcher’, ‘writer’,..) is often an identity which ‘we often create to protect ourselves: the strong one who never breaks, the intelligent one who is always certain... The mask that once protected you can eventually become the cage that prevents you from changing’
An llm is a tool, I had access to Claude too but I didn’t find the bug. The author found it.
Ah, but are you a member of the wondrous, gifted class? Right now, I have to do two things to get access to front line models, with any ability to look at code vulnerabilities. I have to fill out an application, and I have to give my ID to a US firm, Persona, which I will never do, for identity verification.
If you don't do these two things, you cannot use Claude to find that $90k bug.
And it got me to thinking, what might this pattern look like in a decade? Will there be two classes of people? Those that have access to certain types of LLMs, and therefore, are well-to-do? And, what if being well-to-do, makes it more likely you can get access to those certain LLMs?
You wouldn't be able to claim the 90k either without proving your identity too. What's your point? You don't like Persona? Let's not start pretending that LLM Subscriptions are out of reach for the average person with a credit card.
'AI' is Not a Tool: https://www.linkedin.com/pulse/ai-tool-rich-hickey-kjl8e
An invitation for a semantic debate online on a minor point along with a random linked in post stating something about said debate?
No thank you says pretty much everyone.
Yeah, I was about to comment that I’m not reading all that. But good for them though.
Just use Claude to summarize. Ez pz
Then why is it marketed like one? And why can I use it like a tool?
I do have to send input to the thing to get a response, no?
Your dishwasher also takes certain inputs, the dishes and the lid closure. Some inputs are just more mechanical than expressive.
And a dishwasher is also a tool. You seem to be agreeing with the parent comment with a tone of disagreement.
A dishwasher is a machine, likely even more, but certainly not a tool.
I don't have to babysit my dishwasher to make sure it washes my dishes.
What is your point?
It's an unreliable appliance. You load the dishes and close the lid, it's supposed to do the work of washing them, but the actual engineering inside is too sloppy, so you have a non-trivial amount of persistent stains you'd still wash by hand. You might correctly conclude that if this fraction is high enough, the machine is more of a hindrance than an aid.
Sure it is. And it's AI, not 'AI' -- not sure why the need to add quoting.
It's literally the title. It seems every reply to this is just someone saying they didn't read it.
Correct. The headline is incorrect with both the quoting of AI as well as the claim it isn't a tool.
You can hardly blame anyone for avoiding a clickthrough into an auth-walled, user-hostile site to read the point the headline made in an attempt to "gotcha"
At least make your own argument yourself. Dumping a LinkedIn URL is only slightly more appropriate than directly dumping AI slop into the comments
No. Rich Hickey wrote it. Doing a bunch of restatement in order to say the same thing as if it is my own is a waste of time. It's not auth-walled. Read it or don't.
[flagged]
[flagged]
Are you saying I am an llm?
You know that enticing someone to commit suicide is a crime in most of the jurisdictions?
For now, an AI agent is not a 'someone'
It's hard to get a $90k bounty from Google, even with Claude in hand, so we shouldn't be dismissive of that, but it is pretty annoying to read write-ups written entirely in LLM voice.
Shallow dismissals like this contribute nothing to the discussion.
How much more depth is the dismissal supposed to have? The original deception lacks depth.
Are you referring to your own post? If so, accurate
You are underestimating how much effort and prior knowledge it takes to work with claude on this kind of thing.
It really hurts the fingers to type "make no mistakes" in every prompt.
I bet, I could take "Trading places" style, anyone off the street, give them a 100 "Secret" word list and they could spend 6 months discovering all sorts of things.
I agree effort and prior knowledge are required, but a few magic words is like 80% of the effort.
Well the first 80% is easy. It's the second 80% that really gets ya...
100 "Secret" word list sounds statistically more improbable than cracking your password.
OK do it. And make yourself 90K along the way.
This is the typical "ugh that's just rsync" when Dropbox was announced, but for a new LLM age.
_If it's so easy, do it_. Don't post about it. Do it. Fix these bugs at scale. Charge people money for it once you've built up a reputation. Get your own bug bounties.
Perhaps next time they can carve 1's and 0's by hand from a slate tablet.
Same old tired complaining about people not using technology correctly, or the way a _real_ coder would do it.
You mean, using butterfly, right?
For the uninitiated: https://xkcd.com/378/
handing clients access to an LLM gets them beginner level code, easily. same with beginner level exploits. an RCE against google is not a "beginner level exploit". If you've seen AI generated code, I can tell you that AI generated security findings also have that "beginner" smell. an expert who knows what they're doing has to drive them to look at the right places, and use the experts preferred method.
I could have overlooked but doesn't seem there is a mention of Claude in the blog post? Did author say they had used Claude?
I also am wondering if Claude will proceed on a prompt which is about finding a bug like that? I am not sure, has anyone tried?
Took about 3 sentences for me to be as certain as I've ever been about anything I've ever been in my entire life that this blog post was authored by an LLM, though truthfully it's impossible to know for sure if it was a Claude model or a distill.
The cookie is "what caught my eye." not realistic phrasing
The cookie contained a pickle-serialized payload, "which was bad enough on its own." maybe not best practice but not bad on it's own
Flask session cookies are signed", not encrypted," so the... LLMs love to tell you what things aren't, as an emphasis on what they are
Tbf as someone with an infosec background, I would have probably written just about the same.
Pickle payload is a red flag precisely because pickle is not a safe serialisation format format when taken as untrusted input, here the input was trusted but it's still giving someone en easy escalation path (exactly what happened)
And most people would assume the whole cookie payload would be encrypted if they saw just a binary blob, and not realise you can fully read it if it's just signed
But, yes, the post was probably edited by an LLM, as is pretty much everything else you read online these days.
This is a clear example of why it is a shame we cannot block commenters on this website.
I'm sorry mate. You can just ignore me in the future though.
If you're going to make this argument, at least say GLM 5.3 instead of Claude, Claude will refuse to touch anything remotely close to this unless you're in the super special club.
If he used an old school source code scanner would the credit go to him or to the tool? Because Claude is a tool, not a person.
;)
welcome to the club
how does something like this pass code and design reviews. is this code written by ai (the buggy code). Not only was the key that signs the cookie inside the cookie it self but it's using pickle which even model weights are moved away from. I'm a little flabbergasted at the noob nature of this root cause.
pickle is nuts, but I can imagine dumping an entire object into the cookie while troubleshooting something then forgetting to clean it up afterwards.
Pickle is so hard to get write that on my org we forbid ever accepting untrusted pickle input. We basically treat it like it’s an eval.
It was “trusted” input, they validated the signature and everything.
This does not pass automated testing where I work let alone code review.
> how does something like this pass code and design reviews
when the "give a shit quotient" of the employee moves towards "highly engaged with the organization and its mission" to "this is just a job, I'm gonna rest and vest"
I think you're being unreasonably uncharitable. Any and all software contains bugs, and I can assure you the piece of code you wrote has bugs. My code has bugs. Google's code has bugs. iOS has bugs. rocket launching software has bugs.
We can put in systems in place to catch bugs. And that's the system that's working here.
Now to the second part of the question -
How did they not catch it before? If it were "easy", it would not have been worth 90k, and hindsight is 20/20 - once you know where to look for, it is all very very obvious.
From TFA:
> Normal Google Applications (T2), S0 RCE: $75,000 base × 1.2 for exceptional report quality.
"Easy" isn't a factor in calculating the bounty.
If this were Microsoft, they would have received a coupon for 10% off a Thanksgiving turkey, only redeemable in 2030 and only in the month of March, must be purchased with cash, non-transferrable.
And would require a microsoft.com account to redeem.
Do you have any advice for aspiring bug-bounty hunters? Do you have some method for being able to find vulnerabilities like this, or lots of past experience?
> Do you have any advice for aspiring bug-bounty hunters?
We're probably at the end of the era of these kinds of big payouts. AI tools will drive the going rates to approximately the costs of the tokens. Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
That's really not why bounties exist, though people on both sides of the bounty systems claim it is. If it were broadly the case that bounties exist to offset black market sales, you wouldn't pay bounties for vulnerabilities with no market value. This vulnerability is probably a good example of something with a sub-$1000 value on any non-bounty market.
In reality, the modal benefit accruing to a company running a bounty program on receipt of a new serious vulnerability is simply engineering prioritization. You get bug reports from users when you have reliability and UX bugs, but you don't get them for security flaws, because attackers generally have to force vulnerabilities into revealing themselves --- normal users almost never run into them. Bug bounty programs fill that gap.
"This vulnerability is probably a good example of something with a sub-$1000 value on any non-bounty market."
Crazy statement. This would easily be worth more to malicious actors
The only actors who will pay you real money for a serverside RCE bug aren't buying an exploit; they're contracting with you to plan a heist. So the same analysis applies: you yourself could come up with some scheme whereby you yourself use this vulnerability to steal stuff from Google. But that's not what we're talking about when we talk about vulnerability valuations.
There is essentially no market for these kinds of serverside vulnerabilities.
How do you know? You could explain the impact of the vulnerability and let buyers on the black market decide and bid whether they could do something useful with it.
That's not how the "black market" works. There isn't like some bazaar where people describe vulnerabilities and people bid on them. Only a very specific subset of vulnerabilities, those that slot into existing business practices and CNE stacks, have buyers.
How do you know? There were offers on HackForums which did not relate to software components such as smartphones operating systems or webservers but to web sites. Why wouldn't that be possible for Google's developers docs site?
I don’t understand your argument here.
If you’re saying that bug bounties exist for a reason according to both the program runners and the contributors then how is that not the reason they exist?
The people paying and the people working are all in agreement that it exists for that reason.
What part of my logic do you not follow? The way you've posed this question, the simplest response would require me to write that whole comment over again.
That’s an unnecessarily rude response I think.
I detailed my logic above.
Additionally, part of my businesses remediation services include helping companies set up bug bounty programs. Giving researchers an incentive to report bugs for payouts instead of selling or exploiting them for profits is always the intended goal.
This is probably my highest-rated comment on HN:
https://news.ycombinator.com/item?id=43025038
It's the product of direct interviews with bounty people, FAANG/MAG7 blue teamers, and black market participants. If you want more, there's also an interview online that we did with Mark Dowd, who is one of the best known CNE vendors to the NATO IC. You can just Google [SCW Mark Dowd].
I'm happy to go into more detail, but, like I said, you didn't really respond to my comment in any way that I could expand on. Feel free to disagree with me in some kind of specific way.
This really is simple: if you work in this space, you already know that there's essentially no black market at all for serverside web bugs. But there's a $90k bounty for this one, and 5-figure bounties for others. Reconcile those two facts.
Well thats nice and Im happy for you but internet points don’t really mean much here…
What a vulnerability sells for and how much damage someone could do exploiting it are two different things. Someone capable of finding a serious vulnerability is probably capable of exploiting it themselves. They don’t need a buyer.
A company paying big bucks isn’t a contradiction. Its incentivizing disclosure over exploitation which could cost them considerably more. Engineering prioritization is… a minor part of it informed by the ease of exploitation, damage it could do (in the case of an RCE or publicly available keys as in this case quite a bit) etc.
I don’t see why you think these things are mutualy exclusive.
Having given up on the original argument ("Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.") you've now transitioned to a new one: that vulnerabilities can be used by the hackers who find them to plan heists. And that is true. It doesn't set any kind of market-clearing price, and selects for actors who are comfortable ending up in prison, but it is indeed a statement that aligns with reality.
My original comment explicitly said “selling or exploiting them for profits.” You are just ignoring the second half for some reason.
I think you need to breathe a bit and then re-read my comment. I’m not coming at you, I’m saying that what an exploit sells for and what it could cost a company if exploited are entirely different things. That’s where you’re hung up.
Companies have an obvious financial incentive to encourage discovery and disclosure before exploitation.
I just don't care at this point. We're talking past each other. I quoted the argument that I was responding to, in the comment you just replied to.
You think that most security researchers that can find an exploit are all willing to commit a heist with that exploit?
I’m not in the security industry but that strains credulity.
Where did I say that? Being capable of exploiting something and willing to are two different things.
The point is that bounties create an incentive to find and disclose vulns. That doesn’t require a black market for the exploits.
“Someone capable of finding a serious vulnerability is probably capable of exploiting it themselves. They don’t need a buyer.”
I’m responding to this. I would assume the list of people able to find a vulnerability is much much greater than the number of people willing to do a one off heist with it.
What you’re responding to and what you’re saying it says are two different things.
Sounds good. Thanks.
[dead]
> Bounties exist to make it nominally more economical for researchers to report exploits than it is to resell them.
Is this really true? After all, reselling the exploit would be illegal, and most people don't want to break the law to earn income.
most people who care about legality don't work in that field either. whoever gets into this probably starts by being ok with doing shady stuff, I imagine.
Many start in the military and eventually move to private sector. However, an ever growing percentage just straight up go to college to learn the skills.
Cyber companies are in the fortune 100 now, and it's not like it used to be back when you actually had to be interested in it and dedicate your life to learning the skillz from shady forums. Now it's just a job to most practitioners.
what use are 0-days to private cybersecurity companies?
Probably extremely useful for the companies that can sell exploits to state governments.
Huge marketing/PR benefit.
Yes, but also there is also a huge value to individual contributors and their careers. I broke into the field after landing a few relatively high profile CVE acknowledgements for big companies. Responsible disclosure is good for your career.
On the other hand, if the value of that drops you can expect to see some folks start taking the wrong path.
Plenty of people in this space who work on entirely legal things
Most people do not, but it needs only some people who do so for it to be a problem. Most people do not steal, but I still lock my front door.
there are enough people operating in countries that would not charge them for hacking an American company for it to matter, $90k can be a life-changing sum
And before it ends, people with existing reputation can milk the last drops, as mostly their reports gets processed from the thousands of others. Unfortunate status quo.
People who pay bounties for vulnerabilities don’t really care how you find them.
Sorry for the meta, but the discourse in many of the comments is nuts to me.
"Claude did this, not you" - does that mean that humans have no agency at all when they use an LLM? Is that the argument?
Maybe the norm should be to credit the LLM and harness used, like "I used ___ to help me find an RCE", but I can see arguments not to do that in some cases.
edit: The giant thread with this now got flagged, so some users may not see it.
I get you, but ai is different. Me using hand tools to fashion a custom piece of furniture is using tools in an impressive way. While o didn’t find this bug with AI, it’s obvious that AI is a tool altogether different. Yes there is agency and some degree of creativity/finesse in getting it to do what you want, but it does all the heavy lifting. All of it. You just have to poke it and prod it and rub it just right to get the magic gene out of the lamp. But that is such a far cry from knowing what’s needs to be known to do the task yourself, not to mention discovering its feasibility. It’s just wildly different than any tool we’ve known. So obviously people will snark when people don’t give clear and explicit credit.
Taking a step back, I think it's interesting that the emotion in the comments is so strong. AI generally seems to be increasing stress, and we get stress-loaded responses when people see something they disagree with instead of the usual discourse.
The best response to them is "If it's so easy, then when didn't you do it? Easy $90k."
Would Claude have found it without your guidance and knowledge? If yes, Claude found it. If no, you found it while using an LLM.
I suspect a lot of things fall into the latter. It is a common neg to defer everything to the former though.
If we're doing this standard, then let's do it for everything. Claude coded that, not you. Claude built that website, not you. Might make the industry less exhausting tbh.
GPT did the Hugging Face hack, not you (human engineers at OpenAI) :^)
One day soon, this will be like people saying you used a high level language instead of assembly to discover a fault.
People who believe AI is an abstraction layer on top of compilers, are in that middle ground where they know computer science but are far from an expert. Which makes them very susceptible to the Dunning–Kruger effect. Throw in AI sycophantcy and you've got yourself takes like the one above.
Ha.. AI in the form of LLM is not a very good non-deterministic abstraction layer on top of deterministic compilers.
Yet, they are being used with endless surprise as non-deterministic layers of abstraction on top of deterministic compilers.
My point is to use a technology for what it is capable for (where capabilities are discovered weekly and hard to keep up with), not trying to get it to do everything.
Bringing a deterministic lens to non-deterministic software has its dangers too. Non-deterministic software seems to be its own category of software, increasingly capable of non-deterministic approaches that deterministic compilers software couldn’t attempt.
Maybe that’s less compilers and more different type of scaffolding of vectors.
I’m all for skepticism, the issue with cynicism is capabilities often pass those perspectives by sometimes too quickly.
Yeah some people come off as envious they didn't have Claude try it first.
But it needs a disclosure if Claude was used: "The cookie is what caught my eye" - if that was actually Claude's eye then yeah, Simon is claiming to have a keen eye that he in fact does not.
"Claude found a $90K RCE" would have been just as interesting if not more.
Sure, but do we want traditional vuln posts to be “nmap found”?
It is fair an interesting to disclose tools used, but I am with GP in leaving agency with humans, at least for now.
Nmap doesn't write its own blog posts. That's not where I draw the line, but the point isn't at all about "drawing the line." (In other words, don't get caught up in where the line is. There is one, but we're not sitting in some gray area, we've blown past it by hundreds of kilometers by now.)
If I post an image and I say, "I drew this", and someone says, "no you didn't. That is clearly output from Qwen Image 2.1", can I reply, "Qwen Image 2.1 is just a tool, like Photoshop"? I mean, I can, but if I do this, I am missing the point. The point is that I didn't draw anything. I didn't draw. I didn't draw digitally. I didn't draw analog. I didn't draw with my mind. I typed a prompt. This is as close to drawing as photography is to drawing.
So did this person actually find a vulnerability on their own? Hell if I know, but I take issue with LLM-authored blog posts; I don't trust people who post LLM-authored blog posts as if they were written by themselves to be honest about the nature of their work.
If you really are still doing work and understanding and just using LLMs as a tool, show your understanding in your blog post that you are communicating to the world, instead of having Claude or GPT show its understanding. I don't really care whether you used Claude and didn't list it, if it's clear as day you understand what you were doing with it.
I'm dying on this hill. Until HN bans this sort of blog post or bans all of us complaining, these comments are going to be very, very boring. And I am stubborn for longer than most people can stand to read the same comments. Believe me.
That strongly depends on what you consider drawing, right? Sure you didn’t dip a pencil into an inkwell and caused stains to appear on a paper. But you did something that caused stains to appear on the image. You already seem to accept Photoshop as a way for a human to do that. Why not Qwen?
We can discuss the quality of the offering. But that’s a different discussion
If I hire an artist to draw something for me, I don't get to say that I drew it.
Credit is the least interesting part of this whole thing.
If an architect says they built that building, it’s legit to take issue with the semantics and rathole into steelworkers versus electricians, but only insecure people and academics care about precise allocation of credit.
Because it's not drawing. It's not even drawing in the abstract: nobody drew anything. There are no brush strokes. You can watch the steps diffusion models take and at no point is there anything that resembles "drawing".
Drawing with Photoshop absolutely resembles drawing. Photobashing with Photoshop does not. Photography also isn't drawing. These things are all distinct. That's the actual point I was trying to get at here by invoking Photoshop. Image models differ from text in that text models do resemble something closer to what people are actually doing, but you just have to accept that analogies have limitations here. Just like you can draw or not draw with Photoshop, you can use Claude in ways where you're really driving it like a tool to do work, and you can also use Claude in a "fix the bug" sense where you're doing absolutely nothing other than paying $200/mo and typing occasionally.
But whereas Photoshop introduces nuance to what it means to draw or make art by introducing different levels of involvement versus remixing and etc., Claude and Stable Diffusion utterly decimate nuance in a thermonuclear blast that plunges us into a nuance Nuclear Winter, by offering for perhaps the first time a way for the computer to do nearly all of the generalized "thinking" while you sit on your ass and argue on HN. (Reminder for everyone to check on their agents.)
Trust me, I use LLMs, I get the idea. I can "vibe code". On my workbench is a laptop running an entire desktop system including an application suite that I really didn't write. It would be cool to go online and say "Hey guys check out this cool desktop environment I wrote!" but I won't because I really didn't. It was just an experiment. Did I put in effort? Yes. Was it coding? Nope, it was closer to management, though I did write some code. The experiment was some mix of failure and success, but what I would never do is say that I wrote it. I may some day blog about the experience and the pros and cons and how I feel about the resulting code, but for right now I feel the Internet is full of enough slop and most people who I respect are thoroughly tired of hearing about LLMs all day (I am too at this point, though I am not saying I find none of it interesting.)
Do I think all LLM use absolutely must be disclosed? Actually no, and especially right now I understand why some people don't. I know I just said LLMs and diffusion models destroy all nuance, but that's not strictly true. My line in the sand is this: if nobody can tell overtly that you're using LLMs, and you can demonstrate your own understanding of what you are doing, then I don't care. In that case it's more of a concern for the person using the LLM, because the real concern in that case is becoming dependent on LLMs and being unable to write software/etc. without them anymore. But that isn't really my concern.
If I can tell your blog post was written by Claude within the first few sentences you have completely failed to demonstrate anything but your willingness to deceive people. How can you not feel some shame putting yourself on the By-line of a blog post you didn't write? I mean it's written in first person! And I know damn well that not everyone can immediately pick out LLM-generated text, but that isn't entirely the point either. It's one thing to use LLMs to do work, but the point of a blog post isn't just the text, blogging is a form of (formerly...) human expression. It just doesn't make any sense to put Claude-slop blog posts right next to human-authored blog posts, any more than it makes sense to put AI-generated images next to drawings. The point isn't the pixels and isn't the words. That contrasts with code, because with code, the point really is the code, and other stuff is secondary.
I can't tell you whether this person found this vulnerability and triaged it or Claude did. I don't even care. However, I don't trust a blog post they didn't write anyway.
This is a long comment, but it's hard to make it shorter. LLMs and diffusion models are not comparable to tools like Photoshop, in a way that is more extreme than the difference between Photoshop and a pencil. Analogies are nice (limited) shortcuts that can help us reason about things, but in this case analogies alone are insufficient because we are dealing with something that has a brand new shape.
Now do writing. If physical gestures are the important part, can I say I wrote a book when I actually typed it? The muscle activations are totally different.
I mean the process looks very similar even if the physical motions are different. That isn't the point, and honestly, I know you know that.
No, and please don't do the "you're pretending to believe that" thing.
Typing is nothing like writing letters. Fine versus gross motor, static hand positions versus moving left to right, top to bottom.
More importantly, "writing" is commonly understood to mean both the act of forming letters and the act of authoring written content. You write in block or cursive, you write novels.
This idea that drawing is somehow permanently, literally linked to only the physical act and not the creative act? It makes no sense.
> This is a long comment, but it's hard to make it shorter.
I was the parent of this thread, and I just want to really thank you for your reply. Really interesting reading. This whole thing might be the topic of our moment.
> code execution across seven Google domains
What a bargain
Glad to see that Google is actually paying out decent bounties still
Nice catch, but god terrible write-up. Please take the time to write it yourself even if you think you will do worse in explaining it than the LLM. It’s so off-putting to read that. Share your prompt or raw article notes even, it will much enjoyable to read than this
lol thats nice
That's an engineer right there. I really love how you didn't just 'it is what it is'. Claude is never replacing engineers like you.