I think the credit belongs to Sascha still. Look at this:
> The agent surfaced a suspicious issue: the anetd pods in our Google Kubernetes Engine cluster were restarting constantly, around 120 restarts per pod over six days, which is almost one crash per hour. Surely, this couldn't be right!
> Sascha dug into the crash dumps. The stack trace pointed to a concurrent map-access panic, multiple goroutines trying to read and write to the same data structure at the same time without proper locking. But the key detail was where the panic happened: inside the Wireguard module of anetd.
AI: Your anted pod is crashing.
Engineer: Looks in the logs and finds a stack trace.
Your agent didn't find the bug. It's really that simple.
100% but it doesn’t benefit an AI company to properly assign credit. Their AI identified a generic problem, not this problem, and then their AI was guided like a child into the correct spot to start searching for a a bug that it eventually traced.
A new bug appears, it’s in an encryption layer. You solve this by deciding to disable the encryption layer because user experience is better without the errors. You write it up as a recruitment piece for your engineering team.
There may be some good answers and lessons, but they didn’t make it into the article. Saying it’s on a cloud provider’s private network so encryption between your nodes isn’t necessary is a bold choice. Also, what happened to the root cause? Why did it start failing a week ago? Was a downgrade of the offending code not possible?
Not all bug investigations are worth really digging into. Sometimes the right call is to find any fix and move on. But all the nuance, judgement, implications, and lessons learned failed to make it into this post. And they are what make reading incident reports interesting for most engineers.
'Sascha dug into the crash dumps. The stack trace pointed to a concurrent map-access panic, multiple goroutines trying to read and write to the same data structure at the same time without proper locking. But the key detail was where the panic happened: inside the Wireguard module of anetd.'
this is person right? not a agent... and this whole article seems like it was written by AI...
Isn't this like the #1 problem people have with wireguard? I've had clients with the MTU issue every time I've set it up for more than a few clients. Also how on earth is "connection reset by peer" dreaded?
I can't tell if my main source of hate there is the homogeneity itself, or that it's excessively marketing-flavored (fluffy and aggrandizing, mere inches from "our incredible journey...").
came to say this. it's the AI writing cadence, I can smell it from 1000ft:
- Lots of "The" headings
- Always "why it matters"
- Machine gun style cadence of short sentences
I hate that AI has stolen my emdashes and I can't use them without looking like slop. But I will die defending markdown as my preferred note taking format. They're not taking that away from me.
I felt like this with the word delve. Seemed like nobody had ever heard the word before, and that the only possible way it’d be written was if an llm did it - but it’s just a nice word.
Delve was used a lot in corporate writing. A lot of the so-called "whitepapers" that businesses like to publish to show how smart they are were ingested in training models.
Just on HN alone I think I've seen roughly 3948538902748750897520938 mentions of emdashes when others were complaining about slop in the past 2 months. It might as well be the unofficial slop logo. Folks are treating it like a dead giveaway.
I feel like a school friend of mine has been taken from me.
This piece might be a record for how quick it took me to smell the AI-tone and close the tab.. one paragraph! I'm sure it's an interesting bug but I can't stomach reading any more slop.
I think "AI-tone" is a much better way to characterize this stuff than accusing people of using AI. The problem has always been the same. Putting out slop feels disrespectful to the people you want to read/watch your stuff.
Makes me think of how pre-chatGPT I still could barely handle most recipe blogs because of their well known attempts at "filling space". And yea the problem is significantly magnified now everywhere else.
Anyway, my point is, whether or not someone uses AI is almost secondary in a way (even though it can seem pretty obvious to most of us when it's being used). All that matters is if the writing seems like it cares more about throwing words at people instead of actually conveying its points in a way to elicit understanding.
A bug in Wireguard? What did Google change, since it affects only them? Any lessons learned about modifying cryptographic software?
...
Skipping past the investigation bit (minimising my daily slop intake), it's a wrong MTU value causing failing connections when Wireguard is disabled:
> When we disabled WireGuard, we expected the configuration to change to use the full 1500 bytes. However, some nodes in the cluster hadn't been restarted [and were] using the old 1420-byte MTU.
> [paraphrased] This particularly affected Valkey connections because they were distributed across nodes with mismatched MTU settings. So your API pod might not connect. The fix was rerolling all the nodes to get a consistent MTU configuration
I think the credit belongs to Sascha still. Look at this:
> The agent surfaced a suspicious issue: the anetd pods in our Google Kubernetes Engine cluster were restarting constantly, around 120 restarts per pod over six days, which is almost one crash per hour. Surely, this couldn't be right!
> Sascha dug into the crash dumps. The stack trace pointed to a concurrent map-access panic, multiple goroutines trying to read and write to the same data structure at the same time without proper locking. But the key detail was where the panic happened: inside the Wireguard module of anetd.
AI: Your anted pod is crashing.
Engineer: Looks in the logs and finds a stack trace.
Your agent didn't find the bug. It's really that simple.
100% but it doesn’t benefit an AI company to properly assign credit. Their AI identified a generic problem, not this problem, and then their AI was guided like a child into the correct spot to start searching for a a bug that it eventually traced.
just noticed your comment, yea the title is misleading and the whole article is annoying AI slop
A new bug appears, it’s in an encryption layer. You solve this by deciding to disable the encryption layer because user experience is better without the errors. You write it up as a recruitment piece for your engineering team.
There may be some good answers and lessons, but they didn’t make it into the article. Saying it’s on a cloud provider’s private network so encryption between your nodes isn’t necessary is a bold choice. Also, what happened to the root cause? Why did it start failing a week ago? Was a downgrade of the offending code not possible?
Not all bug investigations are worth really digging into. Sometimes the right call is to find any fix and move on. But all the nuance, judgement, implications, and lessons learned failed to make it into this post. And they are what make reading incident reports interesting for most engineers.
am i missing something?
'Sascha dug into the crash dumps. The stack trace pointed to a concurrent map-access panic, multiple goroutines trying to read and write to the same data structure at the same time without proper locking. But the key detail was where the panic happened: inside the Wireguard module of anetd.'
this is person right? not a agent... and this whole article seems like it was written by AI...
Isn't this like the #1 problem people have with wireguard? I've had clients with the MTU issue every time I've set it up for more than a few clients. Also how on earth is "connection reset by peer" dreaded?
This article reeks of desperation. I'm pretty sure Lovable's days are numbered.
hate how it all has the same tone now
I can't tell if my main source of hate there is the homogeneity itself, or that it's excessively marketing-flavored (fluffy and aggrandizing, mere inches from "our incredible journey...").
came to say this. it's the AI writing cadence, I can smell it from 1000ft: - Lots of "The" headings - Always "why it matters" - Machine gun style cadence of short sentences
convergence that we see in all kind of medium when some things are considered “working better” (true or not)
AI writing takes these to an extreme but we have see the same happening everywhere even before AI
I hate that AI has stolen my emdashes and I can't use them without looking like slop. But I will die defending markdown as my preferred note taking format. They're not taking that away from me.
I felt like this with the word delve. Seemed like nobody had ever heard the word before, and that the only possible way it’d be written was if an llm did it - but it’s just a nice word.
Delve was used a lot in corporate writing. A lot of the so-called "whitepapers" that businesses like to publish to show how smart they are were ingested in training models.
the premiere golang debugger is called delve. Also you must not hang out w/ many ttrpgers.
I mean I use it and then it seemed like everyone was acting like it it was a weird llm only word.
How about not caring about what people on the internet say about your writing using emdashes?
They are a false positive signal for identifying AI texts anyway.
Without markdown I don't even know who I would be anymore.
Em dashes aren’t stolen, it’s still clear from the overall voice of the text if it’s AI written or not.
At least, in reasonably long sections of text. I find it can be hard to tell one way or the other in shorter texts (like comments)
Just on HN alone I think I've seen roughly 3948538902748750897520938 mentions of emdashes when others were complaining about slop in the past 2 months. It might as well be the unofficial slop logo. Folks are treating it like a dead giveaway.
I feel like a school friend of mine has been taken from me.
The real dead giveaway is that the writing is bad.
It’s just people who don’t read books outing themselves. Ignore.
Even bookz aee ruined for me now. I was reading careless people and hate all the emdashes even though in a book it's just normal
Indeed: https://news.ycombinator.com/item?id=47921359
and the comments above it
This piece might be a record for how quick it took me to smell the AI-tone and close the tab.. one paragraph! I'm sure it's an interesting bug but I can't stomach reading any more slop.
yup, if you can't be bothered to write a article i can't be bothered to read it... i don't get why people are so lazy
I think "AI-tone" is a much better way to characterize this stuff than accusing people of using AI. The problem has always been the same. Putting out slop feels disrespectful to the people you want to read/watch your stuff.
Makes me think of how pre-chatGPT I still could barely handle most recipe blogs because of their well known attempts at "filling space". And yea the problem is significantly magnified now everywhere else.
Anyway, my point is, whether or not someone uses AI is almost secondary in a way (even though it can seem pretty obvious to most of us when it's being used). All that matters is if the writing seems like it cares more about throwing words at people instead of actually conveying its points in a way to elicit understanding.
Came here to say this. It’s a shame that I’m so exhausted reading slop that I’m probably missing many interesting stories from the industry
A bug in Wireguard? What did Google change, since it affects only them? Any lessons learned about modifying cryptographic software?
...
Skipping past the investigation bit (minimising my daily slop intake), it's a wrong MTU value causing failing connections when Wireguard is disabled:
> When we disabled WireGuard, we expected the configuration to change to use the full 1500 bytes. However, some nodes in the cluster hadn't been restarted [and were] using the old 1420-byte MTU.
> [paraphrased] This particularly affected Valkey connections because they were distributed across nodes with mismatched MTU settings. So your API pod might not connect. The fix was rerolling all the nodes to get a consistent MTU configuration
This article really delves in and and finds the seam - operation reality not operational performance theater