Refreshingly written. And like wading into the ocean on a surprisingly hot spring day, bracing.
At most any given scope, "most" systems you interact with aren't under your own control. For this, consider: because that's true, the fault is yours for not grappling with that reality.
- recognize that my code will interact with systems I cannot anticipate, that change in ways I cannot control, and become coupled more than ever intended
I have a history of asking incredibly basic questions about the assumptions that are going in to a debugging issue for exactly these kinds of reasons. Something changed somewhere; so have we verified our basic assumptions that things are what we think they are... or did something change under our feet without our knowing? Or maybe we actually have a different understanding of the state of the world, and one of us is actually wrong?
It's not fruitful every time, but the number of times you skip asking those questions and chase something around, only to realize that, actually, it _was_ the ridiculous/simple thing... well, it feels a lot better to realize it earlier :)
(personal footnote: do be mindful of the context you're asking in! I feel like in general those sorts of questions are _fairly_ well tolerated by technical people. Sometimes you feel a bit stupid asking the question, but many of the systems I've worked on are large and complicated enough that it's not that weird - and if there's downtime and a healthy team, people genuinely want to hunt every possibility down. People not used to that, though, will sometimes bristle a bit, thinking you're implying that they're stupid and don't understand the basics. There can be ways to be more tactful, but at the very least, it's worth being aware of that reaction so you can follow up with a "just making sure I understand!" or something like that.)
Something I've noticed a lot is the how quickly people jump to complicated answers.
I've seen this a lot.
A few examples:-
- this code isn't logging -> is logging broke -> code was never deployed
- 500s are being thrown -> it's definitely not my code -> revert code 500s cliff face
Literally, every week there is a version of that. I have no idea why or how to tackle it but it is highly prevalent across skill set and roles.
A team manager once called and asked me if anything changed in the hypervisor (I used to support and make local modifications to some virtualization software) because her team's daily data processing job failed, which previously had run with no issues and nothing changed at their end.
I commented back, maybe with a little too much confidence, that if their input file changes daily, then their program changes daily. That is, from one day to the next, it may take different paths (or a different combination of paths) through the code's logic. In that way, I claimed, her program was like a language interpreter, and her input files were like programs.
She didn't seem completely convinced, but apparently they got the job running again because there was no follow up on that particular issue.
If a processing job fails due to the user's input, but the program doesn't let the user know what exactly caused the failure, there may not be any work on your end to get the input corrected, but you should definitely doing work on your program to improve the error handling aspect.
Case 1 seems to be missing something: The upgrade was months before the spike, so something else must have also changed recently. If it was just the optimization in the upgrade it ought to have happened earlier.
I always bring the receipts, so if I say it is them, then the chances are it is them. Pointing fingers before investigation the cause is a waste of precious time.
In many cases, I found that when colleagues said "we didn't change anything" they mean "we didn't change anything relevant" which winds up meaning "we didn't change anything we think could be relevant to this issue" and then "whoops, turns out it was relevant after all".
Refreshingly written. And like wading into the ocean on a surprisingly hot spring day, bracing.
At most any given scope, "most" systems you interact with aren't under your own control. For this, consider: because that's true, the fault is yours for not grappling with that reality.
This has been written about before:
https://ruggedsoftware.org
But even that was missing a bullet:
- recognize that my code will interact with systems I cannot anticipate, that change in ways I cannot control, and become coupled more than ever intended
Nothing changed on our side" and "it's DNS" are the same sentence. Both mean "I haven't looked yet
I have a history of asking incredibly basic questions about the assumptions that are going in to a debugging issue for exactly these kinds of reasons. Something changed somewhere; so have we verified our basic assumptions that things are what we think they are... or did something change under our feet without our knowing? Or maybe we actually have a different understanding of the state of the world, and one of us is actually wrong?
It's not fruitful every time, but the number of times you skip asking those questions and chase something around, only to realize that, actually, it _was_ the ridiculous/simple thing... well, it feels a lot better to realize it earlier :)
(personal footnote: do be mindful of the context you're asking in! I feel like in general those sorts of questions are _fairly_ well tolerated by technical people. Sometimes you feel a bit stupid asking the question, but many of the systems I've worked on are large and complicated enough that it's not that weird - and if there's downtime and a healthy team, people genuinely want to hunt every possibility down. People not used to that, though, will sometimes bristle a bit, thinking you're implying that they're stupid and don't understand the basics. There can be ways to be more tactful, but at the very least, it's worth being aware of that reaction so you can follow up with a "just making sure I understand!" or something like that.)
Something I've noticed a lot is the how quickly people jump to complicated answers.
I've seen this a lot.
A few examples:-
- this code isn't logging -> is logging broke -> code was never deployed - 500s are being thrown -> it's definitely not my code -> revert code 500s cliff face
Literally, every week there is a version of that. I have no idea why or how to tackle it but it is highly prevalent across skill set and roles.
A team manager once called and asked me if anything changed in the hypervisor (I used to support and make local modifications to some virtualization software) because her team's daily data processing job failed, which previously had run with no issues and nothing changed at their end.
I commented back, maybe with a little too much confidence, that if their input file changes daily, then their program changes daily. That is, from one day to the next, it may take different paths (or a different combination of paths) through the code's logic. In that way, I claimed, her program was like a language interpreter, and her input files were like programs.
She didn't seem completely convinced, but apparently they got the job running again because there was no follow up on that particular issue.
If a processing job fails due to the user's input, but the program doesn't let the user know what exactly caused the failure, there may not be any work on your end to get the input corrected, but you should definitely doing work on your program to improve the error handling aspect.
If only for the reason that people stop bugging you about it. Making good UX is self serving.
Case 1 seems to be missing something: The upgrade was months before the spike, so something else must have also changed recently. If it was just the optimization in the upgrade it ought to have happened earlier.
Why does the ACM hate VPN connections?
I always bring the receipts, so if I say it is them, then the chances are it is them. Pointing fingers before investigation the cause is a waste of precious time.
No YOU may not have changed anything. Your colleagues might have. The system might have. The network might have. Windows/Linux updates might have.
Many of those are automated and usually escape proper review. And they can be minor or catastrophic.
I always try to look at "what changed that causes this failure?"
And do so blamelessly. Pointing fingers only gets disdain and blame.
In many cases, I found that when colleagues said "we didn't change anything" they mean "we didn't change anything relevant" which winds up meaning "we didn't change anything we think could be relevant to this issue" and then "whoops, turns out it was relevant after all".
I mean, you can have nothing change on your side, but guess what, your customers didn't talk to you but they are changing things all the time.
[dead]