column: When your best engineers log off for good, don’t be surprised when the cloud forgets how DNS works
Guess who they’ve been replacing them with
“Foreshadowing is a literary device”-ass headline
Pretty common upper management mindset, “Once something is done, its done forever and needs no specialized experience to maintain. So anybody can do it, even young inexperienced newly hired will be able to figure out how to take care of everything without any senior staff around to help them.”
In some cases it’s not even young/inexperienced new hires, just that in a project of any complexity, understanding all/as many as possible of the moving pieces that make it tick becomes more important than just experience or skill. So they hire a few really experienced people to throw at the problem, and manage the young/inexperienced, and they struggle and burn out too
It’s always DNS… or us-east-1. Wait, it was both this time? we should hit the pokies tonight!
deleted by creator
The entire hypothesis is nothing but speculation. It’s based on the assumption that troubleshooting steps live in the mind of some main character dev instead of where it probably lives, inside of a runbook for the service. I don’t doubt that some of the problems at Amazon that the author calls out are real, but sometimes outages aren’t caused by the first source you investigate and sometimes fallback solutions fail.
I’ve never seen work grind to a halt like this where I work
Nah AWS has been having massive outages around the DNS for us-east-1 for the past decade. This is nothing new. Recovery times around this one were particularly brutal because it was Diwali yesterday and all their Indian support engineers were out enjoying the holiday.
Well it’s down again today and I’ve never had this impact my work like this in the past 6 years so I’m not sure how true that is
the cloud forgets how DNS works
The D in DNS stands for centralized
