i wrote this in march 2014 on facebook, responding to a request for opinions from sysadmins about perceptions of fewer graduates interested in ops as a career; the conversation veered into “automate ones’ self out of one’s job” territory, which led me to post this.

I got my current gig [as of May 2024, my previous gig] taking over from a guy who died. Actually died. Actually dropped dead in the middle of the work day in his home office in the south of France a year ago plus a few days from a heart attack. I saw his mailbox while trying to start the transition, it just stops stone cold at like two thirty in the afternoon. And this was a guy in his mid-50s, coming from a mainframe background and who was just learning his way around Linux.
And in the operational inheritance, if you will, I found myself in the middle of some random database stuff, waist deep in a pile of unloved, unmaintained EC2 instances, Win2K3 instance store root instances like the primary — and only — mail server where the AMI doesn’t even show up in the description, 32-bit CentOS instances that can’t be switched to m3/c3 instance types, multiple versions of mysql running on multiple OS types (because reasons), cronjobs rigged to @yearly set to hard stop mysql slaves (PagerDuty and I were best buds at 5 past midnight on new year’s), the scariest randomly organised cascading flood of .BAT scripts, cron jobs with stdout and stderr disappearing into unsent sendmail spools and local Maildirs chewing inodes, shitty Win32 ports of sed and awk, weird five year old EOLed Windows apps that don’t have websites any more and that have been manhandled into working in tremendously wrong ways.
No automation, no documentation, no monitoring, no handover — to the extent that there are still processes operating with no domain knowledge to verify what the fuck is going on and sweet jesus how to get it the fuck out of some bullshit Windows instance and into some kind — any kind — of usable deployable readable version-controllable POSIX-compliant recipe or script or fuck me even a wiki entry.
It’s not all someone else’s fault. I missed some stuff that I didn’t look deeply enough at (partly out of fear … when you stare into the abyss …) And it wasn’t all the dead guy’s fault, because some of the infra pre-dated his influence and the influence of my UNIX-friendly boss. But I found myself deep in the old ways, the dumb ways, the BOFH ways, the jobsworth ways, the engineer-as-remote-keyboard ways. When your DeLorean’s doors open with the plutonium fuel rod expended, you don’t expect to hear Huey Lewis and the News blaring away after you’ve been handed your hoverboard and self-lacing Nikes. Being at the Lab [I worked at Linden Lab from 2007–2009] surrounded by fiercely opinionated, frighteningly smart people turning Debian into Skynet, and then suddenly (well, four years plus or minus suddenly) finding yourself back in the mainframe mindset is like being shot with a taser to the face. And your balls. And then your face again. And then your knee, because fuck you, unix boy.
I’ve had my fair share of production outages, I expect many more until I finish getting our shit together. I look forward to them, because each one is a learning curve.
But I have Tyler Durden standing over my shoulder. You are not your job. You’re not how much storage you manage. You are not the OS you deploy. You’re not the contents of your github repo. You are not your fucking MacBook. I will bring us through this. As always, I will carry you, kicking and screaming, and in the end, you will thank me.
In Tyler we trust.