DEV Community

Cover image for I Almost Ran rm -rf / on Production — And It Changed How I Deploy Forever
Frozen Blood
Frozen Blood

Posted on

I Almost Ran rm -rf / on Production — And It Changed How I Deploy Forever

There’s a specific kind of silence that happens when your terminal cursor blinks… and you realize you’re connected to production.

I didn’t actually wipe the server.
But I was one Enter key away.

Here’s what happened — and the small habits I changed that probably saved me (and might save you too).


The Setup: A “Quick Fix” at 1:17 AM

Classic story:

  • Minor config bug
  • “It’ll take 2 minutes”
  • SSH into the server
  • Fix a path
  • Restart the service
  • Go to sleep

Except I had two terminals open:

  • One connected to staging
  • One connected to production

They looked almost identical.

Same prompt.
Same theme.
Same everything.

I typed:

rm -rf dist
Enter fullscreen mode Exit fullscreen mode

And then I noticed the hostname.

Production.


Why This Was Worse Than It Sounds

That project didn’t just store build artifacts in dist/.
It also had runtime-generated files.

And since I was inside the app directory, not root, rm -rf dist wouldn’t nuke the whole machine — but it would absolutely break live traffic.

It would have:

  • Deleted compiled output
  • Broken running processes on restart
  • Forced an emergency redeploy

All because I didn’t slow down for 3 seconds.


The Real Problem Wasn’t the Command

It was this:

  • Production access was too easy
  • The environment looked too similar
  • Destructive commands had no friction
  • There were no guardrails

It wasn’t a technical issue.
It was a design flaw in my workflow.


What I Changed Immediately

1️⃣ Aggressively Different Shell Prompts

Production is now bright red in my terminal.

Using .bashrc:

export PS1="\[\e[1;31m\][PROD] \u@\h:\w$ \[\e[0m\]"
Enter fullscreen mode Exit fullscreen mode

If I SSH into prod, it screams at me.

No more guessing.


2️⃣ Alias Protection for Dangerous Commands

I added interactive flags to destructive commands:

alias rm='rm -i'
alias mv='mv -i'
alias cp='cp -i'
Enter fullscreen mode Exit fullscreen mode

Now I must confirm before deleting.

Is it slightly annoying?
Yes.
Is it better than downtime? Also yes.


3️⃣ Production Is Now Mostly Immutable

Instead of:

  • SSH → change files → restart

I moved to:

  • Build locally or in CI
  • Deploy artifact
  • Restart via process manager

No editing files on prod. Ever.

If I need to “fix something quickly,” I fix it in code and redeploy.


4️⃣ Reduced Direct SSH Access

Now:

  • Logs are centralized
  • Restarts are script-driven
  • Environment variables are managed via config

If I SSH, something is already very wrong.


The Psychological Lesson

Most production disasters don’t come from incompetence.

They come from:

  • Fatigue
  • Familiarity
  • Repetition
  • “Just this once”

The most dangerous command isn’t rm -rf.
It’s confidence.


The Quiet Truth About DevOps

We talk about:

  • Kubernetes
  • Blue-green deploys
  • Zero-downtime rollouts

But the simplest protection is often:

Make dangerous actions slightly harder.

Friction is underrated.


Conclusion / Key takeaway

I didn’t delete production.
But that near-miss changed how I treat access, commands, and environments forever.

The scariest production story isn’t the one where everything breaks.
It’s the one where you realize how easily it could have.

What’s the closest you’ve come to breaking production — and what guardrail did you add afterward?

Top comments (0)