NASA mistakenly severs communication to Voyager 2

Re:NASA mistakenly severs communication to Voyager 2 by Nifty3929 on Monday July 31, 2023 @01:22PM (#36945901)

I used to do remote work in firewalls quite often, and after locking myself out once or twice, I came up with a new habit: before making any changes I would schedule a reboot for 5min out which would revert any changes. That way if I locked myself out I could just wait for the reboot and get back in.

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by dang on Monday July 31, 2023 @02:12PM (#36946592)

And then if it worked for those 5 min before the reboot, you'd redeploy the change 'for real', without a reboot?

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by 2snakes on Monday July 31, 2023 @02:27PM (#36946807)

Yeah, there are different kinds of memory in firewalls. Like a running-config and a startup-config. If you just change the running-config and don't commit to the startup-config, when the reboot takes place it'll pull the config from the (non-modified) startup-config instead, reverting changes.

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by Nifty3929 on Monday July 31, 2023 @04:00PM (#36947977)

copy run start!

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by Nifty3929 on Monday July 31, 2023 @03:59PM (#36947967)

My typical workflow was:

- Schedule the reboot

- do my changes

- Make sure everything was working properly

- Go get lunch

- Notice a bunch of pages and alarms about a firewall going offline

- Rush back to my office

- Login to the firewall

- Schedule the reboot

- Re apply the changes

- Test it again

- CANCEL THE FING REBOOT THIS TIME

- Eat my now cold lunch

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by booi on Tuesday August 01, 2023 @03:04AM (#36953214)

This used to be my workflow as well. I did make a few improvements though

  - begin change control at 4:55pm on Friday before Christmas
  - schedule reboot
  - paste changes
  - make sure everything is working properly
  - leave security key on desk 
  - go to christmas party
  - firewall goes offline, pages go off
  - remotely log into firewall with phone
  - rush back to office to get security key
  - accidentally type init 1 hanging server
  - discount datacenter remote hands not picking up the  phone
  - rush to datacenter to power cycle server
  - :(

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by prox on Monday July 31, 2023 @02:14PM (#36946628)

This is clever, I like it.

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by knorker on Monday July 31, 2023 @02:41PM (#36947021)

"commit confirmed" from Juniper routers is much better

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by snuxoll on Monday July 31, 2023 @05:16PM (#36948957)

Mikrotik safe mode gets a 3/5 in comparison - it reverts changes you made if you lose connection to the router, so it does it’s job as an anti-lockout mechanism; but I much prefer the atomic nature of a confit commit on junos still.

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by knorker on Monday July 31, 2023 @02:40PM (#36947000)

Standard practice on Cisco routers, where I've worked, is to do "reload 5" before doing dangerous things.

On juniper, it's "commit confirmed".

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by comboy on Monday July 31, 2023 @05:16PM (#36948956)

or safe mode on mikrotik

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by hsbauauvhabzb on Monday July 31, 2023 @03:52PM (#36947876)

‘sleep 300 && init 6’ was my go-to, but since then systemd has made firing init 6 unreliable (it won’t trigger a reboot locally if root has an open ssh session, at least on Ubuntu).

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by stouset on Monday July 31, 2023 @06:08PM (#36949553)

I did a similar thing in the early days of my career, but I actually caused an outage as a result.

In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW.

The script runs, everything looks good. Five minutes later, pagers start going off.

Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.

[ Parent | on HN ]

Re:NASA mistakenly severs communication to Voyager 2 by paulddraper on Monday July 31, 2023 @07:42PM (#36950430)

Kinda like changing display settings in Windows.

Changes will revert in 15 seconds....

[ Parent | on HN ]