I wonder, is there a technical publication elsewhere that has more substantial coverage for interested people?
If there is such an archive, or some approximation thereof, it would surely be fascinating to pore over it.
> there's no reason for the public at large to need it
As a member of said public, I would be curious to know. There's no need for taxpayer-funded agencies to operate in a cloak of darkness.
Most everything done by government should by default be open to the public, with an exceedingly high bar that must be met to be otherwise. Otherwise, you run into nonsensical things like how some details around the assassination of a president 60 years ago are still classified on "national security" grounds.
- NASA informs the public immediately, and then makes the details available later after they've had time to compile the news and information into a format useful for the public
- NASA waits to inform the public until said report is finished
or perhaps you're after option c:
- NASA's network drives are open to the www in read-only mode, because, you know, 'open by default' entails realtime information (even though he doesn't actually care 99.9999% of the time. yet, someone should deliver this functionality, without it costing the taxpayer extra).
NASA routinely makes a LOT of data open to the public. Like, you can get very detailed JWST data directly from NASA. Probably far more detailed than you'd ever care to, because NASA does care about exactly your concern.
Actually, many agencies publish very detailed data if you care to look.
But do you have reason to believe they're working on a detailed public report?
Because if they're not, then you missed option "NASA informs the public immediately, but never makes the details available" which would be unfortunate.
Also they probably already answered a lot of these questions internally during the last week, so it wouldn't hurt to put some of that information out.
This is what the Freedom of Information Act is for:
https://www.nasa.gov/centers/armstrong/FOIA/request.html
The report may not exist yet, so you may need to wait.
I'm also the sort of person who thinks that all code written with public money should be open source.
It's pretty obvious that the people who managed to extend the lifetime of Voyager are very smart, based on all the tricks they had to do.
They are remotely configuring an old-tech device that is billions of kilometers away, with insane lag, and uncertainty that the underlying hardware is even responding properly.
Absolutely anything could have go wrong at this stage.
They'll anyway investigate internally what is going on, in order to hopefully, find a solution.
There is no need to spend resources to make the material public, just to satisfy curiosity (though it's interesting).
But that value is not zero, and replacing it costs quite a bit - both money and time. Asking how and why this happened is a valid inquiry.
For science, I would want to do an enquiry anyway - I'm just commenting on the financial/accountability aspect.
Do you want the military to “work efficiently” if that means little to no oversight? How about Congress?
Oversight and accountability to the citizenry is a foundational principle in a functioning democracy.
Saying that and then countering with your own.
Yeah, the pen dropping is a bit over the top, but as of now the claim is that this situation is planed for and will resolve itself. A report now wont tell us anything of significance. It will get interesting if the realignment fails.
> Oversight and accountability to the citizenry is a foundational principle in a functioning democracy.
I don't see micromanagement in that list.
Is micromanaging that what you're claiming as a strawman in my position? I'm not claiming you are saying the military doesn't need oversight, I'm probing with a concrete example where you draw the line on what constitutes reasonable accountability. Note mine were framed as questions for clarification. That's not a strawman.
Your micromanaging claim is however another strawman statement. I guess I could use clarification on your point. "Micromanaging" would be a direct democratic vote on most or all issues, IMO. That's not what's being asked for here here. What seems to be asked for is transparency. Access to information is not the same as having authority to make all decisions. But it is paramount in a government when people elect representatives who make decisions (or appoint those who do).
>A report now wont tell us anything of significance.
What makes you so confident? A report can tell us if processes were followed appropriately and, if not, if anyone was held accountable for not following them. I'd say that is pretty significant if you care about governmental fraud, waste, and abuse.
You know what you did. Stop it.
They are complaining about the official statement, specifically its lightness on detail.
The unsubtle misparaphrasing of Mark Twain was included as a comedic flourish to provide a light-hearted framing of the comments, but upon review of the subsequent debate, I concede it’s possible that for some, any allusion to statecraft stimulates the adversarial lobes.
Of course there are operational details that we don't need to be made aware of, but for an incident as big as this there's no reason to at least know how it happened and what could be changed to prevent it from happening again.
Also, it is possible that 2 degrees of misalignment still allows some fraction of the signal to be detected, but it not being strong enough to be decoded. The received signal strength and the beamwidth of the antenna could then be used to estimate how far off the mark the Voyager 2 dish is.
Lots of formal processes capture what would otherwise be informal design decisions elsewhere. In this case, they probably have reams of pages detailing a failure mode effects analysis (FMEA). One mode is “oops, we sent the wrong command” and the document would define the specific design mitigation(s) for that outcome until it reaches an accepted risk threshold.
Some failures are fairly common, and individual failures might be fairly inert but have more serious consequences if they are cascaded with another specific failure.. for example, cruise control enable + failure of steering wheel control pad _and_ previously undetected failure of brake sensor/brake light circuit = cruise control stuck ON. Actually, this failure is inert if the cruise control is OFF when it happens. Contrived example but you get the idea ...
I have seen a lot of FMEDA (and other tool) use lately to combat concerns with cascading failure, but not sure what's currently standard at NASA or how they deal with this. I would think cascading failure would be their expected scenario on a 10+ year unmanned mission.
Here's how I would describe the possible analysis approaches in broad terms.
* Start from the hazard of being pointed in the wrong direction and work backwards to identify the causes, forming a tree.
* Start from the event of commanding the wrong direction and work forwards to identify mitigations or the lack thereof, also forming a tree.
* Start from looking at a component or subsystem, list all the ways it can fail without regard for the application, and then consider the application work up towards the causes/events.
* Close any gaps between the top-down and bottom-up approaches.
"Software Fault Tree Analysis (SFTA) is a top-down approach to failure analysis which begins with thinking about potential failures or malfunctions (What could go wrong?) and then thinking through all the possible ways that such a failure or malfunction could occur. Fault Tree Analysis (FTA), is often used by the hardware teams to identify potential hazards that might be caused by failures in hardware components or systems, but with the SFTA, the software isn’t considered the hazard, but it can be a cause or contributor when considered in the context of the system."
"The Software Failure Modes and Effects Analysis (SFMEA) is a bottom up approach where each component is examined and all the possible ways it can fail are listed. Each possible failure is traced through the system to see what effect it might have on the system and to determine if it results in a hazardous state. Then the likelihood of the failure and the severity of the system failure can be considered."
But, to the earlier post, these are driven by hard requirements; specifically adherence to NASA STD 7150.2 and NPR 7150.2. Developers/contractors can tailor/waive them with pre-approval but, in general, they tend to go in the direction of less requirements, not more. This may all be moot because I think Voyager pre-dates any of those requirement documents and I'm not sure what existed in the late 1970s.
Imagine deploying a billion dollar piece of hardware and hoping that it has enough intelligence to keep itself from burning up before you can reestablish contact!
The hardware is the same, but they've updated, patched, and rewritten the software that's running in it throughout the years.
I'm not suggesting that the failsafe mode wasn't originally considered, and implemented, but simply that it doesn't have to be the case. They could have made changes to it over time.
I am paraphrasing what he said as a non-technical person: Voyager has both a dish receiver, and a pole antenna. The dish is the usual mechanism for comms but in an emergency such as this they would send commands to the other antenna. To do this they would turn the main tracking station dish up to max, and send a "TURN AROUND!" signal out.
But prior to that they had to alert the local electricity grid, and the local air traffic control to not have any planes flying over at the time!
I guess the Voyagers are too far away for this manoeuvre now.
https://www.jpl.nasa.gov/news/nasa-mission-update-voyager-2-...
Frankly before your comment I wasn't going to complain because I saw the tantrum you threw when people corrected you on the usage of "bricked" but maybe next time spell the month to avoid ambiguity.
ISO 8601 works for everyone.
It avoids confusing the Americans who otherwise put the month in the wrong place.
It avoids being ambiguous for everyone who may otherwise be worried that it was written by an American with the month in the wrong place, when the day is less than 13.
2023-10-09 is the 9th of October and it's clear to everyone regardless.
It also has the benefit of sorting chronologically if sorted "by name" when used in a filename as it's largest unit on the left, smallest on the right.
I remember when bricking something meant it was totally unrecoverable. Now it means "temporarily not working but will automatically heal".
"Unbricking" will hopefully work automatically, because there is no other option. But that can also fail and there is no way to know, or influence it.
I use bricking in the definition of mobile phone tinkerers .. there are many results for unbricking btw, but I just checked and with the first result it seems that Apple now uses unbricking for activating a new device. Because technically before, it is also just a brick - but here I would agree, that it is not a appropriate term, but rather should be for somehow broken devices.
The JPL doco (https://news.ycombinator.com/item?id=36941433 https://news.ycombinator.com/item?id=36942321) calls it "Command Loss".
In other words it was effectivly a brick to me.
But since it was not a surface pro (I considered buying instead of that one), I could open it and disconnect the battery.
And in effect, unbricking it. Quite trivial fix sure, but nearly impossible with many modern devices, where the battery is glued in.
My point is, not every mode of operation is desired, especially if you cannot change it. Then you might as well have a brick in terms of usefulness.
It was not making its own decisions, to achieve some goal.
It may have seemed that way to you, but actually no. "Bricked" has generally referred to devices that are likely straightforwardly recoverable, but for a lack of documentation from the manufacturer.
Most folks don't really know how to use say Android fastboot or recovery modes either, yet we wouldn't call a device with a wiped system partition "bricked".
Most "bricks" are things like a bootloader getting erased. Reflashing that through the standard process of JTAG or another debug protocol is a straightforward action (after all, the manufacturer has to get the first bootloader on there to begin with). The port pinout and config info just hasn't been publicly documented by the manufacturer, which is what pushes it into the domain of "experts".
Precisely. 'Bricking' something means it is unrecoverable and is irreversible.
No idea a what point in time the definition was changed to mean 'temporarily not working'.
Also, it's not a technical term with a rigid definition, hence "soft-bricking"
What is bricked vs recoverable has always greatly depended on time and effort, individual skill level, available hardware/software tools, documentation, crypto keys, physical access, willingness to replace individual parts etc.
Sometimes, even within an org, some teams e-waste expensive devices that aren't bricked deeper than what other teams recover from as part of everyday workflow.
Taking a typical network device as an example, where do you draw the line? Driving to a remote location to plug the cable into another port, pressing a reset button, booting from USB, flashing a new firmware with TFTP, plugging in an external or internal console cable, opening the case and soldering a header to get access to the console, doing the same with no documentation, or an unknown (but maybe Google-able or reverse engineerable) password, flashing firmware with JTAG, shipping the device back to the engineers (or shipping an engineer to the device)...? It's always been arbitrary.
If you are able to fix it then it is not bricked.
One device may be bricked to one person but not to another. But that must still be the definition, right? Otherwise the word has no meaning.
Find my contracting rates below.
Then "totally unrecoverable" is rare and the term bricked has always been relative. Your bricked device may be as good as new to someone who has a JTAG adaper and knows how to use it.
That's some impressive science there, not like there is a deep-space GPS.
Does it look for the sun and figure out from there?
"... and celestial referencing instruments (Sun sensor/Canopus Star Tracker) to maintain pointing of the high-gain antenna toward Earth"
"- African dung beetles orient to the starry sky to move along straight paths
- The beetles do not orientate to the individual stars, but to the Milky Way"
https://www.cell.com/current-biology/fulltext/S0960-9822(12)...
https://www.science.org/content/article/dung-beetles-navigat...
https://space.stackexchange.com/questions/43803/how-did-the-...
"The high-gain antenna has a beamwidth of 0.5° for X-band, and 2.3° for S-band."
At 130-150 AU, the earth is always within about 0.4° of the sun. Since commands are sent on S-band, pointing directly at the sun gets a pretty good signal.
Being drowned out is harder than you might think. The maximum data rate of a weak signal is 1.4 x [bandwidth] x [signal-to-noise ratio]. If you transmit across a 200MHz band, and your signal is a million times weaker than the noise, you can do hundreds of bits per second.
Second order calculations use careful analysis of the signal pattern in telemetry data- IIRC you can see a slow stretch of the phase which can be used to estimate distance and velocity with high accuracy.
Voyager, along with Apollo, stand as the finest examples of human engineering done yet- we got a bunch of people to the moon and back, and built a probe that still operates 50 years later... farther than anything else humans have launched... I'd be lucky if I can deploy my web app once a week.
In vanilla, all antennas are omnidirectional and as long as there is a working solar panel on a probe, it can be used as a relay sat.
But it does gives you a similar feeling when you schedule a maneuver on a point where the planet/moon blocks your signal, and you only realize after it's too late
Many moons ago when I was hands-on and stressed about migrations & config, my team lead at the time would say exactly the same thing - his wife is a doctor and her job is way more stressful - People die. And I bought into it as a relief for a while.
But... I work on a payroll system. My team does impact people. Mistakes can have important negative consequences to real live individuals - from stress invoked in trying to call help centre and fix their paycheques, to disconnected utilities if they don't get paid correctly/timely, to other downstream consequences.
Any number of other IT systems have significant consequences - e.g. airline ticket systems, airbnb bookings, etc. I feel the "nobody died" is a double-edged sword: it can help relieve people of the daily sense of artificial stress, urgency and grind that management may impose; but also builds a false dichotomy / unreasonably binary threshold on when our job matters / impacts ...
This type of thinking also follows from decades of experience.
For some reason the software engineering world largely abandoned esteem and respect for all of the above.
To imply this was a software bug is a pretty silly representation - the system was poorly engineered and didn't have proper contingencies for sensor disagreement. This is pretty clearly a design/engineering error with a software component.
Besides, the guy said "rarely ever matter" for a reason, not "explicitly never impact things"... Bit of a silly comment from you IMO
In the case of the 737MAX, the software was a design around a physical constraint; that doesn't mean the software doesn't matter. Most software is designed as a workaround of a certain physical or mental constraint.
The pilot couldn't even turn MCAS off originally. That's not a software thing, that's a "who the F designed this" thing.
As the amount of coordination increases, the number of failure modes tends to grow quite fast. That's why software failures in physical, safety-critical systems are not trivially corrected. There are a lot of second order effects that need to be considered.
It fails like buildings near fault lines, because the ground moves under them. Think broken dependencies, operating system obsolescence, et cetera.
An apropos and famous example is the Ariane 5 rocket mishap. The same validated software from the Ariane 4 was used, but the hardware design changed. Specifically, the velocity of the Ariane 5 exceeded that of its predecessor and exceeded the 16-bit variable used.
I would love to see a day when redundancy like this is just a standardized, accepted practice rather than a stand-up debate. Easier said than done of course.
Clients who want NASA quality can have it if they bring NASA budgets and timelines.
Add civil engineering to that nowadays - both buildings and roads.
Sure, there are regulations and licensing, but quite often the entity financing the whole thing cares little about such things.
In cases where fault tolerance isn't as robust, it's for the same reasons as other disciplines you mentioned: budget and importance.
My impression from friends working in other engineering disciplines is that software engineering works similarly to other fields: the more risk to human lives is involved, the more testing, redundancy, etc is involved.
Standardisation - in the big 'E' Engineering world, there would be a recognised international standard for Web Apps that ensured/enforced that all Web Apps supported this functionality, or they would not be approved for use.
Another factor is Accountability. A senior Software 'Engineer' would have to take personal responsibility (liability, accountability) that the software product they are producing and/or overseeing met all these requirements and personally sign off that these standards have been met. If the product were to fail at any point and it was determined that the cause was negligence in following the standard, any damages sought (not common, but not unheard of) would ultimately find their way to the accountable individual and their insurance.
In cases where budgets/importance don't allow for this level of scrutiny, there would still be paperwork signed by the producer of the software and the client acknowledging deviation from the standard and waiving any recourse for doing so.
See the Australian Design Rules (which happens to form the basis of most UNECE and Canadian transport regulations) if you want to see how detailed they are.
To continue the analogy from earlier - standards wouldn't mean all web applications would have to be designed, programmed and work exactly the same way, but it would mean that they would need to be formally tested (to an approved test plan), and to use your example, would need to demonstrate that each of those layers of fallbacks (as dictated by the standard and covered in the test plan) operate correctly in order to be certified.
If anything, I think software has a huge advantage over physical world engineering in that testing can be replicated at virtually no cost whenever a change is made to the design. I shudder to think how many cars get trashed in order to meet vehicle safety testing requirements.
Here is the Australian Standard for Caravan and light trailer towing components, Part 1: Towbars and towing brackets
https://store.standards.org.au/product/as-4177-1-2004
There are thousands of these documents covering everything to do with transport from the vehicles to the reflectivity of street signs.
The regulation (at least in my state) is that only engineers who are registered as Registered Engineers are permitted to carry out professional engineering services in this state.
The main contingency with most software is that you fix it.
It’s the cost when something fucks up.
If I’m holding my phone near a cliff, and I rely on it for navigation and I’m hours from civilization, I’m a little more careful, not because I’m normally super careful. It’s because — in that specific scenario — losing my phone would cost me so much and the chance of it happening is much more likely.
Space companies spend a little extra because the cost is years of development and billions of dollar evaporating in a few seconds.
And there are software teams in certain industries that dot their I’s and cross their T’s as well.
Even on some dumb CRUD app, if it’s a critical piece of code that the rest of the software hinges upon, you spend a little extra time because the cost of fuck up is so major.
Or you’re launching a product and you have a sign up that will seed your user base, you damn well make sure it works.
Jokes aside I think it's mostly a value/cost thing. NASA's software has different requirements and failure scenarios than most software developers (in this context I will not call them software engineers) have to care about. Verifiable correctness is harder to predict, and in most devs' roles it's easier to just try something and see what happens, rather than know what'll happen up front.
it's a lower bar for entry, any kid can run a compiler -- it's harder to acquire a bulldozer and planning permits.
similarly if you look at 'diy' or 'garage' engineering you can find all sorts of hazardous/poorly-built/never-should-have-existed death traps. How many people have died in recent years from fractal burning?
it's still engineering -- they're building their own tools -- but it's within a realm (DIY/maker) that historically has undersold the dangers inherent with the things.
Why? Mostly because they're self-taught, mentorless, and without the direction within their education to be taught the importance of engineering rigor, similar to the kid given the compiler who starts making forkbombs and goofy dialogs to prank their friends.
That's what happens when there's no liability. Even if you're a billion dollar corporation, you can just slap some standard legal boilerplate disclaimer on the license agreement and absolve yourself of all responsibility even when hackers pwn the private information of your customers.
Some version if this legal boilerplate can be found in virtually all software licenses, including open source ones:
THE SOFTWARE IS PROVIDED "AS IS",
WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT.
IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING
THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY,
WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.
An embedded device with a failed bootloader update is considered “bricked”, even if you can open it up and reflash it with a valid bootloader.
I don’t know why folks are so insistent on gatekeeping the word, as if doing so demonstrates some superior personal knowledge.
Weird discussion since we're talking about a piece of hardware that is working fine and doing exactly what it was told to do. They just pointed it in the wrong direction and need to wait for it to recover, which it is set up to do.
We wouldn't say a server with the router IP address misconfigured was "bricked." (or maybe we would... I guess the jargon changes, but that would seem pretty crazy to me)
I would love to see a picture of your computer pulling itself up by the straps on its physical boots the next time you press reset. Bleeding when a process is "killed".
Even something as superficially similar to real-world behaviour a "queueing" is implmented in a very different way in software, for the most part.
sudo wg-quick down wg0
Any time I’m ever doing a pfctl to change my router’s firewall rules I schedule a “revert to the n-1 rule set” job two minutes from now to avoid the trip to the basement & hunt for the serial cable.
I do +2 because once I was too close to the next minute, fat-fingered the update command & instinctively corrected it. But the change had reverted before I hit enter on the new update, so of course I blocked too much traffic and wedged my SSH connection, triggering the cursing and trip to the basement with the serial cable.
He'd set the resolution to 1024x768 on the 800x600 panel, once I fixed that I was good to go!
LOL. Outside of computer dorks what not, it's not very common for people to have a large selection of cables to go digging through. Maybe a drawer full of decrypt USB cables that came as chargers to a phone, but most people I know lose those cables and are constantly buying replacements. So expecting the average person to have a box full of random cables suggests to me you might be living in a bit of a bubble.
hell, even homeless people such as myself tend to have giant collections of random cables, even when they don’t actually have anything that requires charging.
More seriously, Mikrotik routers have a nice feature where they will rollback your config change if the connection you’re configuring one over stops responding to keepalives. Like a lot of Microtik features, it’s probably copied from some Serious Business network OS, but I wouldn’t know.
I wouldn't know who came first, but it's a feature of JunOS (Juniper) as well: every config apply first applies the config, then waits for confirmation on the terminal where it was ran. If confirmation isn't given within X seconds, it reverts the config change.
Not sure who came first, but OpenWRT does this if you make a breaking change in the webinterface, and connectivity is lost for 60 sec, it will rollback the changes.
# ifconfig eth0 down; ifconfig eth0 up
# reboot
and we are charging our phones daily....
I did this exact thing in the small last night - wanted to work on fixing a faulty switch, so my wife and I get on the intercom system on our landline phone so she can tell me when the correct breaker is off.
And of course, breaker #1 is the one that controls the intercom, severing our connection.
But luckily it automatically readjust itself to earth automatically every half year exactly for these events. So on 15.10 we will know, if it is really lost. In either case, the end of its mission is near anyway, because the nuclear batteries are near its end.
edit: Nasa has a blog post on this https://blogs.nasa.gov/sunspot/2023/07/28/mission-update-voy...