News for hackers. Stuff that matters.
Re:NASA Pulls Off 160-Million-Mile Software Patch by dmit on Saturday August 18, 2012 @07:07AM (#4400820)
Related Google Tech Talk: Debugging Code from 60 Million Miles Away https://www.youtube.com/watch?v=_gZK0tW8EhQ&hd=1

Ron Garret talks about DS1, a project in the 90s where a small team used Lisp to send an autonomous craft into space in a third of the time and at a fraction of the cost of previous comparable projects. (Yes, I couldn't have made up a more HN-friendly story if I tried.)

Re:NASA Pulls Off 160-Million-Mile Software Patch by ovi256 on Saturday August 18, 2012 @07:17AM (#4400842)
But did it have paying customers ?
Re:NASA Pulls Off 160-Million-Mile Software Patch by shiven on Saturday August 18, 2012 @07:27AM (#4400858)
Nope. They never figured out the right pricing model for their API :)
Re:NASA Pulls Off 160-Million-Mile Software Patch by sgrove on Saturday August 18, 2012 @10:49AM (#4401266)
Ah, hadn't seen that one, thanks.
Re:NASA Pulls Off 160-Million-Mile Software Patch by SwaroopH on Saturday August 18, 2012 @05:35AM (#4400690)
Impressive (even more for the geek me) but done before. Only relevant because of comparison with hellish OTA process for non-Google Android devices.

https://twitter.com/#!/search/realtime/android%20nasa%20ota

Re:NASA Pulls Off 160-Million-Mile Software Patch by noselasd on Saturday August 18, 2012 @06:51AM (#4400792)
They can even patch Voyager, now at the edge of our solar system: http://www.spaceflightnow.com/news/n1005/11voyager2/

They also patched the MER rovers many times; This bug caused quite a few headlines back in the day: http://www.nasa.gov/offices/oce/llis/1483.html

Re:NASA Pulls Off 160-Million-Mile Software Patch by sehugg on Saturday August 18, 2012 @11:55AM (#4401422)
I seem to remember a Scientific American article from the 1980s where they also had to rewrite code around a dead bit in RAM. They've done some pretty drastic changes since, like using the backup computer to add image compression on the fly (lots more detail here): http://history.nasa.gov/computers/Ch6-2.html
Re:NASA Pulls Off 160-Million-Mile Software Patch by sanarothe on Saturday August 18, 2012 @02:27PM (#4401836)
I once nearly 'bricked' my router playing with some custom firmware. That in itself was terrifying, let alone the prospect of 'bricking the rover.'
Re:NASA Pulls Off 160-Million-Mile Software Patch by ahelwer on Saturday August 18, 2012 @04:56PM (#4402286)
Being responsible for the update would take years off of my life. One slip up and you've thrown away 2.5 billion dollars.
Re:NASA Pulls Off 160-Million-Mile Software Patch by sigkill on Saturday August 18, 2012 @05:11PM (#4402342)
Sure 2.5 billion dollars is a large sum of money. Imagine how you'd break this news to the public. You've practically crushed people's hopes and dreams. "Umm, guys, we just accidentally bricked the rover because of a firmware upgrade gone rogue". I'd end up in tears if I had done this.

Speaking of firmwares, I'm unable to find out information if they have a dual bios kind of system where if the boot fails it can recover from the backup firmware.

Re:NASA Pulls Off 160-Million-Mile Software Patch by skrebbel on Saturday August 18, 2012 @09:02AM (#4401021)
Wait, they're doing this by hand? Production deployments to a computer on a different planet? In the 21st century? If there's one deploy I'd want to automate the hell of (and test, over a 2 baud network), it's this one.

It's not like these guys never did a remote software update before. So I must be missing something.

Re:NASA Pulls Off 160-Million-Mile Software Patch by lisper on Saturday August 18, 2012 @10:43AM (#4401250)
It depends on what you my by "doing it by hand." The update process is "automated" in the sense that it's completely scripted. Every step is known ahead of time, and has been extensively tested on ground-based duplicates of the flight hardware. The only aspect of the process that is done "by hand" is "pushing the big red button" to initiate the next step of the process, and even this is a very stylized and well rehearsed process. The only reason for having even this step of the process be manual is so that humans can assess the situation between steps and satisfy themselves that nothing has gone wrong with one step before proceeding to the next. That's the real challenge in a situation like this: your communications link is operating on the hairy edge of the limits imposed by the laws of physics, so lots of things can go wrong in production that worked in rehearsal.
Re:NASA Pulls Off 160-Million-Mile Software Patch by skrebbel on Saturday August 18, 2012 @02:13PM (#4401788)
i see! Thanks for the clarification. And, makes a lot of sense.
Re:NASA Pulls Off 160-Million-Mile Software Patch by jonknee on Saturday August 18, 2012 @10:38AM (#4401236)
Who said they're doing it by hand? It takes 28 minutes to receive feedback from anything, so they're going very slowly and monitoring what happens.
Re:NASA Pulls Off 160-Million-Mile Software Patch by kbatten on Saturday August 18, 2012 @09:24AM (#4401067)
What is your proposal to automate this? Remember there is very large delay and blackouts that happen.

After they do it by hand several times, sure automate it.

Re:NASA Pulls Off 160-Million-Mile Software Patch by ehsanu1 on Saturday August 18, 2012 @09:29AM (#4401074)
Automate it in a test environment that simulates production as closely as possible. Once the automated software updates work flawlessly under various conditions, use that in production.
Re:NASA Pulls Off 160-Million-Mile Software Patch by vidarh on Saturday August 18, 2012 @10:05AM (#4401151)
Then an unexpected condition happens and the automated response breaks something badly, and suddenly your expensive robot is now an expensive piece of junk.
Re:NASA Pulls Off 160-Million-Mile Software Patch by jonknee on Saturday August 18, 2012 @10:40AM (#4401241)
What makes you think they haven't already tested the hell out of it?
Re:NASA Pulls Off 160-Million-Mile Software Patch by rimantas on Saturday August 18, 2012 @12:00PM (#4401436)
We are not talking about some machine in amazon cloud there. We are talking about the robot on the freaking Mars.
Re:NASA Pulls Off 160-Million-Mile Software Patch by sktrdie on Saturday August 18, 2012 @06:09AM (#4400740)
I find it weird that the rover wouldn't have enough storage to hold both the logics of the landing and the surface mission. I mean, SSDs nowadays can hold hundreds of gigs and are very small.

Does anybody have more details on that? A complete software update just sounds like a really risky thing, for something that you have no physical access to.

My opinion is that they probably have really strict protocols from legacy missions, and since they just work, they're not going to change them.

Re:NASA Pulls Off 160-Million-Mile Software Patch by jgrahamc on Saturday August 18, 2012 @06:13AM (#4400743)
Every time there's a report about Curiosity doing something that overlaps with software or hardware that we at HN are familiar with comments like this appear.

The bottom line is: (a) Curiosity was designed 8 years ago so the tech. on board is old (b) everything that's on board has to be radiation hardened and (c) NASA are conservative about 'new stuff' because they need it to work (e.g. the thrusters on the skycrane were derived from the Viking landers).

Re:NASA Pulls Off 160-Million-Mile Software Patch by angstrom on Saturday August 18, 2012 @09:43AM (#4401096)
Now tack on the fact that it has an initial 2 year mission powered by an RTG. http://en.wikipedia.org/wiki/Radioisotope_thermoelectric_gen...

...but not just any RTG, a MMRTG: http://en.wikipedia.org/wiki/Multi-Mission_Radioisotope_Ther...

It's able to supply a steady 125 W of power for 14 years, before dropping to 100W. The technology has been used reliably on multiple projects. Based on the longevity of the previous rovers that have safely made it to Mars it's worth considering that barring mechanical malfunctions and hardware failures (it has two main computers) the rover could turn power off to non critical components after 14 years and continue to run on 100W. It might not be far fetched to think that this rover could still be running if/when humans arrive in the 2030s.

That's far cooler than what you would get running on the latest and greatest for convenience.

Re:NASA Pulls Off 160-Million-Mile Software Patch by felipemnoa on Saturday August 18, 2012 @02:19PM (#4401808)
>> the rover could turn power off to non critical components after 14 years and continue to run on 100W

That constant source is charging a battery. As long as the battery is still working it would just take longer to charge the battery.

Re:NASA Pulls Off 160-Million-Mile Software Patch by sp332 on Saturday August 18, 2012 @02:41PM (#4401882)
How does the radiation from the power supply not screw with the electronics?
Re:NASA Pulls Off 160-Million-Mile Software Patch by robryan on Saturday August 18, 2012 @07:03AM (#4400809)
I wonder whether it will get to the point where they just send processors was a large amount of redundancy, multicore/ multiprocessor, and just not trust any one result but probabilistically across operations in parallel.
Re:NASA Pulls Off 160-Million-Mile Software Patch by harrywincup on Saturday August 18, 2012 @07:45AM (#4400891)
It must be unbearably frustrating building this stuff to work around the limitations of 10+ year-old technologies when the superior tech inside even the phone in your pocket could achieve so much more.
Re:NASA Pulls Off 160-Million-Mile Software Patch by mturmon on Sunday August 19, 2012 @04:04AM (#4403532)
Almost all of the radiation effects appear to be single event upsets (SEUs). So in principle, it should be possible to design robust software to detect and recover from faults (SEUs do no permanent damage to hardware).

There have been efforts within NASA to design robust OS extensions -- redundancy, watchdogs, heartbeats, checkpoints, etc. -- so that fast/cheap COTS hardware could be used, at least for some functions like image processing. This has never reached critical mass and been carried to completion, however.

So yes, as you guessed, there is significant pain and heartburn that computational capabilities are so far behind. Engineers have to spend a lot of time and cleverness squeezing, say, stereo vision processing, or image compression, into the hardware available.

Re:NASA Pulls Off 160-Million-Mile Software Patch by papalalu on Saturday August 18, 2012 @06:17AM (#4400751)
presumably the 8 year old design would use already-proven tech too..
Re:NASA Pulls Off 160-Million-Mile Software Patch by ballooney on Saturday August 18, 2012 @06:44AM (#4400781)
Indeed, their shopping list 8 years ago was not from Newegg with whatever Intel had at the time, it was whatever BAE had spent many years rad hardening and qualifying by that point, which would itself have been based on an already proven CPU (read: older) when BAE first started looking. All of a sudden it's 1993.
Re:NASA Pulls Off 160-Million-Mile Software Patch by facorreia on Saturday August 18, 2012 @08:35AM (#4400974)
Maybe there's a lesson for us right there.
Re:NASA Pulls Off 160-Million-Mile Software Patch by akent on Saturday August 18, 2012 @06:24AM (#4400757)
The CPU is a radiation hardened PowerPC 750 known as a RAD750 made by BAE. Some more info at http://news.cnet.com/8301-11386_3-57491281-76/slow-but-rugge...
Re:NASA Pulls Off 160-Million-Mile Software Patch by rwmj on Saturday August 18, 2012 @07:03AM (#4400808)
Raises the interesting question: Are there radiation-hardened hard drives (not SSDs)?
Re:NASA Pulls Off 160-Million-Mile Software Patch by Someone on Saturday August 18, 2012 @10:41AM (#4401243)
Hard drives will not work in a vacuum. http://www.dansdata.com/spacecomp.htm:

"Many people think hard drives are hermetically sealed, but they're not - they just have no through-flow ventilation. In vacuum, a hard drive will instantly eat itself when turned on.

[…] Even if you run at full atmospheric pressure, you don't want all your computers to die if you lose some air.

Re:NASA Pulls Off 160-Million-Mile Software Patch by rwmj on Saturday August 18, 2012 @10:46AM (#4401259)
Very good point. I guess the other thing I hadn't thought about is temperature (variability in particular) which must affect hard drives with all their moving/expanding metal parts a lot more than it does SSDs.
Re:NASA Pulls Off 160-Million-Mile Software Patch by Raphael on Saturday August 18, 2012 @12:06PM (#4401455)
Apparently he could make the text look fine, but some of the graphics weren't scaling along.
Re:NASA Pulls Off 160-Million-Mile Software Patch by jlgreco on Saturday August 18, 2012 @06:22PM (#4402518)
It probably wouldn't be too hard to do. I think the problem with that for an application like this would be standard physical harddrive reliability issues. More moving parts, no matter how reliable we normally think of them, probably wouldn't be something that would thrill the people who build these rovers.
Re:NASA Pulls Off 160-Million-Mile Software Patch by Dylan16807 on Saturday August 18, 2012 @04:41PM (#4402244)
The main storage is 2GB. Each camera has 8GB of storage. There is no way in hell this was solely because of space constraints.