They used trios of regular consumer grade disks/servers etc as a cluster and it looked like a single node.
They had to replace a LOT of hardware but this was still cheaper than big iron industrial grades servers.
By their very nature model predictive controllers operate in a world where not everything is perfectly modelled. Engineers do their best and whatever is left is the "error" the controller is trying to deal with.
Sort of like how you can balance a few pitchers of beer on a tray in your hand by remaining aware of the weight, even when people remove one! hahaha :)
If it's no stronger than a sudden wind gust, it's just something the controller has to be able to take care of without a heads-up.
Then the next launch crashed due to slosh induced oscillation - and the one rocket after that had anti-slosh baffles. ;-)
More fluid dynamics
Which is why we use wind tunnels, for example.
These are indeed heavy computations. What I meant is that VoF is one additional equation to be solved besides the N-S equations (either filtered as in LES or Reynolds-averaged as in RANS), the energy equation, your turbulence model equations, and so on. Certainly, not instantaneous at all, but simply an additional "simple" model that we can hook into our current way of doing CFD.
So, my point was, sloshing is a problem that we know how to simulate, although certainly you need HPC resources. Though, looking at those 100k NVIDIA H100 Elon has, I guess they have them! :P
I think it's a huge problem when re-lighting the engines in orbit, though.
Even a real-time simulation should have some measurements to self correct to some degree. Otherwise it'll diverge.
Bug tanks make sense there & they might not be always full. So I can imagine all kind of interesting ways you can work with the fuel in zero go to avoid not only slosh but also the need for ullage thrusters. Eq. some programmable nozzles using in-tank gas to nudge liquid fuel blobs to move in the right direction. Or even some nets or bags that herd in the fuel in the middle of the tank + prevent it from directly touching the side, reducing boil-off or refrigeration requirements. :)
This has been known in the high availability and safety systems industry for a while and a good book to learn these reliability engineering techniques is "Reliability Evaluation of Engineering Systems".
The book is available on amazon: https://a.co/d/1nH824K
https://www.epa.gov/radiation/radiation-basics#:~:text=Gamma....
(Though likely not of course ;)
https://www.wolframalpha.com/input?i=+lead+cube+with+side+le...
So something might be off with your assumption of 1500 usd / kg.
I can't decide if I'm joking or not.
Maybe you could improve the system availability considerably by a bit of gamma radiation protection combined with some more parallelism of the components ..
A second layer blockage for the secondary particles wouldn't have to be as dense or am I missing important physics?
(I guess a lot of gamma radiation would still reach this second layer so please ignore my question :)
But would using redundant systems separated in space connected with each other not offset the chance that they all would be affected at the same time? This is actually not rocket science .. just hard engineering and hardware/software design for redundant systems which is also usable on the ground.
"Hollman also found that creativity got him a long way. He discovered, for example, that changing the seals on some readily available car wash valves made them good enough to be used with rocket fuel."
"Elon Musk" by Vance pg 123
I doubt any Lowes parts made it to space, but you know some went into test articles
Perhaps the key is to be relentless, and resourceful.
The Fukushima and Deepwater Horizon disasters show that this knowledge has not penetrated other industries.
But it turns out, it doesn't matter how many redundant backup diesel generators you've got if a 45-foot wave comes along and they're all left underwater.
the orbital and planetary mechanics kind of suck. They're meant to provide a decent 'arcade realism' for the sake of player/player interactions and pvp/pve.
if you want to experience fuel slosh/weight during a vertical ascent/descent go with kerbal. It models a lot of that stuff without mods -- and mods can make the model even more accurate.
Algorithms can be faulty as well.
Which was never claimed.
That paper is a little bit about Erlang and a whole lot about OTP and other methodology and design technique.
It is still, very much "the paper" for distributed systems, though its applicability to this particular problem is limited.
Does anyone know (or have educated speculation on) what kind of hardware is running these algorithms? Like, do they have a linux machine that's running the control loops? Are we talking megabytes, gigabytes of SRAM?
I would think no -- you would definitely need hard real time for something like this. But my only experience with real time systems is in tiny MCUs with kb of SRAM. That's definitely too small for a controller like this.
Really curious about the nuts and bolts of this.
When you build something like this, you're torn between having a big model that represents everything and a smaller model that is easier to validate and reason about. Based on simulation, you might go for a smaller model that "knows" to stay away from operating areas where hidden variables (like really complicated tank slosh) invalidate the small model.
I doubt the actual control loop is too much processing, but it's certainly possible to build controllers with SDRAM, millions of variables of state, and hard realtime processing, though I wouldn't build it on top of preempt-rt. ;)
>probably involve some kind of fast numerical fluid simulation
https://ntrs.nasa.gov/api/citations/19700023342/downloads/19... (search for "slosh" or "shaping network")
My stepfather worked as a programmer on the Apollo program, and the thing he always talked about as his biggest accomplishment was working on the "slosh problem" -- so yeah, props to the SpaceX team for managing that landing. And props to my stepdad for managing it on hardware that was... a billion times less capable? :-)
I am curious but clueless about these problems, can you expand more?
Well, that makes perfect sense considering that both the spaceport infrastructure, and the booster need to do their calculation on the ground level instead of the highly radiated environment that is space. However, for the rockets themselves, which happen to reach that harsh environment, they may use more resilient and expensive hardware in the future, after passing over the current "let it splash in the Indian Ocean" development and testing phases.
Surely this isn't necessary with a small enough sensor granularity or whatever the terminology is. You can have very dumb software if it reacts quickly enough to changes in perception.
Instead, it's more common to use gasses injected at the top of the tank to push the liquid to the bottom. Falcon 9 uses helium. Starship uses https://en.wikipedia.org/wiki/Autogenous_pressurization as well as small header tanks for the landing propellants.
Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips.