ROS provides a plug-in backplane that allows you to innovate in one area while leveraging existing components for parts that are not your differentiator.
Just make a multi-threaded app, and call functions. If you need to distribute over multiple CPUs, then go ahead and do some IPC. But pub/sub is NOT an architecture. Its a soup of tightly coupled fragments of functionality. It falls apart very quickly in the real world.
I'd be interested, though, in your suggestions on what a real architecture for robotics looks like in your mind? I still remember the time before ROS, 20 years ago, when each robotics team had to designate a sub-team just for building and maintaining the middleware. That was a waste of time and effort. But you seem to suggest that we go back to that? ROS might not be perfect but it's so much better than anything else that exists. It's also open source and we can all work together to make it better rather than reinventing the wheel each time.
Nodelets are also a disaster, which is why ROS2 kinda fixed this by decoupling nodes and processes.
When you're just starting, ROS can be nice for prototyping - you get a batteries-included platform that can do some SLAM and simple motion planning. But as you start adding new features, you need to figure out how to add those features over multiple nodes. This coordination overhead can quickly bring your system to its knees, or at least make it extremely difficult to debug and troubleshoot when things go wrong.
No one should be building or maintaining middleware. Build robots. Read your sensor data, build a model of the world, decide what to do, then send commands to your control systems. This is the hard part of robotics.
ROS solves the easiest part of robotics (plumbing and process management) in the shittiest possible way.
A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates.
The reason ROS has traction is because of university labs, who just need something to work once to be able to publish their paper or write their dissertation. In industry the reliability requirements are much higher, and despite the intensive efforts from the ROS community to "industralize" ROS via adoption of DDS, there seemed to be little understanding that the message protocol wasn't the reason the industry uptake was so low.
This is how I've structured a control system, but using a pub/sub system to share data across those boundaries (and log/inspect in general). "Nodes" that share some resource or fundamentally run back to back based on the data flow can live in the same thread. Higher rate components (eg inner loop controller) live in a thread with a much higher priority. All of this is event driven, deterministic, and testable.
If you have more details about the system you're imagining I'd love to learn more, because so far I don't see what's incompatible with what you've described.
- more should be done by setting up constrained functionality-specific data, and then simply calling a function with just that data. Right now a lot of things are passed state they don't need just because it is part of the message they receive. This makes the code dependencies way harder to separate, because you effectively share function signatures (messages) between different modules. Of secondary concern is the extra memory bandwidth from the extra data passed around, and not being able to pass by const& due to the async.
- lots of things don't need updating at all until they tick over. If you don't update it frequently it has old data. You can try to make sure it works with the latest data by updating it frequently, but that of course has big overhead. I don't see any decent way of making this work unless you either 1) set up global threadsafe state so everything can access it, which is bad from a dependency and locking perspective, or 2) just call functions synchronously with exactly the data they need.
- The issue with message queues, beyond the need for messages to generalize as mentioned above, is that often components need multiple different messages from different sources to perform their tasks. This means every component needs to keep a local copy of the data they need, to translate this async data back into a synchronous paradigm when it runs. In fact, beyond the message passing itself, pretty much everything needs to do its work in a synchronous paradigm, so why even add the async stuff in the middle to begin with? Once the messiness of different sensors' reporting rates is consolidated into the EKF state, and external commands and directives and brought in, after that point pretty much everything could be synchronous. No overhead, no timing issues, no "which message has the data I need", no "why do we have 3 different variants of the same message with slightly different data and which one should I use", etc.
As long as you log the initial input data it should still be able to do replays for reproduction of realtime behaviour, but a) better testing can be done because you can actually tell what every subsystem needs to run correctly just by looking at its function signature, and b) its much easier to refactor (and understand) because of the same properties of the function signature describing all of the inputs and outputs.
"Just" a multi-threaded app and calling functions isn't really a good replacement. For something really basic, sure, but the modularity that ROS introduces, and the core principle of having serialisable recordable messages delivers a lot of value once you start building large systems - particularly those that can span multiple physical machines. But pub-sub is _too_ general as a concurrency model, if ROS had adopted something more constrained that was provably deterministic and was amenable to real time analysis then a lot of its concerns would go away.
But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over? Even modern microcontrollers are blindingly fast, and the majority of the time they spend these days is because of our brain-dead architecture decisions, not due to the underlying problem that actually needs solving.
I haven't had a chance to try it yet, but in theory I like the idea of something like Zenoh Flow to describe the larger data flow graph. https://zenoh.io/blog/2023-02-10-zenoh-flow/
> But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over?
I think your context switch timing is off by several orders of magnitude, but regardless these things aren't one extreme or the other. For sharing data across threads, to an external system, or logging and visualization I still like pub/sub (and I've seen more than my share of horrible abuse), but it definitely shouldn't be treated as one size fits all.
Beyond that, robotics isn't all that special. There are other domains which have soft-realtime requirements and huge scale. AAA video games are a multi-$100-billion industry, which largely ship single-process systems with extreme performance on par with state-of-the-art perception and motion planning use-cases.
AR/VR is another area where this is interesting - https://twitter.com/ID_AA_Carmack/status/1497098684546400260 - Carmack has alluded to this issue multiple times.
Being able to have some shared memory you pass around between safely sandboxed worklets still has any of the same coordination problems. But it at least has much much much less multiprocess jank.
Quests's jank problem is probably in part that multiprocess architecture has to spawn new processes & change priorities of what's running & handle expensive context switches between processes elegantly, which is hard. Not only is non-monolithic code sometimes harder to reason about, it has all kinds of performance boundaries that are very painful to cut through.
Read through Pipewire changelogs & how they've worked so hard to make best use of kernel primitives with less and less overhead, look how hard they've worked to get near Jack levels of efficiency. Great example of the slow march of multi-process optimization.
Sandboxes let you collocate different processes inside the same process. You still have many of the same architectural challenges, but the ability to spawn new code & run it has much much lower overhead. It creates the possibility to make way more multi-process like architectures than we can explore now.
And at that point why bother with an OS? What benefits are and OS providing then anyways? Just deploy with a unikernel.
Elsewhere in the threads here is a discussion of Nodelets, which are often used to load & run trusted code on ROS inside of host processes. This is like a jank special homebrew version, with a lot of pain, of what you get from wasm Sandboxes, where you can quickly spin up cheap fast runtime and connect them as nauseum.
That's really the key. Sandboxes are really about having many (typically different) runtimes in process. Often these will be processes working with processes. As I highlighted already extensively, the benefit is performance & overhead, versus native processes & native ipc. Sandboxes are just a known term for a runtime within the process, for a subprocess.
Likely 1000’s of topics, I’ve never counted.
I am not interested in moving to a single app in a single language at the moment.
Industrial robot arm motor control loop? 1000Hz, single-threaded, dedicated CPU core or even an FPGA.
Robot vision system? 30Hz, tolerance for occasional dropped frames, and some lower priority tasks like saving audit images which are nice to have but not critical. A small monolith with several threads would be a fine choice.
Assigning robot-sorted widgets to customer orders? 1-5Hz, go ahead and use the web coding practices where 200ms is considered a fast response time.
All of these are serious robotics - but they could reasonably be coded in very different ways.
- a large amount of data can be transferred asynchronously. Pub/Sub is a great pattern to achieve that; the cases when you need a synchronous interface are the minority.
- a robotic software has MANY components, you need an approach that incentivize decoupling and lousily-coupled interfaces.
- this lousily-coupled architecture has been a catalyst for innovation and cooperation, because it makes it easier for people to share their code as "building blocks" of a larger system.
Is there a considerable overhead? Sure! But it is much more complicated to write thread-safe code in a huge, monolithic application.
Loose coupling is good if you mainly use open-source packages and modify one component for your research paper. I'm not sure it's a net positive when building most components yourself.
A big multithreaded program can still use queues instead of locks to share data between threads.
Everyone's working on their own ROS nodes in isolation....with no thought to eventual integration testing? It doesn't seem like that is a fundamental shortcoming specific to ROS.
The hard parts were never in the middleware (ROS), but stuff like perception, world modelling, decision making. ROS gives you an ecosystem of decent options to choose from. And improve on if you need to at all.
The standardized interfaces and IPC make it very easy to plug, at runtime, eg. a different localization algorithm into the system.
Heh, yes. At some RoboCup@Home competition (IIRC, Mexico City in 2012 so a while back), a team came in super confident, saying everything worked and they were going to win. Most of it worked ok in the lab, and well in the simulator too, they said. Didn't make it to the 2nd stage I think. And RoboCup is still not exactly 'the wild' for robots.
while(true) { read sensors update world model decide what to do act }
You should only deviate from this when you have a specific reason (concurrency, libraries, IPC, etc). You can attach a debugger. You can deterministically play sensor data through and get great reproducibility for end-to-end testing. Starting with a distributed system is a handicap 80% of the time.
Also - we have used queues to deal with different time scales for a really long time. It works fine here too.
For higher-level behaviors around grasping or manipulation, your point is super valid though. I suppose I'm mostly focusing on navigation-type tasks.
And you mention queues, yes exactly. Abstract a little more and you get pub sub. Abstract a little more and you have the actor model, which is a lovely way of building resilient, reliable, fault tolerant systems — exactly what we want out of robots.
For example, how do you write unit tests for an actor-model system? Without unit tests, how do you properly characterize the code's behaviour? When I last did ROS work, I built the whole thing outside of ROS, tested and validated it worked with tests, and then put some small ROS wrappers on top, and it basically worked first time. But this isn't how ROS-native systems are developed, instead people use Gazebo/Rviz to tweak and add things, and you end up with a system that is grown organically, at the single algorithm level, with all the problems that entails.
As I posted cross-thread, in the actor model, with queues and threads, you inherently encode additional state via the temporal spacing of the messages. Trying to predict what all of these could be so that you can test for edge cases and make sure things are safe is basically impossible. The modularity of ROS lets you set up a giant system pretty quickly, but in order to iron out the edge cases takes about as much time as just rewriting the whole thing as a monolith, because you haven't actually been able to test the system properly and the long tail of hidden state and bugs is impossible to avoid, and also impossible to predict and test for.
From what I've seen of the ROS community, the concept of testing is severely lacking. It usually entails running simulations in lots of different scenarios, which in a testing hierarchy is only really your final integration tests. It doesn't tell you about degradations in various subsystems, eg. control or navigational ineffiencies. It doesn't tell you about regressions based on earlier behaviour. It isn't deterministic, so you get random failures, reducing trust in the testing infrastructure. It takes tons of compute, so your devs wait hours for something they should be able to know in seconds. And because it's slow, devs won't add tests to the same granularity they would otherwise.
In a high reliability environment deterministic code is really important. The actor model doesn't give you that, each and every time you cross its interface. It also makes abstractions for granular testing much more difficult. It isn't a silver bullet, and ROS leans so heavily on it that all of the downsides are effectively unmitigated and impossible to avoid.
It sounds like we're working in a similar space, for me it is drone obstacle avoidance and navigation systems, and I found ROS to be entirely unsuitable for anything more granular than inter-drone coordination.
In an actor model, the units would be the actors. Test that they are deterministic and behave correctly given a message. You can test them for robustness by fuzzing messages and throw them at the actor. Then you use integration tests to test the whole system's performance.
> But this isn't how ROS-native systems are developed
Note that I haven't been arguing for ROS, but for loosely decoupled architectures for distributed systems like robots. I agree that ROS has many shortcomings. Although I would say this is not a shortcoming of ROS, but of ROS developers. Maybe ROS can be blamed for guiding people to work in such a way.
> As I posted cross-thread, in the actor model, with queues and threads, you inherently encode additional state via the temporal spacing of the messages.
Systems other than ROS do it better, but the point I've been trying to get across is that the actor model is great for distributed systems because it makes explicit the inextricable asynchronous, distributed nature of the system. As I've been arguing, you need to pass messages at some point if you want the robot to be a robot -- it has to interact with the world and society at some level, likely many levels. Your obstacle avoiding drone I assume is communicating with a base station, maybe remote compute, and a remote human operator. If we want to properly test this kind of system, we're going to have to make explicit the fact that the network is not reliable, latency is not zero, etc.
In this light, temporal spacing of messages, rather than being an encumbrance, becomes a necessity. It's a means to test and ensure that the system can handle all sorts of timings and orders of messages, just as it would need to do in the real world. By designing and conducting our tests to incorporate this, we can effectively simulate and anticipate the conditions our system will face.
Also, time-deterministic messaging protocols can be used to better manage this temporal aspect.
> you haven't actually been able to test the system properly and the long tail of hidden state and bugs is impossible to avoid, and also impossible to predict and test for.
But does the monolith avoid the edge cases or does it just fall for the fallacies of distributed computing?
> From what I've seen of the ROS community, the concept of testing is severely lacking.
Again, this seems like a shortcoming of the ROS community, and not the actor model.
> this light, temporal spacing of messages, rather than being an encumbrance, becomes a necessity.
And this is the crux of where we disagree. This is a messy part of reality which should be, as far as possible, abstracted away from the algorithms which need to operate on the data presented to them. If I'm running a Kalman filter I don't want to have to design in my filter around frequent gyroscope dropouts because image captures are happening, I want my system to have guaranteed behaviour that this won't happen. Actor model makes this harder by not giving me a way to have explicit guarantees, in fact it moves in the opposite direction by embracing flexibility.
While in general I agree that different components should be independently operable, as a system they will more than likely, in the real world, share various resources and you will need to deal with contention.
Any system which drastically increases overheads via serialisation, context changes, possibly network traffic and finally deserialisation in the place of a few instructions function call is a design which should be used very sparingly.
Actor model makes testing harder, and this results (again in the real world) in testing less. It also makes system level tests nondeterministic. Time deterministic protocols in place of function calls is just a nonstarter IMO. It's giving up control margin, increasing system load, and doesn't leave you any better with regard to system stability in case of failure.
Yes the actor model has its place, but at a very large granularity. Overuse, as in ROS, leads to horrible design constraints, opaque dependencies, difficult or impossible testing, and frankly impossible debugging.
Since you seem to be an actor model evangelist, how would you go about, just as an example, tracing execution flow in a debugger, for example? The data that gets passed into the actor interface is basically runtime-defined GOTOs. Similarly, how would you prove (in a certification perspective) that in certain scenarios the system as a whole behaves in a certain way, and fails in a safe way? Each subsystem can be proved to be safe, but the moment it goes through an async interface all bets are off.
Annoyingly in a lot of real world setups you can't have these guarantees. Your gyroscope, camera, etc are all producing data asynchronously often with slightly different clocks, and they all have different little idiosyncrasies and failure modes.
For example the Mars helicopter almost crashed because it missed a single frame. https://mars.nasa.gov/technology/helicopter/status/305/survi... If possible you absolutely want to fix the frame drop in the first place, but your algorithm should also be able to handle the drop out (or at least reset/recover).
I think that your opinion of the actor model has been particularly colored by ROS, as these constraints aren't necessarily part of the actor model. It's an abstraction, and a formalism that is built around the idea of message passing, but that doesn't mean the actual implementation has to involve literal message passing. If a function call will really do the trick, a sufficient compiler can produce equivalent code.
But the question is... is the function call really synchronous. For instance, you give the example of a gyroscope attached to a kalman filter, but what about a GPS? What happens when the GPS becomes unavailable, and the kalman filter doesn't get any more updates? Indeed many (a majority actually) of my sensors have Ethernet interfaces, and we communicate with the sensors over networks that include routers. Some of the robot's sensors are external to the robot itself, and we communicate with them over a wireless network. So when you say this:
> Each subsystem can be proved to be safe, but the moment it goes through an async interface all bets are off.
I find myself in full agreement! But you cross that async interface as soon as you want data from your sensors, because the sensor interface is asynchronous. So you might as well deal with the asynchrony explicitly.
> Since you seem to be an actor model evangelist, how would you go about, just as an example, tracing execution flow in a debugger, for example?
Typically what I look at are message traces. What's nice about actor model is it lends itself to new ways of debugging, like time travel debugging. It's also a formalism, so we can leverage that formalism to prove properties of the program.
> how would you prove (in a certification perspective) that in certain scenarios the system as a whole behaves in a certain way, and fails in a safe way?
I guess it would depend on what system you're trying to certify and to what standard. If you have something in mind, how would you imagine going it ideally, and then maybe I can try to respond as to how my mind would wrap around it.
Your point is taken about industrial robotics lacking proper tools here —- it’s often the case that industry is about 10-15 years behind research in the field of robotics. But ROS is already being used in industry, and I imagine in 15 years industry will enjoy improved tooling currently being used in research labs to support better testing and reliability in robotic systems.
It'd be like if you designed a video game so that the physics engine (motion planning) happened in one process, while rendering (perception) happened in another process and the world state needed to be exchanged via pubsub. Why do we keep doing this to ourselves?