People who don't like this style (or the code) leave.
The direction for quality has to come from the top and be supported by systematic mechanisms. In fact that would be a question to ask in interviews.
"No-one ever explicitly specified that the bridge shouldn't fall down and kill everyone who was on it at the time", said no engineer, ever.
It’s not that this type of engineering is “not engineering”; it’s that it’s engineering where the engineer themselves (and their ability to actively re-stabilize the system) is considered a load-bearing element in the design, such that the system will very likely fall apart once that element is removed.
Combat engineering is still engineering, in the sense that there are still tolerances to be achieved, and it’s still a disaster if the bridge falls over while it’s in active use for the mission it was built for. It’s just not considered a problem if it falls over later, once that mission is accomplished.
A particular one that caught my eye: Cimarron River Rail Crossing Dover, Oklahoma Territory United States 18 September 1906 Wooden railroad trestle Washed out under pressure from debris during high water 4-100+ killed Entire span lost; rebuilt Bridge was to be temporary, but replacement was delayed for financial reasons.[8][9][10] Number of deaths is uncertain; estimates range from 4 to over 100.[11]
Emphasis mine. How is that different from software engineering?
That said, all bridges have design specifications and limits. If someone drives a convoy of trucks carrying gold bullion over a bridge and exceeds its design max weight by a factor of 10, and it doesn't hold up, that's not the engineer's fault. If a bridge is designed for a geologically active area and is designed to withstand a quake 10x more powerful than the strongest that's ever been recorded in the area, or is predicted by seismologists, and the area gets hit by a 100x quake, that's not the engineer's fault.
And if a bridge designed to last say, 5 years, is neither re-certified for use, or just closed, when its intended lifespan is up then that's not the engineer's fault either.
But the engineer is responsible for finding out what the likely max load of a bridge will be, or how powerful the strongest quake will be, or how long its intended lifespan is - and for adding in safety margins on top of that. They can't just assume that any new bridge will only be needed for 5 years, just because that no-one told them that it will be needed for longer, and then claim "well, it was only temporary" afterwards.
The law takes care of that though. The law and inspection. If it was up to the business, any engineers building bridges who would refuse to project it faster because of that would be fired.
Rather, because software engineering hasn't killed enough people to advance; or when it has, it wasn't obviously the culprit.
In which fields of software engineering do we find actual solid procedures and standards? Avionics. Medical hardware. Fields where the link between bug and death is as short as possible, and so the stakeholders have demanded solid engineering.
What are the consequences of GP's acquaintance writing poor code? Some trading firm becomes slightly less efficient at trading. Impossible to evaluate the net human loss from it (if it's a loss at all).
The civil engineering equivalent would be something like façade design or layout. If your building is ugly or confusing, it will annoy or waste the time of the people who live and work in it, but as long as it doesn't fall on their heads nobody is going to withdraw your certification.
I think the undertone is usually that the software developer at fault had no idea what they were doing, and had no place working on critical systems, rather than that they had ill intent.
and where it had, you get pretty serious control and certification process put in place to avoid it (Boeing notwithstanding)
The MCAS software was modified to read from both angle of attack sensors and to be less aggressive in pushing the nose of the plane down. The software that controlled the indicator light that illuminated when the two angle of attack sensors disagreed was also fixed.
While reviewing the software systems, a number of other software issues were found.
The wiring bundle issue was also found during these reviews.
https://www.barrons.com/articles/these-6-issues-are-preventi...
> The MCAS software was modified to read from both angle of attack sensors and to be less aggressive in pushing the nose of the plane down.
That strikes me as a design issue in the domain of aeronautical engineering, rather than in software engineering. Software engineers aren't the ones with the domain expertise to determine the right aggression parameters.
> The software that controlled the indicator light that illuminated when the two angle of attack sensors disagreed was also fixed.
I thought the issue was that the sensor-disagree warning light was not included as standard, it was sold as an optional extra. [0]
> While reviewing the software systems, a number of other software issues were found.
Interesting, I wasn't aware of that. If I understand correctly these issues aren't thought to have a direct bearing on the crashes.
[0] https://www.nytimes.com/2019/05/05/business/boeing-737-max-w...
But the biggest defect was the decision that the pilots didn't have to be told about the risk (to avoid retraining and certification costs). If they had been told the pilots/airlines may well have protested about the lack of redundancy. All those higher ups in Boeing and the FAA should be in prison.
As the risk of nitpicking, we already can produce reliable code, but as piaste's comment states, it's rare to make the necessary investment.
Avionics software and medical systems software is held to a very high standard, at great expense, but the average website or desktop application is not.
> Webpages will be 100GB of JavaScript though
I agree that the problem of bloat is likely to continue to worsen in the web, but that's just the web. Other sectors, like high-performance game engines, will continue to compete on efficiency.
If it's your job to do what is requested from you, then you do it. It's not like having brittle unmaintainable code is morally wrong. It's not engineers responsibility to judge use case of the customer paying for the bridge.
Actually yes, yes it is! I am a civil engineer and there's a standard of ethics and personal responsibility among engineers that is very, very high. When we graduate, most engineers participate in a ring ceremony. They get a funny, angled ring on their right pinky. It was originally made from the metal from a bridge that fell down and killed people. It was meant to be a daily reminder, worn on the drafting hand, that every single drawing you signed carried the weight of public trust of their lives. Even pedestrian bridges get built with vehicle load standards, because you can't just assume that it will only be used by pedestrians.
Civil engineering is a licensure, and no matter what liability structure you use, when you stamp a drawing, you carry PERSONAL liability for anything that might happen if that design fails. Even small violations of ethics or operating outside your scope of knowledge are dealt with very aggressively by the licensing board.
In school, if we misplaced a decimal or got a calculation wrong or failed to adhere to a very specific standard, it was automatic failure because if you do that in real life, people die.
It's strange too, you hear in other threads about how high the demand is for software engineers, how high their salaries are, how much negotiating power they have with their companies, but when it comes to the actual product content, suddenly they have no power at all and it's just Yes boss, whatever you say, boss. How is this true?
And that is because no one can just go to a 6-month bootcamp and call themselves a Civil engineer. And even if they did, no one would hire them or the employer would go to jail.
If we want Software Engineering to be as rigorous as other Engineering disciplines, we need to erect similar barriers to become a Software Engineer.
Anyway for most systems deployed the customer usually wants to maintain a business using it. If the system constantly falls over, cant be readily changed etc etc that is going to cost the customers business compared to its competitors.
Add that a lot of developers work for the same company as the customer and that is just as much the developers responsibility.
99.999% of bridges built would be expected to hold up 10 years later and require minimal maintenance. I'd wager all bridges even temporary army ones would be expected to not unforeseeably fail. See the Morandi Bridge tragedy.
Do we put the cockpit in front of the boiler or behind?
Do we make a walking steam engine? How about using gears instead of wheels?
Which gauge to pick for the rails?
We'll get there in software engineering at well. It will be very reliable and equally boring.
1. It can be rebuilt in a matter of hours by a single person.
2. The customers that ordered the bridge are also the people on the bridge and are totally fine with the bridge crashing on them from time to time.
I remember the exact turning point. We had a super buggy Windows application that had tons of crashes in it. Instead of root causing each crash and fixing them, I was asked to simply write a launcher app that sat in the background, waited for the application to crash, then re-launch it. That was the great solution. And it was totally acceptable to the customer. Arghhhhh! I remember thinking: I didn't spend four years in university to shit this kind of finished product out.
On the other hand, if someone takes software reliability seriously, either they are going to burn themselves out to meet the crazy deadlines or are going to "not deliver on time", in which case they will be replaced by someone who can "get shit done" cheaper and faster.
If you want to design the software that designs the bridge or the car...just need a 8 week bootcamp.
Compare bridge building to developing software for pace maker.
Now compare number of failures in both cases.
You get what you pay for, if you pay for developer you'll get a developer.
That's way too oversimplified. You can be as good and as thorough as you want, but if the root of the problem is that software is seen as a cost not as an integral part of the solution, you get bad results.
This often starts with unclear/vague requirements that change every other day as new information and understanding is gained.
But it doesn't stop there - unrealistic deadlines, lack of defined processes and quality control as well as disregard (and refusal to budget and schedule) for background tasks (documentation, refactoring, ...) are also contributing factors that can turn even the best and most diligent software developer into a messy code cowboy.
If gaming the system is rewarded more or actually doing a good job is even penalised, why do a good job?
Here is a 1992 truth bomb rewind about it:
https://www.developerdotstar.com/printable/mag/articles/reev...
Basically engineers care about the quality of the code because they know that is the design. As soon as caring about codebase quality leaves the building, so will the quality of the product.
Software engineers often hide behind a contract saying they aren't responsible for anything.
Software engineering is real and does require you to have a real understanding of risk (including through formal methods, model based design, systematic testing etc) in regard to the consequence. If you work in a more regulated industry (e.g. railways, aviation, automotive) these things are taken more seriously. This is why things like 'partial autonomy' driving needs more regulation -- no one is requiring Tesla to have a Chief Engineer sign off on anything meeting any standard.
I would expect that there are plenty of engineers that look at faults in the systems they're working on and say, "management doesn't care, not my problem".
TL;DR: I think the extent to which we are engineers is a choice we individually make.
I remember just refactoring all of foreach-loops/iterators into for-loops and got insane speedups.
Functions with inefficient iterations would get optimized if called directly but deep in the call stack and nothing happens.
These kinds of things are impossible with C++.
It's possible to write efficient Java but you have to not use some of the language features to do so.
What? Anything you can do in C, you can do in C++ at the same performance level (pay only for what you use); just be aware of what you are doing and using.
Wow! Can you please provide a simple example when/where it happens?
> It's possible to write efficient Java but you have to not use some of the language features to do so.
Please tell us more.
An even worse case that made massive speedups is just inlining everything. I believe modern IDEs can automatically inline all invocations of a function. You'd be surprised how much performance you can get with that.
As for writing efficient Java, it comes down to using primitive and value types.
Although, my experience is mostly with Java 8. I have no idea how streams or other new syntax works. After optimizing a really massive Java code base I've never used it since.
You just have to try it and see. I remember it was quite easy in IntelliJ, I believe there's an option to inline all invocations of a function. I just went crazy with that option but did it systematically to really find where the bottleneck is and make a minimal change.
At the super fast speeds you start running into things like:
* why won’t the devirtualizer trigger?
* these object headers are sure wasting a lot of cache
* there’s a lot of forced pointer indirection
And you just end up spending vast amounts of time and effort trying to shave off those few microseconds you’re wasting in the JVM. Once you add in all the effort trying to get around the GC, it’s bleak picture for competing with quality C++ without spending far more effort.
Again, thanks for the response, this was insightful and I'd love to learn more.
Also, an aside - do you have any thoughts on the use of Rust in these same systems? It's a little bit more bleeding edge, but I'm curious to hear an expert's thoughts!
In the JVM I think this can only be done speculatively (you have to double check the type), but it still matters.
In the JVM devirtualising means making a virtual object a full object again, so the opposite of what you want to be happening.
I don't think the JVM really names the optimisation you're talking about, but it does do it, through either a global assumption based on the class hierarchy, or an inline cache with a local guard.
Yes... but let's not say 'broken up on the stack' - the object's fields become dataflow edges. The object doesn't exist reifed on the stack - fields may exist on the stack, or in registers, or not at all.
When we say devirtualize we mean that we know animal is always a cat so we don't have to look up which speak to use.
And that makes literally no sense in-context, you're mapping those words directly to the concept of virtual methods but that's the opposite of the way chrisseaton uses them (and they're clearly familiar with the concept of virtual calls and it's not what they're using "virtual" for), hence asking them what they mean specifically by this terminology.
The answer to both questions is that many people prefer writing in higher-level languages with more safety guarantees.
Yes, of course $HIGH_LEVEL_LANG is preferable for many many use cases. In this context, we're discussing "high speed trading systems", for which native implementations are going to be the favorite.
...but the point of the article is they aren't always the favourite.
So they use regular Java for the build system, deployment, testing, logging, loading configuration, debugging, etc etc. There's a small core written in this strange way... but everything else is easier. And things like your profiler and debugger still work on the core as well.
Depending on the goal, one has to be very careful when choosing what one uses. The good side is, C++ kept its C features: if I'm deciding how I'll do something, I don't have to follow the rules of the "language lawyers." I can do my work producing what is measurably efficient. And compiler can still help me avoiding some types of errors -- others can anyway be discovered only with testing (and additional tools). At the end, knowing good what one wants is the most important aspect of the whole endeavor.
I'm not exactly sure what specifically you are trying to imply here. When you proclaim to ignore language lawyers, it sounds like you are knowingly breaking the rules of the C++ standard. That takes a lot of faith in compilers doing what you meant to do, despite writing code that is incompatible with the standard those compilers implement...
http://www.jot.fm/issues/issue_2003_01/column1/
https://wiki.c2.com/?UmlCaseVultures
One size doesn't fit all. Some solutions to some problems could be and are provably better than those typically promoted or "generally known" at some point of time.
If that all still doesn't mean anything to you, please read carefully and very, very slowly "The Summer of 1960", seen on HN some 9 years ago:
https://news.ycombinator.com/item?id=2856567
Edit: Answering the parallel post writing "When you proclaim to ignore language lawyers, it sounds like you are knowingly breaking the rules of the C++ standard."
No. The language lawyers, in my perception, religiously follow everything that enters the standard and proclaim that all that has to be used, because it's standardized. Including the standard libraries and some specific stuff there that isn't optimal for the problem I'm trying to solve. And especially that whatever is newer and more recently entered the standard is automatically better. It's understandable that they support their own existence by doing all that -- it's about becoming "more important" just by following/promoting some book or some obligatory rituals (it's an easy and time proven strategy through the centuries, and that's why I call it "religiously", of course -- and I am also not surprised that somebody who identifies themselves with being one of the "lawyers" wouldn't like this perspective -- you are free to suggest a better name). But it should also be also very obvious that it's not what's necessarily optimal for me to follow, as soon as I can decide what I'm doing. And yes, it's different in the environment where the "company policy" is sacred. There one has the company "policy lawyers", and typically every attempt of change can die if one isn't one of them.
"Orthodox C++"
E.g. at the whole bottom of the page in some comment is a link to:
"Why should I have written ZeroMQ in C, not C++ (part II)"
where the author writes "Let's compare how a C++ programmer would implement a list of objects..." and then "The real reason why any C++ programmer won't design the list in the C way is that the design breaks the encapsulation principle" etc.
I have indeed more than once used intrusive data structures in non-trivial C++ code, and the result was easy to read, and very efficient. There I really didn't care about some "thou shalt not" "breaking the encapsulation principle" because whoever thinks at that level of "verbot" 100% of times is just wrong.
The "encapsulation principle" is an OK principle for some levels of abstraction, but nobody says that one has to hold to it religiously (exactly my point before). I would of course always make an API which would hide what's behind it. But where I implement something ("the guts" of something), I of course have my freedom to use intrusive elements, if that solves the problem better. I have even created some "extremely intrusive" stuff (with a variable number of intrusive links in the structures). It worked perfectly. Insisting on doing anything as "ritual" all the time is just so, so wrong.
Trading systems trigger almost all their orders based on detecting market changes, which you cannot predict. If a symbol hits a price that triggers one of your trades you want that trade in the market ASAP every millisecond matters. Randomly add 50ms on to that and your trading system is out of business. The customers will go elsewhere.
Typical pauses may be far less than a millisecond in these GCs because they do almost everything in parallel.
Sometimes an arena may instead refer to a region of memory for allocations that are all the same size/type. Because everything is the same size, the data structures for tracking allocations may be simpler.
In a language like Java, you basically preallocate an array of objects of the same type with a bunch of fields set to zero, and then have some data structure to give you a fast interface a bit like malloc. Because you don’t do any extra allocation, the GC can be safely turned off.
If you'd really really care about latency and have the resources, wouldn't it be more effective to just go bare metal? Bare metal as in: no OS (talk to the hardware registers directly from code), possibly a custom compiler, use all the relevant instructions that your CPU offers you. Or at the very least mess with the compiler to optimize it more to the specific make/model of the CPU/hardware that it will run on?
What are the considerations in those companies for or against this?
Compilers already offer flags to optimize for specific processor generations (and use the newly available instructions); you aren't going to be able to do better than that with a custom compiler.
I'm not sure whether it's worth it to go through this trouble (writing for bare metal vs just using an off-the-shelve OS) in a real life trading-app scenario. Hence my wonder :)
VM - the key is to get everything mapped in at the start and avoid page faults subsequently. madvise can help with this and obviously you need to avoid memory leaks that could result in sbrk - but in any case allocating at all in the trading thread is generally unnecessary (after startup) and frowned on. This does mean that the (lock-free) queue back to the shared core needs to recycle memory and both sides need to service it regularly or you need a strategy to deal with exhaustion. Scheduling isn't an issue - you have a single thread per core and (with isolcpus) the kernel scheduler knows to avoid placing other threads on it.
"If you have an unlimited amount of time and resources, the best solution for speed will be coded in FPGA," says Lawrey, referring to the language that directly codes field-programmable gate arrays. "If you still have a lot of time and resources and you want to create multiple exchange adaptors, you'll choose C++. - But if you want to engage with 20+ exchanges, to go to market quickly, and implement continuous performance tuning, you'll choose Java."
It might surprise people that relatively few hfts do latency arb (and the ones that do often have some other structural advantage), and at that the ones are more and more mixing in serious short-term alpha. In that context speed is simply becoming table stakes for playing the game, and not the one true race to win it all.
There are apparently ways to do this with an OS running in parallel as well though when you have multiple cores available apparently (see another comment to this thread).
- raw, unshared access to devices.
- no more interrupts.
Get to OS to set up DMA for you.
> no more interrupts
If the OS has no need to interrupt you, it will never interrupt you. The OS doesn't context switch from a usefully running application on its own core unless you ask it to.
Don't need virtualisation.
The whole of the trading strategy (and capital traded, return profiles, etc) between the first two was fairly different, so it’s hard to compare.
I would say though that the first firm was significantly outperforming its peers in a way the second firm did not, although both were very successful.
The technology wasn’t the whole story in either case, but the first firm had significantly more and better opportunities as a result of just having a better stack all around
There's at least a couple of MMs struggling these days.
I'm still wondering if the businesses would work better if the focus on technical excellency could be instilled and especially which parts of the company would have to suffer for it and what the ultimate outcome would be.
I'm convinced that at least for some companies, it simply can't be the winning strategy.
(well strictly speaking, additionally you could also have hand-written assembly infra).
The main benefits of C per se are ABI stability and availability of compilers.
Rust is worse than C++ in that it struggles to express some idiomatic high-performance code architectures, such as using DMA for I/O, because they violate axioms of Rust's memory safety model, or in other cases because Rust currently lacks the language features. To make it work in Rust requires writing more code and disabling safety features.
Rust fixes many language design and performance issues with Java, while offering some similar types of safety, but (like Java) it is missing some elements of expressiveness of C++ that are important for high-performance architectures.
I've done my fair share of C++, including low latency stuff, and in the grand scheme of things I'd say "expressiveness of C++" is completely overshadowed by its complexity and occasional ambiguity, lack of proper type system and proper generics, lack of proper module system, lack of dependency tracking, lack of a unified build systems, etc. I'm not exactly sure what you mean by expressiveness though.
Instead of using an ownership-based memory safety model, which is clearly broken for systems that use DMA, they can use schedule-based memory safety models, which are formally verifiable. These don't rely on immutable references for safety, and also eliminate most need for locking -- many of the concepts originate in research on automated lock-graph conflict resolution in databases. The evolution away from ownership-based to schedule-based models is evident in database kernels over the decades. It was originally motivated by improved concurrency; suitability for later DMA-based designs was a happy accident and C++ allows it to be expressed idiomatically.
As for expressiveness, beyond the ability construct alternative safety models, modern C++ metaprogramming facilities enable not only compile-time optimization but also verification of code correctness that would be impractical in languages like Rust.
Isn’t everything you wrote from the wrong decade?
Pure latency arb isn’t the only hft trade there is anyways.
I have some sympathy for the idea that the JVM is better since it means you won’t spend all your time chasing crash reports. The thing is, like another comment hinted at, is that this issue is generally more reflective of the environment you build in than the technology choice.
Here’s a good talk on the reasons for and limits of using C++ for low latency systems: https://m.youtube.com/watch?v=NH1Tta7purM
If I had to rank in terms of reliability of trading infrastructure I’ve used/worked on,
* Low Latency C / C++ infra at fully automated trading firm. Very much run by the programmers and quants and also fairly small. By far the fastest (near limits of what you could do) and also most reliable.
* Low Latency Java execution infrastructure. Pretty reliable, not that fast, had some issues with GC battling and manual memory management to avoid GC, etc. There was a pretty clear latency floor (still quite low) even when “doing everything right” that serious native infrastructure beat.
* C++ market making infra at a firm run by manual traders. It was by far the slowest and least reliable. Echos the experience of “spent hours debugging weird crashes”. The culture was very “A trader asked for this and needs it done yesterday. Also this refactoring business doesn’t sound like adding new features, drop it”.
What I saw is that if you hire people who have a good idea of what they’re doing and keep a culture of technical excellence, C++ is definitely a better choice if you care about latency. You really have to maintain a culture of high quality code, testing, and in general caring about the technology.
This is only really possible when all of the stakeholders are involved in the technology, or at least understand the benefits. Once you start down the path of “well this feature could be done a day quicker if...” this goes down the drain, and a few years later you find yourself getting run over on latency AND with an impossible to use trading system. It’s really the worst of both worlds.