News for hackers. Stuff that matters.
Re:How to lose $172k per second for 45 minutes (2013) by wcoenen on Monday April 01, 2019 @09:44AM (#19543133)
I'm amused by the tone. It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse. Welcome to the real world.

We work in an incredibly immature industry. And trying to enforce better practices rarely works out as intended. To give one example: we rolled out mandatory code reviews on all changes. Now we have thousands of rubber-stamped "looks good to me" code reviews without any remarks.

Managers care about speed of implementation, not quality. At retrospectives, I hear unironic boasts about how many bugs were solved last sprint, instead of reflection on how those bugs were introduced in the first place.

Re:How to lose $172k per second for 45 minutes (2013) by seanwilson on Monday April 01, 2019 @10:24AM (#19543503)
> I'm amused by the tone. It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse. Welcome to the real world.

Agree with this, a lot of developers are in a filter bubble where they stick to communities that advocate modern practices like automated testing, continuous integration, containers, gitflow, staging environments etc.

As a contractor, I get to see the internals of lots of different companies - forget practices even as basic as doing code reviews, I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. A lot of the time there's no internal experts there that are even aware there's better ways to do things instead of it being the case they're lacking resources to make improvements.

Re:How to lose $172k per second for 45 minutes (2013) by foldor on Monday April 01, 2019 @10:54AM (#19543864)
I wish I hadn't experienced this exact scenario as well. I actually worked at a publishing company where I discovered the previous team building their major education platform didn't know there was such thing as source control. Their method for sharing multiple team members changes was to email the full project to one person every Friday who would manually open a merging tool and merge all changes themselves. They would then send out the updated code base to all people again. Because of this method, they were afraid to delete any stale code, and just prefixed old versions of functions with XX. As you can imagine, inheriting that code base was a nightmare to deal with.
Re:How to lose $172k per second for 45 minutes (2013) by yibg on Monday April 01, 2019 @12:36PM (#19545161)
I remember having this debate many years ago. Would it be better to introduce a team with no source control knowledge to something like svn first or straight to git?

Svn is easier to understand and use, but then you’d have to break some existing habits to get to git. But going straight to git might be a big step and cause reversion back to whatever system was already there.

Re:How to lose $172k per second for 45 minutes (2013) by ehnto on Monday April 01, 2019 @12:58PM (#19545421)
Straight to git but have a simple and clear branching strategy. As in a full written procedure for the three main events of git. "Get code, commit code, push code".

Then disallow direct commits to master to make people work in feature branches and make merge requests through the platform (github /lab/ bitbucket). I find merging and branching locally is where people normally trip up.

Git GUI tools always make git seem way more complicated than it is so depending on the teams platform I would recommend cli from the start.

Re:How to lose $172k per second for 45 minutes (2013) by jliptzin on Monday April 01, 2019 @01:26PM (#19545740)
I would not start people new to git on cli. That's how you get someone's entire Camera Uploads directory committed to master (I've seen it before). I recommend Git Tower. I use it for most tasks actually even though I am comfortable in the CLI too. It tends to stop me from doing stupid things before I do them.
Re:How to lose $172k per second for 45 minutes (2013) by crooked-v on Monday April 01, 2019 @03:15PM (#19547029)
Also, the git CLI in particular is extraordinarily terrible, given how many conceptually different things are done by the same set of commands. (For example: imagine trying to use git checkout in a repo with a file named "master" at the root level.)
Re:How to lose $172k per second for 45 minutes (2013) by seanwilson on Monday April 01, 2019 @02:25PM (#19546449)
You can probably assume a lot of people who aren't using version control aren't using the terminal either. GUIs are generally much better at CLIs at giving you an overview of what's happening and what actions are available too.
Re:How to lose $172k per second for 45 minutes (2013) by hyperman1 on Monday April 01, 2019 @03:36PM (#19547270)
I've supported a few svn instances. Devs could be trusted with svn, but analysts,testers, system people, ... My god the horrors these guys dream up.

Say you want to edit a few chars in a 250MB file. Why don't you, just to be sure, past that whole file in the comment field? Do that for a few 100 commits. Tortoise really hated that one, and crashed the windows explorer every time you dared look at the logs (out of memory).

Or the time some joker (His CV proudly declares 10 years of developer experience) deletes the root of the tree, doesn't know about history, and goes with his manager straight to the storage admin, who wipes everybody's commit of that day ( a few 100 people). There clearly is no need to contact someone who knows anything about subversion if the data is gone, and maybe this way nobody will notice anything and jell at them.

Or say you want to do an upgrade. Theory is every user leaves, service and network port get shut down, VM instance is backed up just to be sure, you do the svn upgrade. Of course enterprise IT means I have to write detailed instructions to every party involved and under no circumstance allowed tot look at that server myself.

So it turns out: A) some users just keep on committing straight trough the maintenance window. B) The clown who shuts down the service doesn't check if the service is actually shut down, and there is a bug in the shutdown script. So svn just keeps on running. C) The RPM containing the patch is transported by an unreliable network that actually manages to drop a few bytes while downloading from http. D) The guy who should shut down the svn network port is away eating, so they decide to skip that step. E) SVN gets installed anyway (what do you mean, checksum mismatch) and starts commiting all kinds of weird crimes to its storage. F) The VM guy panics, rolls back to the previous version, except for the mount which contains the data files. G) Then they do it all again, and mail me how the release was successful without any detail of what happened.

Let me tell you, svn really loved having its binary change right under it, in the middle of a commit, while meeting its own previous version in memory. Oh and clients having revision N+10, while the server is at version N. A problem that solves itself in a few minutes as N goes up really fast ;-)

Now thats what happens with subversion, which is rock solid and never drops a byte once committed. This company is now discovering the joys of git, where you can rewrite history whenever you feel like it.

Re:How to lose $172k per second for 45 minutes (2013) by srean on Monday April 01, 2019 @01:51PM (#19546026)
Here's a vote for going straight to Mercurial (a lot easier to grok) and link to Joel Spolky's excellent tutorial

https://web.archive.org/web/20180903164646/http://hginit.com...

Re:How to lose $172k per second for 45 minutes (2013) by user5994461 on Monday April 01, 2019 @02:35PM (#19546559)
GitHub can be used with both GIT and SVN so you don't have to choose.

I've introduced a lot of people to SVN over the past decades. Be it programmers, sysadmins, artists, translators, it's fairly quick to learn.

I couldn't begin to imagine introducing anybody to git. It's a horrible nightmare to use, even for developers, there is nothing that come close in how many times it screws up and you have to search for help on the internet.

Re:How to lose $172k per second for 45 minutes (2013) by anon4242 on Monday April 01, 2019 @02:41PM (#19546637)
If you haven't used any source control at all, I don't see why svn would be easier to understand than git? Using git will save you from a lot of pain up ahead so I would definitely go for git.
Re:How to lose $172k per second for 45 minutes (2013) by perl4ever on Monday April 01, 2019 @05:32PM (#19548512)
My limited experience with git (and none with svn) leads me to suggest someone might prefer another source control system, not because it's easier to understand per se, but because you can do simple things in it without fully understanding it.
Re:How to lose $172k per second for 45 minutes (2013) by xorcist on Monday April 01, 2019 @02:11PM (#19546306)
"Straight" to git? If you have already made up your mind, why ask?

Subversion is newer than RCS. But that doesn't mean every use of the latter can or even should be replaced.

Re:How to lose $172k per second for 45 minutes (2013) by mikepurvis on Monday April 01, 2019 @01:00PM (#19545450)
Surely these days that question is answered by the existence of GitLab and several other similar tools.
Re:How to lose $172k per second for 45 minutes (2013) by KSteffensen on Monday April 01, 2019 @04:33PM (#19547916)
SVN and git! Pah!!

IBM ClearCase is the way to go

Re:How to lose $172k per second for 45 minutes (2013) by WrtCdEvrydy on Monday April 01, 2019 @11:07AM (#19544030)
Someone created a jenkins pipeline that would deploy code in a zip file into prod.
Re:How to lose $172k per second for 45 minutes (2013) by mattmanser on Monday April 01, 2019 @11:24AM (#19544277)
Am I missing something? Isn't this exactly how deployment servers actually work?

I'm not into Java development, but this sounds fine on the face of it, without you giving the context of how this pipeline is triggered.

Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @12:00PM (#19544718)
In theory it sounds alright. It's not great, because Jenkins is usually layered with some existing deploy framework that makes "deploy a zip file" pretty suspect. A healthier setup would look more like "build a Tomcat war file Maven, upload and deploy that with Jenkins". But in context, it sounds like the horror is that people were making and transferring a zip from local code rather than building from the tip of source control.
Re:How to lose $172k per second for 45 minutes (2013) by WrtCdEvrydy on Monday April 01, 2019 @11:38AM (#19544452)
Developers had a copy of code in a google drive... they'd modify it, zip it and overwrite the one in google drive, copy it to the network folder which would deploy it and delete it from the network folder... in 2016.
Re:How to lose $172k per second for 45 minutes (2013) by debaserab2 on Monday April 01, 2019 @12:46PM (#19545268)
That’s a pretty standard CI process - zip files are often your deploy artifact (using, for example, git archive)
Re:How to lose $172k per second for 45 minutes (2013) by WrtCdEvrydy on Monday April 08, 2019 @12:31AM (#19602107)
Yeah, except the zip file was... the input.

You pulled the code from google drive, modified it, pushed it to PROD, checked it and moved it back to google drive... and asked the other developers to update.

Re:How to lose $172k per second for 45 minutes (2013) by carlmr on Monday April 01, 2019 @11:20AM (#19544228)
>I've seen companies not using source control

Well I've seen companies that have their own idea of source control. Which is lots of copies on the network drive, and an Excel registry with what is in which file.

It is source control. Just bad source control.

Re:How to lose $172k per second for 45 minutes (2013) by blowski on Monday April 01, 2019 @02:39PM (#19546610)
I worked on a system with “octopus locking”. We a had toy octopus, and you could only change the code (obviously in production) when you had the octopus.
Re:How to lose $172k per second for 45 minutes (2013) by yellowapple on Monday April 01, 2019 @07:12PM (#19549321)
My day job involves multiple third-party systems that enforce their own proprietary version control systems for any custom scripting/programming within those systems. Unsurprisingly, these proprietary version control systems are complete garbage 90% of the time, especially if you're trying to collaborate with someone else.
Re:How to lose $172k per second for 45 minutes (2013) by tetha on Monday April 01, 2019 @02:56PM (#19546811)
This comment is resonating very well with that I've been experiencing at my current job, and with our new head company. It's always nice if you can take a new application, throw it into a clean container build chain with an orchestration - or some AWS stack, and there's a cheap, low-maintenance, HA production system.

However, there's also the skill set of taking such a ... let's call it well-aged development team and approach and modernizing and/or professionalizing it. And yes, sometimes this means to build some entirely lovecraftian deployment mechanism on mutable VMs because of how the application behaves. But hey, automated processes are better than manual processes, which beat undocumented processes. Baby steps.

Re:How to lose $172k per second for 45 minutes (2013) by dba7dba on Tuesday April 02, 2019 @06:09AM (#19552526)
> I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS.

Omg. And here I am feeling ashamed to tell others about my small small personal website with separate dev, qa, production environments on same server (via VirtualHost), code checked into github, deployed via Jenkins self hosted on another VPS, which was initially spun up with Ansible and shell scripts. All done by me for self training purpose. All because I thought businesses would have something more sophisticated with bells and whistles.

And then I hear there are businesses that make changes directly on live production servers...

But I'm not surprised by such stories as I have seen some bad workflow in real businesses that deal with tens of millions of dollars a year.

Years ago, I worked in the NOC of a company that's top in the small niche. They have dozens of employees, and been around for years.

Part of the job responsibility was rolling out patches to production servers. The kicker was the production servers were all Windows servers, running various versions, covering practically all Windows versions ever released by Microsoft. You can see where this is headed.

Rolling out a patch was all done manually, 1 Windows server at a time. Everything was manually done.

The instruction for deploying a change was easily multiple lines long, each with different style of writing/instruction. Often in plain text file format. We would print them out so we could check them off as we went down the list.

The CTO is still there, but everyone IT person under him has left or been let go. Working in the IT there is a struggle because lack of automation and old old stuff, but the CTO just blames bad employees and keeps churning them in/out. When the real issue is the decade or 2 worth of old legacy stuff that need to be cleaned up and/or thrown out, which can only be done by the direction of the CTO. But he knows he won't get that kind of budget from higher-ups so he will just keep hiring/firing employees and/or bring in some H1B workers who's basically trapped once they join. And of the few H1B workers I met there, they were truly completely non-technical. One did not want to learn how to use keyboard shortcuts to do common tasks... Good guy though.

Re:How to lose $172k per second for 45 minutes (2013) by Reedx on Monday April 01, 2019 @11:37AM (#19544437)
> I've seen companies not using source control

> ...all changes are being made directly on the production servers via SFTP

I know this used to be common, but recently? Curious how often this is still the case.

Re:How to lose $172k per second for 45 minutes (2013) by seanwilson on Monday April 01, 2019 @12:24PM (#19545006)
> I know this used to be common, but recently? Curious how often this is still the case.

Several times within the last year for me. Not all companies have big tech departments with knowledgable developers advocating modern best practices. Some big internal systems can start out from someone internal applying some basic self-taught skills for example.

To be fair, the jump to using Git (and especially dealing with Git merge conflicts) is scary. It can be a hard sell to get people to move from a system they already completely understand and have used successfully for years, even if their system looks like ancient history to us.

Literally heard "...but my IDE already automatically uploads to FTP on save, I'm usually the only usually one editing it and I already know what I changed" last week.

Re:How to lose $172k per second for 45 minutes (2013) by frosted-flakes on Monday April 01, 2019 @11:51AM (#19544615)
I have seen it recently. I did my best to change the practice before I left the company, but was mostly unsuccessful. Given that they were still running some of the spaghetti-code PHP scripts that were written in 1999 and still used PHP4 in new development they were stuck in the stone ages. To give a little perspective, support for PHP4 ended in 2008, so they had almost a decade to update, but didn't.
Re:How to lose $172k per second for 45 minutes (2013) by HenryBemis on Monday April 01, 2019 @01:16PM (#19545617)
"If it ain't broken, don't fix it". And then one day the server goes boom, the backup was incomplete, and everyone is trying to find the usb flash disk with Spinrite in it.

Meanwhile the CEO who was rejecting the €¥$£ in yh budget since 2000 is angry at everyone!

Oh the times I have seen this!!!

Re:How to lose $172k per second for 45 minutes (2013) by frosted-flakes on Tuesday April 02, 2019 @01:55AM (#19551418)
Oh, now that you mention backups, that was a nightmare too. Thankfully, the production database was backed up daily on magnetic tape and stored offsite, but the code was generally edited live on the server, and backups consisted of adding ".bak20190402" to the end of the file. Needless to say, losing code wasn't uncommon.

This was for a 100+ year old company with millions of dollars in annual revenue that was owned by the government. So, yeah. 100% the IT director's fault, who'd been there since the early 90s.

Re:How to lose $172k per second for 45 minutes (2013) by taurath on Monday April 01, 2019 @06:32PM (#19549030)
Its both the CEO's fault for either not understanding or not hiring someone who properly understands the risk they're taking on in their tech stack, and whoever's job it was to understand the risk. Part of being a responsible engineer (or IT manager, etc) is to be able to say "no" to new things and to explain that a bad day can and will take you down.
Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @11:48AM (#19544593)
At my first (very small, research-group) employer, I was the one to introduce source control. Another company I saw ~4 years ago had Git, but no one knew how to use it adequately so "lost code" was still a regular event. I haven't seen it since, though; lots of people frightened by and badly misusing Git, but they mostly manage to keep code histories intact.

Having no testing/staging environments remains pretty common, along with its cousin "production work happens on staging". Partnering with not-primarily-software companies and asking about staging infrastructure, you har that regularly. And yeah, SFTP/SCP/SSH is a standard push-and-deploy approach in places where that happens.

Re:How to lose $172k per second for 45 minutes (2013) by user5994461 on Monday April 01, 2019 @02:46PM (#19546687)
On the one hand, source control is really the most basic thing you can find anywhere. It's really hard to find an actual development shop without source control.

On the other hand, outside of a tech company and with less than a dozen developers, don't expect to find any source control. Consultants see a lot of this shit, they work in all industries including where developers don't exist, with a lot of thrown away projects.

Funny thing. Git probably made it worse in recent years by being impossibly hard to use.

Re:How to lose $172k per second for 45 minutes (2013) by z3t4 on Monday April 01, 2019 @04:07PM (#19547640)
Its nice to have a source code history to see why and when something was changed. But a lot of tooling is usually to treat the symptoms of complexity.
Re:How to lose $172k per second for 45 minutes (2013) by cr0sh on Monday April 01, 2019 @11:45AM (#19544538)
Hey, at least they're using SFTP - you know, for the security!

/seriously, though - I...hope this isn't being done any longer - but I bet it is. Sigh...

Re:How to lose $172k per second for 45 minutes (2013) by drablyechoes on Monday April 01, 2019 @12:39PM (#19545190)
Way too common. So many clients I've dealt with have a deployment workflow that is some variation of this.
Re:How to lose $172k per second for 45 minutes (2013) by Defenestresque on Monday April 01, 2019 @10:43AM (#19543734)
There is a really, really interesting breakdown of this document on kitchensoap.com [0]

The writer breaks down why this document is not a post-mortem despite superficial similarities.

>Again, the purpose of the doc is to point out where Knight violated rules. It is not: 1) a description of the multiple trade-offs that engineering at Knight made or considered when designing fault-tolerance in their systems, or 2) how Knight as an organization evolved over time to focus on evolving some procedures and not others, or 3) how engineers anticipated in preparation for deploying support for the new RLP effort on Aug 1, 2012.

>To equate any of those things with violation of a rule is a cognitive leap that we should stay very far away from.

>It’s worth mentioning here that the document only focuses on failures, and makes no mention of successes. How Knight succeeded during diagnosis and response is unknown to us, so a rich source of data isn’t available. Because of this, we cannot pretend the document to give explanation.

He also makes an interesting point related to what you're saying: the SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.

>What is deemed “appropriate”, it would seem, is dependent on the outcome. Had an accident? It was not appropriate control. Didn’t have an accident? It must be appropriate control. This would mean that Knight Capital did have appropriate controls the day before the accident. Outcome bias reigns supreme here.

I'd go into it more but I would instead recommend you take a look at his breakdown as I'd be trying to do a shoddy summary of a really interesting write-up.

[0] https://www.kitchensoap.com/2013/10/29/counterfactuals-knigh...

Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @01:05PM (#19545507)
> He also makes an interesting point related to what you're saying: the SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.

> This would mean that Knight Capital did have appropriate controls the day before the accident.

I think these claims are seriously confused.

SEC fines don't require mens rea, so Knight is simply being punished for having inappropriate controls, their view on the matter be damned. Kitchensoap rightly observes that "this event was very harmful" does not imply "this event was caused by extreme negligence". But the SEC filing focuses on alerting and controls; position limits don't prevent a specific misstep, but they limit the maximum size of any error that does occur. (Knight had position limits on accounts, but didn't use them as fundamental boundaries restricting actual trade volume.) The thesis is that Knight should have prepared to mitigate "unknown unknowns", in which case the size of the error is relevant because the size was exactly what should have been controlled for.

On appropriate controls, SEC fines are certainly outcome-biased, but the claim is obviously that these controls were always inappropriate, and the disaster simply revealed them. Post-disaster punishment creates an ugly system where people who don't take excess risk can be outcompeted before their competitors crumble, but the rule isn't actually conditional on failures.

Kitchensoap asks whether Knight would be judged so harshly if they'd only lost $1,000. Socially, perhaps not, but legally they actually would have! The SEC isn't just punishing Knight for losing money but for disrupting the market with improper trades; it specifically notes that for some "...of those stocks, the price moved by greater than ten percent, and Knight’s executions constituted more than 50 percent of the trading volume. These share price movements affected other market participants...". A smaller loss wouldn't have defended against that charge, while a smaller trade wouldn't have violated SEC rules.

I think the author is basically aware of this, since his fundamental point is that the SEC is describing the legal wrongs rather than the technical mistakes. That's a good point and I'm glad you linked this. But I think his focus on the specific deployment error neglects the fact missing position controls were the larger legal and technical failure.

Re:How to lose $172k per second for 45 minutes (2013) by mannykannot on Monday April 01, 2019 @01:50PM (#19546014)
> Kitchensoap rightly observes that "this event was very harmful" does not imply "this event was caused by extreme negligence".

What makes negligence extreme? Doing things you have specifically been warned against would be one thing, but there are others, including being oblivious to the magnitude of the risk when, with a little thought, it should have been clear.

The inverse of the above quote is equally valid: not-very-harmful outcomes do not imply that the negligence is not extreme, and it was all the days of operating without big problems that allowed the organization to be blind to the risk it was running, day in, day out.

OP wrote: >> Clearly Knight thought [its controls] appropriate or they would have fixed it.

But the problem is that it did not think about it, in a meaningful way: it did not have an informed opinion. Every day in which nothing very bad happened contributed to the normalization of deviance. I am sure there were other days when things went wrong, but without the worst possible outcome, and they became just part of the way things are, instead of a wakeup call.

Re:How to lose $172k per second for 45 minutes (2013) by yellowapple on Monday April 01, 2019 @07:26PM (#19549419)
> SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.

By this logic, we can claim that if a hospital is storing all its employee passwords in plaintext, that's "appropriate" because if it was inappropriate they wouldn't do so.

Or that if a company is neglectful about offsite backups, that's an "appropriate" data retention strategy because if it wasn't, then the company would be taking backups.

In this case, if Knight thought its controls were "appropriate", then that's the problem that needs fixed.

Re:How to lose $172k per second for 45 minutes (2013) by guitarbill on Monday April 01, 2019 @11:59AM (#19544706)
This kitchensoap article may be more correct, but man, it's a arduous read. A bit of brevity and humour can work wonders, conversely complete correctness can be detrimental if people lose attention.
Re:How to lose $172k per second for 45 minutes (2013) by Defenestresque on Monday April 01, 2019 @12:36PM (#19545152)
It is a lengthy slog for sure, but I found it a great way to kill some time. Maybe you just have to be in one of those reading moods.
Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @01:15PM (#19545612)
I'm not hugely fond of this specific article; I think the dissection of the SEC filing misunderstands what the accusation is. (Specifically, it's not having the bug, it's the lack of mitigation for large bugs in general.) To me, the "this is not a post-mortem, here's why" part would have stood better on its own.

But I appreciate the writing style, and I've got his site bookmarked for more systems safety reading in the future. It's a slog, but often this sort of ruthlessly-comprehensive breakdown is the best way to understand exactly how a complex thing went awry. Reading them every so often - even for non-software topics like drug treatments - seems to be a good refresher for my own error-analysis skills.

Re:How to lose $172k per second for 45 minutes (2013) by badfrog on Monday April 01, 2019 @10:44AM (#19543746)
But also for 99% of companies, running the wrong version for an hour won't bankrupt the company and all of its investors.
Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @12:23PM (#19544996)
I've seen a lot of companies and teams with hair-raising practices, but most of them still manage some basic business-logic safeguards like "don't buy for more than $X" or "if we don't sell anything for 10 minutes, shut it all down and sound the alarms". Generally they can't do worse than "making no money" or maybe "paying full expenses without making any profits".

The link here excerpts the full filing, but the real meat wasn't included:

> 16... Because the cumulative quantity function had been moved, this server continuously sent child orders, in rapid sequence, for each incoming parent order without regard to the number of share executions Knight had already received from trading centers... 17. The consequences of the failures were substantial. For the 212 incoming parent orders that were processed by the defective Power Peg code, SMARS sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight inadvertently assumed an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. Ultimately, Knight realized a $460 million loss on these positions.

Knight did $1.4 billion in revenue per year, but there was no safeguard against adopting $7 billion worth of positions in 45 minutes. The end result was that they lost 4 years of profits in under an hour. That's crazy, and not at all standard.

Re:How to lose $172k per second for 45 minutes (2013) by Causality1 on Monday April 01, 2019 @11:22AM (#19544251)
You live by the AI stock trader, you die by the AI stock trader.
Re:How to lose $172k per second for 45 minutes (2013) by siwatanejo on Monday April 01, 2019 @09:46AM (#19543152)
This may be the majority of the industry, true. But some devs filter out employers to not feel miserable. Whenever I find an employer like this, either I try to change the status quo, or I leave.
Re:How to lose $172k per second for 45 minutes (2013) by wDcBKgt66V8WDs on Monday April 01, 2019 @10:44AM (#19543739)
Changing the status quo is kinda fun in these environments. Low hanging fruit, easy work. As long as your head isn't on the chopping block.

And everyone benefits. You, the employer, your colleagues, the fresh grads with starry eyes

Re:How to lose $172k per second for 45 minutes (2013) by rectang on Monday April 01, 2019 @11:39AM (#19544463)
It can be rewarding, but there are a couple caveats.

First, management has to be willing to assign you to cleanup tasks, rather than have you work on new features.

Second, cleaning up spaghetti code safely and turning it into something maintainable is often not easy -- it wasn't written to be maintained! No tests, missing documentation, ill-considered interfaces...

It isn't enough to keep your head off the chopping block: management has to be fully on board.

Re:How to lose $172k per second for 45 minutes (2013) by wDcBKgt66V8WDs on Monday April 01, 2019 @08:52PM (#19550003)
Yeah in my case I've got my manager and architect on board and they love it. My more senior teammates might be threatened though.

I found an impending any-day-now production down scenario last week and got it fixed while learning tremendously and fixing some incidentally related things. It feels amazing.

No tests, no docs, no comments, total spaghetti, global variables everywhere, no logs, and I am loving it. What a job.

Upper upper management is starting to keep tabs on the "productive" output of my team though. We'll see.

Re:How to lose $172k per second for 45 minutes (2013) by WrtCdEvrydy on Monday April 01, 2019 @11:10AM (#19544075)
'Wanna make some enemies? Try to change something'
Re:How to lose $172k per second for 45 minutes (2013) by repomies68 on Monday April 01, 2019 @09:52AM (#19543203)
This. Don't except things to change if you happily stay in a workplace where things are done in a half-assed way. There's endless supply of jobs for software people nowadays.
Re:How to lose $172k per second for 45 minutes (2013) by oconnor663 on Monday April 01, 2019 @01:12PM (#19545585)
> We work in an incredibly immature industry

Whether or not that's true, I don't know if it's the right framing for the problem. Think about how many new cars or new airplanes or new bridges are designed every year. (One or two per large firm? Less? None by any small firms?) Then think about how many new web services are designed every year. (Several for a small firm, dozens for a large firm?)

If "maturing" the software design process means 10x or 100x the cost and time that a project takes today, that isn't going to fly in the market. It might make more sense to point out the discrepancies between the typical software project (low cost for mistakes, high tolerance for downtime) and the atypical finance/medicine/aerospace software project (don't screw up or people die). Maybe the specific part of the industry that needs to mature is the _awareness_ of when you cross that boundary, especially within a single organization. The folks working on the internal HR system and the folks working on the high-frequency trading system need to use different procedures (and different levels of funding and different management expectations), and if they're the same folks on different days that might be very far from obvious.

Re:How to lose $172k per second for 45 minutes (2013) by tracker1 on Monday April 01, 2019 @01:30PM (#19545790)
+1 on this... there's a huge difference in code that may be used by a dozen users to pick where to have lunch and HFT systems that handle millions of dollars worth of capital investment a day. There also is a correlated high burnout.

About a decade ago, I was a contractor in a security services team at a large bank. There were systems in place to minimize the impact of our software, which handled requests for access, and the actual access granted. Our software was developed like most, or worse in the beginning. In the end, by separating workflows and agreeing on interfaces, a lot of issues were prevented altogether.

Other systems should be designed with similar safeguards and separate certain control flows behind well thought out APIs.

Others still should be created much closer to classic waterfall, with adjustments meaning a stop-work, re-evaluate, update design, and proceed process in place.

It should really depend on the situation.

Re:How to lose $172k per second for 45 minutes (2013) by nradov on Monday April 01, 2019 @10:09AM (#19543363)
We do intensive root cause analysis on every serious defect that leaks into production. Look into the "5 why's" technique. Usually there's something to learn in terms of a process or training improvement.

If the analysis shows the defect could have reasonably been caught during the code review phase then management sometimes has to recalibrate the reviewers. It's also important to select the right reviewers based on the scope and risk of the change. For the largest, most complex changes we sometimes pull in up to 6 reviewers including members of other teams and high level architects.

Re:How to lose $172k per second for 45 minutes (2013) by inputcoffee on Monday April 01, 2019 @12:08PM (#19544815)
You guys think this is bad?

In finance, I've seen people compute deals worth billions of dollars using excel spreadsheets and a team of MBAs.

Re:How to lose $172k per second for 45 minutes (2013) by b_tterc_p on Monday April 01, 2019 @01:22PM (#19545699)
There’s nothing explicitly wrong with this. MBA’s aren’t intrinsically stupid, and excel is a reasonable tool for calculations. It’s just a pure functional language using geometric space instead of a namespace.
Re:How to lose $172k per second for 45 minutes (2013) by antoineMoPa on Monday April 01, 2019 @01:05PM (#19545499)
The team of MBA's can probably help avoid 172k$/second types of mistakes no?
Re:How to lose $172k per second for 45 minutes (2013) by Bartweiss on Monday April 01, 2019 @01:21PM (#19545691)
Yep - all the focus on the engineering specifics here seems to miss the sheer magnitude of the thing. Knight risked 4x their annual revenue in under an hour! You can get away with some incredibly crappy systems if you have final-line safeguards like "let's not risk all the money we've ever seen and more without asking a human".

The team of MBAs might screw up an Excel formula and lose money on a deal, but presumably if the math comes out to "let's pay $500 billion for Yahoo", somebody's going sanity-check that before they transfer the money.

Re:How to lose $172k per second for 45 minutes (2013) by perl4ever on Monday April 01, 2019 @05:39PM (#19548578)
I think companies have on occasion lost billions due to a glitch in a contract, sometimes due to delegating work to a novice who didn't fully understand it. And a contract is not totally unlike a program.
Re:How to lose $172k per second for 45 minutes (2013) by spookthesunset on Monday April 01, 2019 @03:22PM (#19547124)
Most business deals are done in part based on results from excel spreadsheets. What other tool would you expect to be used?
Re:How to lose $172k per second for 45 minutes (2013) by paulddraper on Monday April 01, 2019 @04:20PM (#19547771)
This must be ignorance speaking, but what is the issue with that?
Re:How to lose $172k per second for 45 minutes (2013) by darkr on Monday April 01, 2019 @04:42PM (#19548014)
As a tool for rough approximations, spreadsheets are amazing. Beyond that though:

* Lack of unit (or any) testing

* Lack of good versioning practices and code review (e.g emailing around a sheet that's been doing the rounds for 15 years in various different guises and formats and has who knows what horrors lurking)

* Lack of typing (e.g doing a SUM across a dataset consisting of "3", $3, 3, "III", an emoji of the number 3, a jpeg screenshot of the number 3 - might not add up to 18)

* Lack of precision (rounding errors)

Re:How to lose $172k per second for 45 minutes (2013) by dragonwriter on Tuesday April 02, 2019 @01:23PM (#19556492)
> It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse.

Not sure if you are talking about the author of the SEC document (the “bug report”) or the blogger, but in either case, what is okay in 99% of software development may be quite inadequate for critical software used in tightly regulated industries. Context matters.

> Managers care about speed of implementation, not quality.

Managers care about speed of implementation not quality insofar as quality is often hard to unambiguously measure and hard to assess impact on the bottom line (not just quantity, but whether there is really any impact.)

This is a fairly dramatic example of the impact being made concrete.

Re:How to lose $172k per second for 45 minutes (2013) by caprese on Monday April 01, 2019 @02:41PM (#19546634)
> we rolled out mandatory code reviews on all changes. Now we have thousands of rubber-stamped "looks good to me" code reviews without any remarks

with this practice I have gotten a better habit of looking at the code changes and having a mental image of what things do

this coupled with annotation in the IDE helps me really keep up with the code

only occasionally do I get an actual WTF that prompts me to leave any remark or not approve the change

I'm still better informed about the code change and implementation than not

Re:How to lose $172k per second for 45 minutes (2013) by devinjflick on Monday April 01, 2019 @11:29AM (#19544349)
> We work in an incredibly immature industry.

I'd add that it's the the incredibly immature economic system, paired with an even more immature(new) industry that leads to this kind of failure.

I feel like this is a symptom of tech in capitalism. Where the goal is to maximize profits and minimize effort, rather than doing the job correctly. Fitting that this would befall a high freq trading firm.

Re:How to lose $172k per second for 45 minutes (2013) by pwg on Tuesday April 02, 2019 @01:14PM (#19556415)
> I'm amused by the tone.

It reads to me like standard RCA (Root Cause Analysis) language/tone.

Re:How to lose $172k per second for 45 minutes (2013) by hartator on Monday April 01, 2019 @03:20PM (#19547083)
Every industries are immature. We just saw it more clearly because we are in it.
Re:How to lose $172k per second for 45 minutes (2013) by StreamBright on Monday April 01, 2019 @11:34AM (#19544394)
This is true for the bottom ~80% of the industry. The top ~20% of tech companies actually care about quality (as well as speed of implementation). You cannot deliver a shitty solution quickly and call it a day.
Re:How to lose $172k per second for 45 minutes (2013) by ergothus on Monday April 01, 2019 @12:02PM (#19544736)
> bottom ~80% of the industry. The top ~20%

How are you defining "top" and "bottom" here? I can agree with the 80/20 split, but I don't think it necessarily tracks with, say, name recognition or market cap. Amazon, for example, utterly failed on their last Prime day, and from the discussions of those Amazonians I know, it was pretty much inevitable.

The REASON these fast-and-loose habits are habits are because, when they work, they make money. You can't look at a lottery and say the winners "did it right" and the losers didn't - the winners just haven't failed yet. Those companies that AREN'T trying to grow at ridiculous speeds but ARE trying to maintain quality (and are fortunate enough to be in a market niche that supports that) are the ones most likely to have bulletproof reliability. Those won't be the "top" companies by most peoples definitions.

Re:How to lose $172k per second for 45 minutes (2013) by StreamBright on Monday April 01, 2019 @12:58PM (#19545417)
>> , for example, utterly failed on their last Prime day, and from the discussions of those Amazonians I know, it was pretty much inevitable

I am not familiar what happened. I am also ex-Amzn. I think there is also a dimension to this, scale. With Amazon's scale there are quite different challeges then in a smaller company. There are solutions that could not possible scale to their needs.

Fast-and-lose was largely introduced when Google and Facebook got entered the scene. I remember that Google did a survey and most of the users were ok with some breakage and get the newest features ASAP. Many people concluded based on this that all software development is like that. Ironically Facebook lately adopted certain technologies to reduce the amount of bugs in their frontend code (ReasonML). I think there is a large distance between bulletproof and feck-all reliability. Top companies are closer to the bulletproof range of the spectrum while bottom companies closer to the feck-all end.

Re:How to lose $172k per second for 45 minutes (2013) by user5994461 on Monday April 01, 2019 @03:07PM (#19546957)
It's easy to rank companies from top to bottom. Whether there is source control, automated build, tests, reproducible scripted deployment, code reviews, test environments, bug tracker, logs, monitoring of hardware usage, analytics, backup.

Each of these is strongly correlated to quality.

Re:How to lose $172k per second for 45 minutes (2013) by unreal37 on Monday April 01, 2019 @12:58PM (#19545419)
> The top ~20% of tech companies actually care about quality

Most companies are not tech companies, fyi. So you're talking about the top 20% of a very small subset of business.