Agree with this, a lot of developers are in a filter bubble where they stick to communities that advocate modern practices like automated testing, continuous integration, containers, gitflow, staging environments etc.
As a contractor, I get to see the internals of lots of different companies - forget practices even as basic as doing code reviews, I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. A lot of the time there's no internal experts there that are even aware there's better ways to do things instead of it being the case they're lacking resources to make improvements.
Svn is easier to understand and use, but then you’d have to break some existing habits to get to git. But going straight to git might be a big step and cause reversion back to whatever system was already there.
Then disallow direct commits to master to make people work in feature branches and make merge requests through the platform (github /lab/ bitbucket). I find merging and branching locally is where people normally trip up.
Git GUI tools always make git seem way more complicated than it is so depending on the teams platform I would recommend cli from the start.
Say you want to edit a few chars in a 250MB file. Why don't you, just to be sure, past that whole file in the comment field? Do that for a few 100 commits. Tortoise really hated that one, and crashed the windows explorer every time you dared look at the logs (out of memory).
Or the time some joker (His CV proudly declares 10 years of developer experience) deletes the root of the tree, doesn't know about history, and goes with his manager straight to the storage admin, who wipes everybody's commit of that day ( a few 100 people). There clearly is no need to contact someone who knows anything about subversion if the data is gone, and maybe this way nobody will notice anything and jell at them.
Or say you want to do an upgrade. Theory is every user leaves, service and network port get shut down, VM instance is backed up just to be sure, you do the svn upgrade. Of course enterprise IT means I have to write detailed instructions to every party involved and under no circumstance allowed tot look at that server myself.
So it turns out: A) some users just keep on committing straight trough the maintenance window. B) The clown who shuts down the service doesn't check if the service is actually shut down, and there is a bug in the shutdown script. So svn just keeps on running. C) The RPM containing the patch is transported by an unreliable network that actually manages to drop a few bytes while downloading from http. D) The guy who should shut down the svn network port is away eating, so they decide to skip that step. E) SVN gets installed anyway (what do you mean, checksum mismatch) and starts commiting all kinds of weird crimes to its storage. F) The VM guy panics, rolls back to the previous version, except for the mount which contains the data files. G) Then they do it all again, and mail me how the release was successful without any detail of what happened.
Let me tell you, svn really loved having its binary change right under it, in the middle of a commit, while meeting its own previous version in memory. Oh and clients having revision N+10, while the server is at version N. A problem that solves itself in a few minutes as N goes up really fast ;-)
Now thats what happens with subversion, which is rock solid and never drops a byte once committed. This company is now discovering the joys of git, where you can rewrite history whenever you feel like it.
https://web.archive.org/web/20180903164646/http://hginit.com...
I've introduced a lot of people to SVN over the past decades. Be it programmers, sysadmins, artists, translators, it's fairly quick to learn.
I couldn't begin to imagine introducing anybody to git. It's a horrible nightmare to use, even for developers, there is nothing that come close in how many times it screws up and you have to search for help on the internet.
Subversion is newer than RCS. But that doesn't mean every use of the latter can or even should be replaced.
IBM ClearCase is the way to go
I'm not into Java development, but this sounds fine on the face of it, without you giving the context of how this pipeline is triggered.
You pulled the code from google drive, modified it, pushed it to PROD, checked it and moved it back to google drive... and asked the other developers to update.
Well I've seen companies that have their own idea of source control. Which is lots of copies on the network drive, and an Excel registry with what is in which file.
It is source control. Just bad source control.
However, there's also the skill set of taking such a ... let's call it well-aged development team and approach and modernizing and/or professionalizing it. And yes, sometimes this means to build some entirely lovecraftian deployment mechanism on mutable VMs because of how the application behaves. But hey, automated processes are better than manual processes, which beat undocumented processes. Baby steps.
Omg. And here I am feeling ashamed to tell others about my small small personal website with separate dev, qa, production environments on same server (via VirtualHost), code checked into github, deployed via Jenkins self hosted on another VPS, which was initially spun up with Ansible and shell scripts. All done by me for self training purpose. All because I thought businesses would have something more sophisticated with bells and whistles.
And then I hear there are businesses that make changes directly on live production servers...
But I'm not surprised by such stories as I have seen some bad workflow in real businesses that deal with tens of millions of dollars a year.
Years ago, I worked in the NOC of a company that's top in the small niche. They have dozens of employees, and been around for years.
Part of the job responsibility was rolling out patches to production servers. The kicker was the production servers were all Windows servers, running various versions, covering practically all Windows versions ever released by Microsoft. You can see where this is headed.
Rolling out a patch was all done manually, 1 Windows server at a time. Everything was manually done.
The instruction for deploying a change was easily multiple lines long, each with different style of writing/instruction. Often in plain text file format. We would print them out so we could check them off as we went down the list.
The CTO is still there, but everyone IT person under him has left or been let go. Working in the IT there is a struggle because lack of automation and old old stuff, but the CTO just blames bad employees and keeps churning them in/out. When the real issue is the decade or 2 worth of old legacy stuff that need to be cleaned up and/or thrown out, which can only be done by the direction of the CTO. But he knows he won't get that kind of budget from higher-ups so he will just keep hiring/firing employees and/or bring in some H1B workers who's basically trapped once they join. And of the few H1B workers I met there, they were truly completely non-technical. One did not want to learn how to use keyboard shortcuts to do common tasks... Good guy though.
> ...all changes are being made directly on the production servers via SFTP
I know this used to be common, but recently? Curious how often this is still the case.
Several times within the last year for me. Not all companies have big tech departments with knowledgable developers advocating modern best practices. Some big internal systems can start out from someone internal applying some basic self-taught skills for example.
To be fair, the jump to using Git (and especially dealing with Git merge conflicts) is scary. It can be a hard sell to get people to move from a system they already completely understand and have used successfully for years, even if their system looks like ancient history to us.
Literally heard "...but my IDE already automatically uploads to FTP on save, I'm usually the only usually one editing it and I already know what I changed" last week.
Meanwhile the CEO who was rejecting the €¥$£ in yh budget since 2000 is angry at everyone!
Oh the times I have seen this!!!
This was for a 100+ year old company with millions of dollars in annual revenue that was owned by the government. So, yeah. 100% the IT director's fault, who'd been there since the early 90s.
Having no testing/staging environments remains pretty common, along with its cousin "production work happens on staging". Partnering with not-primarily-software companies and asking about staging infrastructure, you har that regularly. And yeah, SFTP/SCP/SSH is a standard push-and-deploy approach in places where that happens.
On the other hand, outside of a tech company and with less than a dozen developers, don't expect to find any source control. Consultants see a lot of this shit, they work in all industries including where developers don't exist, with a lot of thrown away projects.
Funny thing. Git probably made it worse in recent years by being impossibly hard to use.
/seriously, though - I...hope this isn't being done any longer - but I bet it is. Sigh...
The writer breaks down why this document is not a post-mortem despite superficial similarities.
>Again, the purpose of the doc is to point out where Knight violated rules. It is not: 1) a description of the multiple trade-offs that engineering at Knight made or considered when designing fault-tolerance in their systems, or 2) how Knight as an organization evolved over time to focus on evolving some procedures and not others, or 3) how engineers anticipated in preparation for deploying support for the new RLP effort on Aug 1, 2012.
>To equate any of those things with violation of a rule is a cognitive leap that we should stay very far away from.
>It’s worth mentioning here that the document only focuses on failures, and makes no mention of successes. How Knight succeeded during diagnosis and response is unknown to us, so a rich source of data isn’t available. Because of this, we cannot pretend the document to give explanation.
He also makes an interesting point related to what you're saying: the SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.
>What is deemed “appropriate”, it would seem, is dependent on the outcome. Had an accident? It was not appropriate control. Didn’t have an accident? It must be appropriate control. This would mean that Knight Capital did have appropriate controls the day before the accident. Outcome bias reigns supreme here.
I'd go into it more but I would instead recommend you take a look at his breakdown as I'd be trying to do a shoddy summary of a really interesting write-up.
[0] https://www.kitchensoap.com/2013/10/29/counterfactuals-knigh...
> This would mean that Knight Capital did have appropriate controls the day before the accident.
I think these claims are seriously confused.
SEC fines don't require mens rea, so Knight is simply being punished for having inappropriate controls, their view on the matter be damned. Kitchensoap rightly observes that "this event was very harmful" does not imply "this event was caused by extreme negligence". But the SEC filing focuses on alerting and controls; position limits don't prevent a specific misstep, but they limit the maximum size of any error that does occur. (Knight had position limits on accounts, but didn't use them as fundamental boundaries restricting actual trade volume.) The thesis is that Knight should have prepared to mitigate "unknown unknowns", in which case the size of the error is relevant because the size was exactly what should have been controlled for.
On appropriate controls, SEC fines are certainly outcome-biased, but the claim is obviously that these controls were always inappropriate, and the disaster simply revealed them. Post-disaster punishment creates an ugly system where people who don't take excess risk can be outcompeted before their competitors crumble, but the rule isn't actually conditional on failures.
Kitchensoap asks whether Knight would be judged so harshly if they'd only lost $1,000. Socially, perhaps not, but legally they actually would have! The SEC isn't just punishing Knight for losing money but for disrupting the market with improper trades; it specifically notes that for some "...of those stocks, the price moved by greater than ten percent, and Knight’s executions constituted more than 50 percent of the trading volume. These share price movements affected other market participants...". A smaller loss wouldn't have defended against that charge, while a smaller trade wouldn't have violated SEC rules.
I think the author is basically aware of this, since his fundamental point is that the SEC is describing the legal wrongs rather than the technical mistakes. That's a good point and I'm glad you linked this. But I think his focus on the specific deployment error neglects the fact missing position controls were the larger legal and technical failure.
What makes negligence extreme? Doing things you have specifically been warned against would be one thing, but there are others, including being oblivious to the magnitude of the risk when, with a little thought, it should have been clear.
The inverse of the above quote is equally valid: not-very-harmful outcomes do not imply that the negligence is not extreme, and it was all the days of operating without big problems that allowed the organization to be blind to the risk it was running, day in, day out.
OP wrote: >> Clearly Knight thought [its controls] appropriate or they would have fixed it.
But the problem is that it did not think about it, in a meaningful way: it did not have an informed opinion. Every day in which nothing very bad happened contributed to the normalization of deviance. I am sure there were other days when things went wrong, but without the worst possible outcome, and they became just part of the way things are, instead of a wakeup call.
By this logic, we can claim that if a hospital is storing all its employee passwords in plaintext, that's "appropriate" because if it was inappropriate they wouldn't do so.
Or that if a company is neglectful about offsite backups, that's an "appropriate" data retention strategy because if it wasn't, then the company would be taking backups.
In this case, if Knight thought its controls were "appropriate", then that's the problem that needs fixed.
But I appreciate the writing style, and I've got his site bookmarked for more systems safety reading in the future. It's a slog, but often this sort of ruthlessly-comprehensive breakdown is the best way to understand exactly how a complex thing went awry. Reading them every so often - even for non-software topics like drug treatments - seems to be a good refresher for my own error-analysis skills.
The link here excerpts the full filing, but the real meat wasn't included:
> 16... Because the cumulative quantity function had been moved, this server continuously sent child orders, in rapid sequence, for each incoming parent order without regard to the number of share executions Knight had already received from trading centers... 17. The consequences of the failures were substantial. For the 212 incoming parent orders that were processed by the defective Power Peg code, SMARS sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight inadvertently assumed an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. Ultimately, Knight realized a $460 million loss on these positions.
Knight did $1.4 billion in revenue per year, but there was no safeguard against adopting $7 billion worth of positions in 45 minutes. The end result was that they lost 4 years of profits in under an hour. That's crazy, and not at all standard.
And everyone benefits. You, the employer, your colleagues, the fresh grads with starry eyes
First, management has to be willing to assign you to cleanup tasks, rather than have you work on new features.
Second, cleaning up spaghetti code safely and turning it into something maintainable is often not easy -- it wasn't written to be maintained! No tests, missing documentation, ill-considered interfaces...
It isn't enough to keep your head off the chopping block: management has to be fully on board.
I found an impending any-day-now production down scenario last week and got it fixed while learning tremendously and fixing some incidentally related things. It feels amazing.
No tests, no docs, no comments, total spaghetti, global variables everywhere, no logs, and I am loving it. What a job.
Upper upper management is starting to keep tabs on the "productive" output of my team though. We'll see.
Whether or not that's true, I don't know if it's the right framing for the problem. Think about how many new cars or new airplanes or new bridges are designed every year. (One or two per large firm? Less? None by any small firms?) Then think about how many new web services are designed every year. (Several for a small firm, dozens for a large firm?)
If "maturing" the software design process means 10x or 100x the cost and time that a project takes today, that isn't going to fly in the market. It might make more sense to point out the discrepancies between the typical software project (low cost for mistakes, high tolerance for downtime) and the atypical finance/medicine/aerospace software project (don't screw up or people die). Maybe the specific part of the industry that needs to mature is the _awareness_ of when you cross that boundary, especially within a single organization. The folks working on the internal HR system and the folks working on the high-frequency trading system need to use different procedures (and different levels of funding and different management expectations), and if they're the same folks on different days that might be very far from obvious.
About a decade ago, I was a contractor in a security services team at a large bank. There were systems in place to minimize the impact of our software, which handled requests for access, and the actual access granted. Our software was developed like most, or worse in the beginning. In the end, by separating workflows and agreeing on interfaces, a lot of issues were prevented altogether.
Other systems should be designed with similar safeguards and separate certain control flows behind well thought out APIs.
Others still should be created much closer to classic waterfall, with adjustments meaning a stop-work, re-evaluate, update design, and proceed process in place.
It should really depend on the situation.
If the analysis shows the defect could have reasonably been caught during the code review phase then management sometimes has to recalibrate the reviewers. It's also important to select the right reviewers based on the scope and risk of the change. For the largest, most complex changes we sometimes pull in up to 6 reviewers including members of other teams and high level architects.
In finance, I've seen people compute deals worth billions of dollars using excel spreadsheets and a team of MBAs.
The team of MBAs might screw up an Excel formula and lose money on a deal, but presumably if the math comes out to "let's pay $500 billion for Yahoo", somebody's going sanity-check that before they transfer the money.
* Lack of unit (or any) testing
* Lack of good versioning practices and code review (e.g emailing around a sheet that's been doing the rounds for 15 years in various different guises and formats and has who knows what horrors lurking)
* Lack of typing (e.g doing a SUM across a dataset consisting of "3", $3, 3, "III", an emoji of the number 3, a jpeg screenshot of the number 3 - might not add up to 18)
* Lack of precision (rounding errors)
Not sure if you are talking about the author of the SEC document (the “bug report”) or the blogger, but in either case, what is okay in 99% of software development may be quite inadequate for critical software used in tightly regulated industries. Context matters.
> Managers care about speed of implementation, not quality.
Managers care about speed of implementation not quality insofar as quality is often hard to unambiguously measure and hard to assess impact on the bottom line (not just quantity, but whether there is really any impact.)
This is a fairly dramatic example of the impact being made concrete.
with this practice I have gotten a better habit of looking at the code changes and having a mental image of what things do
this coupled with annotation in the IDE helps me really keep up with the code
only occasionally do I get an actual WTF that prompts me to leave any remark or not approve the change
I'm still better informed about the code change and implementation than not
I'd add that it's the the incredibly immature economic system, paired with an even more immature(new) industry that leads to this kind of failure.
I feel like this is a symptom of tech in capitalism. Where the goal is to maximize profits and minimize effort, rather than doing the job correctly. Fitting that this would befall a high freq trading firm.
It reads to me like standard RCA (Root Cause Analysis) language/tone.
How are you defining "top" and "bottom" here? I can agree with the 80/20 split, but I don't think it necessarily tracks with, say, name recognition or market cap. Amazon, for example, utterly failed on their last Prime day, and from the discussions of those Amazonians I know, it was pretty much inevitable.
The REASON these fast-and-loose habits are habits are because, when they work, they make money. You can't look at a lottery and say the winners "did it right" and the losers didn't - the winners just haven't failed yet. Those companies that AREN'T trying to grow at ridiculous speeds but ARE trying to maintain quality (and are fortunate enough to be in a market niche that supports that) are the ones most likely to have bulletproof reliability. Those won't be the "top" companies by most peoples definitions.
I am not familiar what happened. I am also ex-Amzn. I think there is also a dimension to this, scale. With Amazon's scale there are quite different challeges then in a smaller company. There are solutions that could not possible scale to their needs.
Fast-and-lose was largely introduced when Google and Facebook got entered the scene. I remember that Google did a survey and most of the users were ok with some breakage and get the newest features ASAP. Many people concluded based on this that all software development is like that. Ironically Facebook lately adopted certain technologies to reduce the amount of bugs in their frontend code (ReasonML). I think there is a large distance between bulletproof and feck-all reliability. Top companies are closer to the bulletproof range of the spectrum while bottom companies closer to the feck-all end.
Each of these is strongly correlated to quality.
We work in an incredibly immature industry. And trying to enforce better practices rarely works out as intended. To give one example: we rolled out mandatory code reviews on all changes. Now we have thousands of rubber-stamped "looks good to me" code reviews without any remarks.
Managers care about speed of implementation, not quality. At retrospectives, I hear unironic boasts about how many bugs were solved last sprint, instead of reflection on how those bugs were introduced in the first place.