News for hackers. Stuff that matters.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by dfadsadsf on Monday February 05, 2024 @08:22PM (#39269595)
I just do not believe this story. Nobody in the right mind will fire key software engineer 48 hours after system goes live. This is where all the bugs/problems show up and you need all hands on deck to fix them.

Six months after launch when all problems are solved - absolutely, you can fire engineer. 48 hours after launch? absolutely not.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by ericbarrett on Monday February 05, 2024 @09:19PM (#39269982)
I worked for a storage startup in the South Bay in the late 00s which did something like this. After staff crunched to deliver the base operating system and kernel controllers, management gave across-the-board poor reviews to all IC staff to justify no raise. Those with the ability quickly left for greener pastures; Google hired some of the best. Only those who had to (work visas, lack of experience, etc.) remained.

The product, being a v0.1 release implemented in C, was of course plagued with data corruption and poor performance. It read /etc/password from disk on every file access. An uninitialized int caused indirect inode corruption every ~millionth write. And so on.

Because of the poor product quality, they lost multiple opportunities, including a bid for early Facebook photo storage. The company never had more than 250 customers and ended up being sold to a Valley stalwart. The CEO and VP Eng were paid out handsomely, of course; preferred converted stock tranches or whatever. My options from 4 years of employment ended up worth $500.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by sterlind on Monday February 05, 2024 @10:06PM (#39270294)
Sounds like the Igneous playbook, from what I've heard. Immediately after launch they laid off a majority of the dev team, as they'd outlived their usefulness after completing the crunch. I interviewed there but fortunately declined the offer - I felt like I dodged a bullet when my friend (who survived the layoffs, only to be conscripted as a sales grunt) told me about it. I also gather that the CEO had pulled similar shenanigans with Isilon in the past.

If it's any consolation, Igneous died and the execs didn't make out like bandits this time.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by andsoitis on Monday February 05, 2024 @11:49PM (#39270915)
”The product, being a v0.1 release implemented in C, was of course plagued with data corruption and poor performance. It read /etc/password from disk on every file access. An uninitialized int caused indirect inode corruption every ~millionth write. And so on. Because of the poor product quality, they lost multiple opportunities, …”

It sounds like the poor reviews for the ICs was warranted, wouldn’t you say?

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by smsm42 on Tuesday February 06, 2024 @01:13AM (#39271348)
Not necessarily. Big codebase in something like C, written in a rush, pretty much is guaranteed to have such things. We still are finding high-grade security issues in glibc (if you want some library to be clean, it's the one practically everybody is using), and it has been around for 37 years. v0.1 prototype is often plagued with various issues and often doesn't have best performance. You either spend years on polishing it and never release, of you start with v0.1 and fix the issues. Or, having fired all the dev team, don't fix the issues.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by sangnoir on Tuesday February 06, 2024 @01:14AM (#39271357)
> It sounds like the poor reviews for the ICs was warranted, wouldn’t you say

How many crunches have you survived that resulted in a high-quality initial product? If there is enough time to get things right, it wouldn't be a crunch, by definition.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by foobiekr on Tuesday February 06, 2024 @12:29AM (#39271103)
Smells like Igneous or TinTri.

Story shows too much competence for Springpath or Whiptail.

Storage is particularly rife with stuff like this for some reason.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by ericbarrett on Tuesday February 06, 2024 @03:25AM (#39272166)
Was a long time ago, way before them. I agree though, glad to have changed fields.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by foobiekr on Tuesday February 06, 2024 @01:47PM (#39278640)
You would be pleased to know that Whiptail exceeded whatever low bar you might have had. I like to see people achieve the things they set out to do, and I am assuming that was their goal.

Rare to see a product completely canceled and removed from the market as soon as customers started reporting minor things like total data loss due to corruption..

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by gspetr on Tuesday February 06, 2024 @03:20AM (#39272135)
This makes me even more incredulous than the GP commenter.

How did people who "read /etc/password from disk on every file access." get hired by Google?

Or perhaps the right question is for which roles, because with that level of DS&A they shouldn't have passed Google's DS&A whiteboard interview.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by ericbarrett on Tuesday February 06, 2024 @03:45AM (#39272277)
I think you give the whiteboard far too much credit. In the succeeding two decades I have worked at many companies filled with past, present, and future FAANG engineers, and I can definitely say knowledge of algorithms isn’t a defense against writing silly bugs when implementing a complex, novel project from scratch. Just a dumb oversight that would have been caught quickly by user reports—had all the engineers who cared not quit from misanthropic management. There were many such issues, and I only remember these two because I found them myself.

I was in support and only had two good resources for tracking this kind of stuff down, a burnt out but kind SCSI specialist and a brilliant, massively overworked Russian kernel dev on visa. They both lasted an extra year. Bless them both for answering my pestering questions, I learned a lot.

Good times, but all this was about a year too late to make a difference, then the Great Recession hit and it was done.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by me_me_me on Tuesday February 06, 2024 @11:49AM (#39276705)
google hires 27k devs (as per google search)

Do you honestly think all 27k devs are 10x certified geniuses?

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by bitcharmer on Tuesday February 06, 2024 @12:26PM (#39277382)
> The CEO and VP Eng were paid out handsomely, of course

It's the prevalence of scenarios like this that makes me hate the MBA caste with a pasison

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by Cpoll on Monday February 05, 2024 @08:39PM (#39269705)
If you really want to fire someone, you're willing to bleed a little (or a lot) to do it. I've seen key personnel be let go, followed by a scramble to figure out what only they know.

In this case it sounds like a "junior quant" who built an application in "months of hard work" while working 12hrs x 7days, and using technologies they learned on the fly. Sounds like me in my 20s, but I wasn't as indispensable as I thought I was. In any case, that's not as hard to replace as the sysadmin who's been there for 10 years.

From a different perspective, it's kind of like paying a contractor or offshore. You don't usually keep them on board after the deliverable, you hand it over to the in-house team to maintain it.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by gamepsys on Tuesday February 06, 2024 @02:44PM (#39279505)
> From a different perspective, it's kind of like paying a contractor or offshore.

That relationship is much better defined to both parties. It sounds like this person was surprised.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by neilv on Monday February 05, 2024 @10:24PM (#39270401)
There are many possibilities (and I can't really tell from skimming this writeup), but the possibilities absolutely include someone thinking "We got what we needed, now let's drop this person, and free the salary and bonus obligations".

Or they might've decided they didn't want that employee for some reason, maybe even dudebro culture fit mismatch when he showed up looking fatigued, and were "sux2bu" about how they got rid of the person, firing fast. Or that project got scrapped. Or some other person had a look at the code and decided they didn't like it (or didn't like the partner whose pet project it was). Etc.

A factor to keep in mind is that many, many shops nowadays really don't know how to build software that works sufficiently well, or they have low standards for what "sufficiently" means. For example, a few years ago, if your experience was only when the top priority was the appearance of growth, and then the next funding rounds and then exit, and there were all sorts of memes about moving fast and breaking things, etc., and you have no idea how to build systems that work, even if you wanted to... you might well think that developers are disposable and interchangeable commodities. (Someone else here noted the practice of a startup using offshoring contractors for key tech early on, which I think is strong evidence of this kind of thinking.)

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by 1024core on Monday February 05, 2024 @11:16PM (#39270717)
Or maybe they thought: we don't want this fool around who let us exploit him so much.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by gscott on Tuesday February 06, 2024 @03:45PM (#39280320)
People who say no get respect, those who say yes to impossible deadlines, work insane hours, maybe miss an impossible deadline by a few hours-or not, get fired.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by nytesky on Monday February 05, 2024 @08:32PM (#39269664)
Well, Netflix was famous for firing when the business need had passed. You have a seat in company for what you will do, not what you have done.

These high speed trading platforms do a ton of transactions in 48 hours. They assume nothing material will change in 6 months vs 48 hours; and anyways if it breaks why keep paying the dude who built it so flimsy?

Not my way of doing things of all, to be clear, just my interpretation of the “we are a team not a family” implementation. https://www.fastcompany.com/3056662/she-created-netflixs-cul...

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by chii on Monday February 05, 2024 @10:39PM (#39270469)
> You have a seat in company for what you will do, not what you have done.

that's fine, but netflix pays for this privilege by high salary from the get go.

If you worked for a startup, i assume the salary isn't going to be as high. Instead, you'd expect to be paid in equity.

So by firing someone as soon as they're done (meaning their future equity vesting is lost now), they lose a lot of what would've been theirs had the company's long term potential been great as a result of their work.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by mx_03 on Monday February 05, 2024 @08:36PM (#39269687)
>Nobody in the right mind will fire key software engineer 48 hours after system goes live.

Thats the thing. Hedge funds are not managed by people in their right minds

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by missedthecue on Monday February 05, 2024 @09:38PM (#39270102)
Is this cope or are you basing it on something? Because it's actually somewhat of a problem that high finance sucks in so many extremely bright and intelligent people.

I'm sure there are chuds in the HF industry just like any other sector but by and large most people working in those firms are some of the most capable society has to offer.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by mardifoufs on Tuesday February 06, 2024 @12:41AM (#39271173)
To be fair you can be a 'chud' and be very capable still. Especially in finance. Based on my (very short) experience in the field, "competent yet apathetic/morally blank" was the norm.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by mattlondon on Monday February 05, 2024 @11:43PM (#39270887)
Yeah something doesn't add up.

It sounds like they are in the UK. You can't just fire someone out of the blue like that in the UK (assuming they are a FTE), unless they have done something obviously "bad" (proper misconduct etc). If they don't need you anymore there is a defined redundancy process.

If they were not a FTE and were in fact on a temporary contract, then this is totally normal and expected, and the author should have known that as a contractor. Contractors are easy-come-easy-go - that's the whole point of them, and why the pay is higher than FTEs.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by _benedict on Tuesday February 06, 2024 @04:04AM (#39272377)
That’s not true. There’s basically no employment protections in the UK for your first two years of employment. There are pretty good norms in the UK including finance about this, but your statutory protections are pretty weak even after the two years are up.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by ptero on Monday February 05, 2024 @08:42PM (#39269727)
I would agree: while not impossible, it seems like some key information is missing.

On the one hand I would definitely expect zero empathy from a hedge fund. They should pay big bucks (pounds in this case), but if they consider an employee no longer useful it is an instant goodbye.

But firing a key engineer of a successful production system that they plan to operate -- no way!! Neither 48 hours after going live, nor 6 (or 12, 18 or 24) months later. Trading strategies always get refined, changed and tuned and an oops in that process can be very expensive. A hedge fund would fight to keep the key engineer who built the system.

One case where this is possible is if a bigger fund wants to snuff out a new entrant (for example because the smaller fund is messing up the larger fund's strategy). Then the owners get $$ and everyone else gets the boot. My 2c.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by khokhol on Monday February 05, 2024 @10:46PM (#39270507)
Nobody in the right mind will fire key software engineer 48 hours after system goes live.

Perhaps there was a backstory of sorts we're not aware of. But there is absolutely nothing surprising about companies firing people right after handing over huge and/or critical projects. They do it all the time, for both good reasons and bad. Managers in finance are known to be especially "bold" in making decisions like these.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by coliveira on Monday February 05, 2024 @10:39PM (#39270466)
You have no idea of what companies are capable of doing. They probably wanted to do it much earlier but had to wait until the work was "complete".
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by sirspacey on Monday February 05, 2024 @10:29PM (#39270426)
I’ve seen it happen so many times that I envy your disbelief.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by yaseer on Monday February 05, 2024 @08:26PM (#39269628)
I agree, the story doesn't make sense as this person's perspective alone. It's probably only half the story.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by abirch on Monday February 05, 2024 @08:46PM (#39269762)
We definitely don't know enough to make informed opinions. The manager may be new and had not anticipated how many bugs there would be on a version 1.0, the system may not have worked, or many other possibilities.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by fdgadfagfgd on Monday February 05, 2024 @09:35PM (#39270083)
Reading between the lines I just figured it was a crypto quant firm.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by BobbyTables2 on Monday February 05, 2024 @08:50PM (#39269794)
Do people continue paying plumbers and contractors after the job is done? Of course not…

I think it is too easy for an outsider to treat software the same.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by lacrimacida on Monday February 05, 2024 @10:59PM (#39270605)
Plumbers get paid for their work though. Some eager youngster gets exploited, at least once in their lifetime.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by fxtentacle on Monday February 05, 2024 @11:51PM (#39270923)
Or maybe they fired him precisely because of the type of bugs that showed up. You wouldn't want a "move fast and break things" guy to manage billions in cash flow ;)
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by sellmesoap on Monday February 05, 2024 @11:29PM (#39270812)
A friend of mine recently related cleaning up a mess at work where the devops staff was let go 48 hours before the new site deployment - not a tech company - but another case of the left hand doesn't know what the right hand is doing.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by poulsbohemian on Monday February 05, 2024 @11:14PM (#39270706)
Good sir / madam - I was in the middle of fixing a production bug on what was at the time one of the busiest sites on the net when the layoffs came. Like literally was typing out a fix when I got called to the conference room, computer locked when I returned to my desk. Corporate American don’t care.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by Paul-Craft on Monday February 05, 2024 @08:47PM (#39269769)
That's a bold assumption that those involved in firing this person were in their right mind.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by re-thc on Monday February 05, 2024 @08:35PM (#39269683)
> Nobody in the right mind will fire key software engineer 48 hours after system goes live.

Lots of companies and people treat software as physical goods (where it's done it's done) and that everyone is easily replaceable. I've seen many managers not consider uniqueness or the situation but simply we have an engineer with x years of experience so a similar 1 will work just the same.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by j45 on Monday February 05, 2024 @08:43PM (#39269737)
It’s plausible it got to a point where it was done well enough to not allow too much dependency on one person.

The answer seems pretty obvious from the author. They had promised him a cut of the profits and now they don’t have to.

After relying on someone to use their talents to get things done quicker.

Internal politics could also be at play, now that a foundation is built other self interested parties or departments could be angling in to maintain whatever they’re after

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by wahnfrieden on Monday February 05, 2024 @08:28PM (#39269637)
If it saves them a lot of money on paper, yes, they will just hire someone else
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by Grazester on Monday February 05, 2024 @08:33PM (#39269668)
And how long would it take for the person to get up to speed on the code for such software that in production and may have critical bugs that need fixing now?
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by re-thc on Monday February 05, 2024 @08:36PM (#39269692)
> And how long would it take for the person to get up to speed on the code for such software

If only this is more understood. I've been on both ends and as the newcomer you rarely get any ramp up time and management assume you'd know what you're doing in 5 minutes.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by wahnfrieden on Tuesday February 06, 2024 @01:00AM (#39271271)
Business managers don't like to think that way because they know they're already screwed if business continuity relies heavily on "bus factor 1" individuals
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by bugglebeetle on Monday February 05, 2024 @09:00PM (#39269865)
I built a system at an old job that as far as I know has been in operation for over 10 years, completely untouched. I left the company that long ago, interviewed about three years ago for another position in the same department, where someone disclosed it was still up and running, without modification. I’m sure someone had to read the code at some point, but they could have fired me right after it was built and suffered no great loss because of it. I guess I should’ve spent some more time overengineering the whole thing or introducing subtle bugs if I wanted job security, as I have seen others do.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by dirtybirdnj on Monday February 05, 2024 @08:24PM (#39269609)
I think you significantly underestimate the lack of empathy and (lack of) concern for workers that this kind of management has
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by rixed on Tuesday February 06, 2024 @12:04AM (#39270988)
And lack of concern for the end product.

In large enough company, outside of engineering department end product quality or even business long term sustainability is often no one's concern.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by j16sdiz on Monday February 05, 2024 @08:28PM (#39269634)
That's not the empathy issue. It's the system maintenance issues.

You need one or two more release to iron out the bugs, right?

...and this is a trading platform. The market changes, you need program changes for unforseen market changes.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by dirtybirdnj on Monday February 05, 2024 @08:41PM (#39269721)
Lack of empathy is what causes the hubris (stupidity?) that would lead one to believe the person who built the system is expendable.

I have also been in a job where once the project reached a certian point I was fired. They gave me a song and dance about building a tech culture and sustainable development.

It was all lies. They knew when they were going to fire me the day they hired me. Once their demo worked, everything went off the rails. There was no real business plan just hype and bullshit. No wonder they had no idea what to do next.

Sometimes it's a blessing in disguise when you get shit canned at these jobs but you can't pay the bills with moral victories

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by madaxe_again on Monday February 05, 2024 @09:34PM (#39270072)
You assume that they are in their right minds…

This is the U.K. - technology is seen by businesspeople as dirty and bad, as are the people who work with it. Maintenance is unheard of, sustainability is a dirty word.

For instance, I know of a large U.K. medical data processor who wanted ISO270001 compliance, so they got in a contractor to write an ISMS, had it audited, and then immediately fired the contractor upon successful pass of the audit.

What happens next year? Who cares! I’ll be in Marbella with my bag of swag by then. Someone else can figure it out.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by whoknowsidont on Monday February 05, 2024 @09:08PM (#39269913)
>Nobody in the right mind will fire key software engineer 48 hours after system goes live.

lol

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by flandish on Monday February 05, 2024 @09:31PM (#39270051)
You haven’t been around enough middle mgmt who consider “rockstar” engs disposable like capital budget line items.

This is just standard profit over people capitalism.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by BLanen on Tuesday February 06, 2024 @04:51AM (#39272589)
> Nobody in the right mind will fire key software engineer 48 hours after system goes live.

Ah yes, the absolute preposterous idea that nobody, especially not companies, would ever make an irrational choice.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by sunburntboy on Tuesday February 06, 2024 @08:49AM (#39274120)
Could happen if they could find someone cheaper to do the job with the same skill set
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by davedx on Tuesday February 06, 2024 @09:01AM (#39274236)
FWIW I tracked down the company, it's ExodusPoint Capital Management. I think the story is true given by what I found with some basic googling.
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by vamega on Tuesday February 06, 2024 @09:22AM (#39274437)
How did you figure this out? I used to work at ExodusPoint (left about 3 years ago).

It wouldn’t surprise me if it was ExodusPoint, but I’m not sure how one could make that connection from the article.

ExodusPoint had an OMS 3 years ago, so the part in the story about starting trading makes me suspect it’s either an OMS for a new asset class, or a replacement of an earlier system.

Unless the OP joined a portfolio team, and the OMS he’s talking about was to connect that teams strategy to the internal firm wide OMS. I did that for my pod, and I can see a Portfolio Manager making a decision like this.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by mlrtime on Tuesday February 06, 2024 @09:40AM (#39274666)
If you worked at EP then you should also know that they [I've heard] are very litigious. If I were the anonymous writer or op I wouldn't site any sources.

Most likely it was a quant in a pod. OMS could be loosely defined here as well or obscured to not dox themself too much. However EVE trade sounds very specific, I don't know what this means, EVE online trading?

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by davedx on Tuesday February 06, 2024 @09:48AM (#39274744)
I found the person who wrote the article on LinkedIn
Re:I worked 80 hour weeks to deliver a platform for a hedge ... by raxxorraxor on Tuesday February 06, 2024 @11:19AM (#39276158)
I agree that it sounds unlikely. Although if you really invested in someone to develop a platform, you might not want to fire them. Compared to the price of any other software maintenance cost, this is very likely the most bang for the buck you could ever get. Firing someone after 48h seems stupid if they in any way want to use the product that was delivered. Maybe they don't.

While I don't know the specifics about financial software, the job market also is pretty good internationally, so finding something new, as a freelancer or not, shouldn't be too hard.

Re:I worked 80 hour weeks to deliver a platform for a hedge ... by ParetoOptimal on Monday February 19, 2024 @09:45AM (#39430269)
Never seen all engineers able to maintain legacy systems in a profitable business fired because the new half baked system is for sure gonna pan out?

Many CEOs are just playing a video game and don't care if they lose because they have extra lives.