That seems unfortunately true for most low-level infrastructure software. I am very good at those things, and have a medium amount of experience of building web software. Yet I can find a lot more jobs on the latter domain, and they probably won't pay worse (most likely even better).
Besides pay the non-rewarding things in this domain are mostly that the high complexity is often not properly understood by management, rewarded or taken into account in planning. Which then leads to suboptimal products that have been delivered under time pressure.
If a line of code is written to run in a customer's browser, then that line of code may be deployed to millions, maybe billions, of customers. But, if an equivalent line of code goes in to firmware for some widget, then you're doing pretty good to get that line in to a million widgets at all, and it's going to take a lot longer too.
You'll still get paid peanuts for doing it.
I always felt that ISPs suffer a similar problem. Nobody cares if everything works as expected, but we'll get upset of it doesn't. However, there is hardly anything outside of what we take for granted they can do that we will actively appreciate. What could a embedded engineer working on SSDs do that will be noticed, appreciated and not taking for granted by customers?
If a software product takes off, that can happen incredibly quickly, and the new product is primarily composed of code. If a hardware product takes off, the change can't be nearly as fast as it's bound by manufacturing, and the code is just one component in each thing.
I happen to be working on firmware for a VOIP phone today - we'll end up making N million of these things, over some number of years. If I were working on an Android app with similar functionality, that app could conceivably go to N million people tomorrow, or 10N, or 100N...
Anyway, I don't think I have a particularly clear or concise (or even correct) argument here, but it's the only way I've been able to rationalise what we've observed.
Ultimately, whatever we write that goes into firmware is hidden from the customer. The customer pays a price per unit, and other hardware vendors are competing against your product. This competition keeps the overall cost low. Except at the very top level like Intel or Samsung, semiconductor manufacturers seem to be fighting one financial crisis after another.
Competition does not work like that in Software. The products (even in the same domain) are all too different from each other, so though they may be competitors, they are rarely in direct competition.
That simple.
"Why do we need source control? It's all there, right on my laptop. Source Depot is just a bunch of trouble." [rough quote from memory, maybe conflated from a couple of engineers]. I'm happy to report that things got better.
On the software side of Xbox the people were much better compensated, and we wrote lots of firmware, too. It was probably harder to be hired, though.
I've threatened to withhold paychecks from employees who have said this to me. The job isn't done till the code is checked in, building, and backed up.
The solution for this would be to open source such projects so engineers from many smaller companies can collaborate. These companies needs to understand that they will not be able to attract top talent and collaboration instead of competition is the way forward.
To make it look even worse, the GDP tripled but population only rose 50%, so if you adjusted wages against total economic growth instead of productivity median should be around $110,000. Median.
So yes, $90k is pretty much peanuts for how much money an embedded engineer would be on average making for their employer.
> Another problem is that the job while rewarding is not very lucrative.
Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?
Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer.
Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazing.
I left the embedded space and have never looked back.
I predict we will lose all of our engineers - embedded dudes included.
I had zero embedded experience when I started doing embedded dev, and then I had zero data experience -- but a surprising amount of general experience is applicable!
Heaven for generalists that always love doing new kinds of things. Once I learned about it, I knew I probably should've done embedded instead of security research. Of course, now there's significant interest in overlap. Might not be too late to learn all that stuff after all. :)
> Heaven for generalists that always love doing new kinds of things.
Really? That's me. I'm in web development, and I've always felt that embedded was far too specialised for me.Though I do prefer readable, maintainable code.
Yeah, yikes. That does not sound appealing :-).
The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion.
Kids don't routinely grow up with a SAN in the house. They do tend to grow up with lots of internet connected consumer caliber devices and can easily gain exposure to working with these technologies.
I was fortunately able to explore this type of technology in depth because a family owned business let me tinker with their server equipment in high school.
After college I then co-founded a startup back before the cloud became big. That meant we needed to make use of old hardware to provide service to our customers at a price point that made our service profitable. Old drives were not a reliable way to do that. New drives were extremely expensive for old servers back in the day when SCSI was the interface that you expected for a server. We had to get creative and play with JBOD devices. ZFS was an amazing tool for us in those days, and it still is for anyone who wants to tinker.
The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff.
Knowledge is power though. I recently came across an AI startup that I'm now helping. They were spending significant money using GPU computational power to provide artificial intelligence training through a cloud provider. They blew through about $300k in credits within the first year to give you an idea of how much money that type of power can cost.
I am now helping them cut over to their own co-location facility. The first year alone they will save so much money it will pay for the next three years.
Then you read articles like this: https://www.newyorker.com/magazine/2018/12/10/the-friendship...
Reading that helps reinforce the idea that no matter what path you are on in the field there is potential that some random thing you learned about an SSD firmware helps you optimize some growth stage companies product and ultimately that helps you build wealth.
This, exactly.
SSD firmware is opaque, hard to learn from outside. On the other hand, trending web-based framework has all the source code opened, with great documents, and ready to use tools. No wonder young people of today find passion on other things rather than SSD.
Just curious how you did that. Did you disassemble their firmware?
That seems like a completely ridiculous way to try to organize your life. Almost no companies have $10B exits.
If the drive part of a RAID setup I would actually prefer it just reports itself failed and doesn't slow down access to the array by scanning itself for 20 minutes.
As far as I know, that is one of the main differences when buying enterprise or nas drives compared to consumer drives. With nas drives, the firmware gives up very quickly since the drive is assumed to be part of an array with redundancy. Consumer drives will retry reads for a very long time before reporting i/o error.
When designing a NAND memory product, you aim for some max allowed error rate. You choose the error correction algorithm based on that target. Error rates for NAND products are precisely what the designer intended.
Because SSDs are so large and there is such a large number of them, errors can still occur (at a known rate). FTLs should take that into account. Critical data structures can add checksum redundancy which can reduce the error rate for those to an even lower value, which is usually necessary anyway since power disturbances during erase or program can cause programming errors.
There are of course a number of patterns that increase error rates that FTLs have to be programmed to prevent. Encrypting is the first important step, since it makes the data unlikely to be uniform. The second is mitigating read disturb.
The reason you're not getting enough new grads is probably because your compensation isn't competitive enough.
The point being, the solution to this sort of systemic problem is for more corporations to become worker owned. If the guy writing the SSD algorithms has a say in governance and a cut in the profits they will want to stick around. It's the stable version of startups.
is there any feasible way to recover data after a TRIM command has been issued that you can think of? Is there any way to trick the firmware into not returning 0's when reading the blocks of a deleted file? Mostly interested in doing so for Apple
TRIM destroying the entire data recovery and forensics market seems like such a big deal, I still can't believe it although it started years ago
Manufacturing the hardware requires a lot of capitol to begin with. Why these SSD vendors can't afford the high salary to lure the top talents?
Regarding your point regarding languages: I would be interested and motivated, but I probably don't have the skill (probably, only did some x86 ASM/C++) nor the location (Europe). Usually someone doesn't just start with C++ but with a managed language, and once someone lands a job becomes demotivated or just doesn't have enough time.
Why is this a problem?
Let's not forget, we live in a capitalistic society. The job of the capitalists is to exploit labor. The cost of labor is a direct result of the market demand (or not) for a more complex or, by your point, a more robust product.
It need not simply be a market reaction, though. A vendor can create the market for a more complex/robust product. But still, their job is to exploit the available labor so if the people capable of such are available at a lower cost, so be it.
> I have a higher chance working on a web connected gadget to become a millionaire.
As others have noted, perhaps true but the numbers are so small it may as well also be zero. However, as you should now be pointedly aware, the perception doesn't match reality.
Question: do new grads (or let's say up to 3 years experience) working on SSD firmware, making $90k, have to clock in 60+ hours / week and live in insanely high cost regions where their $120k startup salary qualifies them for housing assistance programs and they have to live with at least one if not more roommates? Or is $90k (per another comment) quite good in relation to the total picture?
IOW, my guess is that you are not competing for talent on salary or total comp. My guess is you are competing on general industry attractiveness. The entire mindset around embedded vs web/consumer programming is different. So I think to frame it as a compensation problem is wrong, and by misframing it you will never "solve" it.
Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Procee...
To compensate for all these deficiencies, the SSD architecture and hence the entire FTL becomes very complicated because any part of it can become damaged at any time. We always have to have backup algorithms to recovery from any scenario. Its difficult to build algorithms that can recovery from arbitrary failures in a reasonable time. I cannot have a drive sitting around for 20 minutes trying to fsck itself.
Another problem is that the job while rewarding is not very lucrative. The chance of a multi million dollar payoff for an employee is low. I have a higher chance working on a web connected gadget to become a millionaire. So that means it is really hard to recruit those who are top notch programmers who known how to figure out the algorithms, write the code, debug the hardware. Most new grads these days are interested in python, javascript and machine learning.