To be functional and independent at it, you need:

- a very solid grasp of C/C++,

- a good understanding of the concept of a state machine,

- a decent grasp of digital electronics and communications protocols, and

- some comfort in debugging (or at least diagnosing) electronics issues.

These skills take years to accumulate.

At the same time, I keep hearing the claim online and IRL (particularly among SWEs who try to get into embedded) that "it's just software - you don't need a good understanding of electronics" to be a competent embedded systems engineer. These are also the same people who get hung up on pretty fundamental problems with simple solutions, like:

- I2C buses missing pullups

- Power supplies not delivering enough current (Hi, everyone who's ever tried to drive a huge NeoPixel panel from a RasPi alone)

- IO logic levels not meeting input specs (e.g. driving a 3.3V input with a 1.8V signal)

Maybe I'm needlessly cranky but I've seen a non-trivial number of people go from "It's just software" to "OMG what do I do help" in the face of very basic hangups like these. It highlights to me that folks who have the breadth to do all of these things comfortably are extremely valuable, and hiring companies know this.

There is a "Computer Engineering" major in my university. C/C++ and state machines are taught in a freshmen year course. However, digital electronics / communication protocols / debugging is never taught in class. Practically everyone who know what SPI or I2C means is either an electronics hobbyist or member of a student competition team.

I personally am very interested in the "software for hardware" realm, but it is a little difficult to find people doing interesting approaches. It's usually all Verilog or all VHDL on FPGAs with a smattering of HLS, all C/C++ embedded software, all high-level programming languages with soft real-time on single-board computers or something similar, or somewhere in between with something like LabVIEW. But there's usually almost no overlap between the people that do work in each of those categories.

There's a lot of constraints in these systems and almost all of the innovation has come at sort of the chip level in terms of hardware performance. The design and development process, tools, and experience have seen little innovation, as far as I can tell. Everything remains hard in embedded systems. I don't see any reason why it needs to be that way though.

Because nothing works by default on ES systems, you have to implement it yourself most of the time. If you can clear that hurdle, your company is unlikely to be open source friendly. If you can clear that hurdle, people on other teams won't have time to adopt it for their systems, so it won't spread. If you can clear all of that, all you've won is a single tool. In the meantime, the rest of the industry as written half a dozen.

There are things you can do here, but... yeah.

Also if time is scarce, following pointers is slower than calculating offsets, but that's usually not an issue worth thinking about.

Another reason to keep memory contiguous is that fragmentation can be much more punishing (sometimes), especially when latency is important. On large devices it doesn't matter much if you're under-utilizing the CPU while waiting for RAM, because the OS will do other things and then let you do a lot of work all at once. Embedded processors will have nothing else to do.

It seems like allocating nodes out of a fixed size array wouldn't be too bad, though.

On devices with 1 GB of memory, the contiguous structures are usually better, but embedded developers often still reach for linked lists. On devices with 1 kB of memory, trained software engineers will scoff at linked lists and instead waste hundreds of bytes on large buffers sized for peak usage.

There are alternatives, of course. Your particular target might perform poorly with linked lists, so you might make your malloc use a different internal data structure. Or you might have plenty of flash but not much RAM, so you might trade code size for runtime overhead. Or... There are lots of alternatives, but a linked list is the simple default.

If you've got 6 bytes of free code space and/or you're scrambling for RAM, you look at everything for space savings.

If the hardware team comes to you apologetically and says, "We're sorry, but for cost reasons we have to /double/ the amount of DRAM and flash and give you a faster, more power efficient SOC" then you may not have to worry about this level of detail.

Both of these have happened to me on prior projects. There are no fixed answers.

(Whenever I've gotten one of these types of questions in an interview, I try to have a broader conversation on the subject rather supply the answer they are looking for. If that goes badly, it's a big red flag).

College curricula vary wildly in quality and content. I don't know of one that quite teaches you everything you need to know to be a functional embedded developer.

Is it because linked lists can be slow and can sometimes pocket arbitrary data?

I feel like strong typing and limiting pointers in favor of runners combats this.

Most of the people I know in embedded development are greybeards. There aren't a lot of younger people going into the field. The schools don't seem to be prepping a lot of people for these kinds of jobs. (not sure what we're going to do because a lot of these guys are in their 60s and about ready to retire)

> These skills take years to accumulate.

Indeed. Which is why the low pay is all the more puzzling, but I have theories: As mentioned above, most embedded devs are greybeards. They don't move around much. They probably aren't as familiar with the much higher pay people are getting at the FAANGs, etc. They aren't as likely to have a LinkedIn profile or if they do they don't spend much time there. As an older dev myself, it's not like I'm clamoring for higher pay: my mortgage payment is quite low compared to rent (and will be paid off completely next year), my cars are old but paid off, don't have any debt other than the soon-to-be-paid off mortgage. All this to say that older devs maybe don't make as much money because they're not as aware of market rates and they don't really feel like they need a lot more money if they're happy where they're at.

The pay is there. I'm not sure why there's this persistent notion that it's not.

I generally agree, but "cache line" is a pretty low bar though, no? I'm a web developer with no formal CS background and I've known what a cache-line is since the first year I started to learn to code. Every textbook on my shelf that even tangentially relates to the kind of work embedded devs do makes reference to cache-lines, and god knows how many articles I've read over the years.

Training people from what is probably nothing is a pretty big bet on the part of a team and employer...

I don't disagree with any of this, I just think there's a certain "base" of knowledge for certain jobs that needs to be so well understood that it's automatic. For an embedded engineers I think "cache line" is part of that base in much the same way I would consider it a hard requirement for any incoming frontend web-developer to know what the DOM is.

So if you're hiring for MCU work (bare metal or RTOS) the concept of a cache line isn't important. If you're hiring for application processor work (often Linux), it's very important.

I look around at the 6 figure web dev jobs and it honestly breaks my heart. I look at embedded and engineering roles (ones that interest me anyway) and it's rarely 6 figure. I just don't have the mental fortitude to deal with web dev monotony.

The only other time I got bad pay was game development.

So, my experience is, the more technical the job, the less you get paid (but more interesting/ demanding). Look at middle management, why do they get paid better? (Btw, I've been CTO too)

If you move to be the guy that interfaces between developers and business people, it's not monotonous - each group is retarded in its own unique way.

You either stess over it, or you dont take it too seriously and have fun :D

Any idea how you can do that? I noticed companies are very picky who they put in those spots like need you to have the same experience in the industry or connections in the company.

Business Analysist, solution architects, presales engineers, product owners, sometimes even project delivery managers and developer team leads have to deal with this.

You might stand better luck of getting into one of these positions in a consultancy company, because there each customer has a separate set of requirements and often they don't have enough of these people.

This is why some web devs have 3+ full-time jobs.

Not true at all. Pay has nothing to do with trendiness, but the other way around, being a webdev is trendy because of the high pay, great benefits like WFH and low bar to entry. Meanwhile embedded dev has a higher bar to entry due to the difficulty of the work, low relative pay and low flexibility in terms of WFH, making the jobs untrendy.

And no, the outsourcing of most HW related dev work to China has had a big impact in lowering embedded dev wages in the west, coupled with the fall of great electronics giants in the west. When I started uni to become an EE there was Nortel, Blackberry, Siemens, Ericsson, Nokia, Motorola, Sagem, Philips, etc. developing HW and mobile devices in Europe, or the west, and were hiring like crazy. Fast forward 6 years when I finished my Master's and most of those companies have either went bust, or have become just brand names for Chinese OEMs, or have become sweatshops for far east workforce, keeping only some of their sales and senior management in the west.

The EE market in the west has went way down in the last 10-15 years in comparison to the web dev market which went way up. The only western HW company making insane profits is Apple and semi titans like ASML, Intel, Nvidia, AMD, Quallcomm, while the Japanese, Korean and Chinese companies fighting for the rest of the scraps, and most of the European ones throwing the towel completely. The commodization of HW and FW dev has meant the commodization of dev wages as well.

After graduation, some of my colleagues went into mobile app dev (the iPhone has been out for a few years but were far away from becoming the norm) and are now making several times what I do as an embedded dev at the same level of YoE. Talk about betting on the wrong horse. I still can't stop kicking myself for choosing such a poor career path and wonder if I can still switch as most companies seem reluctant to hire a thirty-someting embedded senior to do junior web dev work usually done by a teen or twenty-something out of bootcamp.

If you are based in UK, drop me a line.

If you are an embedded developer with experience, you probably understand the difference between a linked list and an array, I am interviewing 'senior' web devs and half don't.

Many 'boring' companies like corporate accounting or whatever, need developers badly and pay maybe 70-80% of what 'fashionable' ones do. It could a good place to break in.

From your answer I take it you're well compensated. From another embedded dev who is not well companensated, may I ask in what market you are in?

It was difficult to find a company that compensated well. A lot of the non-tech companies that rely on embedded developers like automotive and consumer electronics companies pay shit.

The large tech companies all have need for embedded developers, for example Apple, Google, Nvidia and Amazon.

That's absolutely insane to me. In EU I make 55K (considered a great mid-senior wage in my area) and could make up to 75K for senior, if I move to an expensive city in Germany and work at a top company but that's the top end of the market. There's no amount of interviewing or job hopping that could get you further as an embedded dev IC here. Higher pay is reserved for management positions which are tough to get. Even making six figures as an embedded IC here is impossible, let alone 200k+

When I was looking for a new job last year I had a hard time finding any company that would even match my compensation. Luckily for me, large tech companies have an absurd amount of money these days are and paying absurd salaries.

Are you comparing apples to apples?

Does a mediocre Java developer make as much as a mediocre embedded systems developer?

Does a elite Java developer make as much as a elite embedded systems developer?

I find your excessive faith in businesses disturbing.

Markets aren't rational [1] and the wage-labour market isn't either. Sometimes there's no good reason for something, only a bunch of chain of consequences that don't necessarily have to make logical sense from an utility perspective.

[1] https://www.cnbc.com/2017/12/21/long-island-iced-tea-micro-c...

Embedded systems are part of tangible products. They have an order of magnitude greater non-recurring engineering costs, then cost of components, manufacture, packaging, inventory, distribution and after-sales service and support. Due to the huge up front costs, volumes are generally limited by capital funds. So the net profits are likely to be 15% - 20% range.

When you look at the differences, it becomes clear why tangible product companies are unable / unwilling to pay like software-only companies.

In contrast, the scale, and thus the complexity and cost, of embedded tends to be limited by physical constraints.

Seems worth the investment from a rational standpoint.

I seriously doubt it. In all my EE gigs, untrained EEs weren't shipping anything but were there learning and helping the graybeard ship. In fact you need a lot of industry experience and learning from the graybeards to ship products that will actually sell for profit and not flop due to some weird edge-case that didn't cross the junior's mind to look for before shipping, since the answers to most difficult issues are had through years of blood, sweat and tears burning the midnight oil with an oscilloscope in hand, not a google search away on stack overflow.

If you're into actual product development and not consulting work, getting untrained EEs to ship products will mean the shortest path to your company going bust in notime.

https://www.wikiwand.com/en/Pointy-haired_Boss

And this is not specific to IT - many academic researchers have to make ends meet with almost minimum wage.

I've also wondered how employers see them from the business perspective. It's hard to determine a 'business profit contribution' value attached to the 'quality of work' they produce except that things don't break down, and stuff gets done on time.

Furthermore, if the embedded code is the sort where once it's done, it's done, and little to no maintenance is required, the business people might see no need for the guy to be around. Maybe someone could clarify if this is true.

It's about in line with what electrical engineers make on average, slightly higher, at least in my location. The market places more value on people who can write code to give you a nice shopping cart interface over the ability to design digital control systems for engine control units to make sure your car can drive properly. It's just what we as contributors in a market value as truly important to us. To give some more insight: Look at the value of Twitter. A control system for a weapons-grade nuclear centrifuge - a weapon of mass destruction - is valued less than a 280 character limit.

So, from that huge pool (big supply) there's plenty for the few that are needed (little demand); enough even that will choose that domain over a better-paying one for interest's sake, so it ends up not having to pay as much.

I think there's been a lot of market distortion over the last 15 years or so due to the way money has flowed. VCs haven't been very interested in funding startups making hardware so they tend to be self-funded as in funded on a shoestring. There's not the money sloshing around for these companies.

Also EE here. Because in EE you need to invest a lot of funds into equipment, prototyping, manufacturing, RMA, shipping, logistics, customer support, etc. only to make pennies in profit when you actually sell your finished products, while it's the opposite for SW dev work, developing it and selling it costs you next to nothing, you just need a laptop and an internet connection, and once you sunk the cost of developing it you can scale it virtually infinitely and sell it across the globe with low costs and high margins.

Long story short, it's a lot more profitable to make apps where you get consumers to click on ads, buy stuff, or invest their savings into meme stocks or crypto, than to build and sell physical products to them (unless you're Apple).

Hardware eng. unsatisfied with $ -> ask their friends in the other 2 cos in the same niche how much they are being paid -> cry

Bootstraping a software business is trivial compared to any other eng. business. That, plus the internet's existence, is what makes dev so well paid.

In hardware world that translates to contracting or hiring a Chinese team to clone your competitor and eat the lower-end of the market. This is especially true for development boards, hobbyist and supply chain stuff. Digikey vs LCSC, Sparkfun/Adafruit vs AliExpress/Taobao, Arduino vs clone boards.

Sometimes the dumb clone company gets big enough and become a legit player in the Anglosphere. Say Seeedstudio or ALINX (FPGA dev boards).

I think it’s because the software is generally simply considered part of the BOM and thus a cost to be reduced. Also, the hardware side typically rule the roost in such projects, and hardware engineers tend to look down on software developers because, after all, how hard is it to just type code in (if you’ve ever looked at code written by EEs…ugh).

As supporting evidence: the pay is better at companies that do take software seriously. Examples: at one class, Cisco; at another, FB, Google, Apple…

Additionally, I'd add that many embedded developers work for hardware companies that don't understand or care about software; why pay so much more, and they already think they're paying "way more" (it's more than we pay EEs!), for something you don't get.

Finally, Embedded Software Engineers do quite well often. We don't pay people based on how challenging their jobs are.

With much of embedded you don't need to worry about updates, maintenance, logs, security, multiple cores.

The quality of code I was submitted was atrocious.

Any halfway decent Chinese firmware/embedded developer is likely going to work for a bigco where they get paid well, like Tencent or Huawei.

So parent's point still stands that Chinese competitors along with customers in the west who prioritized low cost over everything else, destroyed the embedded dev market in the west.

I'd accept "undercutting", but that's a corporate/macro pricing trend, not a trend for salaries among embedded devs in China. I suspect many companies who are undercutting pricing are getting funded by the Chinese government. They're still paying people quite well, just because they need talented engineers just as much as their competitors do.

Maybe you can but they don't exist where I live in EU and that's the only market I can speak of.

Because there is no demand. Because today, you get all the software along with hardware. And that software is already written somewhere in China, by same people who make the hardware.

Google, Facebook, Snapachat, MicroSoft, Amazon all run their hardware R&D in China for obvious reason that the hardware is already being manufactured there. They all copy Apple, and surely think that they can become Apple if they do like them.