nybblesio
What if there are actually two independent markets in play but they're masquerading as one?
Everyone is captivated by Big Tech (a slightly larger, more inclusive, subset which includes FAANG) pay. There's no doubt any engineer would love to be making $250k+/year. However, this market is selecting for something very different from the larger, common, market.
We need to ask why Big Tech pay is as high as it is. I don't believe it's driven by supply and demand in the classic sense. Instead, Big Tech sees its candidate pool as "free agents" [0]. This is an important distinction if you view these free agents as a source of potential competition, either individually or signed to another "team".
What if Big Tech pay is a form of greenmail? [1] Much like sports teams -- where pay is also extremely high -- the owners are aware that they're possibly overpaying for these people to sit on the bench. However, their risk analysis tells them that the cost is worth the small loss. [2]
This, to my mind, explains why Big Tech is focused on a very specific sliver of the engineering talent pool: graduates from top 10 schools. These people tick all the boxes: young, smart, fast, energetic, and unattached (typically). With this model in my mind, Big Tech interviewing practices make perfect sense. They aren't looking for "CRUD-a-day" programmers. They're looking for those few engineers who, even under immense pressure, still rise to the occasion and perform. This signals that they could be serious competition if left alone as a free agent. Note: I'm not saying that any one of these candidates will become competition. I'm sure the probability distribution is low but it isn't zero. Unlike other fields -- where barrier to entry is extremely high -- one of these hot shots might -- once every 10 to 15 years -- pull a miracle out of the ether and disrupt Big Tech to the point of destroying them. [3]
The other market is everyone else who uses and requires software to run their business, but software isn't their business. Unfortunately, this market -- if they could get it -- would just as soon buy something off the shelf instead of hiring software engineers. It's only because such COTS doesn't yet exist that this market still requires software engineering talent.
However, unlike Big Tech, this market doesn't see engineers as competition (and, generally speaking, they aren't). This severely limits the leverage that engineers have when negotiating with these companies. Being blunt: this market is looking for factory workers who shut up, sit in the fishbowl, and do what they're told. The cheaper the better. [4]
It seems to me it benefits Big Tech to blur the differences between these markets. However, engineers need to wise up to the reality of the bifurcation and plan accordingly.
[0] https://www.collinsdictionary.com/us/dictionary/english/free...
[1] https://en.wikipedia.org/wiki/Greenmail
[2] Small loss is relative, of course. Individuals have a difficult time understanding how this math makes sense because a) the numbers are far larger than they're used to seeing and b) they can't see all of the other numbers at play which balance out the strategy.
[3] https://youtu.be/oD65g2RFSHI?t=582
[4] Yes, not every company in this market is like this. However, it's extremely difficult to know this from the outside when you're trying to find a job. You can try to pry the information from them during the interview process but this isn't always successful. My experience is that "good companies" in this market are extraordinarily rare. Plus, there is always a risk that they'll be acquired and the new overlords hate engineers.
What do you mean!? Forethought!? That's BDUF! YAGNI! This is the problem with you damn tech weenies: all you want to do is fix things and think!
There is no business value to fixing this problem. Ship! Ship! Ship! Ship! We need to float the IPO or SPAC or...something so we can all get RICH! Who cares if the damn font system is broken!? I don't remember seeing a story card or epic for that in Jira.
Now, there has to be -- like -- hundreds of blockers in Jira you should be working on right now instead of futzing with the tech bullshit.
</sarcasm>
On a serious note: is it possible there's a reason why everything is just slightly shit?
I built several systems using dBASE III and Clipper. Ah, the Summer of 87. [2]
I know I'm old and washed up when I wish I could time travel back to this era. I miss it.
[1] https://en.wikipedia.org/wiki/Ashton-Tate
[2] https://en.wikipedia.org/wiki/Clipper_(programming_language)
I can only imagine the fight people will have with their insurance. One of the many reasons we can't have nice things.
It was decided that this application would be "modernized". Y2K hysteria was ramping up and the nascent dotcom boom was brewing. I lost count of how many times I heard, "We need to put some lipstick on this pig." The new hotness was Java and applets. Yes, you read that correctly: the new front-end was going to be written in Java and deployed as applets. The AS/400 would remain as the back-end. For a period of time DCE [1] plus C and C++ shims (running on the AS/400) glued the Java front-end to the back-end.
The project did get "completed" but in name only. Dealers hated the new interface: it was slow, it required a mouse, it did not support any kind of type-ahead, and PC equipment wasn't designed for such a brutal environment. My last interface with these systems was in in 2001 for a very short follow-up project. At that time the green screen was still ruling the roost. Perhaps they eventually did kill off the AS/400 and all of the COBOL. Nah, probably not.
The AS/400 wasn't then (and isn't now) sexy. Green screen applications aren't sexy. COBOL, RPG, and other IBM-centric technologies like CICS [2] aren't sexy. That doesn't mean they don't work. Ironically, they tend to work too well.
This is but one example from oh so many over the years. Sometimes, very rarely, an old technology truly requires complete replacement. Often, calling something legacy is used as cover to cargo cult and "keep up with the neighbors".
[0] https://en.wikipedia.org/wiki/IBM_5250
[1] https://en.wikipedia.org/wiki/DCE/RPC
[2] https://en.wikipedia.org/wiki/CICS
EDITED: formatting
Although MonsterMash is more advanced, this reminded me of Martin Hash's Animation Master [1].
[1] https://en.wikipedia.org/wiki/Pratt_parser
[2] Mostly. The Pratt model does introduce potential context around operators (depending on the grammar). It's easy enough to extend the basic Pratt model to support this, but it isn't spelled out in many examples or literature.
What's under test is the design. Way before TDD was a thing, when I worked at IBM, we used to call this "inverted design": write the calling code first to see what the API might look like and then make it work. In the late 80s it would have been considered a massive waste to assert behavior though; we'd just implement it.
Automated functional tests (from the outside in) are where the bulk of does-it-do-what-it-says-on-the-tin testing should happen.
Once upon a time, I worked for a company who rents movies on DVD via kiosks. When I joined the team, pricing was hard coded everywhere as a one (1), because YAGNI. The code was not well factored, because iterations and velocity. The UI was poorly constructed via WinForms and the driver code for the custom robotics were housed inside of a black box with a Visual Basic 6 COM component fronting it. It was a TDD shop, and the tests had ossified the code base to the extent that even simple changes were slow and painful.
As always happens, the business, wanted more. Different price points! (OMG, you mean it won't always be a one (1)!!?) New products (OMG, you mean it won't always just be movies on DVD??!) And there were field operational challenges. The folks who stocked and maintained the machines sometimes had to wait for the hardware if it was performing certain kinds of maintenance tasks (customers too). Ideally, the machine would be able to switch between tasks at a hardware level "on the fly". Oh, and they wanted everything produced faster.
I managed to transform this mess. Technically, I would say it was (mostly) a success. Culturally and politically it was a nightmare. I suffered severe burnout afterwards. The lesson I learned is that doing things "right" often has an extremely high price to be paid, which is why it almost never happens.
On "over-engineering":
I find this trend fascinating, because I do not believe it to be an inherent issue. Rather, what has happened, is that "engineering" has moved ever closer to "the business", to the point of being embedded within it. What I mean by "embedding" here is structurally and culturally. [Aa]gile was the spark that started this madness.
Why does this matter? Engineering culture is distinct and there are lessons learned within we ought not ignore. However, when a group of engineers is subsumed into a business unit, their ability to operate as engineers with an engineering culture becomes vastly more difficult.
The primary lesson I feel we're losing in this madness is the distinction between capability enablement and the application of said abilities.
Think about hardware engineering: I do not necessarily know all of the ways you -- as the software engineer -- will apply the abilities I expose via my hardware. Look at the amazing things people have discovered about the Commodore 64 years after the hardware ceased production. Now, as Bob Ross would say, "Those are Happy Accidents." However, if I'm designing an IC, I need to think in terms of the abilities I expose as fundamental building blocks for the next layer up. Some of those abilities may never be used or rarely used, but it would be short sighted to not include them at all. I'm going to miss things, that's a given. My goal is to cover enough of the operational space of my component so it has a meaningful lifespan; not just one week. (N.B. This in no way implies I believe hardware engineers always produce good components. However, the mindset in play is the important take away.)
Obviously, the velocity of change of an IC is low because physics and economics. This leads everyone to assume that all software should be the opposite, but that's a flawed understanding. What happens today is we take C#, Java, Python, Ruby, etc. and start implementing business functionality at that level. To stretch my above hardware analogy, this is like we're taking a stock CPU/MCU off the shelf and writing the business functionality in assembly -- each and every time. Wait! What happened to all that stuff you learned in your CS undergrad!? Why not apply it?
The first thing to notice is that the "business requirements" are extremely volatile. Therefore, there must be a part of the system designed around the nature of that change delta. That part of the system will be at the highest, most abstract, level. Between, say the Java code, and that highest level, will be the "enablement layers" in service of that high velocity layer.
Next, notice how a hardware vendor doesn't care what you've built on top of their IC component? Your code, your problem. Those high-delta business requirements should be decoupled from software engineers. Give the business the tools they need to solve their own problems. This is going to be different for each business problem, but the pattern is always the same. The outcome of this design is that the Java/C#/whatever code now has a much lower change velocity and the requirements of it are future enablement in service of the tools and abstraction layer you've built for the business. Now they can have one week death march iterations all they want: changing colors, A/B testing, moving UI components around for no reason...whatever.
There are real-life examples of this pattern: Unity, Unreal Engine Blueprints, SAP, Salesforce. The point here isn't about the specifics of any one of these. Yes, a system like Blueprints has limits, but it's still impressive. We can argue that Unity is a crappy tool (poor implementation) but that doesn't invalidate the pattern. SAP suffers from age but the pattern is solid. The realization here is that the tool(s) for your business can be tailored and optimized for their specific use case.
Final thoughts
Never underestimate that the C3 project (where Extreme Programming was born) was written in Smalltalk, with a Gemstone database (persistent Smalltalk). One of the amazing traits of Smalltalk is that the entire environment itself is written in Smalltalk. Producing a system like I describe above, in Smalltalk, is so trivial one would not notice it. Unfortunately, most business applications are not written in environments nearly as flexible so the pattern is obscured. I've held the opinion for a long time that XP "worked" because of the skills of the individual team members and the unique development environment in use.
As I stated at the beginning, this path is fraught with heartache and dragons for human reasons.
- the products I wish to build will take time to do properly and I do not want to be in any kind of rush;
- based on my research, I believe these products could be good earners; however, it will take time and the majority of the revenue will be from the long tail;
- this kind of activity is high risk and I treat it as such. What this means, in practice, is that I put in my 4 hours of focused coding every morning and then go do other, less risky activities;
My experience has taught me to prefer transactional business models. Consultative business models spring from the dark side of The Force ;-).
As a vague teaser: one of the products is a SaaS modernization of a classic desktop database/application dev program complete with a recreation of its programming language; and the other is a game based on the premise of "what is an episode of sleep paralysis and waking dreams like?" I'm excited to be working on both but I have my expectations very firmly in check.
Next, I had to honestly explore other aspects of myself. What else held interest for me besides electronics and software?
Further, of these other interests, which of them were reasonably "safe" from outsourcing, offshoring, and automation? I do not want to transition into new currency acquisition activities that are themselves a race to the bottom. Tangentially, I considered the perceived prestige of each of these. I could give two shits personally, but I've learned the hard way that the perceived prestige of an activity impacts how others will treat you.
Finally -- as I am a software engineer -- how can I use this skill to augment my effectiveness in performing any of these new activities?
This is an ongoing process for me. I've always had an interest in the law. Law school is out of the question, so I've looked into paralegal work and how I could provide a unique service in this space.
Additionally, there are mundane dirty jobs -- specialized trash hauling, electronic recycle brokering, etc. -- that are poorly serviced in many localities. I believe I'll end up doing something here as well.
I view this transition as open ended. It's no longer about "career" but staying in a lateral thinking mode about combinations of possibilities.
However, starting around 2016, finding work started to become difficult. The work I did find was no longer enjoyable. It took a few years, but I finally did enough self introspection to realize: it's not them, it's me. I aged out of the industry. I didn't notice it in the moment, it just happened.
I cannot work in open-office fishbowls. I cannot stomach Agile process and how it has turned something I love into menial factory work (N.B. I get it, your [A|a]gile shop is awesome. I only had such luck once). There are many more things about modern software development shops with which I disagree.
Now, it hadn't occurred to me until later that this was showing through in my attitude. Of course, I really did not want to work on the 500th BBA in my career with six Scrumm Masters all demanding 30 minute meetings every morning. I did not want to write more JavaScript or deal with yet another hotness-of-the-week library that does the same thing as the previous 10 such beasts. I did not want to play Schedule Chicken yet again.
Is it my age? Sure, people change. I'm not bored with the programming I enjoy but I did grow bored of modern corporate software slave shops. Hey, more power to them. It's their shop; they can do what they want.
However, it does mean I have to move on. I'm not saying any of this is true for the OP. Just something to ponder if you're constantly facing rejection.
With that said, if there is interest, I'd enjoy building a clean room compiler for an existing language using EnTT and other techniques I've developed over the years. Thoughts? While I absolutely adore TinyCC, it might be worthwhile giving that space another go?
EnTT has an eventing system as well. These can be triggered in various ways (entity created, entity deleted, component assigned/removed, etc). I've used eventing to control fine-grained compilation pipelines. For example a recent scheme compiler I wrote used events to simplify inlining (both define-macro/define-syntax and straight procedure inlining) and quote/quasi-quote/unquote/unquote-splicing translation.
If you spend a lot of time writing compilers, I find an ECS like EnTT can give you extra flexibility during the design/prototype phase. Try it out on a simple prototype and see if you like it.
From the article:
* The software is not the purpose
Translation: your craft, how you do it, and why are not important. Going meta for your profession is not valued and if people see you doing it, you're in trouble. Let the guilt wash over you, heathen!
* Understand the problem
Translation: going meta (refactoring, egads!) is wrong. We're watching you and if you do this, you're in trouble. If you aren't feeling guilty yet, now is the appropriate time. In addition, you must feel shame for your irresponsibility!
* Perfect is the enemy of good enough
Translation: going meta -- unless it is to realize a clever kludge -- means you are a perfectionist. You're wrong! We're watching you and if you do this, you're in trouble. You're one of those prima donna programmers, aren't you? We have no need for you!
* Choose your battles
Translation: Everyone starts off doing everything by hand, so -- you know -- the whole notion of writing software in the first place is kind of a Faustian Bargain. Hey, why did we hire you anyway? It's clear you're one of those whinging, prima donna, basement dwelling, perfectionist, programmer A-type people. I see we need more money in the "team building" budget. You're fired. Now is the appropriate time to feel like a loser, because you are!
* The best cod (sic) you can write, is no code at all
Translation: We can't even be bothered to spell "code" properly, which shows you how important it is to us. Truthfully, if there were a magic product we could buy that does what you do -- but better! -- we would. Don't you dare start thinking about weighing cost/benefit of buy-versus-build. We know you programmer types. You just want to build stuff and it's never the right stuff and you can't estimate for crap and your profession doesn't matter. Screw it, let's just put out an RFP and see if someone has a silver bullet so we can use it against you programmers! Lead bullets aren't working.
Postscript:
OK, now that I got all the snark out of my system, I feel better. It isn't that parts of this article are wrong but rather it makes extremely broad assumptions that all programmers are doing the same thing, in the same context, for the same reasons.
The current gestalt around software development appears to be that only a select few are allowed to decide the "state of the art" in the profession. These anointed few enjoy autonomy and full creativity but the rest of us must check our brains at the door.
Maybe it's just me, but I find this attitude extremely risky and deleterious to the industry and the human race. Of course, I am a grey beard, basement-dwelling, prima donna, perfectionist, anti-social, programmer so nobody will be surprised.
One area you can quickly see this in action is with user interfaces. So many modern applications construct their UI tediously by coding each component and all of the aggregations by hand. If you take a step back, it's usually very easy to see that the configuration of the UI is just data. Why not just build one layout "engine" and feed it data to generate your UI? The engine part can use whatever hotness you like, but you only have to write that code once. The configuration of the UI (the policy) is what changes frequently (the business requirements); not the mechanism underneath. (This doesn't mean you won't have to change the mechanism, you will. It just requires change at a much slower pace and the changes are bound within the context of the mechanism itself.)
This approach separates layers of a system which have different change vectors and velocities. The mechanism tends to have a much lower rate of change and it's where we programmers like to live. Build tools for "the business" -- whatever that means in your context -- so they can maintain as much of the policy as possible.
If you think through designs like this; however, you will realize that our modern fascination with massive "scaling" of teams is no longer attractive. This may explain why it has fallen out of favor.
[1] https://en.wikipedia.org/wiki/Separation_of_mechanism_and_po...
However, I'm constantly amazed how many programmers fail to see the forest for the trees.
To a large extent, not seeing (or seeing and ignoring) this type of solution is learned behavior. Many environments are hostile to this world view. Writing tools (especially those that generate code...eek!) is seen as "over-engineering" and "unnecessary complexity". If you're in a factory, on a line, your job isn't to climb away from your leaf-node.
Any programmer has the potential to be 10x, they just have to see and act upon it. Few do.
To wit:
- We've evolved into a world with "Agile", which our paymasters have defined as the equivalent of warehouse/factory/shop floor process. Each engineer is "the same"; thus, swappable;
- In this view of the world, the only valuable thing to do is the tiny little sliver of work in front of you. Spending time on anything related to real engineering is verboten. When you're eventually asked why the application is slow, producing work faster isn't possible, or new features are difficult to implement, your "technical" answers about "the code", "the design", and/or the "architecture" are derided as "lazy work" and "not valuable". We just don't understand "the business";
- Since you're a factory laborer now, and there is no expectation that you must "think" to do this job, the notion that you require privacy is laughable. "The business" does not give the shop floor guys privacy, why should they give it to you? Oh, and how long do you suppose it will be before you must ask permission to use the bathroom? Just like factory workers;
- Since you obviously don't need to "think", after all you just have to crank out that next ticket/card/story, then being interrupted isn't an issue. As the foreman of a factory floor, I can reallocate my "resources" however I see fit. After all, they're all assembling the same "widget", right?
- If you're lucky enough to not be in the physical fish bowl of the modern software engineering shop floor, then you're expected to be tethered to some god awful chat client. Again, what you're doing isn't complicated, right? You don't have to plan. You don't need to design. You don't need to think. That's what all the tools, and frameworks, and geegaws they spend big money on are for! So if you get pinged constantly by 20 different people for 8 hours -- and your work isn't getting done -- there is something wrong with you;
- If you are unlucky enough to be in the fish bowl itself, then being interrupted now includes both the physical and virtual;
- Meetings are what "the business" does and what we do isn't important, obviously. So whatever "top of the stack" issue is critical to them at any given moment in time overrides any worthless thing you might be doing.
The simple truth is that software engineering is a low prestige career in the majority of companies. There are a handful of real software vendors that still, on some level, grasp what it is we do. However, the vast majority gave up decades ago and just decided to treat us like factory workers. Welcome to the software industry in the 21st century.
John Cleese has given talks throughout the years about how to be creative, and they're very good. [2]
The Programmer's Stone is also a good source of information about this topic. Instead of "lateral thinking", the authors refer to "mapping" and "packing". Packing is when you're in "step-by-step" mode. When programmers refer to "being in the flow", they're probably referring to the state of mapping or lateral thinking. [3]
[1] https://en.wikipedia.org/wiki/Lateral_thinking
To add to this confusion, IBM ported DB/2 to OS/400 atop the native OS/400 data object model. You can use SQL but that just gets compiled to native operations against what are known as "physical files" (confusing name, yes) and "logical files". Physical files are fixed-length record files with data in them. Logical files are indexes atop physical files. A PC analog to this would be FoxPro, Dbase, Paradox, Alpha, etc.
Many OS/400 programs access these files using the native model; not SQL. Tandem and HP Non-Stop work in very similar ways.
He works 3 to 4 hours per day, every day. He most likely works the same time block every day. As he says in the video, life happens. Don't stress life happening. Attend to things that come up and return to your "six pages a day".
The number of pages isn't important. King even qualifies his answer by saying, "...if things go well." The key component here is:
- every day;
- preferably same time block each day;
- concentrated effort;
- attainable goal with a bit of a stretch each day.
I adopted this model several years ago and my productivity went through the roof. (From my own perspective, measuring against myself; not others.) I can say I'm overjoyed at how much work I can get done using this approach. Five hours each day, starting at 4am, is the sweet spot for me. Note, though, that I wake up at 2am to be ready by 4am and I go to bed, uh, about now.
Here's the rub: it will always appear that you can do more. However, as others have said here, it's an illusion.
Finally, I would make the following observation: if one feels secure and has a stable environment, productivity and motivation don't seem to be elusive. If you're chasing after dollars and you're about to go bust if-it-doesn't-get-done-by-this-Friday, you've already lost. It's been my experience that you cannot start from a foundation of anxiety. It can't be "I have to do this or I'm screwed". I realize this creates a paradox for many folks. My solution was just to accept a much lower standard of living to reduce that meat-computer thread constantly running in my brain saying, "...but where the MONEY going to come from!?" The thread still runs occasionally and I just kill it.
[1] https://www.youtube.com/watch?v=xR7XMkjDGw0
EDITED: formatting
Sure, I'd love to have a "mind palace" like Sherlock. Alas, I do not. I often admit this as early in the interview process as possible to avoid wasting time.
I live in Saint George (very far Southern Utah). It's beautiful, the weather is great (if you like hot and dry), and Washington County is very small compared to Utah County and surrounding areas. Traffic is becoming something of a problem and the area is going to be under constant road construction for the next 20 years if growth continues.
There are very few software companies in the area: PrinterLogic, BusyBusy, Mango Voice, and perhaps a few others. If you do move to the area, make sure you have plenty of remote opportunities or are willing to commute to Las Vegas daily (about 90-ish minutes one way).
I've captured the entire process on video: https://www.youtube.com/watch?v=0_PX2Ue6F_A&list=PLWMUVtnFsZ...
The YouTube playlist is an archive of my daily streams on Twitch.
I understand there are a lot of substandard programmers in the marketplace. However, why is it that this specific criteria is the one the industry is so attracted to? Could it be that kids are proficient at these kind of games? Because I can tell you that in the 1980s, 1990s, and early 2000s, I wasn't being asked to write a regular expression parser under time pressure.
Others in this thread say they've done Triplebyte take-home tests, only to end up in a "go fast-fast-fast!" interview in the end.
Why is speed so important? Every popular [aA]gile methodology today is implicitly -- if not explicitly -- against such "machismo" programming. If you're pair programming, how is this ever relevant?
Whenever I see someone say leetcode, hackerrank, and Cracking the Coding Interview is the "answer" it translates to "only the young need apply" in my head.