When I worked at mega corps, I'd see a "new" tech emerged that was always very similar to a software that our company had developed to solve similar issues. I remember terraform getting released and seeing that it was just an inferior version of the Cloud Formation Template generator someone at my company had invented years prior.

I've seen this happen so many times and every time, I'm surprised that more businesses don't spin off cool internal projects like this into startups. Hashicorp's market cap is roughly 10% of the market cap of the megacorp I used to work for. And right now, my team is working on an internal project to fill the gaps left by various authentication providers we use. But instead of spinning this off into it's own business, they are just kind of waiting around for someone else to develop a better product so they can buy it instead.

It was explained to me that investors generally don't like spinning off internal projects because it can be a distraction, but it does feel like investors are leaving a billion+ dollars on the table. Especially given the absurd P/E multiples software startups enjoy.

To circle this comment back, I think a lot of staff engineers are working on the next generation of cool technology right now. But they might be working at boring old companies, like a massive retailer, and eventually that internal project will get replaced by one built by the engineer who decides to venture on their own and replicate what they were building as a stand-alone product.

There is a simple reason why megacorps don't spin off projects: they would need someone with vision to search for these projects and with high authority to proceed on the spin off. They don't have this kind of people, for the obvious reason that it is not their main concern. Executives in any megacorp are 100% busy making the core business a success, and investing time in small projects that may come to nothing is just a distraction. This may just explain why big corporations are just a waste of time and resources, in the long term.

> They don't have this kind of people

Because the people with vision and conviction would be running their own startups

Not just that they don't have those people. They also don't have the money. Because most companies are operating with someone else's money. It belongs to the shareholders or the investors or the bank. And they have been entrusted with that money in order to conduct a very particular business venture.

If they go off using that money to spin up side-hustles selling engineering tools, they're taking risks with someone else's money to do something they weren't told to do.

I could have a really cool time-travelling database to sell you, but I can't because I signed away my right to invent. So, if you really want to

> built by the engineer who decides to venture on their own and replicate what they were building as a stand-alone product.

We need to change our contracts.

I think this is the key: employment contracts/ip policies should be designed to encourage spinning off internal projects (that are ‘distractions’ front he core product) as separate business units or as open source projects.

FWIW, the founder of HashiCorp created Vangrant when they are in college.

Why did Terraform end up being the dominant product? Product-market fit. That’s not just engineering, but bringing to market a product at the time when people need and understand it. This includes marketing and educating. Also, Terraform has an ecosystem of prebuilt providers. NPM as a package manager is not as well designed as other systems, but it’s arguably been the most successful. While the internal prop tech you mentioned may be more advanced, it’s not obvious you had the right product people with vision to build out that ecosystem. Most managers and product people at large companies aren’t tech savvy beyond buzzwords

My experience at mega corps has been the opposite: instead of a widespread, open source, robust implementation we had to suffer with a home grown, poorly supported, buggy, inferior product because it had the sponsorship of a politically important executive.

Truly dispiriting to be unable to use JSON or Protobuf, because some executive director claims his “standard” is better when it is demonstrably not so. In the end it breeds a culture of mediocrity — management has made it clear that engineering excellence is not a priority. So support the business use case, but do whatever …

That's an unfair characterization. You are low-balling the creativity, discipline and bureaucracy management ability these individuals bring to the table. The right circumstances are only part of the acquisition. Aptidude and attitude are the other parts.

> Something I learned relatively late in my career was that really impressive work is the result of iterations in tiny steps.

I think you are right. Really impressive work is generally the result of a longer journey than you realize. The time and effort put into it is probably underestimated. But my mind immediately tries to think of exceptions (which totally prove the rule). I'm curious if others would agree with this one: Sometimes there are just really great ideas.

Minecraft comes to mind. And I don't mean today's Minecraft. I mean the alpha and beta versions which were wildly popular and essentially just an engine and sandbox world with a tiny fraction of the features today's game has. People have recreated that core game many times with fairly trivial code, which is why I attribute this more to "great idea" than long iterative process that most people cant replicate. And a totally anecdotally and subjective reason (although I've heard others express it too). I found the game immediately nostalgic. It was new but it felt old - like it had existed in some form before but had just now taken shape. That to me seems like a great idea.

Ironically, I find myself questioning if I underestimated how long Notch took to create it. A cursory search says a week for the initial version, but this timeline shows more like 1 to 1.5 years from nothing to alpha: https://minecraft-timeline.github.io/ Even still, the fact that the core game can be recreated without too much effort does suggest that that it was more about being a good idea than being difficult to implement. I don't look at Minecraft and think "wow I could never do this." Instead I look at it and think, "Wow. That was a really great idea."

Although I suppose you could says that great ideas are an iterative process too. But at that point you have to decouple "really impressive stuff" from "I could never do this".

> People have recreated that core game many times with fairly trivial code

Yes, because they have something to reference. It was like Notch took a level editor and made it into the game itself. That isn't an easy thing to figure out and takes time.

> Sometimes there are just really great ideas.

Ideas are easy and usually it's really stupid ideas that turn out to be great ideas. Who would think that an FPS level editor would be a good idea? Or a text box that anyone can edit on a profile page would be popular (facebook)?

There are not great ideas, just ideas that are executed well.

> Notch took a level editor and made it into the game itself. That isn't an easy thing to figure out and takes time.

He built an infiniminer clone for himself when it was discontinued. It certainly grew into more than that, but that's where it started.

>> People have recreated that core game many times with fairly trivial code

> Yes, because they have something to reference.

But you couldn't reimplement podman in a few hundred lines of code.

> Ideas are easy

Not all ideas have the same quality. Should be self evident. Nevertheless you should be able to see that I understand ideas are cheap in some sense - that's what the comment about decoupling "this was an iterative process" from "I could never do this." You can transfer an idea effortlessly - it's more about implementing it. But if its it trival to implement clearly more of the brilliance was on the idea side of things.

> But you couldn't reimplement podman in a few hundred lines of code.

You don't even need a few hundred: https://github.com/p8952/bocker

And then there's 'dokku' which IIRC, started as a bash version of Heroku.

> Not all ideas have the same quality.

They really do. I've heard all kinds of things in my career, but almost none I would want to dedicate a portion of my life building. Not because they are bad ideas or won't work, but because of the person with the idea or it just didn't interest me. Those people went on to be moderately successful (like hundreds of millions worth) but I'm glad I wasn't on that ride.

That would prove my point though. If it actually is that easy to implement then podman is more in the “good idea” than complicated implementation category as well.

It is an idea, and podman wasn't the first (by far) to come up with something. They just had good timing and marketing. That is what is truly important: execution of the idea.

> like it had existed in some form before

I mean, it had. It was called Infiniminer. Admittedly it had only really existed for about a month before, but it was obvious that that first person voxel modification mechanic was a format that could support some really cool games. It got Notch thinking, for sure.

When Notch posted his first tech test video (archive version: https://www.youtube.com/watch?v=UMpv5kZ9-rE), he wrote:

> This is a very early test of an Infiniminer clone I'm working on. It will have more resource management and materials, if I ever get around to finishing it.

Ironically, he never did get around to finishing it.

> Even still, the fact that the core game can be recreated without too much effort does suggest that that it was more about being a good idea than being difficult to implement

As someone that's "recreated the core game" a few different times using a few different techniques (Unity, then Unreal, then OpenGL twice) this just isn't true. Minecraft is very deceptive in the fact that it looks simple, but recreating the core of Minecraft alpha is incredibly time consuming and difficult. Most of the clones you see don't truly recreate the core either, they'll add very simple biome generation, chunking, and some player movement and lighting.

The core of Minecraft alpha consisted of: mobs, "smart" blocks (fences that join to other fences, interactable blocks, animatable blocks etc), water, lava, physics, pathfinding AI, inventory management, crafting system, Redstone (this is actually pretty huge because it's not trivial figuring out how to properly update chunks), multiplayer networking (also not trivial by a long shot), survival mode, biomes and the nether.

Just building the base game with all of that will take around a year of full time dev work, which it did take Notch around that much time. Additionally, making all of these systems good and fast is a whole 'nother story. There are so many optimizations that can be made that add a lot of extra dev time to the mix.

So, sure, creating a cool blocky chunk renderer where you can add and break blocks can be done in like a week. But Minecraft alpha is not a simple endeavor, and I don't think it would have taken off the way it did if Notch just left out all the actual game mechanics.

> I don't look at Minecraft and think "wow I could never do this." Instead I look at it and think, "Wow. That was a really great idea."

And again, I know what you mean by this because I thought the same thing, but this couldn't be further from the truth! It's funny how I wildly misjudge the amount of effort that making a thing takes until I try to make that thing.

And we also have to remember that before you can even begin to create Minecraft you need all the prerequisite knowledge of how the graphics pipeline works and how rendering in general works. But anyways, I just wanted to interject some personal experience here and mention that a lot of the "simple details" cough multiplayer are actually pretty tough to pull off :D

The thing that separates those guys from everyone else is simple.

Its stamina.

No there isn't some higher level of understanding or deep perception. Every one of the all time great programmers had 1 exact thing in common with no exceptions. Stamina. You will not find a single outlier.

They did their craft every day, for many hours, year after year.

So sure, any of us could have taken the individual small steps. But when you look at a whole body of work, you'll see they're cranking through these "small steps" like the Juggernaut running through walls for decades

This is the answer. There's certainly ridiculous talent (especially with ideas), but in my experience it always comes down to stamina stretched through time. Cool things get built by people that stick it out, and recruit others to do the same. And also know _exactly_ when to exit to pursue something new (but that's a different subject).

Interesting, is the ability to know when to exit a prerequisite? I would think there's a distribution between those you spoke of and some that just picked one thing and went forever.

Well what I mean is there's an ability to see precisely when just enough is done -- no more, no less -- before one can hand things over and exit with sanity and / or dignity. It goes hand in hand. Many work very hard and quit before _the_ critical thing is done, which in a sense forfeits all prior work; others continue working and working and burn out. The best understand something about timing, almost in an intuitive sense. I've seen it again and again, and it's impressive.

Some people can build incredibly cool things in two weeks. It's not about stamina really. They typically outsource stamina to senior devs for future iterations and they move onto new more interesting things to create, and senior devs hate them for that ("you did all the cool, we do all the ugly"). But without them nothing of value would be built.

Stamina isn't the word I would go for. It would be commitment. For those that have loads of stamina, it's the commitment to continuous creative encounters that leads to the perceived magic.

From "Courage to Create" by Rollo May: "It is said that the novelist Thomas Wolfe, who was one of the highly creative figures of the American scene, that he was a "genius without talent." But he was so creative because he threw himself so completely in the material and the challenge of saying it-he was great because of the intensity of his encounter"

Apologies going philosophical, its the only way to debate such an answer-science has no real answers for this title we call "staff engineer"

“Nothing in this world can take the place of persistence. Talent will not; nothing is more common than unsuccessful people with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and determination alone are omnipotent. The slogan "press on" has solved and always will solve the problems of the human race” - Calvin Coolidge

And with Coolidge's great persistence, he helped build the foundation for the Great Depression.

It is possible to "press on" in the wrong direction.

Yes. Determination is essential for success but does not guarantee success.

Kind of like how virtually all great cooking requires salt, but use of salt goes not guarantee a great dish.

Sure, but we're talking about "persistence" and dedication on the scale of "I built a complex piece of software"

After a good time of reflection, questioning mentors etc… Maybe it’s better to press on (in the wrong direction) and learn from it than do nothing and contemplate what we could do or achieve or debate for hours/weeks/months to know if it’s the right decision.

> The separation of talent and skill is one of the greatest misunderstood concepts for people who are trying to excel, who have dreams, who want to do things. Talent you have naturally. Skill is only developed by hours and hours and hours of beating on your craft.

- Will Smith

My take is that talent is a potential, without practice it is left unrealised and never turns into skill, nor success.

Then people look at success and say "well person P has natural talent, I don't, so I can't do that", which is very sad, because ultimately it's mostly about a shit ton of work, and dedication, and commitment, and persistence.

Commitment from the individual, and commitment from your organisation!

I work in a relatively 'flat hierarchy' so there are few organisational structures and proceedings that keep projects on track. It's only me that keeps the marble on the track, there are no organisational tracks. Once I lose interest or stamina the marble is gone.

Other organisations with regular check-ins, project tracking etc. have a system that at least keeps the marble in place.

Maybe, maybe not. At the end of the day it takes people interested and motivated to head in the right direction.

it's not even commitment. it's discipline. many of us know what we should be doing, and how to do each of these small steps. most of us (including me) don't have the daily discipline to pull it off.

How about grit?

Perhaps the word you're looking for is perseverance?

Almost each of such guys didn’t have to respond to Jira tickets and answer to shit asked by PMs and TPMs on a daily basis while standing in a circle and showcase/demo “progress” weekly, be part of a trillion meetings in a day, while also preparing documents for things you provably never understood the why of, and in general justify your existence there in a quantifiable way to every TDH in the team and the company.

So when you’re looking for that role after you think you’ve got that stamina/perseverance/commitment, do remember that maybe you still have to find that right place to work at.

Tall, dark and handsome?

probably "Tom, Dick and Harry" in this context, meaning everyone

Understanding theory helps a lot. It's a hard limit on what your capable of making.

Maybe tenacity instead of stamina?

I beg to differ. I have high endurance but since a few years I make minuscule steps and plateau way below my expectations. Meanwhile some people are gliding with their hands in their pockets and will only need 7 days to exceed their initial goal.

> Something I learned relatively late in my career was that really impressive work is the result of iterations in tiny steps.

I agree 100%

> I am definitely of the opinion that given then right circumstances most people could produce something that other people believe is at that sort of level of complexity.

This is the "if my grandma had four wheels, she would be a truck" argument and in my experience it is not accurate. A quote from Ratatouille puts it very eloquently: "A great chef could be anyone, but anyone could not be a great chef"

> They're not that different to the rest of us

I beg to differ. Again, with the caveat of "in my experience", they are very different. Maybe not in ways that would make a great Forbes article, like having superhuman IQ, but for example "having grit" or simply "ability to obsess over the right things" is a key ingredient that most people don't have enough of.

I can’t relate to this, as it seems to imply you just don’t believe that really talented people exist and that these people can accomplish things other simply can’t. Doesn’t jive with my experience.

I used to be awe inspired by the jvm code base and other “major” pieces of software. Then one day I realized that these code bases were the product of literally 10s of thousands of people years.

It’s not something anyone could do, but often it’s the product of what many can do.

> people who make amazing things. They're not that different to the rest of us.

Clearly that can’t be true. “Amazing” by definition cannot be done by the average person.

Linux is a good example - Linus literally iterated on it as he was learning stuff in University, with the help of one of his classmate/friend. And IIRC, the very start of it was just printing a stream of characters or numbers which represented the output of parallel/asynchronous task handling. They just kept adding one feature after another, until they felt that it was "done" - but of course kept on building.