This is called "first principles thinking" and is one of the most reliable methods for producing novel insights into a problem space. It is uncommon in practice because it has a high cost both economically, because you are re-deriving everything you think you know from scratch instead of "standing on the shoulders of giants", and socially, because you are deviating from orthodoxy promoted by high-status individuals.

This kind of relentless indifference to conformity is a rare quality in people.

There's also the cost that you are likely to just spin your wheels retreading old ground. Examples to the contrary are notable, but notable because they are rare.

I made this assumption for many years, and yes, that cost would be high. I eventually discovered that, more often than not, everyone assumed that someone must have already investigated a particular hypothesis but if you tried to identify that "someone" it turned out that they didn't actually exist. After doing this exhaustive search a few times in a few different domains and coming up empty handed, it changed my perspective on the matter, to my great benefit.

Searching for evidence that someone has actually done the work is relatively efficient. It never fails to astonish me the number of times that everyone believes a particular bit of ground has been thoroughly tread yet, if I try to find concrete evidence that someone has done the work, there is no evidence that anyone actually has. There is a strong cognitive bias (I don't know if it has a name) where everyone assumes that someone else has already tried every obvious or reasonable approach and that belief is treated as factual.

>if I try to find concrete evidence that someone has done the work, there is no evidence that anyone actually has

Unfortunately, the fact that negative results tend to get little if any publicity works against you here. And it's probably worse outside, not inside of academia. How often would a project team in some big company, or a couple of guys in a garage try out some promising alternative approach to something, fail to realize an advantage over the conventional approach, and then go out of their way to publicize that failure? That would be extra work for no - or even negative - gain.

It might just be meant to be humorous, but even "If at first you don't succeed, destroy all evidence you even tried" sounds more likely than "If at first you don't succeed, put some extra effort into telling everyone".

Would you be willing to give an example of an area where it was widely thought that "someone" had already checked all the obvious possibilities, but you were able to find an obvious possibility that actually hadn't been explored?

I think you're arguing for doing surveys of literature, which I would also argue for. My argument was against the veneration of From First Principles, which is largely incompatible with Maybe Someone Already Did That, More Completely, More Correctly, A Long Time Ago, And It's Easy To Check First.

This here is the real danger. Many people choose to reinvent the wheel and end up finding out that their approach was actually horribly flawed compared to what already existed, or they turned out not to have the resources to see it through. It takes a brilliant person with a lot of luck to make this work in an advantageous way.

Two great examples come to mind right away, both from Elon Musk. SpaceX was founded on first principles and now he's been able to undercut competitors by an order of magnitude. Conversely, his attempt approaching car manufacturing from first principles in the gigafactory has proven to be a complete disaster.

I think the older and more competitive a space is, the less likely a first principles approach will bring success.

Perhaps it’s not the age per se, but the amount of players over time. (Number of car manufacturers >> space companies).

That was my general point (without making the wording too complex)

>Conversely, his attempt approaching car manufacturing from first principles in the gigafactory has proven to be a complete disaster.

Nonsense. What is that based on?

>Conversely, his attempt approaching car manufacturing from first principles in the gigafactory has proven to be a complete disaster.

Lol yes, manufacturing one of the best selling cars in the country was a great disaster.

There are also the times when you discover a tech, a library, or a language that supposedly solves your issue, but in reality only goes 80% of the way.

Unfortunately, it can take a lot of work to then a) find workarounds (which in my experience tend to become a maintenance nightmare) b) try alternates (which may have other but similar limitations) c) pull out and start doing it yourself anyway.

So this is a worthwhile investment when the existing solutions are insufficient to the task and there is great opportunity in reshaping the landscape.

I find that mentoring a novice apprentice (both teaching general domain knowledge and describing specific architecture decisions in projects they’re getting involved in) is a great way to get back to first principles.

The expected outcome is increased human resource so it somewhat offsets the economic costs.

One caveat is that this process requires the willingness to rethink your own established models, and recognising when they might’ve been outdated or incorrect takes some self-discipline.

Mentoring is the fastest way for one to learn or relearn first principles. Explaining an idea to someone requires that you speak to their level of understanding. Thus, explaining a complex idea to someone with understanding of only fundamental truths (novices), requires that you only speak in first principles. If you cannot, you will provide a bad explanation. When you eventually are able to provide a good explanation, you will know you have grasped the fundamental truths of the idea.

It is common for the young, remember? It is also why you should vote for the young and idealistic; not cynical old farts like ourselves.

and how is that different to the NIH syndrome?

It’s the exact same thing, but successful. Part of being “brilliant” is knowing when to apply first principles and when to build on what exists.

I put “brilliant” in quotes because unless you can do this consistently, you can also just get lucky...

Many insights to be made by this comment. One that is easily missed is how hard it is to fight against the existing ecosystem/assumptions. I've found that a lot of people either don't, or make a point of challenging everything. A strategy they've worked fairly well for me so far is to challenge one(!) assumption after a lot of careful analysis, and then try to adapt to the ecosystem as best as possible. Most revolutionary things are not made in one big bang, but in many incremental steps. They only look like big bangs in hindsight.

Even the nuclear bomb was fairly incremental, with truly a big bang in the end.... (pun intended ;)

The problem is that multi-layered problem solving is superhard to execute. It's not that it's hard to find a solution that would be orders better, but it's just that it's risky, as it's an all-or-nothing path. It at least requires a lot of confidence for the existing market.

With waterfall-kind of planning, that would work. Unfortunately it got a bad name in the '90s and early '00s. The wild-west of inexperienced developers, who needed to solve business problems they didn't understand, combined with clients who didn't understand technology.

I believe this is a great period to do bigger things, and you see this happening with for example SpaceX, Tesla and Apple: space is happening again, companies are creating super specialized and complex ICs, there's actual business value and consumer value being produced, and projects are reasonably on time with these companies. This in constract with companies who, what it seems, do more of an iterative approach: Facebook, Google, Amazon. No huge innovation going there.

Aren't those companies great examples of this though? They have a great vision, but really try to play inside the existing system except for one thing. Tesla for example haven't yet done much innovation that the other car companies don't, except for betting hard on electricity (or more specifically, batteries). A more radical approach challenging the whole system on all fronts would be to re-think personal transportation (perhaps cars VS horses is a better example of that). Sure, they realize that in order to make electrical cars viable, they have to put up chargers. But that's really just a means to an end (batteries/electricity as fuel). Same goes with how they sell cars - they have to in order to break through the market, it's not something that is necessarily core to their business or why they exist. They exist to replace ICEs with battery driven mirrors. The rest can stay. (autonomous driving might be more revolutionary if they succeed and create something different than what cars are today, but that's a lot of ifs and buts)

Apple is also very much in that corner. The smartphone existed long before the iPhone. They just packaged it and polished the concept so it made sense for a broader market (and they are/were really great at that), but how radical were they really? Revolutionary after a while perhaps (if you can be a slow revolutionary - seems a little bit contradictory), but technologically they just put a lot of existing techs together and packaged it differently than the existing players for a different market/crowd and had tremendous timing (I bet the smartphone revolution was coming either way, they just helped define the concept and bridge the gap from early adopters to mass market).

You, like many others, me sometime ago including, are missing the point that can be illustrated with Apple - Jobs broke the carrier control over software that is running on the device which, until Jobs, had been a given unquestionable fact of life like the sky is blue. That is what unleashed the revolution, not the nice piece of hardware which would be just a brick under ATT control like many before.

Consider that Tesla is still an abject failure on their goal of converting the world’s transport to sustainable energy.

I don’t think the strategy you outlined will be enough. In addition to “advance batteries” I believe “full self-driving” as well as “mobility as a service”, “radically production-driven design”, and possibly “car bundled with energy production and storage devices” will be required for Tesla to hit their goal.

If you take those 4-5 things together I do believe that constitutes a total reinvention of the automobile.

An interesting theory as to why we never evolved with wheels is the idea that an evolutionary improvement requires a viable intermediate state...something that works well enough in the transition from inferior to superior. It is worth noting that human inventions can evolve in much faster timelines than biological evolution, but no matter how fast they come, there does need to be a way to fulfill intermediate needs, or else there is no path forward.

What follows is my own personal views on current technological evolution, which may be wrong. Regardless:

Electric vehicles have been evolving extremely rapidly, and now they are viable enough for most privately owned passenger vehicle use cases. Through continuous evolutionary investment, they may make their way to some commercial vehicles, maybe even small planes. But the chasm is so extremely large for large commercial vehicles like airliners and cargo ships, that we would need massive (several orders of magnitude) technological improvements in energy density in order to even start considering them.

SOFCs are inferior to batteries and supercapacitors from an efficiency standpoint. They may never be the power system of choice for passenger cars or other small scale and lightly used systems. But the thing they have going for them is their ability to evolve as the ecosystem evolves. They can run on diesel fuel, JP8, even crude. They can run on biodiesel, or renewable ethanol or methanol. And they can run on pure hydrogen. They can even run on a mix of all of those fuels. At every path in the transition to hydrogen, there is a viable intermediate state. For this reason, I think you'll see fuel cell powered cargo ships and airliners long before you see battery powered.

> An interesting theory as to why we never evolved with wheels is the idea that an evolutionary improvement requires a viable intermediate state

Not really; wheels aren't an improvement in the first place so there is no need to explain why they didn't develop. Note that it's easy to make robots that use wheels, and difficult to make robots that use legs, but we make legged robots anyway so that they'll be able to handle environments other than dedicated roads.

(More recently, we make flying robots, sidestepping the issue that we don't really know how to do legs well.)

Okay, but evolution has given us some pretty expansive diversity in traits across the biological world. No single trait is pareto optimal, but relatively superior/inferior depending on context. Wheels have some pretty extreme efficiency advantages, useful for both speed as well as endurance. They come with disadvantages too, but there are plenty of animals who never need to leave environments where legs are optimal or even necessary for survival. There are some animals that will never leave grassy plains, for example. Yet none of them have evolved wheels either. We don't have marine animals that have evolved propellers, despite an edge in efficiency.

Evolution is content with a very long iteration cycle and a very high failure rate.

Human engineering is much more efficient - building things we understand, we accomplish in decades what takes nature millions of years or is straight up not viable.

I sometimes think about evolution and its relation to human engineering, and I don’t know if it is a useful thought but, evolution created humans, and therefore, human creation is also in fact a product of evolution in nature itself.

In that way, evolution evolved itself by creating humans, and through us evolution is now happening at an accelerated rate in some aspects.

I try not to get all philosophical, because I know that other people, who are actual philosophers, have thought about these things already and I can’t compete with them but I can’t help but think about such things anyway.

Have you considered how these elements essentially detached from the body (it's why they'd be able to spin around) would grow and the like?

"Many bacteria are equipped with a flagellum, a helical propeller that allows bacteria to travel."

Humans didn't evolve with wheels because excepting the last few hundred (possibly thousand) years roads didn't exist.

How useful do you expect a set of wheels would be on the tundra? Or are you conceptualising a theoretical monster truck human? We'd be better off with tracks...

"Ideas that diverged too much from the existing successful tech had a huge obstacle in their way: they had to recreate all of the existing solutions in their new model.”

You’re so right! There are great examples of companies that preserved the existing ecosystem of solutions (Facebook with PHP, Microsoft with C++, Google with Java and C++, maybe also Amazon with Perl 5 and C++).

There are also great examples of successful companies that in a certain sense did or had to do things in an entirely new model (WhatsApp with the Open Telecom Platform, F5 Networks with their data center FPGA-powered hardware load balancers, Tesla and SpaceX, and so on).

There are also great examples of companies that did both, where they preserved an existing ecosystem but also added another ecosystem on top to fix flaws (NeXT Computer with Smalltalk and C, Stripe with the modern version of that as Ruby and Go, Twitter with Ruby and Java, so on).

Microsoft is the best example of a company that can succeed even with what seems like a crazy choice. Almost everything being written in C++ sounds insane to the point of being business suicidal, but they worked hard enough to make it work. If Microsoft can succeed in the ways that they have with C++, and if Facebook could succeed for as long as they did with PHP, and Instagram can allegedly succeed with Django[1], then anyone can succeed with anything as long as you can endure the stress that a seemingly peculiar decision might cause you. And those standard ecosystems that they chose all had immense benefits, they just also had some pretty immense weaknesses as well (Powering 500 million daily users primarily with Python scripts?? That just seems pretty crazy to me, if it’s true).

[1]Still very unclear about how much Python actually powers the web services behind Instagram. <https://instagram-engineering.com/tagged/python>.

I don't see why writing an operating system and surounding tools in C++ is a crazy decision (I'm assuming you meant windows & desktop apps by saying microsoft). Certainly not in the early 90's, but if you have a big codebase of that already, sticking with it doesn't seem crazy to me.

People make too much of a fuss over the differences between programming language. There are some important differences (whether or not memory safe is a big one, but it is hardly critical to a succful bussiness) but by and large the difference between using say php, python, etc for your program is superficial in the extreme.

The fact that many large and lucrative code bases are written in c++ should disabuse you of the idea that c++ is suicidal.

“Many” is almost underselling it. It’s literally EVERY web browser that anyone uses. I don’t know where people get these ideas. OP has a shallow understanding of programming languages, software development, and their history.

> It’s literally EVERY web browser that anyone uses.

And is a fine example of path dependence.

No one can write a browser in any other language not because C++ is necessarily better but because the activation energy to bootstrap to a useful browser is so large (for example, one must provide a smoking fast Javascript engine before one even considers the web page rendering pipeline).

It will be interesting to see how much of the Firefox codebase gets taken over by Rust as time passes.

Basically, C++ developers (like me) are stupid. We're like primitive species, stuck with C++ because we can't understand superior languages. The Lisp developers tried to bring us the light, but we chased them away by throwing rocks at them. Now the Rust evangelists are doing the same thing, but I'm afraid we might take a break from writing video games, stock exchange software, safety-critical software, CAD packages, web browsers, video codecs and the like, to throw rocks at them.

My point is: maybe C++ actually _is_ the superior language. It can't be a coincidence that all the best software projects tend to be written in C++, and not in Haskell or whatever.

> My point is: maybe C++ actually _is_ the superior language. It can't be a coincidence that all the best software projects tend to be written in C++, and not in Haskell or whatever.

If we're going by best software written, I suspect that means C is the better language than C++ by a wide margin. It would be interesting as to whether Visual Basic would be superior on that axis as well.

C++ is the better language--except that we always have to expose a C FFI because nobody seems to have enough critical mass to stabilize the library ABI. C++ is the better language--except that Apple wrote another language because they don't believe that and that Mozilla wrote Rust because C++ wasn't good enough. C++ is the better language--as long as you have a new codebase that only uses the latest features and Satan help you if you have stuff from pre-2005 because God won't be enough. C++ is the better language--as long as compile time isn't an issue.

I can go on if you wish...

I don't think C++ developers are stupid and those choosing to start new projects in it generally have really good reasons for doing so. I also think that many older projects are in C or C++ via path dependence because C++ was the superior choice when the project started and now they have far too much code to switch.

I also believe this will be why Rust eventually becomes a very important language--Rust allows you to modernize that old codebase in a piecemeal fashion.

Dunno. As a mainly C++ developer I think Rust is a real alternative because it tries to solve the same problems better. Lisp and Haskell do not solve the same problems. Huge Lisp codebases probably become hard to understand quickly, and both Lisp and Haskell don't produce very efficient code (or make it hard to produce efficient code).

Time will tell. At the moment I'm not convinced. Rust is a marginal improvement which is not worth the trade-offs in terms of immaturity, lack of libraries, books, standardization, etc. It might end up just like D. Or it might actually find itself a viable niche.

AFAICS D screwed up with its "optional" but really not optional if you want to use the standard library garbage collection. Rust contains no such mistakes, and seems to contain fewers mistakes than C++. Any experienced C++ programmer should know that C++ contains plenty of mistakes. Exceptions are very awkward, the string class is useless, hash maps are much slower than they could be (the std API enforces that), modules should have appeared 20 years ago, non-const references as stealth pointers are bad for readability, [...]

Why C++ string class is useless?

It's basically no better than vector<my_char_type> - i.e. it has no Unicode support whatsoever and very few string-specific convenience methods. Using algorithms is possible, but tedious.

Because I often work with Qt, I'm comparing it with QString, which is much more than a vector of chars. https://doc.qt.io/qt-5/qstring.html

Well, you just kinda answered your question. There is STL, Qt and Boost. Like all other languages. They do the same thing but in different capacities, you should select based on your needs. I think std::string can be supplemented for Unicode support, I am gonna say the fmt library was the supplement but I feel like I am wrong.

I used std::string and QString extensively for 4-5 years and find QString inexplicably bloated and un-intuitive for my use cases.

I don't understand your problem with QString, but anyway, AFAIK the common way to get Unicode support with std::string is ICU. ICU is quite large (~15-20 MB binary) and it breaks binary compatibility with every release.

I don't do C++ anymore professionally, but from I remember, std::locale was very useful: https://en.cppreference.com/w/cpp/locale

And QString had no magic to it other than wrapping its underlying data in ytf-16 which std::u16string does too.

It's not perfect but it gets the job done, IIRC.

> "[Lisp and] Haskell don't produce very efficient code (or make it hard to produce efficient code)."

Curious here. Are you speaking from experience -- i.e. you tried and failed -- or is this simply something you guess must be true? People write high-performance software in Haskell, and it's their tool of choice.

> People write high-performance software in Haskell, and it's their tool of choice.

I've heard of those but have never managed to see one. Do you have some examples ? (to give you my "baseline" - handling >500k messages/second on a desktop cpu for instance as this is something quite easily achievable in C++)

Modern c++ is pretty good, honestly. With the power of STL, you can so many neat things, and make beautiful code. Example: https://www.youtube.com/watch?v=pUEnO6SvAMo

It's almost like language choice doesn't matter at all...

Language choice is an implementation detail. As long as the language isn't technically unable to do the job (ie you can't write a device driver in Python), it doesn't matter much. It may be intensely interesting to practitioners, but it's not a make or break decision. Use what makes the team comfortable.

You're describing what economists and systems theorists call "path dependency". It is common that someone has questioned the assumptions, but that the cost to overcome the path dependency is higher than the gained efficiency or capability.

Meaning: the rareness of disruption is to be expected, because most disruptions fail in stifling silence.

Another hurdle to examining alternate models is “theory induced blindness” where new approaches are either unseen or written off because they don’t fit well within a more traditional framework. It’s especially hard to convince those who have staked a career on the theory that blinds them.

Which means that this is what the exploration pattern ends up looking like in practice: https://www.smbc-comics.com/index.php?db=comics&id=2866#comi...

Aside: This SMBC shall be long etched in my memory: "All the best work has been done over here! [...] The funding is here too!"

---

Here's a teaser challenge for someone who has access to HN comments data for analytics: How often does an early comment come to dominate the discussion, rather than a better comment which comes later? Assume that there is some IID stochastic process which generates comments of varying quality (play with your distribution of choice), and plot what the time distribution of "best" comments looks like under that model -- compared to the time distribution of most upvoted/discussed comments on HN (can try different composite metrics here too).

---

Provocative meta question for PG: Are (venture-funded) startups the "fashionable" idea that is over-subscribed? How would one know? (suppose we restrict the discussion to Silicon Valley)

Isn't this just evolution versus revolution?

Most of the time new products evolve from the current state. Sometimes there's pent-up demand/frustration/hostility that allows for a clean break to a new solution (even if that solution has just as many problems, at least they're not the same problems).

Probably nine times out of ten people are looking for a better buggy whip. Without any other information, improving what you have is statistically the better bet. You have a very strong predictor, but no guarantees.

Inertia seems a good metaphor too, although that too lacks a bit of the specificity you're getting at.