The 100x plus speedup is no exaggeration either, I've noticed some incredulity from others at the numbers.

For a very simple comparison, I recently was testing a (poorly) custom built data-oriented Entity-Component-System for usage in games with a more typical "componentized" object approach. No multithreading or anything complicated.

On my system, the typical approach could generate about 1000 new objects and attach a single component in about 1 millisecond.

The data-oriented approach could generate about 100,000 new "objects" and attach a single component in about 0.5 milliseconds.

Same thing in the end, but one is roughly 200x faster in the same time frame. It's pretty stunning when you see stuff like this in benchmarks.

I don't get this at all - you're talking about generating objects - I'm assuming you're talking allocating new objects. What's your bottleneck ? Allocator or do you have some poor data structure to update it ? Where is the speedup coming from ? The kind of speedup you're talking about sounds like moving from allocating each object separately to some arena and avoiding allocator overhead.

I could be wrong in the assumptions - but OP is talking about fitting stuff in CPU cache don't really see how that translates to your scenario.

    -----s.-ms.-us.-ns|----------------------------------------------------------
                     0.1 ns - NOP
                     0.3 ns - XOR, ADD, SUB
                     0.5 ns - CPU L1 dCACHE reference           (1st introduced in late 80-ies )
                     0.9 ns - JMP SHORT
                     1   ns - speed-of-light
        ?~~~~~~~~~~~ 1   ns - MUL ( i**2 = MUL i, i )
                   3~4   ns - CPU L2  CACHE reference           (2020/Q1)
                     5   ns - CPU L1 iCACHE Branch mispredict
                     7   ns - CPU L2  CACHE reference
                    10   ns - DIV
                    19   ns - CPU L3  CACHE reference           (2020/Q1 considered slow on 28c Skylake)
                    71   ns - CPU cross-QPI/NUMA best  case on XEON E5-46*
                   100   ns - MUTEX lock/unlock
                   100   ns - own DDR MEMORY reference
                   135   ns - CPU cross-QPI/NUMA best  case on XEON E7-*
                   202   ns - CPU cross-QPI/NUMA worst case on XEON E7-*
                   325   ns - CPU cross-QPI/NUMA worst case on XEON E5-46*
        |Q>~~~~~ 5,000   ns - QPU on-chip QUBO ( quantum annealer minimiser 1 Qop )
                10,000   ns - Compress 1K bytes with a Zippy PROCESS
                20,000   ns - Send     2K bytes over 1 Gbps  NETWORK
               250,000   ns - Read   1 MB sequentially from  MEMORY
               500,000   ns - Round trip within a same DataCenter
        ?~~~ 2,500,000   ns - Read  10 MB sequentially from  MEMORY~~
            10,000,000   ns - DISK seek
            10,000,000   ns - Read   1 MB sequentially from  NETWORK
        ?~~ 25,000,000   ns - Read 100 MB sequentially from  MEMORY~~
            30,000,000   ns - Read 1 MB sequentially from a  DISK
           150,000,000   ns - Send a NETWORK packet CA -> Netherlands
        1s:   |   |   |
          .   |   | ns|
          .   | us|
          . ms|

(https://stackoverflow.com/a/33065382)

However,

    0.001 ns light transfer in Gemmatimonas phototrophica bacteriae

biology has much more performant/optimized machines, therefore, yes, plenty of room for improvement in silico.

The problem with the "plenty of space down there" remark is that practical computers have to have their input states programmable, ie., there must exist some causal deterministic process to set the state of the input.

It's not clear that organic solutions at that level can do programmable computational work, nor that their work is at all deterministic.

At best, it would seem the organic direction for computing will be about building robots rather than CPUs.

Oh, but the "organic" solutions do highly deterministic, extremely programmable computational work: 99.99999+% of newborns have 2 hands, 2 legs, and 1 head, and they all started development from a single cell [1]. It's just that the "organic" solutions are written in a 4+ billion year-old highly redundant, distributed, resilient, evolved language whereas our CPUs are not on the same phylogenetic tree.

The quotation marks around organic are just there to point out that there is something wrong with the dichotomy organic (various pro/eu-karyotes from bacteria to humans)/inorganic (from thermostats to CPUs).

[1] Michael Levin: Anatomical decision-making by cellular collectives https://www.youtube.com/watch?v=Z-9rLlFgcm0

> extremely programmable computational work: 99.99999+% of newborns have 2 hands, 2 legs, and 1 head, and they all started development from a single cell [1].

Just on the risks of early miscarriage from wrong number of chromosomes I'd say your numbers are way off.

> Miscarriage is the most common complication of early pregnancy.[21] Among women who know they are pregnant, the miscarriage rate is roughly 10% to 20%, while rates among all fertilisation is around 30% to 50%.

https://en.m.wikipedia.org/wiki/Miscarriage

So 30-50% failure rate.

You are nitpicking, nevertheless, newborn, noun, a baby that was born recently [1], hence the 99.99999+% figure is applied for the full term pregnancies, once the fetus is decoupled from the mother and has been born as a, well, newborn.

[1] https://dictionary.cambridge.org/dictionary/english/newborn#...

I'm not nitpicking - I'm saying the reason most babies are born with 2 legs hands and a head is because genetic defects die off before birth (plus the screening we have for early termination nowadays) - and the failure rate starting from a single cell is huge.

The genetic defects failure rate is irrelevant, TSMC also throws away bad batches [1], mistakes happen, life is complex and multifaceted, etc. The issue is if biology is a deterministic computational platform, and the argument I was making, using the hyperbolic figure of 99.99999%, I will grant you this, is that not only biology is a deterministic computational platform, but it is better than the one we can achieve currently in our CPUs: no CPU is able to regenerate a melt down core, axolotls can grow back a limb as if it was never gone.

[1] https://www.anandtech.com/show/13905/tsmc-chip-yields-hit-by...

"But at that point your argument is almost tautological"

Yes, that is the point: biodevelopment is deterministic and computational. Watch the Michael Levin video linked above: they cut the head of a planarian worm, it grows back a head; they cut the tail, it grows back a tail; they cut the tail and the head and change the bioelectric gradients, it grows back two heads or two tails.

But at that point your argument is almost tautological - all children born alive are fit to survive ? Being born with such huge asymmetry (missing limbs) and still surviving is low probably (impossible without a head).

I'd say a more convincing argument for deterministic machinery is identical twins - I don't know how much variation there is to put in numbers.

"But at that point your argument is almost tautological"

Yes, that is the point: biodevelopment is deterministic and computational. Watch the Michael Levin video linked above: they cut the head of a planarian worm, it grows back a head; they cut the tail, it grows back a tail; they cut the tail and the head and change the bioelectric gradients, it grows back two heads or two tails.

Don't forget the 0.5% infant mortality rate

https://www.cdc.gov/nchs/fastats/infant-health.htm

Number of infant deaths: 19,582

Deaths per 100,000 live births: 541.9

Leading causes of infant deaths:

– Congenital malformations, deformations and chromosomal abnormalities

– Disorders related to short gestation and low birthweight: not elsewhere classified

– Sudden infant death syndrome

> 99.99999+% of newborns have 2 hands, 2 legs, and 1 head

This number is far too high. The rate of conjoined twins (violating "1 head") is about 1 in 50,000 [1], and the rate of "limb reduction defects" (violating "2 hands and 2 legs") is about 1 in 1,900 [2].

Those correspond to 99.998% and 99.94% respectively. 3-4 nines is still impressive for such a complex system, but let's not claim it's 7+ nines.

[1] https://www.chop.edu/conditions-diseases/conjoined-twins [2] https://www.cdc.gov/ncbddd/birthdefects/ul-limbreductiondefe...

"The occurrence of conjoined twins is rare. Its actual prevalence is unknown, but it is estimated to range from 1:50,000 to 1:200,000" [1]. 1 in 200,000 would raise it to 99.9995%. But as pointed again and again in the other comments, the pointless, hyperbolic figure is irrelevant. When cutting the planarian worm head, the regeneration is always, 100% a head, if no change in the bioelectrical gradients. The argument was about the deterministic computation done by biology in the morphospace.

[1] Importance of Angiographic Study in Preoperative Planning of Conjoined Twins Case Report, https://www.sciencedirect.com/science/article/pii/S180759322...

> pointless, hyperbolic figure is irrelevant

Then why not simply give the correct, still impressive, figure, as I suggested?

> the regeneration is always, 100% a head, if no change in the bioelectrical gradients

This is also a meaningless statement. It's correct 100% of the time, except when something goes wrong and it's not.

Can you quantify the likelihood of something going wrong with the "bioelectrical gradient"? I'm not familiar with this organism but I suspect it's several nines, but less than 7.

In general, probabilities less than a certain amount stop being meaningful, because it's more likely that the model used generate the probability fails to reflect reality. See https://www.lesswrong.com/posts/AJ9dX59QXokZb35fk/when-not-t...

The figure you suggested is also wrong, as per the article I linked, and I can no longer edit the original comment.

The change in the bioelectrical gradient is a human intervention over the organism. Watch the video I linked above. There is 0% chance of "something going wrong with the bioelectrical gradients", it's at the experimenter's will. If you are not familiar then why do you suspect? Your statement is not even meaningless.

> I can no longer edit the original comment

Okay, great, we're getting somewhere. So you concede that the true number of human birth defects is on the order of 4-5 nines.

We know this because we've observed a huge sample size of human births. Meanwhile, the experiment you reference only observed a small set of planarian worm amputations. So we can't conclude there are even 4-5 nines of reliability there, let alone "100%".

Otherwise, we could simply observe a few hundred human births, observe no defects, and conclude that human births are also "100%" reliable.

In our original debate, we were both slightly wrong about the number of nines of reliability in human births. However, you are now infinitely wrong by claiming an infinite number of nines of reliability in planarian worm amputations. I don't know whether the actual number of nines is 5, or 10, or 20, but I can be certain that it's not infinity, because that would violate the laws of probability.

Concede? Debate? Since you linked to pseudo-philosophical mindholes such as LessWrong I suppose it's only natural you would see it as a debate. I will no longer reply since your worldview is irreconcilable with learning and understanding beyond "I am right/less wrong, you are (infinitely) wrong".

Again, you have no idea what you are talking about, as you admitted you are not familiar with the planarian worm organism and regeneration research, and it's not a problem, we are all ignorant about various things, that's why we learn: too bad your learning appetite has been a casualty to the illusion of LessWrong "rationalism". Nevertheless, it is really funny to see you being "rational" and speculating upon things you have no understanding and no desire to learn about. I really laughed reading your now deleted comment starting with "Zero is not a probability."

Just to make it clear for anyone else who might read this: it is impossible to throw a ball in the air and see it flying in the air forever. There is 0% chance of that ever happening. There are no "laws of probability" to be violated in this "experiment". Just the same, when you amputate a planarian worm head, regardless if you did it once, never, or 100,000 times before, it will always 100% regenerate a head, if you, the experimenter, haven't altered the bioelectrical gradients of the worm [1]. The planarian worm regeneration is still being researched and it is revealing biology as a deterministic computation in the morphospace with abilities far exceeding what we currently can muster with our CPUs.

[1] Planarian regeneration as a model of anatomical homeostasis: Recent progress in biophysical and computational approaches, https://www.sciencedirect.com/science/article/abs/pii/S10849...

I don't think the next step is to expose more complexity of underlying hardware to the developer. If at all, then the next step is to have compilers deal with that. Let them rewrite the program.

You make a good, general point. But I found that a data oriented approach has benefits outside of performance and resource usage, because it nudges you towards:

- normalized, small/tight data structures

- data structures that are closely related in computational terms

- smaller interfaces and functions that tend to be more re-usable and general

- fewer if/else branches and more existence based branching via loops

- fewer "business level" generics, macros and similar abstractions, because you can dispatch easily via tagging to concrete types

- less code that "digs"/"drills" into data and more code that composes data

- generally a simpler (less coupled) end result

This all comes with a cost of having to do upfront design and exploration in order to decompose and lay out your data. And while it reduces the mental overhead of understanding the individual pieces of your program on a day by day basis, it might increase learning curve of seeing the big picture, especially at the beginning. So it is a tradeoff.

But I think it would be too dismissive to say that the programmer is doing compiler work here. It is design work, and it is unlearning some of the notions of how to structure programs that carried on since the 90's. Some of which are performance related and some of which are about sensible code structure.

APL-like array-oriented languages can shine here. And do not complain about syntax; think numpy or pandas instead.

Caches are much more limited than RAM though. If every program under the sun starts targetting them directly, is the 100-1000x speedup hold true?

Legit question, I have no idea about how this would behave.

Reserve cache for performance sensitive programs? (OS support?)

How does the OS know for which programs I require better performance?

What if we just put all the RAM on the CPU die? Imagine the speedups for everyone!

I know, sounds crazy. Who’s gonna make such a drastic change to the industry?

You don’t even solve the problem by moving the memory on-die, as the majority of time is not spent sending the signal on traces, it’s waiting for the relatively slow memory to find, read, and send the data on the bus.

Expanding the cache so everything fits in it is one way to achieve the performance uplift of cache hits, but cache is expensive compared to memory, and current cache sizes are tiny compared to RAM. If cache gets 50x bigger tomorrow, chips will get at least 10x as hot, power-hungry, and expensive.

think about it, L1, L2 and L3 are already on die, yet they are progressively slower. Why do you think it is the case?

Distance from the CPU. And traditionally, RAM sits waaaaaay across the motherboard. If you put it on the CPU die, it’s much closer and much faster. Still slower than L1 and L2 cache, sure, but much faster than waiting on the fetch across the motherboard.

Look into Apple’s unified memory.

Imagine not being able to upgrade or have a memory configuration outside of what the manufacturer provides! Incredible!

Apple machines are already pretty much not RAM-upgradeable. And they design their own chips too. So if there were benefits to putting RAM on the die (I don't know myself, others in this thread suggest there are in fact not), Apple would seem like a possible first mover, they don't seem subject to the downsides.

This is exactly what "unified memory" is on Apple systems, the RAM is on the silicon with the CPU.

> ... the available RAM is on the M1 system-on-a-chip (SoC).

Judging by the replies, it seems no one is looking at Apple's "Unified Memory." RAM is on the same silicon with the CPU.

Apple RAM is not on the same silicon[1] as the CPU, just on the same package. Also it doesn't have significant latency advantages. It has higher throughput, possibly because of a custom controller. The biggest advantage is that the GPU doesn't have to go through the PCI bus to access it.

[1] my understanding is that RAM and CPU processes are very different and it is hard to produce a chip with both features while remaining optimal.

A 100x speedup makes sense because a cache miss costs around 100 nanoseconds, compared to arithmetic operations themselves costing under one nanosecond. If you traverse a data structure in which everything's a pointer, every access may well be a cache miss, especially if the rest of the code is also pointer-heavy and thrashes the cache.

Are there any available practical examples of what this actually means, how to do it, and what the limits of this technique might be with programs that your average programmer may encounter/write?