mota7
Written by Claude AI. Which is a problem because Claude text is riddled with engagement bait and filler, making it exhausting to read.
The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).
The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")
MESI takes a non-trivial amount of time to run. It's usually mediated at the L3 cache, which is frequently clocked slower than the main core, so it's a non-trivial number of cycles for a MESI state transition to happen (think at least ~40 cycles).
Compare that with a cacheline that's already Exclusive on a core: Writing to this cacheline is a pure L1 cache write at ~1 cycle, no MESI involved, no L3 cache involved.
Note that the typical userspace space mutex (some memory location that's modified by compare-exchange) is implicitly relying on MESI whenever two cores race for the lock, or whenever a lock is release on one core and then taken on another, etc etc.
So if your userspace coherency can be handled via some sharding that avoids needing MESI to run, it can go much faster than relying on the CPU's "internal mutex" aka MESI.
To directly address "is it possible that a hardware implementation of an algorithm could be slower than its software variant": The point is that with RSEQ, you can write a _different_ algorithm that's faster than an implementation relying on hardware mediated MESI. Instead of having memory that's accessed for a bunch of different cores, there's a chunk of memory per core, and each core almost always writes only to it's allocated chunk of memory.
So GPUs have ~120 small systolic arrays, one per SM (aka, a tensorcore), plus passable off-chip bandwidth (aka 16 lines of PCI).
Where has TPUs have one honking big systolic array, plus large amounts of off-chip bandwidth.
This roughly translates to GPUs being better if you're doing a bunch of different small-ish things in parallel, but TPUs are better if you're doing lots of large matrix multiplies.
So to make a drastic change in direction (aka, a very different orbit):
1. First burn to move far away from the center of the orbit so that you're going very slow.
2. Then burn to make a large change in direction (orbit).
3. Then wait until you cross your desired final orbit, and burn again to close it.
The tradeoff is that these types of orbit change are very slow (because you want to be going very slow for the middle burn, which means you take ages to get there).
So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns out, machine guessing apparently doesn't care much about the difference.
They're effectively doing softmax with a fixed temperature, but it's unclear that this work is going to do better than just learning a per-head temperature parameter.
c.f. https://arxiv.org/abs/2010.04245 which shows an improvement by learning per-head temperature.
The other way to think about this is that it looks like a hacked-up kinda-sorta gated attention. If that's the case, then doing softmax(alphaq_1k_1^T - log_sigmoid(betaq_2k_2^T)) might be better? (where alpha,beta are learned temperatures).
ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.
It's a delicate balance based on the locations that rows are being read and written. In the case where a row being repeatedly written from only one location and not being read from different location, the writes can be significantly faster than would be naively expected.
Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely because of the need to avoid the memory blowup).
Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.
5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers.
As a data point: 5nm lithography equipment is > $100M/unit, needs megawatts of power to run, and they handle about 60-80 wafers per hour.
90nm lithography equipment is ~ $1M/unit, needs kilowatts of power to run, and they handle about 60-80 wafers per hour.
So a wafer scanned on 5nm equipment that could have been scanned on 9nm equipment is carrying a very large capital and operational cost burden that's pointless.
People tend to forget just how hard the edge cases in vision are!
You could also think of this as: We start with a terrible signal to noise ratio. So we need to average over very large areas to get any reasonable signal. But as we increase the signal, we can average over a smaller area to get the same signal-to-ratio.
In the beginning, we're averaging over large areas, so all the fine detail is lost. We just get 'might be a dog? maybe??'. What the network is doing is saying "if this a dog, there should be a head somewhere over here. So let me make it more like a head". Which improves the signal to noise ratio a bit.
After a few more steps, the signal is strong enough that we can get sufficient signal from smaller areas, so it starts saying 'head of a dog' in places. So the network will then start doing "Well, if this is a dog's head, there should be some eyes. Maybe two, but probably not three. And they'll be kinda somewhere around here".
Why do it this way?
Doing it this ways means the network doesn't need to learn "Here are all the ways dogs can look". Instead, it can learn a factored representation: A dog has a head and a body. The network only needs to learn a very fuzzy representation at this level. Then a head has some eyes and maybe a nose. Again, it only needs to learn a very fuzzy representation and (very) rough relative locations.
So it only when it get right down into fine detail that it actually needs to learn pixel perfect representation. But this is _way_ easier, because in small areas images have surprisingly very low entropy.
The 'text-to-image' bit is a just a twist on the basic idea. At the start when the network is going "dog? or it might be a horse?", we fiddle with the probabilities a bit so that the network starts out convinced there's a dog in there somewhere. At which point it starts making the most likely places look a little more like a dog.
Peak insolation varies widely, but 1000W/m^2 is a typical value. 5.8L/hr/m^2 means that it's using something like 180kWh/m^3 on raw solar insolation.
For comparison, reverse osmosis is around 3kWh/m^3. This means that 20% efficient solar panel would produce around 67 L/hr/m^2 (aka ~11x more).
Obviously this is passive versus active, but it's still a surprisingly large difference.
It's not quite as simple as just 2 bits per digit, because the last digit can only be large if the earlier digits are small.
This makes the worst case is (1,2,6) which encodes as 48, for 5.58 bits.
The worst case of (1,2,3) is respectively 4, 3 and 2, resulting in log_2 24 = 4.32 bits.
The first choice isn't picking from 6, it's picking from 4. (because if 5 or 6 are chosen, then there's not enough following numbers to be able to choose 2 more numbers). That's the piece about '(n-k+1) choices for the first number'. 5 or 6 are not possible choices given the ordering requirement.
0. set y = 1. set x = 0.
1. How many possible choices are there _given what we've picked so far_? Call it N.
2. Pick one of them. Call it 'a'.
3. set x = x + a * y. set y = y * N.
4. Repeat steps 1-3 until the number of choices drops to be less than 2.
Decoding is the same thing in reverse. Start with x. mod it with the first N. that's the first choice. divide by N, mod with the number of remaining choices. that's the second choice. etc.
For the case of n choose k, we get the rule that we can't pick anything lower than what has been picked so far. This means that first choice is one of (n-k+1) items. (We can't pick higher than that as we'd run out of choices before we reached k).
With this construction, we can see your case is easy. You just need to be able to signal that some choice was the last choice. So the each choice is from a set of N+1, being the N possibilities plus "picked nothing". Picking nothing means there's no remaining choices so it terminates.
The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.
Now in same cases that may indeed be actual process noise! But there are many fully predictable data series that are difficult for deep networks to learn and in those cases actual signal will be silently removed.
So there's very little loss in using single-precision floating point, and a lot of gains in smoothly handling larger numbers that arise from addition et al.
edit: I'm half wrong. The bit in mantissa isn't wasted, the only bit that's wasted in the sign bit in the exponent, so the efficiency loss is ~1.5%