anon291

Meh what's the point of even writing binary, just give it access to the system bus directly and be done with it. general purposes almost seem unnecessary at this point. Just matrix multiplies

So I work on low-level compiler backends and such for chips (mostly AI at this point). I completely disagree that this is something humans never do, because I have spent years doing it. However, it's monotonous and boring and LLMs do it better.

> For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.

Correct. For a long time people wer obsessed with 'increasing context length' as if humans have some great context length system in our brains. We don't. I doubt our equivalent of 'context length' (short-term memory) is very long at all. We just have a good training process that puts these memories into long term storage.

You also have intuition on the weights, you just have better things to do with your time than to ponder them.

However, humans can easily do what Astra did to all the weights. It's not difficult to interpret the weights. It's just tedious, and LLMs excel at tedium.

So the logical follow up here is, since the model can obviously interpret the weights in ways we cannot, to simply take the weights of some LLM itself and then write a program and algorithm that just implements the algorithm the LLM follows. It would seem that, since the language models are obviously over-parameterized, such a system would be able to be written in a 'conventional' way with much simpler code.

I work on AI and I oppose a bailout for an ai bubble or whatever else.

If you're lucky enough to be in this field and you're not managing your finances, that's not the governments problem to fix

Why is this a cloud service instead of a local emulator? I don't get it

At each cycle, a cpu is a simple function from the previous state to the next state.

While we were having this discussion, someone went ahead and did this:

https://divagr.com/blog/qwenputer

> There’s no such thing as a read/write head in transformers. You can use it as a metaphor but that’s a leaky metaphor, hence all these misunderstandings.

The input to a transformer is a list of embeddings. The output is a list of embeddings of the same length. At the end of the day, you can represent the entirety of the transformer's matrix multiplies and such as a DFA (IEEE 754 float multiplication and addition is a simple DFA, and you're just scaling this up that many times). Thus, you take a list of embeddings (i.e., the tape) and output another, modified, list of embeddings. You have successfully read (ingested) and written (the transformer output) N positions on a tape.

Yes. Aesthetic things are good for the soul in subtle ways. It's like making a split pea soup without a bay leaf. You wouldn't be able to figure it out, but you know something is missing and it makes most people sad even if they can't identify it.

> but they won't increase the value of the house.

But they will make your life better.

The papers describing lean and it's core operations are deliberately small. The kernel of a theorem prover is meant to be such that a human can check it for correctness. The kernel does not include things like type inference or execution. It literally checks the proof and that's all and everything needs to match exactly. It's fairly trivial to write such a thing.

Humans have to establish trust in this.

After that, the rest of the language is written. Yes it is possible the type inference engine is broken but in that case the kernel will reject it

Hey now... If you have a formulation in type theory (lean is not set theory), then you can translate to homotopy type theory fairly easily

of course it has a read write head. In fact it has N read / write heads equal to the size of the context since each latent state is updated with the results of the cross attention mechanism at each step.

Alternatively, you can view the '<unknown>' token that is dynamically driven to the embedding of the output token to be the write head. Either way, the simple truth is that you can train a transformer system (i.e., an iteratively applied transformer) to simulate an arbitrary turing machine.

It's not contradictory at all. The dfa at each step of the turing machine model is also o(1).

For the umpteenth time a transformer is an arbitrary turning machine due to the fact that it has a theoretically limitless tape (the context).

A single transformer step is simply O(1) (for a particular parameter size), but it applied a potentially infinite number of times.

Thus while a single transformer cannot solve any problem, a transformer system applied in the normal way can solve any computable problem.

This is why I bought real estate in the bull market of the last decade.

That's a surmountable political problem

No because companies trade at multiples of revenue. A $10/yr in revenue company would trade at a total market cap of like $200. That $190 is new money.

Completely false. If I start a company spending $5 in capital and issue ten shares worth 50 cents each, and later on my revenue from that company reaches $10 through my labor (or the labor of the employees I paid), then the valuation of the company is at least $10 (certainly more) and money has been created because those ten shares are worth more.

Unlike central bank printing or even loan creation, this method of wealth creation is not zero-sum and not inflationary.

why do we care though? If everyone has a high base standard, what does it matter

Wave function collapse is a misnomer. We now know that weak measurements do not collapse the wave function but rather drives the probability amplitudes of the states towards an eigenvalue. For those familiar with linear algebra recall that power iteration of a matrix with a random vector will yield the matrices eigen vector given enough iterations. Collapse is similar to this phenomenon but non deterministic.

My theory: The reason we perceive collapse as irreversible is because the computational complexity of reversing the equations is very high. Especially as the state converges towards an eigenvalue of the measurement operator, it becomes more difficult to reverse. This is similar to how a hash function is reversible in principle but not practice. My two cents.

Many worlds and such are just pseudo religious nonsense.

I mean ... The thing about formal verification is that you can write the extremely slow but correct version of your code and then have an optimizer write a much more efficient version with a proof of correctness.

For things like programming languages we often have tedious, but slow, semantics that can be directly implemented in a suitable language. Then really all that a compiler or interpreter is is a faster version of those semantics.

If you are able to show correspondence between the fast and the the correct version, then you can toss the correct and slow version without even ever running it and just defer to the provably correct fast version.

Most software is ultimately like this

Triangles are the simplest 2-d polytope?

Nice performance. Just looking at the triangles, some of them seem a bit tiny. Does that cause rendering problems at all?

I mean I've been part of many community organizations so I'm not surprised, but interesting to note nonetheless!

You mean you joined the party leadership by going to meetings and such? I'm curious. Can you expand on what you mean by disengagement?

Welcome to fatherhood.

And in that time, if not for COVID, people would have been choosing to live in cities for one of the first times in American history.

Before COVID America was seeing a resurgence of city life.