fxnn
And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.
It was an example, yes. But he also referred to Luhmann’s Zettelkasten, a huge collection of notes, systematically interconnected, see https://en.wikipedia.org/wiki/Niklas_Luhmann#Note-taking_sys.... Looks like the author followed a similar approach in Obsidian, a tool to store markdown notes in an interconnected manner.
He wrote: "Roam Research turned bidirectional links into a cult. Obsidian let the cult go off-grid. The lore deepened. You weren’t taking notes. You were building a lattice of meaning. ... A quote would spark an insight, I’d clip it, tag it, link it - and move on."
But it didn't work out: "the insight was never lived. It was stored. ... In trying to remember everything, I outsourced the act of reflection. I didn’t revisit ideas. I didn’t interrogate them. I filed them away and trusted the structure. ... The more my system grew, the more I deferred the work of thought to some future self who would sort, tag, distill, and extract the gold. That self never arrived."
Hence, what he mostly wrote down were thoughts, ideas, quotes, hoping that insights and value will come from this huge collection. Turns out, it wasn't so easy for him. I believe he'll need more focus, more curation, a more targeted and heartfelt approach.
Well, it sounds like it did work out back then for Niklas Luhmann. He said that all his famous publications wouldn't have been possible without his structured note taking approach. His 90,000 cards were digitized and are now available online, see https://niklas-luhmann-archiv.de/bestand/zettelkasten/zettel...
It's true: often enough AI struggles to use libraries, and doesn't remember the usage correctly. Simply adding the go doc fixed that often.
The use case the article describes is exactly the idea behind CouchDB: a database that is at the same time the server, and that's made to be synced with the client.
You can even put your frontend code into it and it will happily serve it (aka CouchApp).
Yes, but then, when we talk about understanding in LLMs, we talk about existence, but not necessarily about determinism.
Remember that, while chess engines are (I guess?) deterministic systems, LLMs are randomized systems. You give the same context and the same prompt multiple times, and each and every time you get a different response.
To me, this, together with the fact that you have an at least 1-in-10 chance of getting a good move (even for strange scenarios), means that understanding _does exist_ inside the LLM. And the problem following from this is, how to force the LLM to reliably choose the right „paths of thought“ (sorry for that metaphor).
I’d guess it’s exactly the same with humans: a human that received good general education can quickly learn specific things like C.
Because what companies want to hide are usually secrets, that are available to (nearly) no one outside of the company. It’s about preventing accidental disclosure.
What AIs are trained on, on the other hand, is publicly available data.
To be clear: what could leak accidentally would have value of course. But here it’s about the single important fact that gets public although it shouldn’t, vs. the billions of pieces from which the trained AI emerges.
The importance of the individual painting diminishes at this scale.
However, on the other hand we also have the scale at which they learn, which kind of makes every individual source line of code they learn from pretty unimportant. Learning at this scale is statistical process, and in most cases individual source snippets diminish in the aggregation of millions of others.
Or to put it the other way round, the actual value lies in the effort of collecting the samples, training the models, creating the software required for the whole process, putting everything into a good product and selling it. Again, in my mind, the importance of every individual source repo is too small at this scale to care about their license.
So why do we care from where LLMs learn?
I use GitHub Autopilot mainly for smaller chunks of code, another kind of autocompletion, which basically safes me from typing, Googling or looking into the docs, and therefore keeps me in the flow.
KV persistence is covered in Chapter 3, right from the start. Redis is also mentioned as an example for an in-memory store with „weak durability by writing to disk asynchronously“.