marviel

In fact in Gu's blog post (linked in a post below) it's mentioned that they created a Mamba model that used this in place of the tokenizer.

paper

> 1. H-Nets scale better with data than state-of-the-art Transformers with BPE tokenization, while learning directly from raw bytes. This improved scaling is even more pronounced on domains without natural tokenization boundaries, like Chinese, code, and DNA.

> 2. H-Nets can be stacked together to learn from deeper hierarchies, which further improves performance.

> 3. H-Nets are significantly more robust to small perturbations in input data like casing, showing an avenue for creating models that are more robust and aligned with human reasoning.

I wrote more about Spec Driven AI development here: https://lukebechtel.com/blog/vibe-speccing

I agree these things don't think like we do, and that they have weird gaps, but to claim they can't reason at all doesn't feel grounded.

Yeah it isn't a panacea but it has afforded me less frustration than the alternative of jumping straight in.

> Since what I've been working on is just a mash-up of other people's ideas

Totally, I find most work I do, if I'm honest, is in this bucket. LLMs are pretty good at "filling in the gaps" between two ideas like this

this is the idea behind my recent post actually[1] where I recommend people use AI to write specs before they code. If all you have to do is a human is edit the spec, not write it from scratch, you're more likely to actually make one.

[1] https://lukebechtel.com/blog/vibe-speccing

Changing these buttons to "Rent" would force people to acknowledge what they're getting for their $60 bucks -- effectively a rental.

Once the problem is identified en masse, capitalism can do its work more efficiently.

it's only useful if it'll be used!

The md files are actually pretty great for shareability, versioning, and picking up where you left off.

I talked about a similar, but slightly simpler workflow in my post on "Vibe Specs".

https://lukebechtel.com/blog/vibe-speccing

I use these rules in all my codebases now. They essentially cause the AI to do two things differently:

(1) ask me questions first (2) Create a `spec.md` doc, before writing any code.

Seems not too dissimilar from yours, but I limit it to a single LLM

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

Which embedding types did you try? I'm surprised that embeddings weren't able to take you further with this.

"The Emotion Machine", by Marvin Minksy (the AI view)

&

"The Art of Empathy", by Karla McLaren (the Internal / Emotional View)