lettucegoblin
Re:Fmllm: 4mb training data, 100mb model, Fibonacci embeddings, near-c...
on Friday August 22, 2025 @07:24AM (#44983164)
I think the coherence might be coming from the filtering, not the fibonacci intervals.
my thinking is that the process looks like this:
your fibonacci script finds a big list of candidate words -> your grammar rules arrange them -> sentencetransformers aggressively filters this list and picks the single most semantically coherent sentence.
so is it possible that sentencetransformers is doing all the heavy lifting for creating coherence? have you ever looked at the raw, unfiltered candidate words/sentences to see if they have any coherence on their own?
on the "french toast" example, could this be a case of language pareidolia(e.g. seeing faces in clouds)?
the model selects "piney connubial produit" because its math is the closest to the input -> your brain, being a great pattern matcher, creates a story to connect the words.
so is the meaning actually being generated by the fibonacci part, or is it just being found by the filter and then interpreted by you?
with the punctuation model, i'm guessing it's just learning simple surface-level patterns, right?
like, it learns: text contains "but" -> place comma before "but".
how does it handle complex sentences that don't follow the simple patterns in your 4mb of training data? does that break down? the comparison to bert seems off because they're solving the problem in fundamentally different ways.
tofigure out if the fibonacci part is actually meaningful, have you thought about running some controls?
for example, what happens if you replace the fibonacci intervals with random intervals? does the quality of the final, filtered output get worse?
what if you just looked at the raw word lists themselves? is a list generated by your fibonacci model measurably more coherent than a list of words just pulled randmly from the text?