entilzha

Great to see our paper here again! Since the paper release, we've also released model weights here for anyone interesting in building on top of it: https://huggingface.co/facebook/blt. We also added HF Hub code to easily load the model https://github.com/facebookresearch/blt?tab=readme-ov-file#l....

At least I wasn't aware of this work, but thanks for the refs! I'm always curious to read papers from 10-20+ years ago that have similarly inspired ideas. If it makes sense, we'll mention those in the next related work update.

I don't believe so, or at least if someone tried it didn't work well enough that I remember :). Some of the motivation for the architecture changes in encoding patches stemmed from finding FLOP efficient ways to express relationships between byte sequences. E.G., having a long context window makes sense when dealing with tokens, but you don't need as long as an attention window if you're attending byte sequences to make patch representations, since the patch representations will implicitly be part of a longer context window in terms of number of patches.

(Author here)

If I understand your question right, this is one of the reasons BPE is nice and the parent liked it. For any character sequence, provided the characters are in the alphabet used to create the BPE vocab, there are no unknown words/sequences. One downside of some previous tokenization methods is you could have unknown/UNK tokens, EG dictionary based methods.

In our paper with bytes, we also avoid the UNK issue, since we can have an embedding for every possible byte, since it’s not that many (and for sequences of bytes we use hash embedding, although we did test n-gram lookups for the top K frequent byte n-grams in the training data).

(Author Here)

Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tried a bunch of patching schemes, EG, keep a running total of entropy until the total exceeds a threshold, but ended up finding simple things worked better.

I’ll see if we can add more information about the small CNN in a next update to arXiv paper.

(Author Here)

Related thought, I think BPE is quite a good, cheap inductive bias to have in a model, which is part of what made it challenging to scale better against. I also suspect this is part of why with less training FLOPs BPE is better (left side of figure 1), BLT has to expend some of its FLOPs budget to recover/learn some of this useful bias. With more training FLOPs this becomes a smaller fraction of the budget though leading to better scaling.

(Author Here)

There is at least some work on character based modeling, but it hasn’t scaled well before. The challenge I think with something more adhoc for exceptional tokens is that it’s hard to see gains since they are by definition, infrequent. If the text is rare enough, BPE should produce many single byte tokens, so current models actually expend more compute on these rare sequences.

BLT scales well because it expends less compute (by patching) on more predictable (low entropy) byte sequences. Current models only to some degree get this benefit, if it’s a larger BPE token, but that only goes so far.

So it’s really two related, but different motivations.

(Author Here)

In editing we couldn’t find a good place for this so cut it in the current version, but at one point had discussed a parallel with information density of speech as described by one paper. Essentially the paper found that in languages that were less information dense per syllable, speakers spoke faster to achieve similar information density as languages with higher density per syllable. You could see patching by entropy paralleling this if you consider that low entropy bytes in terms of Shannon entropy are less information dense.

(Author Here)

Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which means you’re pretty FLOP inefficient so scale worse. You could make the model smaller to compensate, but then the model isn’t as good.

Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But ya, there are many directions to go from here!

I tried a few a while back. What I really want is as close to 1-1 to obsidian UI as possible. I found with some of the plugins that it could be hit/miss on working correctly. If I were doing only markdown notes, then wouldn’t need obsidian ;)

While you’re here, a killer feature for me would be the ability to privately host obsidian sites (similar to publish). Even if it required subscribing to publish to download a tarball of the site (that isn’t public), it could still be worth it. My use case is sharing obsidian notes with non-users (eg coworkers) in a private way.

Any thoughts on how to access/modify on mobile without making it too cumbersome? I often think about todo on walk/train, but could see making it a computer only thing.

Totally agree on not trusting any one doctor. Nowadays, I basically assume doctors are narrow minded experts and do the broader thinking myself (by reading widely). It’s the old saying that when you have a hammer everything looks like a nail. I wish I realized the extent to which this is true earlier.

My issue happened to be with sesamoid/metatarsal fracture, flat rootedness, and poor gait. No doctor cared about my gait which turned out to be the root cause of my other symptoms. They all wanted to do surgery/orthotics and basically thought that would fix it alone vs combined with PT.

Not entirely true. For good utilization you need both GPU/TPU ops to be fast (written in C), but that won’t get you far if your input pipeline (possibly written in python) is slow. I could imagine if all the TF calls work in PyPy, that it would help with throughout by speeding up the input pipeline and keeping the GPU/TPU saturated more effectively. One solution is to write everything to TFRecords, but for experimentation that’s kinda annoying.

I’m a co-author and would be happy to answer questions about our work!

Anyone know if puppy works with deep learning libraries like pytorch/tensorflow, or if there are plans to do so? Not looking for numerical speed ups, but for speed ups in preprocessing code

A similar issue got me from recommending android and google services to friends/family to actively discouraging them (and migrating off of every service I could feasibly do). The WiFi chip on my nexus 6 burnt out and neither Motorola or google would replace it without paying several hundred dollars (motorola said it was a software issue and google said it was a hardware issue). Looking at forums this was a known issue across every nexus device.

I have zero patience for poor customer service.

You could actually simplify even more with the trick used in a comment farther down (https://github.com/0101/pipetools). That way you would implement __or__ on for example `pipe` and not have to wrap each function.

As it turns out, implementing that different syntax in pyfunctional wouldn't be too hard or API breaking I think. Mainly it would require 1) wrapping/exporting functions to a module you could bring into scope (eg `from functional import functions as F` or `from functional.functions import *`), 2) Writing wrapper code to provide something functionally similar to `pipe`.

On libraries, I swear I saw something a while back, but my googlefu just now didn't help me find it.

True, but for better or worse python has support for spark/sklearn/pandas/numpy/plotting/deep learning/web which makes it a good jack of all trades language for data/research science. Its strength is that it can do all of those at levels varying from proficiently to best-in-class.

Seq doesn't return a generator. At its core there is a concept of Lineage taken from Apache Spark. Essentially, when you do something like seq(data).map(func) it builds on a list of operations to execute and holds a copy of the base data. When asked for a value only then will it compute the result.

So for parallelization operations that are embarassingly parallel (map/filter etc) are paralelized with multiprocessing. If the data is heavy and serialization is expensive it might be slow, but for operations where the bulk of the work is done in parallel it can help a lot.

Data interchange between the two is relatively seeemless. seq(df) converts a pandas dataframe into a sequence of tuples/namedtuples. Calling .to_pandas(columns=cols) will convert a sequence of tuples/namedtuples to a dataframe with cols optionally being the column names

For operations where vectorization pandas will be much faster since its based on numpy. That being said, there are things which are very awkard to do in pandas which are very easy in pyfunctional

Thanks on name!

Fair enough on docs being a bit scarce. The main intent is to being able to easily make a pandas dataframe a sequence of tuple/namedtuples, and convert a sequence of tuple/namedtuple to a pandas dataframes.

Not OP, but author. Line continuation is pretty annoying. Adding custom methods wouldn't be hard (https://github.com/EntilZha/PyFunctional/issues/113), I just don't have time right now to add it.

With respect to _, this would make these two equivalent: a = lambda x: x + 1 b = _ + 1 a(1) == b(1)

Is there an expanded example I could look at for the pipe stuff, I haven't used clojure before

It would be great to have more native support for FP (flat map is a glaring example I run into a lot). It would help if there were 1) a less verbose anonymous closure syntax 2) support for a multi-line closure.

Author here. 1.0 has been out for a while now, and I'm very committed to API stability. 1.1 (or 1.0.1) should be out as soon as I have some time with mostly minor fixes here and there.

I hadn't considered doing a comparison matrix, but I haven't done a '_' lambda since it seems redundent with the one in fn.

1password is a bit spendy, but I have found the convenience it provides to be valuable (browser integration, TouchId on iOS, and a few other things)

After Rice joined I actually completely stopped using Dropbox, transferred files, and deleted my account.