This didn't originate from within Facebook. It is an evolution of the work done in late 2013/early 2014 around FSE[1]. Facebook later hired Yann to support his work around this (pretty amazing!) tool.
The ability to define your own dictionary is useful beyond simply large indexable datasets. Anything application-specific can usually benefit from user-defined dictionaries.
[1] http://fastcompression.blogspot.com/2013/12/finite-state-ent...
Re:Zstandard v1.4.7
by kieckerjan on Thursday December 17, 2020 @12:29PM
(#25457784)
The dictionary helps with the boilerplate stuff that is shared between records but not repeated inside them. So it works like a charm for relatively small fragments of structured data like e.g. HTML pages, XML-docs or SQL statements: lots of "SELECT * FROM" fragments that can be replaced by a single token. I suspect it works even better for this kind of data than for natural language data, which tends to be skewed in a Zipf-like way.
If I had to guess, since this originated from within Facebook, they're using the feature for user definable dictionary to maintain a large corpus of sample, representative user uploaded content.
Which would accomplish the purpose of tuning its performance for their own specific internal usage.