w4nderlust

We got very similar findings: we published a paper that show that smaller LLMs (3-7b) when finetuned with LoRA can match or outperform GPT-4 on a variety of tasks (29 out of 31) including classification, summarization, info extraction, "reasoning". https://arxiv.org/abs/2405.00732 (Predibase cofounder and coauthor of the paper)

Predibase ( http://predibase.com ), also referred in the article, is a platform specifically designed for exactly that. It also has "repos" for finetuning multiple models and comapre their performance and keeping things organzie. It also allow you to query any of the finetuned models on the fly from a single GPU with multi-lora serving. (Predibase founder here)

Here is an example of the Predibase platform, referred in the article for the Solar model, but that can train also Llama-3, Phi-3 and Mistral. https://www.youtube.com/watch?v=R2JQhzfaOFw&themeRefresh=1 I think you can assess by yourself if it's easy enough to do for you. (Predibase founder here)

Speech is coming to Ludwig soon, we are currently working on that.

You may be right about that, but also it depends on the requirements that you have. Ludwig gives you a lot of options for those tricks, like for instance gradient clipping or regularizers, or learning rate and batch size scheduling, but those things usually are useful for squeezing that extra 3% performance, and ieven in those cases, having them already implemented is an advantage. My personal experience is that in many cases doing the first step, getting 80% of the final performance, is enough then to convince someone of the value of what you are doing and then you can spend time later improving over it, and with this regard, Ludwig gets you from 0 to 80% really quick.

Text is just one of the possible types of features supported in Ludwig. For those features, no, you can train models on any language, with a couple little caveats: you can train both character based models and word level ones. For character ones you don't really need anything, and you can train also of languages without explicit word separation like chinese. For word based ones, you need a function to separate your text into words. By default a regular expression is used, which generically kinda works for most languages that separate words, but the tokenizer from the spaCy library can also be used. SpaCy provides models for a few languages, and at the moment we are wrapping just the english ones, but it would be extremely easy to add also the other ones supported in spaCy.

That experience you talked about, being able to really quickly come up with solutions starting from an already established and working configuration, was one of the main motivating factors behind the making of Ludwig too! Basically I had a model, and they asked me first to add an input feature, then to add another one, then to add an output feature. Every time I did it with generality in mind so that the following time they would ask me it would require less time to do it. In the end the final solution is that it just costed me to add a for in a YAML file. :)

Your configuration file looks really good! Why did you stop pursuing it? To your points:

* That may be true, but I think it also depends on the level of abstraction. Fon instance in Ludwig the encoder, although configurable, are pretty monolitic. It's a trade off with flexibility, in Caffe or in your configuration file you are super flexible in specifying each single operation, while in Ludwig that is abstracted from you, but the advantage is that, as long as you trust the encoder implementation, those encoders mimic papers / state of the art models and require much less configuration to be written in order to run (in many cases if you are happy with the default parameters you don't have to configure them at all). So if the field moves and a new text encoder comes in, one can easily add a new encoder to Ludwig too. It's a dangerous game to play catch up, but hopefully releasing it as open source and making it easy to extend may help spontaneous contributions from the community.

* debugging is an issue, that is true, I answered to another post about that, but again, it's a matter of tradeoffs. Debugging is kind of a nightmare in SQL too for instance, or is in TensorFlow (even if the tfdbg improved things a bit).

* extending requires coding, that is also true, but if you have an idea for a new encoder for instance, all you have to do is implement a function that takes a tensor of rank k ad input and output a tensor of rank z as output, and all the rest (preprocessing, training loop, distributed training, etc.) comes for free, which kind of a nice value proposition imho and let you focus on the model rather than all the rest.

Thanks for the interesting discussion!

You have a good point about debugging. So far my approach has been to write quite long and explanatory error messages (like this https://github.com/uber/ludwig/blob/9c9e5de56dcad89461c5d2ec... for instance) and as detailed documentation as I can and, at least when people used it internally, that has been enough. But I'm definitely open to suggestions here.

We are considering a bunch of options re feature store. I personally like EuclidDB, but I'm not sure it fits the notion of feature store that we need (while it's great for ANN inference). Anyway we are looking into it.

Regarding the code-free vs declarative, I mean, it really depends where you draw the line :) Is calling a command line program with some parameters declarative programming? What if there are a lot of parameters? And what if those parameters have a nested structure and are contained in a YAML file? And what if the parameters become so many and so detailed that you can write almost each single operation than the command line program will perform (like in Caffe configurations files for instance)? Do you see what I mean? Anyway, if it's not code-free you may agree that it's the closest open source tool to be code-free :)

I had the chance to read a preprint, it's a great survey, it will be published soon.

I edited the page. Now there's a download button and a preview button. If you click the preview button, the javascript document viewer embedding the pdf will be loaded, otherwise it won't be loaded.