roborovskis
roborovskis on Hacker News
What would you define as 'distillation' versus 'learning'? How do you know that what a LLM is doing is 'distillation' vs a process closer to a human reading a book?
From my perspective, pretraining is pretty clearly not 'distilling', as the goal is not to replicate the pretraining data but to generalize. But what these companies are doing is clearly 'distilling' in that they want their models to exactly emulate Claude's behavior.
Posted by
roborovskis on Friday December 12, 2025 @01:44PM
from the thinkingmachines.ai dept.
Posted by
roborovskis on Tuesday January 07, 2025 @11:25AM
from the brianfitzgerald.xyz dept.
Posted by
roborovskis on Monday December 02, 2024 @10:05PM
from the brianfitzgerald.xyz dept.
Posted by
roborovskis on Thursday November 14, 2024 @01:41PM
from the twitter.com dept.
Posted by
roborovskis on Sunday June 09, 2024 @09:02PM
from the github.com dept.
Posted by
roborovskis on Wednesday April 17, 2024 @10:26AM
from the stability.ai dept.
You could definitely use this for upsampling negative prompts, though I haven't tested that much. In theory, future T2I models shouldn't need to be negatively prompted as much; I find it's better to focus on really high quality positive prompts, as that is closer to the captions the model was trained on.
You can take a look at the dataset here: https://huggingface.co/datasets/roborovski/upsampled-prompts... Roughly 5k samples were needed for the smaller ones at a minimum, filtered from the 95k total generated.
Yup, the model will still forget details sometimes. This is a common issue with prompt upsampling methods, but I'm hoping to improve this with the next version.
Thanks for the kind words! I started with the 780M param flan-t5-large model, and kept trying smaller and smaller base models - I was shocked at how good the output was at 77M. As you go smaller, though, it's much easier to accidentally overfit or collapse the model and produce gibberish. Had to be very careful with hyperparams and sanitizing / filtering the dataset.
As Invoke is open-source and already has transformers as a dependency, it should be pretty easy to add.
I haven't tested extensively with non SDXL based checkpoints but there's nothing really SDXL specific about the model; if you're using a fine-tune that's trained on booru-style tags, it will probably not work as well - but otherwise it should work just fine. And in that case, just fork the project and tune it on however your fine-tune prompts best :)
will fix these, thanks for the heads up!
Posted by
roborovskis on Thursday March 14, 2024 @12:35PM
from the brianfitzgerald.xyz dept.
Posted by
roborovskis on Thursday December 07, 2023 @01:10PM
from the stability.ai dept.
Posted by
roborovskis on Tuesday November 21, 2023 @02:01PM
from the stability.ai dept.
Posted by
roborovskis on Saturday July 31, 2021 @04:43PM
from the comma.ai dept.
Posted by
roborovskis on Wednesday July 25, 2018 @02:29PM
from the jalopnik.com dept.
The fact that they can see what plugins I have makes uTorrent wanting a browser plugin make total sense. Just thinking.