Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by d23 on Monday April 10, 2023 @02:14PM
(#35515408)
Yeah I'm having trouble understanding why this isn't something that's already accomplishable using SD + ControlNet. I guess Segment Anything lets you not need to manually segment an image? The example images don't seem to be particularly better than what one could generate without segmenting using autogenerated canny or scribble or depth controlnet images to me though.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by ulrikhansen54 on Tuesday April 11, 2023 @07:39AM
(#35523647)
Segment Anything is orders of magnitude more powerful than anything we've ever seen before within the real of image segmentation and the first 'foundation model' in that regard, so this is a real step-change in background removal/editing, etc. -- hell, we might not even need green screens anymore!
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by jonplackett on Monday April 10, 2023 @07:03PM
(#35518471)
Segmenting really well, like how a human would do it has been for some reason really really hard for a computer to do. I guess because it requires understanding of what objects actually are. So that part is amazing that it works so well.
And then the control net part seems more magical just because you have a REALLY accurate input
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by forgingahead on Monday April 10, 2023 @04:09AM
(#35510189)
Nice work! Did you figure out the training/fine-tuning code for Segment Anything? There's an on-going discussion on the official repo[0] but no clear results yet.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by jack_riminton on Monday April 10, 2023 @04:26AM
(#35510271)
I wonder if Segment Anything can work on things that aren't 'real looking' e.g. if I created a semi abstract piece in the style of Matisse on MidJourney, would it be able to segment the wiggly vase that doesn't look like a vase?
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by runnerup on Monday April 10, 2023 @08:00AM
(#35511323)
It works fine for me on very abstract stuff. Seems quite well generalized. Just wish they might release the CLIP encoder for it so you can prompt it with text prompts.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by okamiueru on Monday April 10, 2023 @05:39AM
(#35510611)
The readme.md has some image examples. If I interpret them correctly, the first one is the input image, the second the segmented output, and the rest are example outputs using the prompt text shown above the collage.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by GaggiX on Monday April 10, 2023 @08:50AM
(#35511644)
It's a controlnet model trained on SAM segmentation maps. The final model takes a prompt and a segmentation map as input and generates an image conditioned on them.
Controlnet is a neural network added to an already trained model so they can be conditioned on new stuff like canny edge, depth map, segmentation map. Controlnet let you train this model on the new condition "easily", without catastrophic forgetting and without a huge dataset. In the repo linked by OP, they have trained a controlnet model on the segmentation map generated by SAM: https://segment-anything.com/
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by nonbirithm on Monday April 10, 2023 @08:51AM
(#35511652)
Sounds like a good idea. Some recent Stable Diffusion toolkits like ComfyUI are modular so in theory you'd be able to build a pipeline like this inside the application instead of being constrained to one workflow.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by fkarg on Monday April 10, 2023 @12:29PM
(#35514136)
they described it in their paper: they created it themselves. they had/have a loop of having humans annotate images their system was uncertain about, but the released one was fully automated.
Re:EditAnything: Segment Anything + ControlNet + BLIP2 + Sta...
by dgellow on Monday April 10, 2023 @11:51AM
(#35513751)
Great, that was obviously coming! Stable diffusion with controlnet was already so good, with segment anything we start to really have a fantastic experience! Really cool, I will definitely play with this during the week, thanks for sharing
And then the control net part seems more magical just because you have a REALLY accurate input
[0]https://github.com/facebookresearch/segment-anything/issues/...
But I will try it out with label studio and this alone with a classic training workflow should speed up the process tremendously anyway.
https://www.artsy.net/article/artsy-editorial-guide-painting...
Controlnet is a neural network added to an already trained model so they can be conditioned on new stuff like canny edge, depth map, segmentation map. Controlnet let you train this model on the new condition "easily", without catastrophic forgetting and without a huge dataset. In the repo linked by OP, they have trained a controlnet model on the segmentation map generated by SAM: https://segment-anything.com/
: excluding non-public ones