Fine-Tuning VLMs for Data Extraction

Re:Fine-Tuning VLMs for Data Extraction by cvaman on Friday October 04, 2024 @08:39AM (#41740777)

Is it possible to host multiple fine-tuned VLMs on a single machine? like multiple models sharing the GPU(s) for inference?

[ Parent | on HN ]

Re:Fine-Tuning VLMs for Data Extraction by spikyspider on Friday October 04, 2024 @09:23AM (#41741118)

Yeah. If you have a large enough GPU you can use vanilla pytorch to load as many models as required. Docker is a good option if you have isolated services. Triton, TensorRT, TorchServe, RAY are also good services to checkout, especially when you want to load multiple adapters for the same LLM/VLM backbone. Is there anything specific you are looking to serve?

[ Parent | on HN ]