News for hackers. Stuff that matters.

A comment on Launch HN: Outerport (YC S24) – Instant hot-swapping for AI model weights.

Re:Launch HN: Outerport (YC S24) – Instant hot-swapping for ... by zackangelo on Thursday August 22, 2024 @10:31AM (#41320711)
Is this tied to a specific framework like pytorch or an inference server like vLLM?

Our inference stack is built using candle in Rust, how hard would it be to integrate?

[ Parent | on HN ]
Re:Launch HN: Outerport (YC S24) – Instant hot-swapping for ... by tovacinni on Thursday August 22, 2024 @12:30PM (#41321955)
We’d just need to write a Rust client for the daemon and load the weights in a way that is compatible with candle- we can definitely look into this since parts of what we are building is already in Rust!
[ Parent | on HN ]