tovacinni

We’d just need to write a Rust client for the daemon and load the weights in a way that is compatible with candle- we can definitely look into this since parts of what we are building is already in Rust!

The advantage of loading from a daemon over loading all the weights at once in Python is that it can support multiple processes or even the same process consecutively (if it dies or something, or had to switch to something else).

Loading from disk to VRAM can be super slow- so doing this every time you have a new process is wasteful. Instead, if you have a daemon process that keeps multiple model weights in pinned RAM, you can load them much quicker (~1.5 seconds for a 8B model like we show in the demo).

You _could_ also make a single mega router process, but then there are issues like all services needing to agree on dependency versioning. This has been a problem for me in the past (like LAVIS requiring a certain transformer version that was not compatible with some other diffusion libraries)

Awesome to hear- that sounds like an application we'd love to help with!

(Please feel free to reach out to us too at [email protected] !)

Yep, it'll work for multi-GPU as well!

Thanks!!

Maybe! Many people don't want to be vendor locked-in though and there are new GPU cloud providers gaining traction. Some still prefer on-prem.

We hope to make it easier to bridge the multi-cloud landscape by being independent and 'outer'.

That's a super neat idea- we should in fact be able to use this same system to support the orchestration of a 'system prompt caching' sort of thing (across deployments). I'll put this on my 'things to hack on' list :)

Variants do not have to be the same architecture- the demo (https://hotswap.outerport.com/) runs on a couple of different open source architectures.

That being said, there is some smart caching / hashing on layers such that if you do have models that are similar (i.e. a fine-tuned model where only some layers are fine-tuned), it'll minimize storage and transfer by reusing those weights.

Thanks! If you mean the costs for users of Outerport, it'll be a subscription model for our hosted registry (with a limit on storage / S3 egress) and a license model for self-hosting the registry. So mainly storage, since the idea is to also minimize egress costs which are associated with the compute tied up in it!

Thanks!

I think both are fits- we've gotten interest from both types of companies and our first customer is a "OSS model host".

Our 40% savings result is also specifically for the 5 model services case, so there could be non-trivial cost reduction even with a reasonably small number of models.

Render times are listed on table 3 of the paper- interactive, but perhaps not high-performance!

Here's a Twitter thread that explains what this paper is about: https://twitter.com/yongyuanxi/status/1354478763065528320?

It sort of can; something I observed while I was still a student at a "top CS university" is that there were certain individuals (superstars) in the school that a lot of people looked up to, and whichever companies they chose to do internships at or work full-time at, everyone else would want to go there. This happened even if the company was a mostly no-name startup.

So purely from a recruiting standpoint; if this sort of phenomenon happens at an undergraduate level, I can believe that just having John Carmack on your team alone can bring more talent in, for potentially cheaper just because of increased supply.

Sorry if this is a stupid question, but can't you do this without converting the image into an RGB 3D space? (i.e. iterate through the pixels and count the ones within a certain range of what you want)

It's hackernews clickbait for sure :p

Worked for me - just searched "retail jobs" and came up with results from around my area

Does anyone actually think that HR is on their side? :p

On the other hand, Kiva Systems which is now Amazon Robotics is apparently now a terrible place to work. :(

Your argument sounds vaguely similar to the one that Jonathan Hurst (the founder of OSU's DRL/Agility Robotics) likes to make- he insists that Japanese robots focus on looking good while American robots focus on being practical.

I think this is an oversimplified way of looking things.

ASIMO isn't the only robot that Japan has made. The market for Six Axis Robot Arms, the industrial robots that are used in factories everywhere, is dominated by Yaskawa Motoman and Fanuc, both Japanese companies.

Meanwhile, the robots you cite (e.g. Cassie) are far from actually being practical- sure they're awesome and generate a lot of hyped up headlines, but they won't have much traction on the market for a long time.

I think it really depends on the atomicity of the said "tasks" and how often branches get merged back into the master branch. It's very easy for branches to get abused and merging long-standing branches in with the master branch is always expensive task that's bound to have scary issues.

In an organized enough team with a large enough project, tasks should be able to be carried through concurrently on a single branch for the most part.

Correct me if I'm wrong, but for VCS, isn't it generally bad practice to unnecessarily branch off and introduce complex structures?

In my experience, branches generally lead to people feeling like they have free reign and introducing a slew of issues (reduced code quality, difficulties in merging, broken builds, etc)

Really curious how the sound quality optimizations based on room acoustics work.

I'd imagine it's a lot of signal processing and such, but I'm curious how they quantified optimizations and tested to see that it actually enhanced the audio based on room acoustics.

I don't know why this makes you so mad, but the title definitely helped me be interested in the article and I ended up learning a few things about myself through the article.