zhwu

Yes, SkyPilot Platform built the scheduler on top of Kueue. There is some shortcoming of Kueue's TAS that you have to resolve, e.g., you cannot do TAS with a Elastic job, which makes the hot swapping not possible for large-scale jobs on NVL72 racks: https://kueue.sigs.k8s.io/docs/concepts/elastic_workload/#li...

"No Topology-Aware Scheduling (TAS) support."

There is another blog talking about this coming soon.

This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps.

Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"

RL is not just an AI engineer problem but also a problem for AI infra: many components that can lands on different places with special requirements on scheduling during launching and recovering.

We put this blog to dive into the infra challenges with some good visual. Would love to hear people’s thought on it!

It seems that cursor now supports your own sandbox

The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.

Cloud services, such as autoscaling EKS or AWS Batch are mostly limited by the GPU availability in a single region. That limits the scalability of jobs that can run distributedly in a large scale.

AI batch inference is one of the examples, and this post found that by going beyond a single region, it is possible to speed up the important embedding generation workload by 9x, because of the available GPUs in the "forgotten" regions.

This can significantly increase the iteration speed for building applications, such as RAG, and AI search. We share our experience for launching a large amount of batch inference jobs across the globe with the OSS project SkyPilot.

TL;DR: it speeds up the embedding generation on Amazon review dataset with 30M items by 9x and reduces the cost by 61%.

This recent blog actually looks into the case with multiple writers and the distribution for the time for a writer to take the lock: https://blog.skypilot.co/abusing-sqlite-to-handle-concurrenc...

Dealing with all the Kubernetes pod configs / deployments is too much for an AI engineer. Being able to focus on the real model work would be super important.

The finetuning can tailor the model to have more customized knowledge, just like the identity knowledge of itself shown in the blog post. If you ask the original llama model, it should know nothing about SkyPilot or Vicuña, as it is trained on old knowledge from the internet.

However, finetuning still cannot get rid of the hallucination problem that all the chatbot suffers from. It depends on how accurate you expect the chatbot should be. The retrieval might be considered more accurate, as it will not make up solutions, but just return irrelevant answer in the worst case.

Great reference!

Just want to add about hosting your own LLM vs using ChatGPT. Cost is definitely a thing to consider, but it also depends on whether it is ok to share the requests to your product with OpenAI.

Also, something you cannot do with ChatGPT is to custom it with your own data, such as internal documents, etc. As shown in the blog, the model trained by ourselves can easily know its identity.

It is the underlying operational guide of the latest release of Vicuna-1.5: https://twitter.com/lmsysorg/status/1686794639469371393

This is cool! The Llama 2-70B can be hosted in my own cloud environment.

It seems training the Vicuna on custom dataset could be quite easy as well, according to the following: https://github.com/skypilot-org/skypilot/tree/master/llm/vic...

Very interesting! Quite surprised to see PaLM-2 ranked even lower than open-sourced Vicuna.

SkyPilot is actually the tool that helps you find the resources on any cloud, including AWS, GCP, Azure, IBM (comming soon) or even Lambda Clouds. It can automatically search for the spot instances across all the regions and clouds, based on the availability and prices.

You need to use the transformers from the main branch instead of the pypi version, because the llama support is recently added. According to the readme of the repo, you need to install transformers with: pip3 install git+https://github.com/huggingface/transformers

Wow, that is very interesting. Would you mind sharing the prompt you used to query the model?

Yes, you need to convert the original LLaMA model to the huggingface format, according to https://github.com/lm-sys/FastChat#vicuna-weights and https://huggingface.co/docs/transformers/main/model_doc/llam...

If you follow this command in their instruction, the delta will be automatically downloaded and applied to the base model. https://github.com/lm-sys/FastChat#vicuna-13b: `python3 -m fastchat.model.apply_delta --base /path/to/llama-13b --target /output/path/to/vicuna-13b --delta lmsys/vicuna-13b-delta-v0`

It is mainly because of the legal issues caused by the license of llama model weights. We need to figure it out with Meta's llama team before releasing.