Hi folks,Is there any public information on how Apple Intelligence optimizes the KV cache? For instance, in their cloud PCC, they claim to use stateless computing, where the context is sent to the cloud each time. Given that the context can be lengthy and the KV cache quite large, what magic might Apple use to handle this?
Hi folks,Is there any mature/widely-accepted autonomous LLM system today? I found "GoEx" from Berkeley and is there anything else that the community are working on? Thanks!
Hi folks,I'm going to set up a local LLM, e.g., Llama2 for our team (10 people), but I'm not quite sure about the process. I'm planning to run the model on single/multiple H100s and which is the best tool/framework to handle multi-users at the same time? I did a bit research into vLLM but still I wish I could hear from the experts who did this local LLM thing before. Our team members have the...
Another thing is that I'm a little bit confused about all kinds of "LLMs". I looked into hugging face but there are so many different models and they may have variant size or structure. Which model can be a good start? I guess gpt4 is too large to handle in a lab environment (sorry I don't really know much about LLM :).
Hi folks, AI newbie here,How to run LLM-based experiments in lab environment? I'm a PhD student focusing on system security, and recently we are trying to catch the trend and find some security issues in LLM training. However, it seems tricky to me on how to set up the environment or simulations. We can buy few A100/H800, is there a standard way to get started?