Anyway thanks for the response.
I'm curious to know, based on your experience in production, how much does Chain-of-Though Reasoning typically cost in terms of tokens for frameworks like LlamaIndex, LangChain, CrewAI, etc.?I understand it depends on many different factors including the complexity of the product and the architecture of the agents involved, but I'd love to hear about your experiences in creating...
Hi everyone,I'm planning to build an AI workflow using open-source tools like models, vector databases, and frameworks (e.g., crewAI, langchain, llamaindex) without doing any training or fine-tuning. Can anyone share the average computational cost to go from an idea to a finished product using cloud-based services? I'm specifically interested in the average total cost for developing...
I know about FHE and TFHE, but as you said, they need a lot of computational resources.
Ignoring the training process and just looking at inference, what are the "technical" drawbacks of this idea? If the first and last layers of the network run on the local machine, how could a malicious cloud reverse engineer the inputs and discover the raw data?
Hello,One of the biggest challenges with cloud-based inferencing for LLMs is keeping user data private. Is it possible to use both local and cloud machines together to solve this?For example, could we run the first and last layers of an LLM on a local machine to protect the data privacy and use the cloud for the rest to speed things up? We could fine-tune the first and last layers locally to...
Hello all, I am very curious to know how feasible it is to use a cluster of Versal HBM Series (e.g. VHK158 evaluation board) for training an LLM like LLaMA-2 70B in terms of 'Performance/Power/Cost.' Are there any papers regarding the comparison of a cluster of VHK158 evaluation boards and a cluster of, say, A100s?Thanks.
Hello all,I am very curious to know how feasible it is to use a cluster of Versal HBM Series (e.g. VHK158 evaluation board) for training an LLM like LLaMA-2 70B in terms of 'Performance/Power/Cost.' Are there any papers regarding the comparison of a cluster of VHK158 evaluation boards and a cluster of, say, A100s?Thanks.
Hi guys,I saw lots of people clustered their RPis for different purposes (e.g. data mining, machine learning, big data storage and Hadoop and so on ...). As you know RPi has it's own drawbacks. For example lack of enough RAM or high speed IO busses (e.g. USB 3 or GbE). So I was thinking about a brand new board with these kind of capabilities at the same price and size. However as a trade off we...