Ask HN: Privacy-Aware Inferencing for LLMs

Re:Ask HN: Privacy-Aware Inferencing for LLMs by koutetsu on Saturday June 08, 2024 @06:41AM (#40616674)

I had a similar idea some time ago but didn't implement due to its complexity and the need to juggle different parameters. On top of that, I don't think there are any guarantees when it comes to privacy. Your users will have to trust that no mistake will be made in handling the raw data or the preprocessed data and that no malicious actor will be able to access the original weights.

You should instead try looking into Homomorphic Encryption:

https://huggingface.co/blog/encrypted-llm

It is resource intensive and slower but it serves your purpose better, in my opinion.

[ Parent | on HN ]

Re:Ask HN: Privacy-Aware Inferencing for LLMs by manili on Saturday June 08, 2024 @06:57AM (#40616753)

Thanks @koutetsu,

I know about FHE and TFHE, but as you said, they need a lot of computational resources.

Ignoring the training process and just looking at inference, what are the "technical" drawbacks of this idea? If the first and last layers of the network run on the local machine, how could a malicious cloud reverse engineer the inputs and discover the raw data?

[ Parent | on HN ]