Re:Web LLM – WebGPU Powered Inference of Large Language Models
by summarity on Saturday April 15, 2023 @06:23PM
(#35585202)
How big is the "runtime" part? My use case would basically be: run this in a native app that links against webgpu (wgpu or dawn). Is there a reference implementation for this runtime that one could study?
[ Parent
| on HN ]
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by crowwork on Saturday April 15, 2023 @07:35PM
(#35585733)
tvm runtime is pretty decent(~700k-2M level depending on dependency included), you can checkout tvm community and bring up the question there, i think there might be some common interest. There are impl of runtime for vulkan, metal that can be used as reference.