Re:Web LLM – WebGPU Powered Inference of Large Language Models
by brucethemoose2 on Saturday April 15, 2023 @03:14PM
(#35583663)
> For example, the latest MacBook Pro can have more than 60G+ unified GPU RAM that can be used to store the model weights and a reasonably powerful GPU to run many workloads.
...for $3.5K minimum, according to the Apple website :/
Is there any chance WebGPU could utilize the matrix instructions shipping on newer/future IGPs? I think MLIR can do this through Vulkan, which is how SHARK is so fast in Stable Diffusion on the AMD 7900 series, but I know nothing about webgpu's restrictions or Apache TVM.
Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by summarity on Saturday April 15, 2023 @04:06PM
(#35584065)
It's also interesting that this opens up the full saturation of Apple Silicon (minus the ANE): GGML can run on the CPU, using NEON and AMX, while another instance could run via Metal on the GPU using MLC/dawn. Though the two couldn't share (the same) memory at the moment.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by brucethemoose2 on Saturday April 15, 2023 @04:21PM
(#35584179)
The GPU's ML task energy is so much lower that you'd probably get better performance running everything on the GPU.
I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by brucethemoose2 on Saturday April 15, 2023 @04:19PM
(#35584162)
Technically any newish laptop with 64GB of RAM has 64GB of "VRAM," but right now the Apple M series and AMD 7000 series are the only IGPs with any significant ML power.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by sroussey on Saturday April 15, 2023 @05:13PM
(#35584577)
I’m not sure what you mean. Typically, an iGPU slices off part of RAM for the GPU at boot time, which means it’s fixed and not shared. When did this change?
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by lights0123 on Saturday April 15, 2023 @08:21PM
(#35585999)
For Intel, it seems that per their chart under "What is the maximum amount of graphics memory or video memory my computer can use?" and discussion under "Will adding more physical memory increase my graphics memory amount?" at https://www.intel.com/content/www/us/en/support/articles/000..., iGPUs included with 5th gen/Broadwell processors were their first to do so in 2014.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by m1el on Saturday April 15, 2023 @04:21PM
(#35584178)
If you're doing inference on neural networks, each weight has to be read at least once per token. This means you're going to read at least the size of the entire model, per token, at least once during inference.
If your model is 60GB, and you're reading it from the hard drive, then your bare minimum time of inference per token will be limited by your hard drive read throughput. Macbooks have ~4GB/s sequential read speed. Which means your inference time per token will be strictly more than 15 seconds.
If your model is in RAM, then (according to Apple's advertising) your memory speed is 400GB/s, which is 100x your hard drive speed, and just the memory throughput will not be as much of a bottleneck here.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by ratg13 on Saturday April 15, 2023 @06:23PM
(#35585207)
There will be LLM specific chips coming to market soon which will be specialized to the task.
Tesla already has already been creating AI chips for their FSD features in their vehicles. Over the next years, everyone will be racing to be the first to put out LLM specific chips, with AI specific hardware devices following.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by dan_mctree on Sunday April 16, 2023 @01:19AM
(#35587473)
What exactly is the ideal sort of hardware to be able to run and train large models? Do you basically just need a high end version of basically everything?
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Saturday April 15, 2023 @04:41PM
(#35584334)
To clarify, running this WebLLM demo doesn't need a 3.5k MacBook Pro which costs $3.5k :-)
WebGPU supports multiple backends, besides Metal on Apple Silicon, it offloads to Vulkan, DirectX, etc. It means a windows laptop with Vulkan support should work. My 2019 Intel MacBook with AMDGPU works as well.
Our model is int4 quantized, and it is 4G in size, so it doesn't need 64GB memory either.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Saturday April 15, 2023 @05:59PM
(#35585008)
This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-(
We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Saturday April 15, 2023 @09:26PM
(#35586404)
In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Sunday April 16, 2023 @03:13AM
(#35587952)
Would be nice if anyone could help us benchmark! Our primary focus though is not model performance, but to demonstrate the capability that TVM Unity generates code targeting WebGPU and allows them to run with client GPUs :-)
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Saturday April 15, 2023 @09:22PM
(#35586378)
To share some fun stuff, here is the response generated by this model: As an AI language model, I would respond by acknowledging that the model discussed in the message is indeed smaller than some of the larger language models like GPT-3/4. However, I would also point out that size is not always a direct indicator of performance, and that the model may still be able to perform well on certain tasks or in certain contexts. Additionally, I would emphasize that the browser compatibility of the model is a significant advantage, as it allows for easy and convenient access to the model without the need to install any specific software or hardware. Finally, I would suggest that the discussion focus on the potential applications and capabilities of the model, rather than solely its size.
Back to the topic, we don't make much innovation on the model, so I am probably not the best person to evaluate how a model compares with SOTAs. There are indeed lots of super cool techniques being explored lately that makes it possible to deploy smaller and smaller models, for example, LLM.int8() [1] and int4 quantization [2] without loss of zero-shot accuracy. Can't predict the future, but maybe one day there will be something really powerful but small enough to fit in the pocket of everyone :-)
[1] Dettmers, Tim, et al. "LLM. int8 (): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022).
[2] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit Inference Scaling Laws." arXiv preprint arXiv:2212.09720 (2022).
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by junrushao1994 on Sunday April 16, 2023 @03:30AM
(#35588018)
Their model says:
Dear [Name],
Thank you for your message. We understand that the model you are referring to is a simple and basic model. However, it is important to highlight that this model serves a specific purpose and can be useful for certain applications.
Regarding the comparison with larger models like GPT, it's important to note that different models have different strengths and weaknesses, and the choice of model depends on the specific task and use case. While larger models like GPT may be more powerful and capable, they also come with greater computational and memory requirements.
We appreciate your concerns and feedback, and we will take them into consideration as we continue to develop our models. Our aim is to provide solutions that are tailored to the needs of our clients and meet their requirements for accuracy, efficiency, and performance.
Thank you for your input, and we hope to have the opportunity to work with you in the future.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by visarga on Sunday April 16, 2023 @02:39PM
(#35592970)
Tricks like Speculative Sampling show we can use small models to do useful work for large models, or use large models as correcting devices for small models. So I see a mixed future - both small and large models - one with low latency and fast, the other slow and sparsely called, working together to achieve the qualities of both. Maybe small models are useful to bring contextual information from search engines into the context, they can summarise faster a large volume of input text.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by jimmySixDOF on Saturday April 15, 2023 @11:33PM
(#35587038)
I'm also not sure what the real use case is here other than an intellectual experiment. I may be missing the point but as I understand the world, running a Local LLM has nothing to do with the web in any way while it is doing so and so what's the benefit of WebGPU ? If you are streaming some constantly changing live data 3D experience in real time served over the internet by a website you just clicked on then WebGPU is making life easier. If you have a local pile of training data and weights sitting there on your hard drive for serious crunching then its like tying your shoelaces together just because developers want to have write once run everywhere dx which is nice for them but value sould always favor the user. Happy for someone to change my mind.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by slimsag on Saturday April 15, 2023 @05:53PM
(#35584934)
Nice work! I knew it wouldn't be long before someone put this together :)
I'm curious if given a different language (like Zig) with WebGPU access if you could easily translate that last-mile of code to execute there or not? In specific I wonder if I can do it, and if you can give me an overview of where the code for "Universal deployment" in your diagram actually lives?
I found llm_chat.js, but it seems that doesn't include the logic necessary for building WGSL shaders? Am I wrong or does that happen elsewhere like in the TVM runtime? How much is baked into llm_chat.wasm, and where is the source for that?
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by crowwork on Saturday April 15, 2023 @06:15PM
(#35585140)
The WGSL are generated and compiled through TVM and embedded into the wasm.
I think what you mean is wgpu native support. At the moment the web gpu runtime dispatches to the js webgpu environment. Once TVM runtime comes with wgpu native support (like the current ones in vulkan or metal), then it is possible to leverage any wgpu native runtime like what Zig provide.
Additionally, currently tvm natively support targets like vulkan, metal directly which allows targeting these other platforms
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by slimsag on Saturday April 15, 2023 @06:17PM
(#35585154)
OK that makes sense; so basically if I want to give this a shot then I would just need to read llm_chat.js and the TVM docs, and translate llvm_chat.js to my language of choice effectively?
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by crowwork on Saturday April 15, 2023 @06:19PM
(#35585176)
I think instead what would be needed is a wgpu native runtime support for TVM.
Like the implementations in tvm vulkan, then it will be naturally link to any runtime that provides webgpu.h
Then yah the llm_chat.js would be high-level logic that targets the tvm runtime, and can be implemented in any language that tvm runtime support(that includes, js, java, c++ rust etc).
Support webgpu native is an interesting direction. Feel free to open a thread in tvm discuss forum and perhaps there would be fun things to collaborate in OSS
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by summarity on Saturday April 15, 2023 @06:23PM
(#35585202)
How big is the "runtime" part? My use case would basically be: run this in a native app that links against webgpu (wgpu or dawn). Is there a reference implementation for this runtime that one could study?
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by crowwork on Saturday April 15, 2023 @07:35PM
(#35585733)
tvm runtime is pretty decent(~700k-2M level depending on dependency included), you can checkout tvm community and bring up the question there, i think there might be some common interest. There are impl of runtime for vulkan, metal that can be used as reference.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by bhouston on Saturday April 15, 2023 @05:22PM
(#35584654)
It is funny how we have WebNN API coming, but it is so slow to arrive that we are misusing graphics APIs to do NN again. Anyone have a clue when WebNN API will arrive? It would be significantly more power efficient than using WebGPU/WebGL.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by paulgb on Saturday April 15, 2023 @08:32PM
(#35586065)
I wouldn’t consider this to be “misusing” a graphics API in the same sense that tensor operations on top of WebGL are. WebGPU intentionally provides compute shaders with the goal of enabling non-graphical pipelines.
> [WebGPU use cases include] Executing machine learning models efficiently on the GPU. It is possible to do general-purpose GPU (GPGPU) computation in WebGL, but it is sub-optimal and much more difficult.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by rektide on Saturday April 15, 2023 @10:13PM
(#35586638)
I'm more worried that it'll end up kind of inferior to like, Google's other work, OpenXLA & iree, which seem to be the cross-target ML toolkit.
https://news.ycombinator.com/item?id=35078410
I really want this to be good. It's hard to really trust the web spec is going to live up to what the other really really good web ML work is. But it'll at least unlock some good speed up, whenever it lands. I genuinely don't mind more delay, if it helps get things into a better position for long term wins.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by archerx on Sunday April 16, 2023 @03:28AM
(#35588010)
This is really cool and I had no idea about it. I tried the live style transfer on live video demo and was amazed, even the with gpu polyfill I was getting 5fps on my underpowered laptop. It brought me back to the days of deepdream and deepstyle but live and on the web.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by dwheeler on Saturday April 15, 2023 @06:23PM
(#35585201)
> Thanks to the open-source efforts like LLaMA ...
Is the LLaMA model really open source now? Last I checked it was only licensed for non-commercial use, which isn't open source software at least. Have they changed the license? Are people depending on "databases can't be copyrighted"? Are people just presuming they won't be caught?
There's lots of OSS that can use LLaMA but that's different from the model itself.
This is a genuine question, people are making assertions but I can't find evidence for the assertions.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by dwheeler on Sunday April 16, 2023 @03:50PM
(#35593666)
Ok, all evidence I can find is that the LLaMA model is not open source, and the article is just wrong. Software != model. If someone has evidence otherwise, please post.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by tantony on Sunday April 16, 2023 @04:00AM
(#35588123)
The speed at which this stuff is developing makes it very hard to keep up and contribute in a meaningful way. Especially when you have a separate full time job.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by Hyzer on Saturday April 15, 2023 @08:15PM
(#35585962)
Are there any other projects/libraries that can run Llama models on Apple Silicon GPU? This is the first one I've seen.
Comparing it to llama.cpp on my M1 Max 32GB, it seems at least as fast just by eyeballing it. Not sure if the inference speed numbers can be compared directly.
vicuna-7b-v0 on Chrome Canary with the disable-robustness flag: encoding: 74.4460 tokens/sec, decoding: 18.0679 tokens/sec = 10.8ms per token
llama.cpp:
$ ./main -m models/7B/ggml-model-q4_0-ggjt.bin -t 8 --ignore-eos = 45 ms per token
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by omneity on Monday April 17, 2023 @02:00PM
(#35604228)
Yes and it's definitely a step in the right direction. It's beyond me whether this is enough of an effort to support the current LLM movement.
I expect Apple to be more proactive in these regards to capture the minds of a lightning-fast growing market increasingly slipping through its control (ChatGPT can be accessed from anywhere) and offer more incentives to developers, co-marketing being the lowest starting point.. probably something they're already working on and we didn't see the results of yet.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by firatsarlar on Sunday April 16, 2023 @12:07AM
(#35587185)
Is this chasing impossible - not criticize, and love effort- ?
But is it -a little- really possible to run an LLM in a single machine ?
I want to believe :)
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by ranguna on Sunday April 16, 2023 @04:35AM
(#35588253)
I don't get where your question is coming from, you can already run LLMs on a single machine. Checkout llama.cpp, tabby, text generation webui, gpt4all, AI Dungeon open source models like clover-edition, and know this we gpu based app.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by firatsarlar on Monday April 17, 2023 @05:16AM
(#35598407)
The question comes from a kind of confusion. We know the requirements of LLMs. How can we run the hardware it is currently working on, only the big LLM, with an 11Gb graphics card? I really didn't mind!
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by Tepix on Sunday April 16, 2023 @02:56PM
(#35593167)
GTPQ has been the missing piece, it allows quantizing the model weights from 16 to 4 bits with only a small loss in quality.
That it turn allows running even the large 65 billion parameter version of the LLaMA model in ~33GB of RAM or VRAM.
With VRAM that requires two 24GB GPUs which is no longer completely out of reach.
The model running in the browser is a smaller version with 7 billion parameters, which is good enough for some things.
Re:Web LLM – WebGPU Powered Inference of Large Language Models
by quickthrower2 on Sunday April 16, 2023 @08:34AM
(#35589467)
Works well on my Dell XPS13 which is not particularly powerful but is modern. Answers are slower than chatGPT but not painfully slow and the response quality is great.
...for $3.5K minimum, according to the Apple website :/
Is there any chance WebGPU could utilize the matrix instructions shipping on newer/future IGPs? I think MLIR can do this through Vulkan, which is how SHARK is so fast in Stable Diffusion on the AMD 7900 series, but I know nothing about webgpu's restrictions or Apache TVM.
Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.
But yeah, AMD/Nvidia are never going offer huge memory pools affordably on dGPUs.
I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
I have been wanting to get a beefier Mac Studio/mini m2 the more
I’m seeing Apple Silicon specific tweaked packages.
But 64G of VRAM is not the same as GPU mem, apples and oranges
AMDs IGP are way less attractive because they use rather slow DDR4/5 memory while the M2 has blazing fast memory integrated in the package.
We're talking about 50 GB/s vs 400 GB/s. Nvidia's A100 has 1000 GB/s.
Memory bandwidth is usually the bottleneck in GPU performance as many kernels are memory-bound (look up the roofline performance model).
The Pro/Max have double/quadruple that bus width. But they are much bigger/more expensive chips.
I can find very little about it (or other 7000 series integrated gpus). Is this usable at all for running LLaMa in some way?
This is the only thing I can think of, not everyone will have the latest high end GPUs to run such software..
The comment to which you replied was asking about the need for a GPU, not the need for a lot of RAM.
I guess it will take hardware and software a while to catch up to compete with ChatGPT..
Tesla already has already been creating AI chips for their FSD features in their vehicles. Over the next years, everyone will be racing to be the first to put out LLM specific chips, with AI specific hardware devices following.
WebGPU supports multiple backends, besides Metal on Apple Silicon, it offloads to Vulkan, DirectX, etc. It means a windows laptop with Vulkan support should work. My 2019 Intel MacBook with AMDGPU works as well.
Our model is int4 quantized, and it is 4G in size, so it doesn't need 64GB memory either.
We didn't do much evaluation because there isn't much innovation on model side, but instead we are demoing the possibility of running an end-to-end model on ordinary client GPUs via WebGPU without server resources.
Compare using the loss function?
* BoolQ
* PIQA
* SIQA
* HellaSwag
etc...
Any model you can trivially load in your browser will be significantly smaller than those models, and broadly speaking smaller = worse.
This example is a 4 GB model, that’s (I guess) based off some smallish model like the llama 7B.
It’s a proof of concept, not a chat-gpt replacement.
There’s nothing here that’s new other than “runs in the browser”; so it won’t be better than any other model you can get your hands on.
This kind of thing should be label ByoM (bring your own model). The model isn’t the interesting part of this.
Back to the topic, we don't make much innovation on the model, so I am probably not the best person to evaluate how a model compares with SOTAs. There are indeed lots of super cool techniques being explored lately that makes it possible to deploy smaller and smaller models, for example, LLM.int8() [1] and int4 quantization [2] without loss of zero-shot accuracy. Can't predict the future, but maybe one day there will be something really powerful but small enough to fit in the pocket of everyone :-)
[1] Dettmers, Tim, et al. "LLM. int8 (): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022).
[2] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit Inference Scaling Laws." arXiv preprint arXiv:2212.09720 (2022).
There you go, summarised for you.
Dear [Name],
Thank you for your message. We understand that the model you are referring to is a simple and basic model. However, it is important to highlight that this model serves a specific purpose and can be useful for certain applications.
Regarding the comparison with larger models like GPT, it's important to note that different models have different strengths and weaknesses, and the choice of model depends on the specific task and use case. While larger models like GPT may be more powerful and capable, they also come with greater computational and memory requirements.
We appreciate your concerns and feedback, and we will take them into consideration as we continue to develop our models. Our aim is to provide solutions that are tailored to the needs of our clients and meet their requirements for accuracy, efficiency, and performance.
Thank you for your input, and we hope to have the opportunity to work with you in the future.
Best regards, [Your Name]
> I would also point out that size is not always a direct indicator of performance, and
Yes. It is.
This thread is a direct response to a comparison to GPT, and your response (generated or not) is dishonest.
I can’t be blunter than that.
If you want amortise your responsibility by posting generated responses, go for it. Do whatever you want.
My response is directly to the parent comment about the comparison to GPT, for anyone who is unclear about the comparison.
I'm curious if given a different language (like Zig) with WebGPU access if you could easily translate that last-mile of code to execute there or not? In specific I wonder if I can do it, and if you can give me an overview of where the code for "Universal deployment" in your diagram actually lives?
I found llm_chat.js, but it seems that doesn't include the logic necessary for building WGSL shaders? Am I wrong or does that happen elsewhere like in the TVM runtime? How much is baked into llm_chat.wasm, and where is the source for that?
I think what you mean is wgpu native support. At the moment the web gpu runtime dispatches to the js webgpu environment. Once TVM runtime comes with wgpu native support (like the current ones in vulkan or metal), then it is possible to leverage any wgpu native runtime like what Zig provide.
Additionally, currently tvm natively support targets like vulkan, metal directly which allows targeting these other platforms
Then yah the llm_chat.js would be high-level logic that targets the tvm runtime, and can be implemented in any language that tvm runtime support(that includes, js, java, c++ rust etc).
Support webgpu native is an interesting direction. Feel free to open a thread in tvm discuss forum and perhaps there would be fun things to collaborate in OSS
https://www.w3.org/TR/webnn/
Seems to be in development over at Chrome: https://chromestatus.com/feature/5738583487938560
> [WebGPU use cases include] Executing machine learning models efficiently on the GPU. It is possible to do general-purpose GPU (GPGPU) computation in WebGL, but it is sub-optimal and much more difficult.
https://gpuweb.github.io/gpuweb/explainer/
I really want this to be good. It's hard to really trust the web spec is going to live up to what the other really really good web ML work is. But it'll at least unlock some good speed up, whenever it lands. I genuinely don't mind more delay, if it helps get things into a better position for long term wins.
Is the LLaMA model really open source now? Last I checked it was only licensed for non-commercial use, which isn't open source software at least. Have they changed the license? Are people depending on "databases can't be copyrighted"? Are people just presuming they won't be caught?
There's lots of OSS that can use LLaMA but that's different from the model itself.
This is a genuine question, people are making assertions but I can't find evidence for the assertions.
https://news.ycombinator.com/item?id=35344787
https://github.com/facebookresearch/llama links to
https://forms.gle/jk851eBVbX1m5TAv5 which contains LLaMA license agreement below the form.
Ie. Here’s a diff. Go find your own copy of the llama model and apply this patch.
…and hoping that’s good enough to get away with; even though you can’t really argue it’s not distributing a derivative work in part.
Distribution of the actual model (eg. Running it in your browser) seems to usually result in “and then Meta slaps you with a takedown notice”.
Comparing it to llama.cpp on my M1 Max 32GB, it seems at least as fast just by eyeballing it. Not sure if the inference speed numbers can be compared directly.
vicuna-7b-v0 on Chrome Canary with the disable-robustness flag: encoding: 74.4460 tokens/sec, decoding: 18.0679 tokens/sec = 10.8ms per token
llama.cpp: $ ./main -m models/7B/ggml-model-q4_0-ggjt.bin -t 8 --ignore-eos = 45 ms per token
https://github.com/apple/ml-ane-transformers
I expect Apple to be more proactive in these regards to capture the minds of a lightning-fast growing market increasingly slipping through its control (ChatGPT can be accessed from anywhere) and offer more incentives to developers, co-marketing being the lowest starting point.. probably something they're already working on and we didn't see the results of yet.
With VRAM that requires two 24GB GPUs which is no longer completely out of reach.
The model running in the browser is a smaller version with 7 billion parameters, which is good enough for some things.