News for hackers. Stuff that matters.

A comment on Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words. Parent comment

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by iknowstuff on Thursday May 11, 2023 @02:01PM (#35905845)
Even a small startup, a researcher or a tinkerer can get a cloud instance with a beefy GPU. Also of note, Apple's M1 Max/Ultra should be be able to run it on their GPUs given their 64/128GB of memory, right? That's an order of magnitude cheaper.
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by mejutoco on Thursday May 11, 2023 @02:55PM (#35906644)
I am confused. Those amounts are ram, not gpu ram, aren‘t they? Macs cpus are impressive, but not for ml. A most realistic one for a consumer is a 4090 rtx 24 GB. A lot of models do not fit in that, so A6000 48GB and over for some professional cards. That might be around 9000€ already.
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by codedokode on Thursday May 11, 2023 @03:55PM (#35907589)
> Macs cpus are impressive, but not for ml

On Mac GPU has access to all memory.

[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by piperswe on Thursday May 11, 2023 @03:58PM (#35907614)
Apple Silicon has unified memory - all memory is accessible to both the CPU and GPU parts of the SoC.
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by karmasimida on Thursday May 11, 2023 @04:36PM (#35908041)
But they comes at max 32GB model?
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by mkl on Thursday May 11, 2023 @05:48PM (#35908849)
Mac Studio (desktop) is up to 128GB, and Macbook Pro is up to 96GB.
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by himlion on Thursday May 11, 2023 @02:55PM (#35906654)
I overlooked the unified memory on those machines. Can it really run this performantly?
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by ukuina on Friday May 12, 2023 @02:03AM (#35912515)
I run Vicuna quite well with my M1 Pro, 32GB.
[ Parent | on HN ]