Apple introduces M4 chip

Re:Apple introduces M4 chip by threeseed on Tuesday May 07, 2024 @06:32PM (#40292304)

There really isn't enough emphasis on the downsides of server side platforms.

So many of these are only deployed in US and so if you're say in country Australia not only do you have all your traffic going to the US but it will be via slow and intermittent cellular connections.

It makes using services like LLMs unusably slow.

I miss the 90s and having applications and data reside locally.

[ Parent | on HN ]

Re:Apple introduces M4 chip by whimsicalism on Tuesday May 07, 2024 @07:20PM (#40292641)

Unusably slow? It's like 0.3 seconds to first token and then pretty much all of the tokens can follow within a second.

I find it hard to understand the edge usecase for text-based models.

[ Parent | on HN ]

Re:Apple introduces M4 chip by EarthMephit on Tuesday May 07, 2024 @10:11PM (#40293692)

Even in Australia is the LLM lag to a server noticable?

Generally an LLM seems to take about 3s or more to respond, and the network delay to the US is a couple of hundred milliseconds.

The network delay seems minimal compared to the actual delay of the LLM.

[ Parent | on HN ]