News for hackers. Stuff that matters.

A comment on Mistral ships le chat – enterprise AI assistant that can run on prem. Parent comment

Re:Mistral ships le chat – enterprise AI assistant that can ... by aukejw on Thursday May 08, 2025 @06:04AM (#43924724)
How do you determine peak memory usage? Just look at activity monitor?

I've yet to find a good overview of how much memory each model needs for different context lengths (other than back of the envelope #weights * bits). LM Studio warns you if a model will likely not fit, but it's not very exact.

[ Parent | on HN ]
Re:Mistral ships le chat – enterprise AI assistant that can ... by simonw on Thursday May 08, 2025 @08:12AM (#43925366)
MLX reports peak memory usage at the end of the response. Otherwise I'll use Activity Monitor.
[ Parent | on HN ]
Re:Mistral ships le chat – enterprise AI assistant that can ... by aukejw on Thursday May 08, 2025 @08:46AM (#43925575)
I'm also trusting `get_peak_memory` + some small buffer for now.

Still, it reports accurate peak memory usage for tensors living on GPU, but seems to miss some of the non-Metal overhead, however small (https://github.com/aukejw/mlx_transformers_benchmark/issues/...).

[ Parent | on HN ]