News for hackers. Stuff that matters.
Re:Ask HN: What framework should I learn? by jononor on Sunday April 07, 2024 @06:34PM (#39964559)
The library which runs your ML model will do multithreading. Depending on model size, it may perform the best when passed multiple samples at a time to process. For this reason, it can be useful to implement "micro batching", which combines multiple samples from independent requests into batches that are processed together. In such a model one would have n_processes << n_cpus. Which tends to help with RAM usage also.

There are no comments on this comment yet.