News for hackers. Stuff that matters.
Re:Ray: A Distributed Framework for Emerging AI Applications by vvladymyrov on Sunday July 04, 2021 @02:43PM (#27731713)
Are you running inference on CPU or GPU?
Re:Ray: A Distributed Framework for Emerging AI Applications by ramoz on Monday July 05, 2021 @11:06AM (#27738712)
CPU, GPU doesn't work out well in our case. There is the data transfer cost as well as memory constraint & the model blows up in memory for every inference call.
Re:Ray: A Distributed Framework for Emerging AI Applications by ramoz on Monday July 05, 2021 @11:10AM (#27738754)
x2 Gunicorn workers, MKL mapped to half physical cores for each... for some reason the model (tensorflow) performs better on half vs 1 worker