News for hackers. Stuff that matters.
Re:GPU Open Analytics Initiative by aub3bhat on Friday May 12, 2017 @07:07PM (#14328630)
I am using Celery workers which run a single tensorflow model in solo concurrency mode. The current workaround is to keep track of per process gpu fraction and then incorporate information about Total GPU memory available when assigning workers/celery process to individual GPUs.
Re:GPU Open Analytics Initiative by jacquesm on Friday May 12, 2017 @08:02PM (#14328871)
Right, so essentially you are doing per process memory management because the layers lower down aren't smart enough to handle this transparently. I'm fairly sure that the nvidia libraries do support releasing memory and from what I see on my own machine memory usage of the GPU does go up and down when running longer jobs. So maybe it isn't so much a structural problem as a problem in one particular module that you exercise a lot?