News for hackers. Stuff that matters.

A comment on High performance models in TensorFlow.

Re:High performance models in TensorFlow by nyamhap on Thursday May 04, 2017 @10:40AM (#14265146)
My bottleneck is still speed of reading data from json. I wonder whether I should wait for features to be built out here or go down the path of writing a custom data reader in C++
[ Parent | on HN ]
Re:High performance models in TensorFlow by jacksnipe on Thursday May 04, 2017 @10:43AM (#14265170)
If your data has a little extra structure that isn't shared by JSON in general, you could probably get serious performance gains by rolling your own.
[ Parent | on HN ]
Re:High performance models in TensorFlow by sirfz on Thursday May 04, 2017 @03:19PM (#14267835)
Utilizing multiprocessing for reading and processing jsons (or any type of data) then feeding the output into a shuffle_batch* op works great for me.
[ Parent | on HN ]
Re:High performance models in TensorFlow by jamesblonde on Thursday May 04, 2017 @10:50AM (#14265215)
You could use Tensorflow-on-Spark to read your JSON into a RDD in Spark. Then the Tf-RDD-Reader will be in-memory and can feed your training.
[ Parent | on HN ]