News for hackers. Stuff that matters.
Re:DeepMind moves to TensorFlow by eoinmurray92 on Friday April 29, 2016 @01:07PM (#11597100)
TensorFlow is the machine learning codebase, but typically how do machine learning research teams manage their training sets, dataset metadata and collaboration on these large datasets?
Re:DeepMind moves to TensorFlow by cosmoharrigan on Friday April 29, 2016 @05:49PM (#11598996)
Here are a couple of examples from the Python deep learning ecosystem:

IDSIA, affiliated with Juergen Schmidhuber and many other leading ML researchers, has released Sacred, "a tool to help you configure, organize, log and reproduce experiments." https://github.com/IDSIA/sacred

MILA, affiliated with Yoshua Bengio and the Theano project, offers fuel, "a data pipeline framework for machine learning": https://github.com/mila-udem/fuel

Re:DeepMind moves to TensorFlow by barneso on Friday April 29, 2016 @02:20PM (#11597651)
Most teams I have seen have either template scripts or boilerplate that generates datasets, and share both the generated data and the scripts via normal ways that people share data and code: disk, S3, github, emailing of notebooks, etc.

It requires a fair amount of set-up, but works surprisingly well once there is a core team and problems established.

We are building mldb.ai to help bring the data and the algorithms for ML together in a less ad-hoc manner and to help move things out of research and into prod once they are ready. Many of the hosted ML solutions (Azure ML, Amazon ML, Google Data Lab, etc) and other toolkits (eg Graphlab) are working on similar ML workflow and organizational structure problems.

Re:DeepMind moves to TensorFlow by sysreader2016 on Friday April 29, 2016 @04:46PM (#11598637)
Which projects you know use "disk, S3, github,..." to share their datasets? I'm curious what you think because I haven't read about any ML projects actually using hosted ML solutions like Amazon ML+S3. I've only seen Amazon recommend Amazon ML.
Re:DeepMind moves to TensorFlow by HappyTypist on Saturday April 30, 2016 @12:19AM (#11600332)
S3 is a good way to share files
Re:DeepMind moves to TensorFlow by flatfilefan on Saturday April 30, 2016 @03:59PM (#11603309)
Re:DeepMind moves to TensorFlow by sysreader2016 on Friday April 29, 2016 @06:13PM (#11599101)
I'm working toward moving my deep learning project DeepSentry into e-commerce. DeepSentry was developed in Python3 and C. After any significant amount of time 60+ days the dataset per host is TB-PB. So, I've been digging into implementing a Big Data solution. I've found that Apache Spark on Hadoop-MapR with Postgres is popular.