Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by berto99 on Thursday January 09, 2014 @11:49AM
(#7031488)
Wow, this looks really nice. One thing that isn't clear is how I get the information out. so for example, if I'm building a recommendation engine, is there some kind of api that my webapp can use to get the information (sorry, new to all this).
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by tristanz on Thursday January 09, 2014 @11:57AM
(#7031548)
Right now Sense is best for ad-hoc interactive analysis and batch/scheduled jobs. You can run long-running services that expose something like a REST endpoint, but we have plans to make exposing services much easier, so I'd probably hold off until we have an "official" solution.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by tansey on Thursday January 09, 2014 @11:43AM
(#7031451)
Looks great!
Do you have full support for numpy, scipy, matplotlib, pymc, and pandas?
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by tristanz on Thursday January 09, 2014 @12:03PM
(#7031595)
Yes. I'll also point our that Anand (apatil) was a core developer for PyMC. We have big plans around Bayesian computation on Sense.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by apatil on Thursday January 09, 2014 @11:51AM
(#7031509)
Thanks! Yes, we do, and you can install packages yourself and/or let us know if anything is missing.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by siganakis on Thursday January 09, 2014 @03:56PM
(#7033167)
Is it possible to move data around between languages / engines?
E.g. would it be possible to run a query against Red Shift (step 1), then cleanse it in Python (step 2), run R scripts over the Python output (step 3) then dump the results back to Red Shift (step 4)?
If I then decide I need to change the Red Shift query (step 1), can I then re-run the whole pipeline?
Munging data between different tools is what I seem to spend most of my time doing, so anything that helps that would be a big productivity boost.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by apatil on Thursday January 09, 2014 @06:46PM
(#7034062)
At the moment, not really. In your scenario, you could have a Python dashboard launch a Redshift dashboard with startup code that runs the initial query, then clean the result, then launch an R dashboard and pass the clean data to it either over the shared filesystem or a messaging system such as 0MQ or Redis, then save the results to S3 for consumption by Redshift.
You're probably looking for something smoother, though. We definitely intend to have a good solution for workflows like you describe in the future.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by ironchef on Thursday January 09, 2014 @06:54PM
(#7034091)
I think you may want to think about treating the analyses like an ETL pipeline (think dependent jobs in chronos or some such) using some intermediary (S3, whatevs). That would probably be useful to a lot of analysts.
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by pwang on Thursday January 09, 2014 @12:38PM
(#7031855)
Looks great! Is it really just you and Anand that put this together?
Re:Show HN: Sense - A New Cloud Platform for Data Science an...
by tristanz on Thursday January 09, 2014 @12:51PM
(#7031952)
Yup. It's just the two of us. So far. If anybody wants to join: [email protected].
Sense supports R, Python, JavaScript and SQL out of the box, but is fully extensible to new languages and tools:
https://github.com/SensePlatform/sense-engine
We have Julia, Hive, and Spark engines in development.