News for hackers. Stuff that matters.

A comment on Lesser-Known Python Data Analysis Libraries.

Re:Lesser-Known Python Data Analysis Libraries by dguaraglia on Tuesday April 19, 2016 @10:32AM (#11527217)
My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Elastic MapReduce than MRJob itself.)

I love the concept and ease of development, but I can't shake the feeling that the infrastructure is so shaky it almost amount to instant technical debt (sorry if this offends anyone, I'm just a dumb customer.)

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by __derek__ on Tuesday April 19, 2016 @12:57PM (#11528313)
It looks like mrjob development has been re-started, but there was a disconcerting period (nearly two years) without a release.[1] I used it for rinky-dink projects, and it seemed fragile at the time, so I can understand your inclination to divest from it.

[1]: https://github.com/Yelp/mrjob/releases

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by stevejohnson on Tuesday April 19, 2016 @01:52PM (#11528776)
In case anyone's curious, what happened was that Dave (@davidmarin) and I (@irskep), the mrjob maintainers, left Yelp within about a month of each other. (There's no story there, just coincidence.) There was never any momentum with new maintainers, going by the release history.

But now Dave is working on mrjob regularly again, hence the pace of recent improvements.

Grandparent is correct about the second-class support for non-EMR production Hadoop usage. Like any open source project, the code only works well if a major stakeholder invests in improving it. Few non-EMR users spend much time contributing, so the situation doesn't improve.

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by dguaraglia on Tuesday April 19, 2016 @01:55PM (#11528790)
Hey guys, for what its worth, MRJob has given us around 3 years of working (if sometimes clunky) EMR, so thanks for that :)
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by gdulli on Tuesday April 19, 2016 @03:29PM (#11529505)
I have the opposite experience with MrJob. Classifying it as an inactive project is demonstrably false. The rest are EMR complaints, I use it on my own Hadoop cluster.
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by dguaraglia on Friday April 22, 2016 @02:01PM (#11551107)
Just read the comment from one of the creators: https://news.ycombinator.com/item?id=11528776
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by zfrenchee on Tuesday April 19, 2016 @10:36AM (#11527247)
Do you know of any good alternatives? Any way to write MapReduces in python?
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by ymt123 on Tuesday April 19, 2016 @11:13AM (#11527537)
It's not quite the same (since it doesn't become a Map-Reduce job) but if you're mostly interested in the programming paradigm/scalability the Python API for Apache Spark might be a good alternative
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by tanlermin on Tuesday April 19, 2016 @12:17PM (#11527992)
Yes! Check out dask: http://www.slideshare.net/continuumio

Its free with a permissive license.

It is also capable of native HDFS integration, Yarn etc and can do more complex and granular parallel patterns than just map reduce. Also has a API for distributed dataframes and arrays with linear algebra ops.

DISCLAIMER: I don't work for continuum. I just want to see its projects succeed because I was a user will benefit.

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by _dark_matter_ on Tuesday April 19, 2016 @10:39AM (#11527274)
https://hadoop.apache.org/docs/r1.2.1/streaming.html
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by pvnick on Tuesday April 19, 2016 @11:23AM (#11527599)
This is likely the best answer for those who wish to code within the map/reduce paradigm by hand and would prefer to use python.
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by pwang on Wednesday April 20, 2016 @11:12PM (#11539401)
BUT WHY

Your performance is going to be complete and utter crap because you're paying for serialization on every single data element.

Dask is higher performance and more pythonic: http://matthewrocklin.com/blog/work/2016/02/22/dask-distribu...

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by hamilyon2 on Tuesday April 19, 2016 @11:43AM (#11527743)
Luigi does decent job. It is relatively easy to start with and powerful enough to do almost anything
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by rch on Tuesday April 19, 2016 @12:24PM (#11528045)
I've been using Luigi for a few months, with no complaints. It supports running Python jobs on Hadoop and Spark, but it's not really a MapReduce framework unto itself.

However http://discoproject.org/ might be worth a look as a standalone alternative.

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by sitkack on Tuesday April 19, 2016 @12:36PM (#11528139)
I have used Disco extensively in the past, nothing but good things to say about it. Fast job launch, easy to write, the DFS has been stellar. This was only using Python for job code.
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by dguaraglia on Tuesday April 19, 2016 @10:53AM (#11527389)
Unfortunately, no. We are slowly moving away to a streaming infrastructure, so I've been mostly trying to "keep it running" until we are done replacing it. Sorry.
[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by tanlermin on Tuesday April 19, 2016 @12:17PM (#11527994)
Check out dask: http://www.slideshare.net/continuumio

Its free with a permissive license and actively growing.

It is also capable of native HDFS integration, Yarn etc and can do more complex and granular parallel patterns than just map reduce. Also has a API for distributed dataframes and arrays with linear algebra ops.

DISCLAIMER: I don't work for continuum. I just want to see its projects succeed because I was a user will benefit.

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by gshulegaard on Tuesday April 19, 2016 @12:29PM (#11528084)
Andrew Montalenti did a great talk about scaling out Python at Parsely at the last PyData conference: https://www.youtube.com/watch?v=gVBLF0ohcrE

But TBH, after a certain scale you should really be asking whether or not you should be using Python.

[ Parent | on HN ]
Re:Lesser-Known Python Data Analysis Libraries by mring33621 on Tuesday April 19, 2016 @12:07PM (#11527922)
Apache Flink? http://www.kdnuggets.com/2015/11/getting-started-python-apac...
[ Parent | on HN ]