Joining a billion rows 20x faster than Apache Spark

Re:Joining a billion rows 20x faster than Apache Spark by edw on Thursday February 09, 2017 @07:04PM (#13612208)

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their commitment to [wave hands again] all that stuff. The tragedy when this happens is that sometimes that small team of people is not aware what their function at the company is—being the human props for the sales and marketing teams. The alternative, I suppose, is far more depressing—that the "data scientists" are fully aware of their role in the grand scheme of things.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by emmanuel_1234 on Friday February 10, 2017 @12:11AM (#13613374)

Woah, that was exactly my previous position.

Upside: I had one of the most paying position among the technical people. I also got to play with expensive stuff.

Downside: it was soul crushing, I was delivering no value whatsoever and had a really hard time looking at my colleagues in the eye, as they were making a third of my salary (at best).

I got out, joined a new company with that in mind and now have a very exciting job. They do have a dedicated R&D, Data Science team which is a shit show: absolutely brilliant people completely wasted as they can't build anything for lack of programming/technology experience, in an environment where their theoretical skills are mostly useless. I'm genuinely sad for them.

[edit: one of the company still has a Hadoop/Spark cluster for their whooping 500Mb of data]

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by FractalNerve on Thursday February 09, 2017 @08:43PM (#13612645)

You really just described me. I've to do such a show-case project to complete my MSc Thesis for a minimum pay. And make all the proofs, so that the C-Suite guys can use during their presentations to sell what I made for a lots of money. (Even unaware if they'll charge their customers in the hundred thausands or millions range).

However, I'm really glad to find out about SnappyData.io, that's gonna save me a lot of time waiting. It would truly be my perfect dream, if they allowed running any programming language inside an environment like Jupyter.org or BeakerNotebook.com, but with Pandoc.org Markdown. So that I can essentially work fulltime programming, while I can also document it and also be able to export my documentation to a good looking latex thesis.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by sudsmenon on Friday February 10, 2017 @01:21AM (#13613618)

SnappyData has a Zeppellin interpreter and the code is open source. So if adding a interpreter for jupyter is something that can be easily added, I am sure someone from the community would find it a interesting project to undertake. Agree that it would be useful

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by wakkaflokka on Friday February 10, 2017 @09:57AM (#13615759)

This is so true. For what it's worth, there are probably a lot of people who would give anything to be a human prop earning a salary significantly above $100k (even far outside of SF). For someone who's actually interested in data science, this would be miserable.

[ Parent | on HN ]

Re:Joining a billion rows 20x faster than Apache Spark by dandermotj on Friday February 10, 2017 @06:10AM (#13614574)

This is excruciatingly accurate.

[ Parent | on HN ]