News for hackers. Stuff that matters.

A comment on Show HN: How to analyse 100 GB of data on your laptop with Python. Parent comment

Re:Show HN: How to analyse 100 GB of data on your laptop wit... by drblah on Monday December 02, 2019 @10:37AM (#21683259)
I am currently doing an internship as part of my masters degree where I am analyzing ~30 GB of data. I'm using Postgres + Python and it is working quite well, even on my 2014 MacBook Air.

It would indeed be interesting to see how this approach with Vaex compares to Postgres. Though, I would be quite sad giving up SQL in favour of Pandas DataFrame indexing and Python looping :)

[ Parent | on HN ]
Re:Show HN: How to analyse 100 GB of data on your laptop wit... by maartenbreddels on Monday December 02, 2019 @10:44AM (#21683356)
No Python looping happening in Vaex :), otherwise, we wouldn't get this performance.

We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html

I think GraphQL is easier in combinations with front end development, and you can tab-complete your way out. Early days for this sub-project, but I think very promising.

[ Parent | on HN ]
Re:Show HN: How to analyse 100 GB of data on your laptop wit... by drblah on Monday December 02, 2019 @10:57AM (#21683508)
Ah, of cause. It makes sense you don't loop in Python.

This all seems pretty interesting. I will give it a go.

[ Parent | on HN ]
Re:Show HN: How to analyse 100 GB of data on your laptop wit... by mooneater on Monday December 02, 2019 @12:48PM (#21684747)
Ive been switching an reporting system that did analysis in postgres, to analysis in pandas (mostly business-stats type summaries).

It feels like growing wings and a jetpack. Almost everthing is waaay easier and faster.

[ Parent | on HN ]