Analyzing 1.1B NYC Taxi and Uber Trips

Re:Analyzing 1.1B NYC Taxi and Uber Trips by thro1237 on Tuesday November 17, 2015 @03:30PM (#10583741)

Was the 300GB of data processed on MacBook Air? How as the response time for queries?

[ Parent | on HN ]

Re:Analyzing 1.1B NYC Taxi and Uber Trips by lil_tee on Tuesday November 17, 2015 @03:44PM (#10583817)

Yep, all on a 2012 MacBook Air. Data size was over 400GB with indexes

Simple queries on indexed columns of the trips table take a minute or two, more complicated queries that require a full sequence scan can take up to a few hours

[ Parent | on HN ]

Re:Analyzing 1.1B NYC Taxi and Uber Trips by ufmace on Tuesday November 17, 2015 @05:16PM (#10584397)

All on internal storage? What was the max internal storage then, 512GB? Sounds like that's pretty tight, barely possible if you aren't doing anything else that takes up much space.

I wonder if it would be easier to stick the Postgres on a server, maybe AWS or local, and just do the queries from the laptop. Or maybe on a Tmux on the server, so you can let a long query run without having to keep the laptop up.

[ Parent | on HN ]

Re:Analyzing 1.1B NYC Taxi and Uber Trips by lil_tee on Tuesday November 17, 2015 @06:25PM (#10584775)

Yes, the database is all on the machine's local 512 GB hard drive. I did store the downloaded flat text files to an external drive and loaded them into the db from there.

[ Parent | on HN ]