News for hackers. Stuff that matters.
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by ktamura on Sunday February 14, 2016 @02:19AM (#11097472)
Quite frankly, for data of this scale, PostgreSQL is more than adequate on a modern machine (although the OP's step-by-step guide is surely help for those new to Redshift)

For those interested in funny and insight analyses of the data, check out this blog: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta...

Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by JimmyAustin on Sunday February 14, 2016 @03:53AM (#11097625)
I'm not sure I would want to try analyzing a 267 GB database on most modern machine. Redshift, on the other hand, is optimized quite well for rapidly querying flat sets like this.

On the other hand, since Redshift is PostgreSQL 8.0.2 under the hood, any code you write for analysis should be easy to move over (with the exception of having to add dist and sort keys).

Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by manigandham on Sunday February 14, 2016 @06:01AM (#11097825)
> I'm not sure I would want to try analyzing a 267 GB database on most modern machine.

Why? Modern servers come with more RAM than that. Even multi-terabyte datasets are not a problem. Today's hardware is way better than most think.

Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by lwhalen on Sunday February 14, 2016 @08:17AM (#11098145)
Could not agree more. Production data needs to be bigger than a petabyte or three before my default response is anything other than 'let's load it into RAM.' :-)
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by sokoloff on Sunday February 14, 2016 @08:36AM (#11098208)
You mean terabyte, right?
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by lwhalen on Sunday February 14, 2016 @03:05PM (#11099739)
Well shucks, this is why I try not to HN at 3am on the regular... Yes, terabyte, not petabyte. And I'm well past the edit interval at this point; how embarrassing, mea culpa :-)
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by JimmyAustin on Sunday February 14, 2016 @02:51PM (#11099692)
I meant in the context of people playing at home who might be working off a laptop or a desktop with only 16-32 GB of ram, not someone with half a terabyte of RAM on a multi-socket server.
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by manigandham on Sunday February 14, 2016 @04:23PM (#11100075)
That's still fine, it will take longer but will still run without problems. A good SSD is recommended though.