News for hackers. Stuff that matters.

A comment on Does my data fit in RAM?. Parent comment

Re:Does my data fit in RAM? by manigandham on Thursday February 13, 2020 @03:30AM (#22316204)
You don't need to store everything into RAM to get fast results. Data warehouse relational databases are designed exactly for this kind of fast SQL analysis over extremely large datasets. They use a variety of techniques like vectorized processing on compressed columnar storage to get you quick results.

Google's BigQuery, AWS Redshift, Snowflake (are all hosted), or MemSQL, Clickhouse (to run yourself). Other options include Greenplum, Vertica, Actian, YellowBrick, or even GPU-powered systems like MapD, Kinetica, and Sqream.

I recommend BigQuery for no-ops hosted version or MemSQL if you want a local install.

[ Parent | on HN ]
Re:Does my data fit in RAM? by arnon on Thursday February 13, 2020 @03:32AM (#22316213)
Some of those aren't designed for data that doesn't fit in RAM. They even have it in their name (MemSQL for example).
[ Parent | on HN ]
Re:Does my data fit in RAM? by manigandham on Thursday February 13, 2020 @04:05AM (#22316333)
All of those support datasets that don't fit in RAM, or else they would be useless at data warehousing.

MemSQL uses rowstores in memory combined with columnstores on disk. Both can be joined together seamlessly and the latest release will automatically choose and transition the table type for you as data size and access patterns change.

[ Parent | on HN ]