New in PostgreSQL 10

Re:New in PostgreSQL 10 by wjossey on Thursday September 21, 2017 @10:04PM (#15308898)

I’m not sure I agree with this statement. BigQuery is not in memory, nor is redshift (not super familiar with snowflake). Whereas memsql is and would be prohibitively expensive to operate on datasets that are analytics based, because it is in memory based. It’s my understanding that citus is best suited to analytics workloads and parallel queries, and really isn’t comparable to memsql as it is attempting to provide scale out support with disk, which memsql doesn’t natively prefer.

Can you share some docs with me around how citus is more OLTP based? Looking to be educated.

[ Parent | on HN ]

Re:New in PostgreSQL 10 by manigandham on Thursday September 21, 2017 @10:47PM (#15309080)

MemSQL has in-memory rowstores and disk-based columnstores. Every columnstore table has a rowstore to aid in fast data ingest. Rowstores are meant for access of individual rows, lots of data updates, and smaller reference tables while columnstores are for the bulk of data requiring large scans and analysis. We have tables with 20 billion rows that take up < 10gb of memory.

BigQuery streaming ingest also uses in-memory buffering into BigTable until it's written to disk, just like Cassandra with memtables flushed into on-disk SSTables.

Citus is more manual work than MemSQL and doesnt support columnstores native, although you can use it with their cstore extension to get both. It's just a different design giving you a postgres database that automatically shards vs MemSQL's many optimizations around HA, replication, and very fast analytics performance.

Try the features page: http://www.memsql.com/features/

[ Parent | on HN ]

Re:New in PostgreSQL 10 by wjossey on Friday September 22, 2017 @10:38AM (#15312562)

Thanks for the education on MemSQL. I wasn’t aware of the analytics use case (had never come up in discussions I was a part of for that tech), so it was news to me.

This article had a nice overview. Somewhat outdated but mostly still up to date. https://lnkd.in/g4anCf5

I’ll say it seems precarious to me to mix and match row store in memory data with a columnar store. Is it never a performance issue to have these two systems potentially in contention with one another for resources?

[ Parent | on HN ]

Re:New in PostgreSQL 10 by manigandham on Friday September 22, 2017 @01:25PM (#15314054)

It's just data in memory. Rowstores are read from memory and columnstores are read from disk. The architecture uses leaf nodes and aggregator nodes so aggregrator memory is used for assembling the final results of a query.

Version 6 is even faster, you can try out the beta: http://www1.memsql.com/beta-6.3-features.html

[ Parent | on HN ]

Re:New in PostgreSQL 10 by ddorian43 on Friday September 22, 2017 @09:47AM (#15312098)

Every BigQuery query, actually loads all the data (for that query) in-memory until it finishes.

[ Parent | on HN ]