News for hackers. Stuff that matters.
Re:Updating a 50 terabyte PostgreSQL database (2018) by manigandham on Monday March 22, 2021 @07:00AM (#26539144)
All modern columnnar/analytical databases have no problem in pulling back older data; in fact, that's the point of running large scale analytics.
Re:Updating a 50 terabyte PostgreSQL database (2018) by KptMarchewa on Monday March 22, 2021 @09:08AM (#26540253)
I agree, but analytics is orthogonal topic.

Unless you want to use columnar database to retrieve 15 transaction that particular user did month ago...

Re:Updating a 50 terabyte PostgreSQL database (2018) by manigandham on Monday March 22, 2021 @09:16AM (#26540334)
There is nothing wrong with that use case. It works just fine. It may not have the same latency as an OLTP database but there are many factors (indexing, sort/partition keys, compression, etc) that can return similar performance for those queries.
Re:Updating a 50 terabyte PostgreSQL database (2018) by KptMarchewa on Monday March 22, 2021 @09:21AM (#26540393)
Just to clarify what you advocate for: you want to run this type of queries on analytical databases (like Snowflake, Redshift etc) to display 10 rows on frontend device?

    select * from transactions where user_id='asdf' and completed_date <'2021-02-01' and completed_date >= '2021-01-01'
Re:Updating a 50 terabyte PostgreSQL database (2018) by manigandham on Monday March 22, 2021 @09:25AM (#26540439)
Sure. Again the database and data layout make a difference. Snowflake has startup latency since the "warehouse" might not be running but Redshift will do fine, along with many others that are always on.
Re:Updating a 50 terabyte PostgreSQL database (2018) by darksaints on Monday March 22, 2021 @09:56AM (#26540809)
Redshift has some pretty significant and fairly unpredictable per-query overhead. For example, all queries are compiled, then code is distributed to every node in the cluster, then executed and gathered before returning. And most analytics databases use a scheduled execution model, which can delay execution from a couple seconds to several minutes, depending on database loads.