News for hackers. Stuff that matters.

A comment on Updating a 50 terabyte PostgreSQL database (2018). Parent comment

Re:Updating a 50 terabyte PostgreSQL database (2018) by pupdogg on Monday March 22, 2021 @08:56AM (#26540132)
Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format or data archiving capabilities available in 2021. It's OK! I was in the same boat until I needed to implement such a solution for my use case. What I'm suggesting does not limit their customers from searching any historical data. Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested solution. I strongly believe that mission critical transactional databases (postgres in this case) MUST be run very lean to keep their app running at hyper speeds at all times. 50TB overhead seems very inefficient when you take into account the the low cost solutions available in this day and age.

Based on my personal experience of achieving 94% compression on 2TB of data using snappy parquet file format, they could be looking at a final dataset size of 3.5TB on Clickhouse.

[ Parent | on HN ]
Re:Updating a 50 terabyte PostgreSQL database (2018) by dang on Monday March 22, 2021 @02:19PM (#26544601)
> Your comment clearly illustrates that you have no working knowledge

Please omit swipes like that from your comments here. The rest of your comment would be find without that bit.

https://news.ycombinator.com/newsguidelines.html

[ Parent | on HN ]
Re:Updating a 50 terabyte PostgreSQL database (2018) by KptMarchewa on Monday March 22, 2021 @09:09AM (#26540267)
>Matter of fact, it might be 100x to 1000x faster for them to do so with the suggested solution.

That must be trolling. 1000x faster than single digit millisecond indexed query retrieving 15 rows?

The fact that you keep talking about storage size means that you're talking about analytics not transactional needs.

>Your comment clearly illustrates that you have no working knowledge of Clickhouse or parquet file format or data archiving capabilities available in 2021. It's OK! I was in the same boat until I needed to implement such a solution for my use case.

Also, fuck off with that condescension.

[ Parent | on HN ]
Re:Updating a 50 terabyte PostgreSQL database (2018) by dang on Monday March 22, 2021 @02:20PM (#26544615)
> fuck off with that condescension

Please follow the site guidelines, regardless of whether or not someone else has broken them. Otherwise we just get a downward spiral.

https://news.ycombinator.com/newsguidelines.html

[ Parent | on HN ]
Re:Updating a 50 terabyte PostgreSQL database (2018) by KptMarchewa on Monday March 22, 2021 @07:58PM (#26549171)
Sorry, I get way too heated when I see this type of condescending argumentation.
[ Parent | on HN ]
Re:Updating a 50 terabyte PostgreSQL database (2018) by dang on Monday March 22, 2021 @09:26PM (#26549856)
Yes, I experience that too and I bet most people do. One way I often explain this that seems to work well is this:

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

[ Parent | on HN ]