News for hackers. Stuff that matters.
Re:Be good-argument-driven, not data-driven by ramesh31 on Thursday September 01, 2022 @10:17AM (#32676526)
>Would you say the big data threshold moves every year?

It moves with Moore's law. Big data is anything that cannot reasonably fit into memory for a single server, so yes that number is well over 1TB now.

Re:Be good-argument-driven, not data-driven by orangepurple on Thursday September 01, 2022 @11:00AM (#32677259)
I know this isn't the correct definition but I think of "big data" as the set of data which takes me more than 15 minutes to query on average with a moderately complex Postgres SQL join on well indexed information. I use JSONB in Postgres regularly and have indices on that too. So far I have gotten really far with increasing Postgres work_mem to a gig or more, a fast SSD, and strategically placed materialized views. These kinds of operations in Pandas make my computer billow smoke by comparison.