It's an important point to remember that a lot of things involved in human society have not exploded in size or complexity in the last 30 years.

Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has moved from the millions into the billions.

It's important, because if a government needs to, say, calculate something involving "every building in the country", or "everybody with a criminal record" they need to understand that this task, in 2019, can in fact be done by a single programmer parsing flat text files on their MBP, and does not need a new department.

This is a bit like Grace Hopper always pointing out the difference between a microsecond and a nanosecond - https://www.youtube.com/watch?v=JEpsKnWZrJ8

Awesome point well put.

Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it.

Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.

I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.

This doesn't follow:

1) One reason to run a cluster is to use flaky commodity hardware instead of high-reliability specialized hardware. Note the transition from specialized hardware and computer rooms of the '80s and '90s to cloud computing on preemptible instances.

2) They might just be straight-up wrong / misguided. http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... As a professional developer I've seen tons of distributed systems that could be replaced by a well-designed non-distributed system, but ELK and Mongo and etcd and Kafka and more generally cloud servers are easy off-the-shelf tools.

You can get a lot of milage out of parallel xargs and other of the shelf tools.

If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.

80% of bioinformatics.

This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."

Or they don’t. Not all data are cloud-scale aggregations.

Also most people don't deal with terabyte sized data sets

Isn't it logical to have historical summary info and then the full log for say the past month?

What if you want to change what's contained in your summary?