Modern Data Practice and the SQL Tradition

Re:Modern Data Practice and the SQL Tradition by gigatexal on Friday November 08, 2019 @10:34AM (#21483107)

SQL is indeed a powerful too. But it's just one of many. Just today I used: SQL to dump data to csv, python to modify it, and gRPCurl to upload it to another service. Different tools, different problems, use the right one.

That being said ... RDBMs still suffer from scale and there are many efforts to fix this with sharding to disaggregating the data layer from the query layer with examples like AWS's Aurora and GCP's BigQuery etc -- in all likelihood data will likely live in many different places and will need to be adapted into things like a datastore or queried from things like Hive and Presto, or even a traditional ETL + EDW setup -- my point being there's no one thing that will solve everything as most single solutions break down when the number of rows is measured in the billions or more.

[ Parent | on HN ]

Re:Modern Data Practice and the SQL Tradition by nomel on Friday November 08, 2019 @12:33PM (#21484560)

> SQL to dump data

Is this another way to say you use SQL to query specific data from a larger dataset?

[ Parent | on HN ]

Re:Modern Data Practice and the SQL Tradition by chrisjc on Friday November 08, 2019 @01:33PM (#21485241)

I imagine they mean SQL to unload from whatever SQL storage engine out to blob(s3,azure,etc), HDFS or local-file system.

COPY INTO s3://mybucket/foobar/ from (select a, b, c from ...);

[ Parent | on HN ]

Re:Modern Data Practice and the SQL Tradition by gigatexal on Friday November 08, 2019 @01:28PM (#21485190)

Yeah sorry. I used the cli front end to my database from a container and had it dump the results to a file that I then massaged with python

[ Parent | on HN ]