Running Awk in parallel to process 256M records

Re:Running Awk in parallel to process 256M records by co_dh on Tuesday June 02, 2020 @03:55PM (#23395593)

I like the idea of using AWK for this. But you can give kdb/q a try. 250M rows is nothing for kdb, and it seemed that you can afford the license.

[ Parent | on HN ]

Re:Running Awk in parallel to process 256M records by greggyb on Tuesday June 02, 2020 @05:13PM (#23396380)

kdb/q, or many other mature data platforms.

250M records isn't big for a lot of data platforms. I've seen plenty of solutions in traditional RDBMSs that would easily scale an order of magnitude larger than this, if not several.

Moving into anything analytically focused, like an OLAP engine, or just a columnstore in an RDBMS easily gives you another order of magnitude or few.

[ Parent | on HN ]

Re:Running Awk in parallel to process 256M records by lenkite on Tuesday June 02, 2020 @04:29PM (#23395949)

kdb/q is amazing. I have seen some experienced folks make jaw dropping data computations at the drop of a hat on gigabyte sized data. Makes everything else look laughably puny.

I suspect if it was open source, it would probably be the most popular big-data storage and computing platform.

[ Parent | on HN ]

Re:Running Awk in parallel to process 256M records by labelbias on Wednesday June 03, 2020 @04:49AM (#23401376)

Hmm... 256M records is still not that much for postgres or mysql so it shouldn't be too much for kdb/q. Several gb of data also shouldn't be too much.

[ Parent | on HN ]