adren123

I wonder how ClickHouse would compare against DuckDB given that the latter usually performs very well on OLAP benchmarks such as TPC-H

Wikidata.org the database behind Wikipedia is a good alternative as a general database including films and related crew for instance with Avatar (2009) https://www.wikidata.org/wiki/Q24871

Although is cannot be queried in SQL, but SPARQL instead https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service

Indeed, the 2013 Spike Jonze movie is the first idea that popped-up to my mind when I saw those videos amazing to see this movie 10 years after it was released in the light of those "futuristic" tools (AI assistant and such)

If you want to win on both counts (no buggy pipes and coloring), and if you're certain of the order, you can use the following:

    rg nokogiri.*linux

An initial import with DuckDB from all the 15 files takes only 36 seconds on a regular (6 years old) desktop computer with 32GB of RAM and 26 seconds (5 times quicker than QuestDB) on a Dell PowerEdge 450 with 20 cores Intel Xeon and 256GB of RAM.

Here is the command to input the files:

CREATE TABLE ecommerce_sample AS SELECT * from read_csv_auto('ecommerce_*.csv');

There is also git-bug [1], an offline bug tracker fully embedded in git which can bridge to Github, Gitlab, Jira

[1] https://github.com/MichaelMure/git-bug

I also use regularly miller/mlr with files having 1M+ lines and I never had problems with the hardcoded processing (i.e. "verbs") nor the DSL language which is know to be much less efficient compared to verbs.

Both visidata and miller are essential tools to process/view CSV/TSV files, way better than LibreOffice or Excel in terms of performance on large files.

What is also interesting is that while strace seems to report much less overhead in "./empty" compared to /bin/true

   $ strace -c ./empty 
   strace: exec: Erreur de format pour exec()
   % time     seconds  usecs/call     calls    errors syscall
   ------ ----------- ----------- --------- --------- ----------------
     0,00    0,000000           0         1         1 execve
   ------ ----------- ----------- --------- --------- ----------------
   100,00    0,000000           0         1         1 total


   $ strace -c /bin/true
   % time     seconds  usecs/call     calls    errors syscall
   ------ ----------- ----------- --------- --------- ----------------
    54,05    0,000060          15         4           mprotect
    24,32    0,000027          27         1           set_tid_address
    10,81    0,000012          12         1           set_robust_list
     8,11    0,000009           9         1           munmap
     2,70    0,000003           3         1           prlimit64
     0,00    0,000000           0         1           read
     0,00    0,000000           0         2           close
     0,00    0,000000           0         8           mmap
     0,00    0,000000           0         1           brk
     0,00    0,000000           0         4           pread64
     0,00    0,000000           0         1         1 access
     0,00    0,000000           0         1           execve
     0,00    0,000000           0         2         1 arch_prctl
     0,00    0,000000           0         2           openat
     0,00    0,000000           0         2           newfstatat
   ------ ----------- ----------- --------- --------- ----------------
   100,00    0,000111           3        32         2 total

the multiple execution seems to be much slower with the empty executable than with /bin/true using the multitime tool https://tratt.net/laurie/src/multitime/

   $ multitime -n 10 ./empty
   ===> multitime results
   1: ./empty
               Mean        Std.Dev.    Min         Median      Max
   real        0.006       0.000       0.005       0.006       0.006       
   user        0.001       0.001       0.000       0.002       0.002       
   sys         0.000       0.001       0.000       0.000       0.002       
   
   $ multitime -n 10 /bin/true
   ===> multitime results
   1: /bin/true
               Mean        Std.Dev.    Min         Median      Max
   real        0.002       0.000       0.001       0.002       0.003       
   user        0.001       0.000       0.001       0.001       0.001       
   sys         0.000       0.000       0.000       0.000       0.000

the full monty: alias c='echo -e "\033c" ; stty sane; setterm -reset; reset; tput reset; clear'

doesn't only clear the screen, but also escape any "broken" terminal configuration such as output of a binary on the tty