Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on ParaText: CSV parsing at 2.5 GB per second. Parent comment
Loading a 156mb csv file in kdb 32-bit free version, single thread:
\t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0:`t.csv 1850
>>> timeit.timeit('paratext.load_csv_to_dict("t.csv",num_threads=4)', setup="import paratext", number=1) 3.1176819801330566
Loading a 1.5GB csv file in kdb:
\t quote:`sym`time`ex`bid`bsize`ask`asize`mode!("STCHXHXC";",")0:`q.csv 14135
>>> timeit.timeit('paratext.load_csv_to_dict("q.csv",num_threads=4)', setup="import paratext", number=1) 12.962939977645874
However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.
There are no comments on this comment yet.
Loading a 156mb csv file in kdb 32-bit free version, single thread:
in paratext, 64-bit: No improvement.Loading a 1.5GB csv file in kdb:
in paratext: Not too shabby! An almost 10% improvement over KDB by turning my fans on and burning my lap!However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.