News for hackers. Stuff that matters.
Re:Four MLs (and a Python) by jkldotio on Thursday April 30, 2015 @03:43AM (#9463767)
I wanted to have a look at how much PyPy might speed up the task so I generated a one million line x 20, 287M, big-data.csv of my own with the following.

    import random
    with open("big-data.csv","w") as f:
        for b in range(1000000):
            f.write(','.join([str(random.random()) for x in range(20)])+"\n")  
Each line looks like: "0.47509825737,0.525866136528,0.167956183215...0.888687040645".

The script from the blog takes 7.3 seconds to chew through it on my machine with Python 2.7.8, and 4.9-5.5 seconds with PyPy, a reasonable improvement. Out of interest, taking all the boilerplate and checking off, the following is 7 lines versus the 24 lines or so in the blog and runs in a similar 4.9-5.4 second range on PyPy (although performance blows out under Python 2.7.8 to 13 seconds or ~6 seconds slower than the blog). Somewhat more Pythonic, and more likely do be done live in the REPL without even bothering to write a script, to my mind.

    from collections import Counter
    c = Counter()
    lines = (rawline.strip().split(",") for rawline in open("big-data.csv"))
    for line in lines:
        for colname, colval in enumerate(line):
            c[colname]+=float(colval)
    print c  
Above on PyPy.

    time pypy sum5.py ./big-data.csv

    real	0m4.972s
    user	0m4.946s
    sys	        0m0.024s  

wc on the same machine.

    time wc big-data.csv  

    real	0m3.701s
    user	0m3.672s
    sys	        0m0.028s
Bonus one-liner (6.6 seconds under PyPy).

    print map(sum,zip(*([float(x) for x in rawline.strip().split(",")] for rawline in open("big-data.csv"))))
Re:Four MLs (and a Python) by dagw on Thursday April 30, 2015 @04:55AM (#9463970)
Bonus bonus easy to read one-liner (using numpy):

   numpy.loadtxt("big-data.csv",delimiter=',').sum(axis=0)
Re:Four MLs (and a Python) by nine_k on Thursday April 30, 2015 @01:15PM (#9466321)
This is a nice data-processing trick, but obviously runs very little Python code.
Re:Four MLs (and a Python) by dagw on Thursday April 30, 2015 @04:35PM (#9467479)
Well over 99% of the run time is spent in loadtxt which is pure python. In fact this code is slower than the original code due to the overhead of the checks that loadtxt does.