Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e...
It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :)
*
Here's a simple easy code that is incredibly slow in Python:
interesting = set(line.strip() for line in open('interesting.txt'))
total = 0
for line in open('data.txt'):
id,val = line.split('\t')
if id in interesting:
total += int(val)
interesting = set(line.strip() for line in open('interesting.txt'))
total=0
for c in chunks: # im lazy to actually write it
df = pd.read_csv('data.txt', sep='\t', skiprows=c.start, nrows=c.length, names=['id','val'])
total += df['val'][df['id'].isin(interesting)].sum()
Reading in chunks is not bad (and you can just use `chunksize=...` as a parameter to `read_csv`), but pandas `read_csv` is not so efficient either. Furthemore, even replacing `isin` with something like `df['id'].map(interesting.__contains__)` still is pretty slow.
Btw, deleting `interesting` (when it goes out of scope) might take hours(!) and there is no way around that. That's a bona fides performance bug.
In my experience, disk IO (even when using network disks) is not the bottleneck for the above example.
There's also code in genetic_distance() function that IIUC is meant to handle the case when sample1 and sample2 are not similarly-indexed, however (a) you essentially never use it, since you only pass sample1 and sample2 that are columns of the same dataframe (what's the point then?), and (b) your code would actually throw an exception if you tried doing that.
P.S. I like the part where you've removed the comment "note that this is a slow computation" :)
scikit-allel: http://scikit-allel.readthedocs.io/en/latest/index.html
scikit-allel example: http://alimanfoo.github.io/2015/09/21/estimating-fst.html
with open('interesting.txt') as interesting_file:
interesting = {line.strip() for line in interesting_file}
with open('data.txt') in data_file:
total = sum(int(val) for id, val in map(lambda line: line.split('\t'), data_file) if id in interesting)Also, if the data you're reading is numeric only - or at least non-unicode / character data - you might be able to get a speed boost reading the data as binary not as python text strings.
Given his code you referenced, could you elaborate on what makes it look slow at a glance, and how you might speed it up? :)
if snp_taxID not in samples_of_interest.keys():#Check if Genome is of interest
Numba does not support dictionaries and has limited support for pandas dataframes (only underlying arrays, when convertible to NumPy buffers, if I understand correctly). This limits usefulness for many non-array situations, as well as some existing code-bases (the dictionary is fundamental in Python and typically used everywhere -- often for performance).
"Despite the examples and docs, Numba is voodoo. Damned cool voodoo, but still voodoo"
If you provide me with a self-contained code example (with data required to run it) that is “too slow”, I’d be willing to try and optimise it to support my point above.
Also, have you tried Numba? It maybe a matter of just applying a “@jit” decorator and restructuring your code a bit in which case it may get magically boosted a few hundred times in speed.
[1] https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post....