Re:The scope of index variables in Python's for loops
by cnvogel on Sunday January 18, 2015 @04:00PM
(#8908861)
Obviously, when comparing different methods of computation with negligible cache/memory footprint, I'd run it on hot caches to maximize the visible effect on the implementation. File here was about 1.5 GB on a 4GB RAM machine, and I repeated each test to have a stable duration for each.
Would I have tested, say, different patterns of data access (reading files sequentially, parallel read from multiple disks/partitions), of course I would have made sure that either caches are cleared (/proc/sys/vm/drop_caches), datasets will exceed memory, and actual CPU load of the algorithm is according to the real-world-usage scenario (pure read() throughput is meaningless when CPU is bogged, and L1-3 caches swamped by actual algorithmic work).
Re: your additional tests: Interesting, what python-version and size of files are you running it on?
Re:The scope of index variables in Python's for loops
by acadien on Sunday January 18, 2015 @04:11PM
(#8908897)
I'm trapped using 2.7! On this machine it looks like I'm running 2.7.8. Files are roughly 1GB each. They're on a relatively slow file system, just a single HD on this machine... no raid, no ssd.
Edit; I just tested your code and my own and against a set of files of uniform size.
time wc -l GIANTFILE1 -> 16.7 seconds
cProfile.run("for i,v in enumerate(open("GIANTFILE2")):pass") -> 18.1 seconds
cProfile.run("sum(file_block_count_lf(open('GIANTFILE3','rb')))") -> 22.9 seconds