Re:Ask HN: Premature optimization – when to use?
by nekopa on Tuesday August 11, 2015 @12:17PM
(#10042217)
Thanks for this, I am very new to python, so I will try this out.
But I think at the moment my bottleneck is the scraping part. The current program hits 365 pages (each day for a year) and managed to pull down the 20 million results.
I am now trying to figure out how to pull more info from each day, and that's why I am thinking about trying to optimize. Does cProfile measure things like time on a webpage (via bs4)?
Do you have any links on optimization "thinking", as in, I don't know if it is my approach that needs optimizing, vs my code.
But I think at the moment my bottleneck is the scraping part. The current program hits 365 pages (each day for a year) and managed to pull down the 20 million results.
I am now trying to figure out how to pull more info from each day, and that's why I am thinking about trying to optimize. Does cProfile measure things like time on a webpage (via bs4)?
Do you have any links on optimization "thinking", as in, I don't know if it is my approach that needs optimizing, vs my code.