Ask HN: Premature optimization – when to use?

Re:Ask HN: Premature optimization – when to use? by aburan28 on Tuesday August 11, 2015 @11:09AM (#10041602)

Find your bottleneck and see what function calls are the slowest/fastest and find out the root cause of any perceived lack of performance. Just run python -m cProfile <yourscript.py> and you'll get that information

[ Parent | on HN ]

Re:Ask HN: Premature optimization – when to use? by nekopa on Tuesday August 11, 2015 @12:17PM (#10042217)

Thanks for this, I am very new to python, so I will try this out.

But I think at the moment my bottleneck is the scraping part. The current program hits 365 pages (each day for a year) and managed to pull down the 20 million results.

I am now trying to figure out how to pull more info from each day, and that's why I am thinking about trying to optimize. Does cProfile measure things like time on a webpage (via bs4)?

Do you have any links on optimization "thinking", as in, I don't know if it is my approach that needs optimizing, vs my code.

[ Parent | on HN ]