Ask HN: Does anyone have experience scraping dynamic content?
http://www.nfl.com/scores
For example, for this URL, there is a section on the right for "Big Play Highlights" and the first one listed is "L.Brown 7-yard TD pass from..."
It looks like this data is being loaded via some kind of AJAX call. Do you have any ideas on how I can scrape this stream of highlights data? I've never tried to scrape any dynamic content in the past.
Ideas?
(
4 comments
| 3 points
| on HN
)
I have used Selenium to scrape dynamic content before by waiting for new DOM elements to be populated by AJAX, so I know this sort of thing is possible in a way you couldn't do with wget.
just make your own tstamp for random