Sec.gov is blocking all requests from AWS?

Posted by atlasspring on Saturday September 25, 2021 @10:41PM
from the self-post dept.

I noticed sec.gov is blocking all requests from AWS since the 4 days ago. Has this been the same for you? How are you able to bypass this issue? Currently I can scrape sec.gov only from local computer. My ideal solution is that I can have a program that runs on some remote servers

( 2 comments | on HN )

Re:Sec.gov is blocking all requests from AWS? by talolard on Saturday October 02, 2021 @03:43PM (#28731401)

Add a user agent to the request

See here: https://www.sec.gov/os/accessing-edgar-data

[ Parent | on HN ]

Re:Sec.gov is blocking all requests from AWS? by arthurcolle on Saturday September 25, 2021 @11:24PM (#28658154)

You could use a proxy run on another provider, Linode, DigitalOcean, etc. Or you could scrape sec.report

What are you using SEC sites for, just EDGAR or something else?

[ Parent | on HN ]