News for hackers. Stuff that matters.

A comment on Web Scraping 101 with Python. Parent comment

Re:Web Scraping 101 with Python by craigmi on Wednesday February 10, 2021 @06:38PM (#26096343)
"I send a command at a random time between 11pm and 4am to wake up an ec2 instance."

Any chance you could tell me your setup for this?

[ Parent | on HN ]
Re:Web Scraping 101 with Python by rinze on Wednesday February 10, 2021 @07:48PM (#26096866)
Not my project, but if I had to do it I'd try something like the following:

* Set an autoscaling group with your instance template, max instances 1, min instances 0, desired instances 0 (nothing is running).

* Set up a Lambda function that sets the autoscaling group desired instances to 1.

* Link that function to an API Gateway call, give it an auth key, etc.

* From any machine you have, set up your cron with a random sleep and a curl call to the API.

And that should do the trick, I think.

[ Parent | on HN ]
Re:Web Scraping 101 with Python by ses1984 on Wednesday February 10, 2021 @11:02PM (#26098125)
>From any machine you have, set up your cron with a random sleep and a curl call to the API.

You might as well just call the ASG API directly.

[ Parent | on HN ]
Re:Web Scraping 101 with Python by warsheep on Thursday February 11, 2021 @01:37AM (#26098888)
Why use autoscaling and not just launch the instance directly from lambda? The run time is short so there's no danger of two instances running in parallel
[ Parent | on HN ]