If you want to build stuff like this in future, check Page.REST a simple API I built. You can pass a query string `contains=TEXT` to check the existence of a word or a phrase.

Here's a Postman collection of all possible options with the API https://app.getpostman.com/run-collection/620bbe44b6ecc6c2e3...

You can maybe add a glitch repo to make it easier to get started.

Thanks! I looked at Glitch but didn't quite get how I can make a shareable sample.

https://visualping.io/ can do this, although it sends email alerts rather than SMS messages.

I used it to notify me of new Hololens events, listed on this page https://www.microsoftevents.com/profile/web/index.cfm?PKwebI...

Other possible services include https://uptimerobot.com/ and https://www.pingdom.com/

Just set it to check for the presence of a word on the page, etc.

How do you deal with the legality of scraping?

I was hired to scrape some economic related pages and build an excel file and email it, I got that done but not sure if I should try to host this and turn it into a service or just set it up for the client and let them deal with it. It's just personal use on their part.

I would not equate scraping with periodic searching for a keyword.

The problematic part is when you scrape data off of websites, and the owners don't want you to do it; as in, they would not even be happy if you manually copied that stuff into an excel file for fun or profit.

I can see that, although automating to decrease views of a page(from checking by refresh)... I suppose the end result is sign up in this case.

So for me I'll set it up for them and them run it.

> How do you deal with the legality of scraping?

How did Google deal with it, when they started their search engine business?

My counter to this is, without Google's "algorithm" the search results would not be great/find what you want. And in this case websites want to volunteer/agree to index their site for visibility but then you still have to visit the site because conveniently The site summary ends before the content that you want. I don't know how I would find websites unless referred on some website.

I kinda felt the same way about how Netflix started out but I think they probably had a deal to pay a portion of revenue from dvd's that they rented out.

robots.txt

Because this was very important to me and my life forward, I thought it was better to roll my own custom notifier. Moreover, the phone call thing was the most important part which will get attention any time of the day than an email.

So bounce your (unique) e-mail to an SMS-forwarder :)

I like the telegram API :)

Cool! Thanks for that Already using it to track whenever a couple stores here in Japan put the Nintendo Switch back in inventory or lottery sale

This is awesome, thank you.

No Hololens events in London, is that by design?

Wait a while...it took about six months for one to happen here in Sydney. Well worth the wait - half an hour of exclusive use of a Hololens. All the while being recorded for market research purposes, but that seemed like a fair trade.

Did the same thing to get a wedding registration appointment with the Berlin city administration, but used Ghostery instead of AWS+Twilio. The responsible administration department currently doesn’t give out appointments and advises not to plan weddings because they are so understaffed, but I didn’t feel like waiting. Got a notification email from Gostery in the middle of the night, registered the only available slot et voilà!

Isn’t Ghostery an ad-blocker?

I can’t see anything else with that name. All the references in this thread are great tips, so I’d like to track this one down too.

Thanks for commenting - it was Ghost Inspector

Can you please elaborate on the state of your bureaucracy. It'll make me feel better about American politics.

I think it is a passing thing. Germany recently legalized Gay marriage and presumably that lead to a huge backlog of gay people finally wanting to marry.

So we both had life changing events. Its the real winning moment of a programmer when he is able to use his/her skills for a "non-work" task automation. Its more satisfying.

I did a very similar thing to check if my H1B was approved. My use case was simpler - I regexp matched a keyword on the webpage and converted the script to an alexa skill deployed locally. Unfortunately, i wasn't home for the week when the approval happened :P. But it was fun asking alexa to check the status!

I’ve tweaked Gmail filter settings so some Meetup invites appeared as a phone alert, ie nearly everything else filtered into no alert.

London Meetup fill up within minutes that many data science ones allot tickets “randomly” via lottery.

Surely this is a market opportunity?

The problems are too fragmented for a generic alert service to solve.

Did something similar to check availability of a special deal by my ISP.

No need to write code, you only need a shell account on a box somewhere. Put a line like this in your crontab:

    20 9,12,15,18,21 * * * curl -s http://example.com/special-deal-page | grep 'something that needs to appear on the page' | mailx -E -s "it's happening!1!" $MAILTO

This will check every three hours during daytime (which is what I wanted). "mailx -E" will only send an email if the input is not empty. Don't forget to define MAILTO and to test the contraption.

This is pretty much exactly how my https://github.com/lamby/gumtree-sniper works

Pretty concise. I needed a loud phonecall, so used twilio free developer account for that.

I used this same technique to apply for an early interview for the U.S. CBP's Global Entry program. Someone already open-sourced the tool, then I set up a cron job to run it: https://github.com/davidofwatkins/ge-cancellation-checker

Did you make it to SK? There's a few of us here, but not that many...

Yes, made it to SK, but moved to Montreal for better job prospects. Probably will have to move to Toronto because my spouse's can't find a good job due to the strict French language requirements for non IT jobs in Quebec.

I've heard that you need to stay in that province (min 2 yrs) if you go through pnp program. then how come you moved to Quebec?

It’s express entry after provincial nomination. Once PR is given and landed, you are free to move and work anywhere in Canada.

Congrats. It is for sure a life changing hack.

Not sure why you choose to do diff? You could have just searched the dom for the existence of 'Apply' button? Am i missing something?

Frankly, I didn't think much. Just wrote this script in an impulse of a moment when I missed a slot. During that time, I didn't think about how it was done, but the end result. Sure diff would have triggered false positives, luckily didn't get any.

Perhaps you didn't know how the button was supposed to look like? Seems like it's better to get false positives than to miss it.

To be clear, H1B isn’t an immigrant visa. It’s a temporary skilled worker visa. There are other visas designed for skilled immigrants. Someone on an H1B isn’t an immigrant so of course the system would be “broken” when expecting H1B to result in a green card. It’s no different than expecting a tourist visa to be the first step to getting a residency visa. H1B was never implemented as a pathway to permanent residency.

Still that aside, great ‘hack’ and congrats on getting your Canadian residency!

Thats wrong information. H1B is a “dual intent” visa.

Its broken because it was really intended for exceptionally skilled workers. But currently its a lottery. USCIS just takes the word of the company applying for it and grants the visa. This loophole is being exploited by numerous body shops. That is reason why there are low quality H1B workers. There are various other problems like employee mobility, spouses of H1B is not free to work in the land of the free. Country of birth based quotas for permanent residency, rather than skills or points based quotas.

Could you please upload your code on GitHub? May I ask your github profile?

You could write this with simpledomhtlm + php + cron or a raspberry pi (cron) and then send yourself a webhook notification via slack or configure mailgun (mailing client) or firebase android notification (haven't done this).

Inspect the page,find the structure, isolate or if it is easier to just dump the entire plaintext and use strpos to find a specific word.

It made me so happy to read your post! That is such an amazing idea!

Is that for Saskatchewan Immigrant Nominee Program?

Yes. SINP in short.

Why don't you convert this into a paid service? Sounds amazing!

I did, hosted at sinpnotifier.com, Saskatchewan province is not that attractive for skilled immigrants. Had one sale though, then I shut it down.

> Why don't you convert this into a paid service

and

> I did, ...Had one sale though

Yeah, was about to say, it's too small of a niche. Plus the risk that sooner or later, Saskatchewan would update their site to provide the same feature.

Yes, too small of an audience to sell it to. Also I didn't care about the timezones in the script. It will make the call no matter in which timezone you are in (you will get the call in the middle of the night), because Saskatchewan's site do not care, it will close when the quota gets filled and it gets filled fast, within hours !

Well, you'd want to make it a /bit/ more generic, surely? Like, "load this url and send a text to this number if the (page text | diff) matches this regexp"?

Even allow people to create presets that would then be added as a generic thing. You could use it for camping all sorts of intermittent online things (web site specials, online ticket pre-orders, etc.)

There are many sites which already does this. During that situation when I missed a slot, I didn't want to take any chances again by using sites like visualping. I trusted my script and twilio API to work.

First I thought to ring a loud alarm connected with a Pi, but then it will be limited to my apartment. So I went with twilio to give me a phone call. Because I should act immediately and time was of the essence. Emails or SMS might go unnoticed.

Maybe you didn't market it to right crowd? I know a lot of South Asian people who go to Saskatchewan just to get their PR.

There are a lot of Software Engineering/Development jobs in Toronto or bigger cities so people in those fields wont leave these bigger cities.

The problem with monetizing this service is because there are many immigration watsapp groups with many members in it. So when one member buys this service, he can simply broadcast this notification to his watsapp groups. I even made the service free, but no takers. Anyways it worked for me, changed my life !

Ah fair enough. Glad it all worked out for you. I did choose Canada ( well, my dad) for undergrad because of easier immigration policy.

You took the right decision. (well, your dad).

This is interesting - as there do no longer exist equal opportunities between the different castes with this.

Saddle the lawyers, we ride at dawn!

Using one's intelligence and skills to better your life should be celebrated, not lamented as evidence of burdening some other "caste."

I get what you mean, but the same might be said for someone being able to check manually every 10 minutes. Then there is no equal opportunity between the working and workless classes.

Hey, I would be willing to pay for such service. How should I contact you? My contact details are in my profile. Plshlpme