News for hackers. Stuff that matters.
Re:Show HN: Dffer – Get notified when a website changes by awill on Friday October 06, 2017 @11:29AM (#15417545)
There are many, many competitors in this space (visualPing, changeTower, versionista, followThatPage, trackly.io). Most have similar pricing to Dffer, but some are completely free (possibly with worse UI).

I wonder whether this sort of freemium model works, because there are some people who's use case (like mine) absolutely isn't worth spending money on, but I'd like more than 3 checks for free. Obviously I'll take a worse UI that's free over spending money on such a ridiculously simple app. I could literally buy an EC2 instance just to run 10-20 daily cron jobs for less than signing up for Dffer.

My use case is to use trackly.io to see when new firmware/software updates come out for devices I own. But of course, that's never something I'd be willing to pay for.

Re:Show HN: Dffer – Get notified when a website changes by aakilfernandes on Friday October 06, 2017 @01:14PM (#15418421)
There are some users who would actually rather pay $5/mo than use a free service. Thats because paying money gives the impression (real or not) that the service is more likely to stick around and be fixed when it goes down.
Re:Show HN: Dffer – Get notified when a website changes by devdad on Friday October 06, 2017 @03:39PM (#15419770)
For me, $5 for not having to setup cronjobs is worth it.
Re:Show HN: Dffer – Get notified when a website changes by awill on Friday October 06, 2017 @04:18PM (#15420072)
I don't think price is about the task (cronjobs), it's about how you value the output.

If I'm using cronjobs for some trivial unimportant task (like checking for firmware updates), I am not willing to pay. If I'm using cronjobs for super important competitive analysis, sure I'd pay.

Re:Show HN: Dffer – Get notified when a website changes by alexdean on Friday October 06, 2017 @05:10PM (#15420514)
Do any of these companies offer webhooks, or are they all oriented at human users reading emails?
Re:Show HN: Dffer – Get notified when a website changes by rahimnathwani on Friday October 06, 2017 @11:33AM (#15417580)
Does this work with content that is loaded dynamically with JS? I've used http://www.changedetection.com/ in the past, but it only scrapes the HTML.
Re:Show HN: Dffer – Get notified when a website changes by bluepeter on Friday October 06, 2017 @07:14PM (#15421140)
We offer “full page” rendering using PhantomJS (soon to be headless Chrome) for Versionista, a competing product. It works rather well, but it introduces its own set of problems... eg you need timers to ensure all the AJAX calls have completed, and so on.
Re:Show HN: Dffer – Get notified when a website changes by rahimnathwani on Sunday October 08, 2017 @08:53AM (#15428020)
Thanks - just signed up.

I found the 'crawl' settings a little confusing, as the first item is about loading additional URLs (which is what I thought I wanted, as JS is often loaded from separate URLs rather that being embedded in the HTML), but then there's a later option about a full browser. I chose this latter option, and left the other one unchecked. I hope that's correct.

Re:Show HN: Dffer – Get notified when a website changes by CHANCECHANEL on Friday October 06, 2017 @11:23AM (#15417493)
I wonder how its implemented. I think its possible to do run this type of service at a very large scale for extremely low costs using AWS Lambda and Cloudwatch events.
Re:Show HN: Dffer – Get notified when a website changes by bluepeter on Friday October 06, 2017 @07:12PM (#15421130)
That’s exactly what we use for a competing product. The difficulty is less in the crawling, and more in the scheduling and difference visualization (including filters). Presenting salient results without a ton of false positives is hard work.
Re:Show HN: Dffer – Get notified when a website changes by fabiospampinato on Friday October 06, 2017 @02:38PM (#15419230)
I've made something similar a while ago called "rssa" [1]. You give it some urls, some way of extracting information from them (via a jQuery selector, regex etc.) and if things change it tells you.

I use it for tracking prices on Amazon, some stats about my repositories and so on.

[1] https://github.com/fabiospampinato/rssa

Re:Show HN: Dffer – Get notified when a website changes by jdc0589 on Friday October 06, 2017 @03:26PM (#15419660)
seems like a lot of us did this. my version let you write little JavaScript modules to extract data from a page (which had been parsed by jsdom already), and then the change tracking picked things up.
Re:Show HN: Dffer – Get notified when a website changes by jvagner on Friday October 06, 2017 @04:42PM (#15420308)
Here's something I've wished I'd had over the 20 years of web development efforts I've been involved in: for production sites, a site that crawls a path regularly and saves a copy of the site in a timeline fashion.

How many times I've thought, "Man, we've made a lot of changes in the last 9 months and no one has a visual narrative of all of those changes..."

Re:Show HN: Dffer – Get notified when a website changes by dewski on Friday October 06, 2017 @04:50PM (#15420385)
I built exactly this once, was fun to set and forget and look back on it in a few months. It calculated the diff and you could see obvious spikes when big changes were rolled out. You saw a lot of big spikes on say Apple.com when new products were announced or marketed, whereas Reddit.com was always changing and couldn't detect any meaningful patterns.

It also had ffmpeg in the backend that would create a timelapse for you on demand. I built it when we rolled out our some rebranding/homepage changes on GitHub.com and nobody really knew off the top of our heads when exactly the last time we updated it was. Git only tracked code changes, no visual changes.

It was originally built on top of http://www.paulhammond.org/webkit2png/

Re:Show HN: Dffer – Get notified when a website changes by wingerlang on Sunday October 08, 2017 @03:42AM (#15427356)
I implemented an auto-screenshotter for my app that takes a screenshot every time the app launches after a few seconds. It works really well to give a visual progression of the apps style and features. I'm at 3k screenshots over 1.5 years of development. The idea is to make a timeline when the app is done.

I liked it so much I will add it into each app I work on from now on.

Re:Show HN: Dffer – Get notified when a website changes by hobofan on Friday October 06, 2017 @05:32PM (#15420640)
I'm currently building something that goes somewhat in that direction. Shoot me a mail, and I'll get back to you :)
Re:Show HN: Dffer – Get notified when a website changes by tectonic on Friday October 06, 2017 @02:25PM (#15419085)
I use Huginn for this - https://github.com/huginn/huginn
Re:Show HN: Dffer – Get notified when a website changes by veritas3241 on Friday October 06, 2017 @02:20PM (#15419034)
What I really want is something that can track changes to documents (PDFs, but any doctype really) hosted on some site. The URL may or may not change, but the location of the document URL on the page probably won't be changing. Really I'm just interested in the MD5, so no need to track specific changes.

I've got some custom scripts but paying somebody would be nice. Management of these becomes a pain after 10 or so.

Re:Show HN: Dffer – Get notified when a website changes by dmoena on Friday October 06, 2017 @04:26PM (#15420143)
Hey, can you tell more about the context on why you need this? It would help to know if there's a more general need and if so, maybe someone gets motivated to implement it...
Re:Show HN: Dffer – Get notified when a website changes by bluepeter on Friday October 06, 2017 @07:08PM (#15421106)
Hey, I run Versionista (https://versionista.com) and we offer PDF change tracking and visualization.
Re:Show HN: Dffer – Get notified when a website changes by maturz on Friday October 06, 2017 @11:58AM (#15417809)
I've been using a Firefox add-on for this

https://addons.mozilla.org/en-US/firefox/addon/update-scanne...

It works quite well but there's not much in terms of excluding content. When I needed to do that I just ran it through a simple script that filtered for me (Node.js).

Re:Show HN: Dffer – Get notified when a website changes by mrskitch on Friday October 06, 2017 @01:37PM (#15418633)
Curious if this works on SPA's, and if so what are you using?
Re:Show HN: Dffer – Get notified when a website changes by bluepeter on Friday October 06, 2017 @07:10PM (#15421119)
It’s posaible to do this for SPAs, yes. We use PhantomJS for rendering JavaScript at Versionista... soon we will swap to headless Chrome.
Re:Show HN: Dffer – Get notified when a website changes by jdormit on Friday October 06, 2017 @11:15AM (#15417423)
What is the use case for this? Why would I pay for it?
Re:Show HN: Dffer – Get notified when a website changes by aldo712 on Friday October 06, 2017 @11:55AM (#15417789)
About a year back I had written a script that would scrape a page every 15 minutes, and notify me when the content changed. It was basically a Band Tour page, and I wanted to be the first to know when the tickets got announced so that I could grab 'em before they got sold out.

P.S. : It was an amazing concert. :)

Re:Show HN: Dffer – Get notified when a website changes by ropeladder on Friday October 06, 2017 @11:26AM (#15417518)
Assuming this is reliable it would be great for job searches. (if you're targeting smaller organizations with static-enough job pages) Seems a bit expensive though--especially when I might rather sign up for, say, daily checks on up to 240 websites instead of hourly checks on 10 websites.
Re:Show HN: Dffer – Get notified when a website changes by jldugger on Friday October 06, 2017 @02:28PM (#15419113)
I actually do this. It's far more effective to apply for new postings than whatever happens to be live at the moment, as companies will happily advertise a position even as they're waiting for a candidate to accept an offer they've made, keeping new applicants around 'just in case'.

But I just use cron, shell, and github.

Re:Show HN: Dffer – Get notified when a website changes by devdad on Friday October 06, 2017 @04:16PM (#15420046)
I'm building a product in kinda new market. I'll use something like this to keep up with other landing pages.
Re:Show HN: Dffer – Get notified when a website changes by mbrd on Friday October 06, 2017 @11:24AM (#15417504)
I've set it up to monitor an "out-of-stock" string on a product I'd like to buy, but is going to be hard to get in the run up to Christmas, so I guess that's one.

If I was very keen, I guess I might pay a little to check more often than daily (although I haven't done that in this case).

Re:Show HN: Dffer – Get notified when a website changes by jnmandal on Friday October 06, 2017 @11:45AM (#15417683)
I've used a script I wrote that does this basically to try and snag react conf tickets https://github.com/jnmandal/sitewatcher
Re:Show HN: Dffer – Get notified when a website changes by ams6110 on Friday October 06, 2017 @01:49PM (#15418754)
One interesting case is tracking news sites. The details added or removed from new stories over time can sometimes support a claim of bias or agenda-driven reporting. Of course sometimes they are just corrections.
Re:Show HN: Dffer – Get notified when a website changes by vira on Friday October 06, 2017 @03:40PM (#15419774)
It could help with competitor analysis - point it to the websites of competition in your industry to stay on top of where they are going.
Re:Show HN: Dffer – Get notified when a website changes by PascLeRasc on Friday October 06, 2017 @11:55AM (#15417785)
Monitoring companies you want to work for for new job postings is what I use similar tools for.
Re:Show HN: Dffer – Get notified when a website changes by always_good on Friday October 06, 2017 @12:42PM (#15418176)
I wouldn't wait for a "formal opening" before applying. In fact, if you do, you're competing with even more people.
Re:Show HN: Dffer – Get notified when a website changes by PascLeRasc on Monday October 09, 2017 @04:52PM (#15437356)
How has that worked for you?
Re:Show HN: Dffer – Get notified when a website changes by wu_tang_chris on Friday October 06, 2017 @01:25PM (#15418517)
So when do you apply?
Re:Show HN: Dffer – Get notified when a website changes by always_good on Friday October 06, 2017 @04:34PM (#15420229)
Immediately. There's no harm, and it's more likely that a company is always interested in good applicants. You can always follow up again when the position does show up on their open position listing.

Also, if you apply during the "offseason", it's more likely that technical hire has more time to respond to you in more detail. I've developed a couple relationships that way that paved the way for an eventual hire in one case and a technical relationship in another.

Re:Show HN: Dffer – Get notified when a website changes by wu_tang_chris on Friday October 06, 2017 @05:22PM (#15420571)
So you just submit your resume to places you'd like to work, even if they're not hiring? Do you even look at job postings? That's interesting...
Re:Show HN: Dffer – Get notified when a website changes by graysonk on Friday October 06, 2017 @11:20AM (#15417460)
> What is the use case for this?

I've written something similar for a company who wanted to monitor their rival's website for news about expansion.

> Why would I pay for it?

I'm not sure either. A quick script in a cron job could also do this.

Re:Show HN: Dffer – Get notified when a website changes by sho on Friday October 06, 2017 @11:30AM (#15417551)
> A quick script in a cron job could also do this

I love comments like this. You couldn't hope for a better example of "shit's easy syndrome".

Yeah, a quick script in a cron job. Oh, but now we need 2 pages monitored. Now we need 100. Now provide an interface for marketing to add and remove pages. Make sure it emails these N people (different for each page of course!). Oh, this has to be up 24/7 so we'll need a server and monitoring and a test environment and deploy strategy. Your script is giving false positives because of changing asset timestamps. Oh, your script needs to provide a visual indication of what changed and when. Etc etc etc et cetera.

That's why you pay for things like this. Because it's actually a shitload of work.

I recently used a similar site (visualping.com) to score a pair of airpods, which are in tight supply, by watching my local apple store for availability. Worked beautifully. Write my own script, are you kidding?

Re:Show HN: Dffer – Get notified when a website changes by Karrot_Kream on Friday October 06, 2017 @03:57PM (#15419900)
> Yeah, a quick script in a cron job. Oh, but now we need 2 pages monitored. Now we need 100. Now provide an interface for marketing to add and remove pages. Make sure it emails these N people (different for each page of course!). Oh, this has to be up 24/7 so we'll need a server and monitoring and a test environment and deploy strategy. Your script is giving false positives because of changing asset timestamps. Oh, your script needs to provide a visual indication of what changed and when. Etc etc etc et cetera.

> I recently used a similar site (visualping.com) to score a pair of airpods, which are in tight supply, by watching my local apple store for availability. Worked beautifully. Write my own script, are you kidding?

You are totally conflating use cases. For most people, monitoring 2/3 things, a cron script works fine. In fact, I have one that's running right now and it Just Works (TM). By the time I need to provide visual diffs and have bulk email logic, I'm probably going to use an external tool (or whip it up as a side project if I think it's worth my time).

You can't equivocate the n=1 case and the n=10000 case.

Re:Show HN: Dffer – Get notified when a website changes by sho on Saturday October 07, 2017 @03:01AM (#15422653)
> You can't equivocate the n=1 case and the n=10000 case.

Fair enough, but the comment I was replying to was equivocating those cases too. I was simply pointing out that maybe, just maybe, a service like this might have some value above and beyond a 5 minute script and a cron job. And indulging in a small rant against the "i could do that in a script/i could write that in a weekend" etc mindset.

Re:Show HN: Dffer – Get notified when a website changes by jldugger on Friday October 06, 2017 @02:37PM (#15419218)
I mean, I've written scripts like this, complete with Chef server deployment and filtering criteria. It's not surprising to see HN mention how feasible it is; we're not really the target market.
Re:Show HN: Dffer – Get notified when a website changes by uptown on Friday October 06, 2017 @11:43AM (#15417659)
"I'm not sure either. A quick script in a cron job could also do this."

Except probably 99% percent of the planet couldn't tell you what a cron job is.

Re:Show HN: Dffer – Get notified when a website changes by koolba on Friday October 06, 2017 @11:46AM (#15417700)
I think 99% is low balling it. Also, not all of the remainder could actually set one up either.
Re:Show HN: Dffer – Get notified when a website changes by olkid on Friday October 06, 2017 @03:25PM (#15419651)
So, I am part of the 1%! Gosh, I didn't even know...
Re:Show HN: Dffer – Get notified when a website changes by graysonk on Friday October 06, 2017 @11:54AM (#15417771)
I get that, but we are on hackernews right now. Presumably a larger portion of the users here know what cron is.
Re:Show HN: Dffer – Get notified when a website changes by lightedman on Friday October 06, 2017 @03:00PM (#15419438)
Long ago, web browsers used to actually have this capability built-in. I remember doing it with Netscape Navigator.
Re:Show HN: Dffer – Get notified when a website changes by bootcat on Friday October 06, 2017 @01:40PM (#15418661)
Wanted to do something similar for so long !! I am also interested if you do ajax sites, and if so how ?
Re:Show HN: Dffer – Get notified when a website changes by laktek on Friday October 06, 2017 @04:55PM (#15420422)
If you want to build something like this on your own, check Page.REST.