News for hackers. Stuff that matters.
Re:Downloading a file regularly - how hard can it be? by moe on Monday April 09, 2012 @01:51PM (#3818236)
I assume changes are usually small, you may want to try serving diffs?

I.e. have the clients poll for the md5 of their _current_ list-version.

On the server store the diff that will upgrade them to the current version under that filename. If a client requests an unknown md5 (e.g. because he has no list or his list is corrupted) default him to a patch that contains the full file.

This requires a little logic on both ends (diff/patch), but would probably slash your bandwidth requirements to a fraction.

A little napkin math:

25 lists * 150kb * 1mio fetches = ~3.75T

vs

25 lists * 1kb (patch) * 1mio fetches = 25G (0.025T)

Re:Downloading a file regularly - how hard can it be? by pjscott on Monday April 09, 2012 @06:52PM (#3819702)
This is probably the Right Way, but it would be more work than minor tweaks to the delay logic.