Re:Downloading a file regularly - how hard can it be?
by moe on Monday April 09, 2012 @01:51PM
(#3818236)
I assume changes are usually small, you may want to try serving diffs?
I.e. have the clients poll for the md5 of their _current_ list-version.
On the server store the diff that will upgrade them to the current version under that filename. If a client requests an unknown md5 (e.g. because he has no list or his list is corrupted) default him to a patch that contains the full file.
This requires a little logic on both ends (diff/patch), but would probably slash your bandwidth requirements to a fraction.
I.e. have the clients poll for the md5 of their _current_ list-version.
On the server store the diff that will upgrade them to the current version under that filename. If a client requests an unknown md5 (e.g. because he has no list or his list is corrupted) default him to a patch that contains the full file.
This requires a little logic on both ends (diff/patch), but would probably slash your bandwidth requirements to a fraction.
A little napkin math:
25 lists * 150kb * 1mio fetches = ~3.75T
vs
25 lists * 1kb (patch) * 1mio fetches = 25G (0.025T)