Re:Fallout 76 Day One Patch Is Larger Than the Game Itself
by mjevans on Wednesday November 14, 2018 @05:36PM
(#18454847)
Basic idea:
If the file is smaller than 10x the base patch unit size: don't bother just add the whole file as a single unit / send it all for each update.
Else: Create a manifest of the file: Size on Disk, Checksum of file, Name of file mapping to (one or more) list of segments within file by segment offset, segment size, segment checksum. A custom list might be provided by a resource packer that is aware of assets / code segments within the file.
A website will provide an interface that has a list of file names to target size/hashes.
There will also be a list by size/hash that enumerates the segment lists within the file.
To update, the file size/checksum would be computed and the manifests obtained from the server; these would be compared to the files on disk and invalid segments/files discarded.
The target version would then be selected and the target manifests collected. First limited by filename and then optionally project wide, segments would be matched first checking size then checksum. There are a couple different methods that might then be optimal depending on the desired outcome, which depend on disk space, operating system support for hard-linking/de-duplicating subsegments of a file, if the media is known to be an SSD or spinning rust, etc.
However, in any event, that infrastructure allows for easily checking and upgrading / repairing to any version from any version at the expense of storing a small quantity of metadata and having full copies via the actual delivery service for segments on file. The backend might also have de-duplication of the stored data as a bonus.
This is probably a very naive implementation of compression via de-duplication and should be obvious to anyone skilled in this form of art.
If the file is smaller than 10x the base patch unit size: don't bother just add the whole file as a single unit / send it all for each update.
Else: Create a manifest of the file: Size on Disk, Checksum of file, Name of file mapping to (one or more) list of segments within file by segment offset, segment size, segment checksum. A custom list might be provided by a resource packer that is aware of assets / code segments within the file.
A website will provide an interface that has a list of file names to target size/hashes.
There will also be a list by size/hash that enumerates the segment lists within the file.
To update, the file size/checksum would be computed and the manifests obtained from the server; these would be compared to the files on disk and invalid segments/files discarded.
The target version would then be selected and the target manifests collected. First limited by filename and then optionally project wide, segments would be matched first checking size then checksum. There are a couple different methods that might then be optimal depending on the desired outcome, which depend on disk space, operating system support for hard-linking/de-duplicating subsegments of a file, if the media is known to be an SSD or spinning rust, etc.
However, in any event, that infrastructure allows for easily checking and upgrading / repairing to any version from any version at the expense of storing a small quantity of metadata and having full copies via the actual delivery service for segments on file. The backend might also have de-duplication of the stored data as a bonus.
This is probably a very naive implementation of compression via de-duplication and should be obvious to anyone skilled in this form of art.