Much faster incremental apt updates

Re:Much faster incremental apt updates by legulere on Sunday December 27, 2015 @01:35PM (#10798272)

> The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines. This changed on December 24, by adding read buffering for reading lines, vastly improving the performance of rred.

Why not use memory mapped files and let the operating system deal with caching in a way that is most efficient for that specific case?

[ Parent | on HN ]

Re:Much faster incremental apt updates by julian-klode on Tuesday December 29, 2015 @12:11PM (#10807643)

Not all our files are uncompressed, so directly using mmap does not make sense for us, because we also need to be able to use files compressed with gzip, bzip2, xz (and soonish probably lz4).

[ Parent | on HN ]

Re:Much faster incremental apt updates by ars on Monday December 28, 2015 @01:01AM (#10800102)

I believe mmap is not fully portable without extra work, so they avoid it - debian runs on a LOT of different systems.

[ Parent | on HN ]

Re:Much faster incremental apt updates by legulere on Monday December 28, 2015 @01:28AM (#10800169)

It was developed inside the original BSD and is part of POSIX. So it's supported on all unixes today. The only place it isn't supported is Windows.

[ Parent | on HN ]

Re:Much faster incremental apt updates by icefox on Monday December 28, 2015 @09:56AM (#10801333)

It is everywhere even Windows. There is a little bit of weirdness on some older versions of Windows CE and some undefined behavior with edge conditions on some OS's if I recall, but you can find the capability everywhere. I added the map API to QFile back in the day and went through checking different OS's for compatibility. A quick google search here is a link to the windows method:

https://github.com/radekp/qt/blob/master/src/corelib/io/qfsf...

unix version: https://github.com/radekp/qt/blob/master/src/corelib/io/qfsf...

[ Parent | on HN ]