News for hackers. Stuff that matters.
Re:Building a Distributed Log from Scratch, Part 1: Storage ... by ww520 on Friday December 22, 2017 @12:57AM (#15985543)
One question about the index file, "in Kafka, the index uses 4 bytes for storing an offset relative to the base offset and 4 bytes for storing the log position." Isn't the relative offset to the base offset already pointing to the physical location of the message in the segment file? What's the purpose of the second 4-byte field log position?
Re:Building a Distributed Log from Scratch, Part 1: Storage ... by foota on Friday December 22, 2017 @02:09AM (#15985771)
I think that the offsets are both in terms of number of messages, whereas the log position is the byte in the file that the message occurs at.
Re:Building a Distributed Log from Scratch, Part 1: Storage ... by kungfooguru on Friday December 22, 2017 @11:37AM (#15988710)
Right, I think earlier Kafka it was actually the position as well, but then they moved to the current method.