> The order of log entries does not matter.

This is surprising, Kafka-like logs are all strictly ordered.

* within a partition

The reason I made ordering not matter is so that multiple peers don’t have to worry about keeping the exact same order when appending to each other’s logs.

The log entries do have timestamps on them. You can sort by timestamp, but a peer has the right to append an new entry that’s older than the latest timestamp.

How do you manage log size for high-transaction systems?

It’s not terribly high transaction, yet. If it becomes that way, I would partition the one by entry type so all the high transaction stuff gets stuffed in a particular log, and then ensure it can be summarised regularly.

How do you know summary entries are valid if order doesn't matter?

I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)

IME you have to be willing to recalculate the summaries up to some kind of consistency window.

Yes you may be changing history and you may have a business reason not to address that revision immediately (you've already billed them?) - but the system can still learn it made a mistake and fix it (add activity from Jan 30 evening that comes in late to the Feb bill?)

Realistically, you never store summaries in the log. Instead, you store what it took to calculate them. So you won't store "we have 10 customers on this date with this range" but instead store "we found these 10 customers on this date with this range". This assumes you can store infinite sized lists in your log, but realistically, this is never a concern if you can keep your time windows small enough. Then, you periodically do a reconciliation and log corrections (look for entries not summarized -- easily done via a bloom filter which can tell you what entries are definitely NOT in your set) over a longer period.

For example, we had a 28-day reconciliation period at one company I worked at (and handled over 120 million events per day). If you appended an event earlier than 28 days prior, it was simply ignored. This very rarely happened, but allowed us to fix bugs with events for up to 28 days.

Calling a WAL a ledger, why? Ledger sounds fancier but why would it be a ledger in this case?

I believe "ledger" implies commutative property (order does not matter).

I am not aware of any such implicit connection of ledger and commutative property, also couldn't find anything as my google-fu is letting me down. Anything I can refer to? Generally curious to know use of term ledger outside of accounting and blockchains.

I have seen it used to mean WAL before, so I am taking this with a dose of skepticism.

In a double-entry ledger the order of transactions doesn't matter, the balance is the sum of entries.

Depending on the data model of your log, if calculating the current state is a commutative operation, I think it's fair to call it a "ledger".

We called it a ledger since we stored financial data and basically used the “ledger” format from plain text accounting, initially.

That answers my question, thanks! Ordering does not matter part has me curious too, I will read other comment and come back to you l.

The lack of ordering is surprising. Without that you can’t stream without a buffer.

The idea is that all peers eventually have the same collection of log entries, with an agreed-upon sort method. Currently that sort method is time stamps.

When it’s time to have identical replicas, a log entry must be appended that says the log is closed. Once all peers have this, the log is now immutable and can be treated as such. An example of this (for a payroll system) is a collection of entries for time clock in and outs, payroll taxes withheld, direct deposits to employees, etc when it’s time to do a payroll. The log is closed and that data can never be altered, but now the data can be relied on by someone else.

Conceptually this is surprisingly similar to the type of internal logs a typical database like PostgreSQL keeps.

> a log entry must be appended that says the log is closed

Related concepts: 'epoch' (distributed consensus), 'watermark' (out-of-order stream processing)

if order does not matter how do you implement deletes?

You don’t do deletes. The log is append only.

If you want to get rid of an entry, you create a new entry that negates the existing entry.

Yes, but if order doesn't matter, the delete might appear before the create.

Might be misunderstand something

exactly my point ;)