> ZFS has caching for writes

Not really.

It will accumulate synchronous writes into the ZIL, and you put the ZIL on a fast SLOG vdev. But it will only do so for a limited amount of time/space, and is not meant as a proper write-back cache but rather as a means to quickly service synchronous writes.

By default asynchronous writes do not use the ZIL, and hence SLOG vdev at all. You can force it to, but that can also be a bad idea unless you have Optane drives as you're then bottlenecked by the ZIL/SLOG.

"Fast" is meaningless.

A SLOG needs to have: good throughput and low latency at a queue depth of 1.

I should point out that the SLOG only caches synchronous writes, which are written twice with ZFS.

Also, the L2ARC is great, but does still have RAM overhead. There are also useful tunables. I had a workload on a RAM limited machine where directory walking was common, but data reads were fairly random and a L2ARC configured for metadata only speed it up by a large amount.

You can also have a special metadata only/ small files dedicated special VDEV. ZFS can pull of many tricks if you configure it well. Of course, the L2ARC is better, if you don't trust the caching device that much (e.g. only have a single SSD).

You can also configure an L2ARC to only hold metadata, which is safer than a metadata device.