Update: seems that "fast size" feature was cancelled or is not yet ready: https://eclecticlight.co/2019/02/06/how-big-is-that-folder-w..., https://forums.developer.apple.com/thread/79901

So the problem is likely with size caching in Finder.

I tried getattrlist on my system, and ATTR_DIR_ALLOCSIZE is always 0 and ATTR_DIR_DATALENGTH is some unrelated small number (probably size of directory entry in FS, not size of its files).