thinxer
S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.
Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.
[1]: https://github.com/pytorch/examples/blob/main/imagenet/extra...
Text to Shareable Web Apps
Site Generator for Apple Notes
Sparks of Artificial General Intelligence: Early Experiments with GPT-4
Apple AMX Instructions
MLIR CodeGen Dialects for Machine Learning Compilers
Programming Language Memory Models
Where Are Pixels? – A Deep Learning Perspective
Accelerate
OctoML raises $28M Series B for its machine learning acceleration platform
Unoptimized (more portable) versions of theses routines exists, but I suspect they will perform well even with M1.
Vcpkg: C++ Library Manager for Windows, Linux, and macOS
The core value behind monorepo (and monorepo-like approaches) explained in the book is that dependency management is harder than version control.