sthoward
Re:The Future of Large Files in Git Is Git
on Saturday August 16, 2025 @08:05AM (#44922514)
We built `oxen` to solve the issues we (and many others) had with DVC and LFS. The highlights: open source cli and server, mirrors git for easy learning, handles large files and millions of files, performant af. Would love feedback if you check it out.
https://github.com/Oxen-AI/Oxen
or check out the performance numbers https://docs.oxen.ai/features/performance
Re:Live Dive into How to Finetune DeepSeek R1 on Synthetic Data
on Wednesday January 29, 2025 @10:16PM (#42874429)
DeepSeek... so hot right now.
Re:Show HN: GitLFS taking forever? No prob, heres the best AI data ver...
on Monday December 30, 2024 @05:55PM (#42554475)
Really well written Mathi.
Re:We put 1M files into DVC, Git-LFS, and Oxen.ai
on Tuesday November 05, 2024 @05:42PM (#42055844)
Great thought. Right now we are optimizing for moving data from machine A to machine B, but getting data to the GPU is interesting. We're on it.
Re:We put 1M files into DVC, Git-LFS, and Oxen.ai
on Sunday November 03, 2024 @06:49PM (#42037113)
Hey all,
If you haven't seen the Oxen project yet, we have been building an open source unstructured data version control tool.
We were inspired by the idea of making large machine learning datasets living & breathing assets that people can collaborate on, rather than the static ones of the past. Lately we have been working hard on optimizing the underlying Merkle Trees and data structures with in Oxen.ai and just released v0.19.4 which provides a bunch of performance upgrades and stability to the internal APIs.
To put it all to the test, we decided to benchmark the tool on the 1 million+ images in the classic ImageNet dataset.
The TLDR is Oxen.ai is faster than raw uploads to S3, 13x faster than git-lfs, and 5x faster than DVC. The full breakdown can be found here.
https://docs.oxen.ai/features/performance
If you are in the ML/AI community, or rust aficionados, would love to get your feedback on both the tool and the codebase. We would love some community contribution when it comes to different storage backends and integrations into other data tools.
Re:Using Llama3.1 405B to generate political synthetic data
on Friday August 02, 2024 @12:55PM (#41140512)
This has the legitimate possibility to be the most telling spam or ham (and funniest) yet. Very interested in the state by state breakdowns.
Re:Fine Tuning a Diffusion Transformer (DiT) from a Single YouTube Video
on Thursday May 30, 2024 @10:04PM (#40530758)
So how many different prompts did you try it on?
Re:Can the Data Fit in My Mouth
on Thursday May 30, 2024 @09:58PM (#40530728)
Lol, should we add this as a metric for repos over on www.oxen.ai?
Re:Instruct-Tuning BitNet 1.58
on Tuesday April 09, 2024 @10:47AM (#39980031)
This was wild... especially the live session :) The convo after was truly great. Great video and paper recap!
Re:A HuggingFace Datasets Alternative
on Thursday February 29, 2024 @03:02AM (#39547311)
More than just an alternative - built better, faster workflows.
Re:Ask HN: AI/ML papers to catch up with current state of AI?
on Friday December 15, 2023 @04:52PM (#38659265)
Would suggest our weekly paper club called Arxiv Dive - https://lu.ma/oxenbookclub. You can see past ones on our blog (https://blog.oxen.ai/) - have covered papers like Mamba, CLIP, Attention is all you need, and more. We also do a "hands on" session with live code, models, and real world data on Fridays!
Re:Deep Dive into the Vision Transformers Paper (ViT)
on Friday December 01, 2023 @08:41PM (#38494974)
Sign up here: http://lu.ma/oxenbookclub