News for hackers. Stuff that matters.

anon389r58r58

You mean like Modular?
Re:How to solve computational science problems with AI: PINNs on Wednesday January 22, 2025 @04:22PM (#42797666)
The general rule of thumb to go by is that whatever Karniadakis proposes, doesn't actually work outside of his benchmarks. PINNs don't really work, and _his flavor_ of neural operators also don't really work.

PINNs have serious problems with the way the "PDE-component" of the loss function needs to be posed, and outside of throwing tons of, often Chinese, PhD students, and postdocs at it, they usually don't work for actual problems. Mostly owed to the instabilities of higher order automatic derivatives, at which point PINN-people begin to go through a cascade of alternative approaches to obtain these higher-order derivatives. But these are all just hacks.

Re:Micrograd.jl on Friday August 30, 2024 @04:49PM (#41404285)
I think you are asking an ill-posed question in parts. Julia has a lot of great things, and needs to continue evolving to find an even better fit amongst the many programming languages available today and sustain itself long-term.

Emulating Python's ML ecosystem is not going to be a viable strategy. The investment into the Python-based standard is just too large.

What I could see happening though is that the continuous evolution of the ML ecosystem will further abstract components of the software stack down to an MLIR/LLVM abstraction level at which point something like Julia could also use this componentry. Sort of a continuum of components, where the "frontend" language and associated programming style is the remaining choice for the user to make.

Re:Micrograd.jl on Friday August 30, 2024 @12:28PM (#41402230)
Almost feels like a fallacy of Julia at this point, on the one hand Julia really needs a stable, high-performance AD-engine, but on the other hand it seems to be fairly easy to get a minimal AD-package off the ground.

And so the perennial cycle continues and another Julia AD-package emerges, and ignores all/most previous work in order to claim novelty.

Without a claim for a complete list: ReverseDiff.jl, ForwardDiff.jl, Zygote.jl, Enzyme.jl, Tangent.jl, Diffractor.jl, and many more whose name has disappeared in the short history of Julia...

Fully agree, and I think the tinybox is great if you put only one of them somewhere in your local office.

I just don't think it makes sense to connect multiple of them into a "cluster" to work with bigger models, as the networking bandwidth isn't good enough and you'd have to fit multiple of these big boxes into your local space. Then I might as well put up a rack in a separate room.

For that you'd probably be better off removing one of the GPUs, and replacing it with a networking card.

The problem of the form factor will remain. The tinybox is 15U big for compute that you'd normally expect to find in a 4U form factor.

The networking of the tinybox is woefully inadequate. I.e. it only has an OCP 3.0 interface which is unoccupied. If you can fit everything onto one tinybox, then you'll be good, if you cannot, then you'd be better off by having a more professional workstation solution like e.g. NVIDIA RTX cards which have more memory.
Re:PyTorch is dead. Long live Jax on Saturday August 17, 2024 @02:36PM (#41276804)
So the answer is not Jax?

Because JAX is not designed around a mature compiler stack. The history of Jax is more so that it matured alongside the compiler...