smhx

If you're the author, unfortunately I have to say that the blog is not well-written -- misinformed about some of the claims and has a repugnant click-baity title. you're getting the attention and clicks, but probably losing a lot of trust among people. I didn't engage out of choice, but because of a duty to respond to FUD.

> > torch.compile is 2 years old, XLA is 7 years old. Compilers take a few years to mature

> That was one of my major points - I don't think leaning on torch.compile is the best idea. A compiler would inherently place restrictions that you have to work-around.

There are plenty of compilers that place restrictions that you barely notice. gcc, clang, nvcc -- they're fairly flexible, and "dynamic". Adding constraints doesn't mean you have to give up on important flexibility.

> This is not dynamic, nor flexible - and it flies in the face of torch's core philosophies just so they can offer more performance to the big labs using PyTorch. For various reasons, I dislike pandering to the rich guy instead of being an independent, open-source entity.

I think this is an assumption you've made largely without evidence. I'm not entirely sure what your point is. The way torch.compile is measured for success publicly (even in the announcement blogpost and Conference Keynote, link https://pytorch.org/get-started/pytorch-2.0/ ) is by measuring on a bunch of popular PyTorch-based github repos in the wild + popular HuggingFace models + the TIMM vision benchmark. They're curated here https://github.com/pytorch/benchmark . Your claim that its to mainly favor large labs is pretty puzzling.

torch.compile is both dynamic and flexible because: 1. it supports dynamic shapes, 2. it allows incremental compilation (you dont need to compile the parts that you wish to keep in uncompilable python -- probably using random arbitrary python packages, etc.). there is a trade-off between dynamic, flexible and performance, i.e. more dynamic and flexible means we don't have enough information to extract better performance, but that's an acceptable trade-off when you need the flexibility to express your ideas more than you need the speed.

> XLA's GPU support is great, its compatible across different hardware, its optimized and mature. In short, its a great alternative to the often buggy torch.compile stack - if you fix the torch integration.

If you are an XLA maximalist, that's fine. I am not. There isn't evidence to prove out either opinions. PyTorch will never be nicely compatible with XLA until XLA has significant constraints that are incompatible with PyTorch's User Experience model. The PyTorch devs have given clear written-down feedback to the XLA project on what it takes for XLA+PyTorch to get better, and its been a few years and the XLA project prioritizes other things.

the author got a couple of things wrong, that are worth pointing out:

1. PyTorch is going all-in on torch.compile -- Dynamo is the frontend, Inductor is the backend -- with a strong default Inductor codegen powered by OpenAI Triton (which now has CPU, NVIDIA GPU and AMD GPU backends). The author's view that PyTorch is building towards a multi-backend future isn't really where things are going. PyTorch supports extensibility of backends (including XLA), but there's disproportionate effort into the default path. torch.compile is 2 years old, XLA is 7 years old. Compilers take a few years to mature. torch.compile will get there (and we have reasonable measures that the compiler is on track to maturity).

2. PyTorch/XLA exists, mainly to drive a TPU backend for PyTorch, as Google gives no other real way to access the TPU. It's not great to try shoe-in XLA as a backend into PyTorch -- as XLA fundamentally doesn't have the flexibility that PyTorch supports by default (especially dynamic shapes). PyTorch on TPUs is unlikely to ever have the experience of JAX on TPUs, almost by definition.

3. JAX was developed at Google, not at Deepmind.

it's not a new PyTorch feature.

It's just a showcase of existing PyTorch features (including libtorch) as an end-to-end example.

On the server-side it uses libtorch, and on mobile, it uses PyTorch's executorch runtime (that's optimized for edge)

that's a direct implication that they're waiting for a liquidity event before they speak

most of the GenAI players use both PyTorch and JAX, depending on the hardware they are running on. Character, Anthro, Midjourney, etc. are dual shops (they use both). xAI only uses JAX afaik.

You've created a superior llama/mistral-derivative model -- like https://old.reddit.com/r/LocalLLaMA/comments/17vcr9d/llm_com...

How can you convince the world to use it (and pay you)?

Step 1: You need a 3rd party to approve that this model is safe and responsible. the Purple Llama project starts to bridge this gap!

Step 2: You need to prove non-sketchy data-lineage. This is yet unsolved.

Step 3: You need to partner with a cloud service that hosts your model in a robust API and (maybe) provides liability limits to the API user. This is yet unsolved.

>What's a fair benchmark?

the absolute golden benchmarks are https://github.com/pytorch/benchmark They are a diverse set of userland code taken from github as-is and made into benchmarks.

thanks for the heads-up, looks like we didn't yank the CPU wheels on those dates. will get to them in the next set of working hours, as its an unlikely scenario (not only do you have to install the wheel of a specific date, you also have to specify the undocumented feature flag [dynamo])

> So technically, if you are pulling the older version of pytorch-nightly (specifically 2.0.0.dev20221230), it will still pull that compromised dependency (because torch have explicit version lock to it).

All PyTorch nightlies with this dependency have been deleted

This is the last one. It was also the first one.

this only affects the nightly pytorch. the stable pytorch build doesn't depend on `torchtriton`.

the nightly pytorch moved to depend on our own secured `pytorch-triton` now, secured on PyPI and our nightly channel.

the version numbers are different, even though the package names are the same. A stable version will have a version number such as `1.13.0`, where as a nightly version will have the date in the version number, such as `2.0.0.dev20221230`. You can check this either with `pip list | grep torch` or via `python -c "import torch; print(torch.__version__)"`

If you installed `torch` via the instructions to install the nightly version specifically, then you get the nightly version. By default, you get the stable version of `torch`.

ShardedTensor was merged / subsumed into DTensor: https://dev-discuss.pytorch.org/t/rfc-pytorch-distributedten...

Lots of development and traffic happening here: https://github.com/pytorch/tau/

It can run much faster. For example, using the PyTorch nvFuser JIT gives a 50% speedup:

https://old.reddit.com/r/MachineLearning/comments/xa75km/p_p...

counter-example: CPython, NumPy, SciPy and many critical well-used open-source projects that have tiny volunteer teams and operational budgets that are minuscule compared to something like React / PyTorch

Whether using Lua or Python, for GPU-based scientific computing, the need to drop into a C call is the same. The overhead of Python vs Lua never really mattered.

While we were based on top of LuaJIT, we couldn't use the JIT for anything because we had to always call into the C library for GPU kernels (and LuaJIT can't JIT through an opaque C call, incase that C call changes the interpreter stack).

Where Python really helps is with its ecosystem. The entire data science and ML ecosystem is in Python.

The difficulty of calling out to C is not much greater in Python, things like PyBind11 make it pretty natural.

We ship everything needed for userland -- including parts of CUDA/CuBLAS and CuDNN that we need (which is why our binaries are so fat).

GPU drivers would be kernel-land and I don't think we actually can install GPU drivers as part of a `pip install`. Will look into what Flux is doing, but I doubt they ship GPU drivers.

Separately, thanks for flagging the Poetry issue, we might prioritize it, especially if the fix is easy.

Author here. Being conscious about speed and performance is different from making that your competitive advantage or USP.

Our main focus is usability, and one of our secondary focuses is to not look like clowns in the performance department.

So, we try to take more decisions that trade off performance for usability than vice versa.

another good resource that's more specific to machine learning is https://paperswithcode.com/datasets

I've been helping on the project, it's lead by Chris Cummins and Hugh Leather.

Just a heads-up for folks, we haven't fully cleaned up and gotten ready for public attention yet, we are 90% there.

Once we are golden, we're going to write a note with a way to submit the results of your own agents, compare with baselines (random, actor-critic), etc.

And as others noticed, in it's current form we're focusing on code size and phase ordering, but we will be expanding over time to other optimization problems like runtime.

@m0zg if you could comment on the issue you quoted with any details, it would be really helpful to us. Unlike what is reported in the github issue, `permute` isn't the regression.

For reference, one of the core devs added more details based on where we are with our investigation: https://github.com/pytorch/pytorch/issues/37142#issuecomment...

PyTorch maintainer here: we're looking into that and if needed will issue 1.5.1 asap. It didn't show up in release testing, which among other things does end-to-end imagenet runs with ResNet50 and a few other models (i.e. time and memory didn't regress). Will also figure out how to catch this early.

notable co-releases along with PyTorch 1.5:

- TorchServe: model serving infrastructure for scalable model deployment

- TorchElastic w/Kubernetes: fault-tolerant "elastic" neural network training, allowing nodes to join and leave (for eg. to leverage spot pricing)

- Torch_XLA: updates for PyTorch TPU support

- New releases of torchvision, torchaudio and torchtext

Summary blogpost at https://pytorch.org/blog/pytorch-library-updates-new-model-s...

Hi I am a PyTorch maintainer.

NVIDIA has dropped CUDA support for macOS: http://www.cgchannel.com/2019/11/nvidia-drops-macos-support-...

This was pretty evident for a few years, and it's one of the top reasons for us to not provide official binaries with CUDA support -- the maintainer overhead was way too much. We did work to make sure it still builds with CUDA support from source (with a contbuild) but once CUDA 10.3 or 11 releases, we have to drop that too.

There's also fastapi, which is well-respected in Python. Someone wrapped fastapi / rabbitRPC to serve PyTorch models (with auto-batching to increase serving efficiency) in https://github.com/catalyst-team/reaction