antinucleon

AVO’s paper author (ex-NVIDIAN) is here. This work was done half a year ago for GPU kernels, and the same approach has now been applied to ARC-AGI-3. I think people are still underestimating the evolution progress; e.g., recently we made a self-improving evolution harness that generated an entire inference stack and is better than SGLang/vLLM on various tasks: https://int21.ai/insights/addressing-the-inference-bottlenec...

Yes

Mojo is trying to create a new language to solve the problem, and specialized for CPU. We are using a more pragmatic way to solve GPU AI computation problem.

We haven't compared yet.

We developed AITemplate majorly for Meta's focus at that time, eg Ads/Ranking need. For HippoML is startup we are building for Generative AI. HippoML is not using AITemplate.

It is actually non-trivial to get GPU run fast, especially on SoC with strong CPU like M2.

Hippo is faster than AITemplate, and supports more generative models. We haven't compared vs TVM, but for absolute token/s on M2 Max, Hippo is able to running decoding on LLAMA with datacenter level GPUs (with other SW).

We will disclose more details very soon.

Yes. We support >= 1bit <= 16bit models out of box for various of models.

AITemplate's original designer is here. We quit Meta in January and start HippoML (https://hippoml.com/). We just disclosed our new engine's performance on LLM: https://blog.hippoml.com/large-language-model-inference-from... On Apple M2 Max our new engine encode/decode is 13.8X/2.4X faster than llama.cpp

CuDNN v7 was used in the experiments, in the experiments parts each comparison was listed with version or commit number.

Summary: Tensor program is able to be optimized by using machine learning and transfer learning. The numerical program optimization model is trained on feature from low-level AST of the program. Experiments: Tasks: ResNet, MobileNet, LSTM LM, DQN Hardware: CUDA/ARM GPU/ARM CPU Speed up compare to CUDNN, TensorFlow Lite and ARMComputeLib: ~from 1.2X to 3.8X faster in end-to-end test.

The version tested in paper should not have P2P,so it was much slower than current version.

There is a huge distributed performance advantages vs TensorFlow. You can get a hint from Prof. Carlos Guestrin's keynote talk at Data Science Summit 2016. Also, CMU CS Dean Andrew Moore cited MXNet as "is the most scalable framework for deep learning I have seen"

The author is here :) Thanks for your suggestion. The detail of network modification will be released in my thesis, there is a "fast good net" but currently I don't have time to make it faster, so I simply use fast poor net.