jacobgorm

The people behind Toit are the original people behind the V8 VM in Chrome. They know their stuff.

Can you describe in detail what these powerful institutions are, and how they manage to hold back change?

The model architecture might be ten old, but it seems it was trained recently, so the title is a bit misleading.

Land of the free...

It should come with a flight attendant and an obligatory five minute safety instruction before each ride. I've ridden in multiple Teslas, including many Ubers, and no one has even given me the manual to read or shown me how to escape in case of a fire.

The register is hosted by US-owned DXC (formerly CSC.), who, apparently, did not know how to rate-limit the lookups.

"A little plague will be good for your natural immunity, by my favorite Danish team of crackpot researchers have just received $10M in express funding to test in African children whether the GCB vaccine helps, when we expose them to the bacterium"

Even this week following new mediocre PISA results for Danish pupils, I see organizations like the Danish Industry Association and the Danish Chamber of Commerce posting calls for more computers in classrooms, to combat the problems caused by computers in classrooms. They say kids just need to be tought better “technology understanding”.

I prefer to wirelessly charge my phone because the charging port is the part that typically wears out before the rest of the phone. My first EV's charging cable also wore about after around seven years of use, so I had to replace the entire charger. Charging port doors on some EVs also also known to develop faults in the locking motor, and the ports and/or its cover tends to fall of, which is is costly to repair as you need to repaint the new part. And then come the issues with having to choose EV models to suit the layout of your garage, as well as issues with the cable getting in the way when trying to pass the charging car with your bike, etc.

I cannot afford the Cayenne, but once this gets to cheaper models this will be an option I would be willing to pay for.

Neso is an ahead-of-time compiler backend for Triton. It lowers Triton TTIR/TTGIR into MSL or HLSL shader code, to allow modern LLM compute kernels such as Flash Attention II to run on GPUs in consumer devices such as Macs and Windows laptops. A companion project, NSpeech https://github.com/dropbox/nspeech, built in Rust with Candle, demonstrates how Neso can be used to power local ASR and TTS models.

I can program all my non-CUDA GPUs use completely open non-proprietary toolchains. And if that ceases to be the case on one platform I can switch platforms without having to rewrite all my code.

If that business makes it money selling a cross-platform AI inference engine, as was the case for my previous startup, it is bad business to tie yourself to single platform. I managed to build a single code base that would support CPUs, OpenCL, Metal, CUDA, D3D12, and WebGPU from a single set of kernel sources. As a single developer, there was no way I would have been able to, at the time, maintain separate code paths and GPU kernels for each of those platform, in addition to training the models etc.

It is currently faster than the stock Candle / MPS shaders it replaces on MacOS/ARM64, and IIRC a bit slower than OpenVINO/CPU on my old Windows laptop, where I never got OpenVINO/GPU to compute correctly. Candle didn't have support for GPUs on MacOS/Intel, and OpenVINO ceased to be supported there.

Compared to OpenVINO (I tried ONNX runtime too, but never got it produce correct outputs with my quantized models) it is very nice to be able to build the exact kernels I need, at the quantization settings and precision that works for the models I have and with the custom operators required (speech models do a lot of non-standard stuff), run from a single set of sources, and not have to ship a hefty third-party DLL, and having to deal with their memory leaks and other stability issues.

As it happens, I just got my employer's permission to release as open source a Triton back-end for Metal and D3D12 GPUs here: https://github.com/dropbox/neso .

As an example of you how can use it to deploy real models there is this project doing ASR and TTS: https://github.com/dropbox/nspeech .

Finally, I am also going to be switching the inferencing part of Witchcraft from current Candle on MacOS and OpenVINO on Windows to just Candle with Neso; https://github.com/dropbox/witchcraft

What reasons would you have to prefer Rust over C for compute kernels? I am a great fan of Rust, but I don't see any benefit for kernels, due to their relatively simple nature.

For learning that may be a fine approach, but CUDA (in C++) really tries to hide what is going on behind the scenes, which is roughly:

1) code gets split between a host part that goes through your normal compiler, and a device part that goes through the GPU compiler. You may as well write the kernels separate and compile them via a separate compilation step, and keep your trusted host compiler for the host-side code.

2) data needs to move between the host and devices via explicit buffer transfers and synchronization steps, CUDA tries to hide this with annotated pointers, but it is really easier to think about those as just buffers that you allocate and transfer IMO, instead of trying to transparently share pointers between host and device like CUDA does.

3) kernel launches can we wrapped in a function similar to:

void RunKernel(const char *kernel_name, size_t width, size_t height, size_t depth);

Instead of the funky <<< >>> syntax that CUDA for C/C++ imposes. The problem is that once you start putting that in your code, it stops being C++ and stops being portable to non-CUDA GPUs. The launching and grid settings can be a bit hard to grasp at first, but sugarcoating that in bastardized C++ syntax does not absolve from having to understand it eventually.

So a good place to start might be an OpenCL or Metal primer, depending on the hardware you have available. D3D12 (and probably Vulcan too) makes this much harder than it should be, with too much boilerplate but is overall a mature and well-designed API should you wish to develop for Windows. Starting with WebGPU might also be good these days. It has a very different shader language than the others, but the rest of the concepts are similar, and it has a strong emphasis on making things async, which is what you want for performance anyways.

Claude/Codex should be able to get you moving very quickly.

Because I know it is not technically crap, a lot of competent people worked on it, most with good intentions. I suppose it is better described as a cleverly designed Trojan horse than can infect your software and make that software become crap, in the sense that it becomes harder to maintain, increases code duplication, messes with your build system, ties your build system to platforms that have their toolchain binaries available, etc., etc., without bringing any long-term benefits over learning things the hard way.

I haven't felt this popular since then 1990s when I was opposing Visual J++ and IIS.

You can call their APIs without needing to compile your code with a proprietary compiler or adopt a bastardized version of C++.

No. CUDA allows you to write all the code in a single file, and uses a preprocessor to split it back out and pass it through separate compilers, one for host and one for device.

CUDA is not an API, CUDA is a language, so you cannot make that comparison.

Nobody cares if kernels are written in Rust. Kernels were meant to be written in C, but if you want to go more high-level try Triton or a similar DSL that nicely abstract tile sizes etc.

I strongly dislike CUDA. Once you have allowed that proprietary cr*p into your C++ codebase, it is very hard to get rid, and you end up with code that is either tied to a single vendor or an #ifdef hell, probably both.

The best way to program GPUs is face up to the reality that they are not the same machine as the CPU, write your kernels in separate files, and launch them manually, like in Metal, OpenCL, and D3D12, etc. These days we even have DSLs like Triton that make kernel writing much more ergonomic than anything you would hope to achieve in Rust.

I invented VM live migration to solve the problems with traditional process migration, and my work was inspired by both Sprite and MOSIX. I believe that VM migration is in wide use today, especially at hyperscalers. Mike Nelson, who worked on Sprite, invented VM live migration in parallel with me, but secretly within VMware.

I've been on Mastodon for a few years, and though it has less drama than Twitter/X, it does seem to have reached critical mass of users and does cover most of my needs for random gossip. I am surprised that that nobody has mentioned it here.

You were lucky to get out alive.

You can always hide in your own country, as long as you or the people who protect you are able to stay in power, but at least you cannot use the rest of the world as playground or shopping center. Sucks to be a rich kleptocrat and not be able to have a yacht in Nice or be able to take your wife shopping in London.

There is no memory overhead, in that everything resides on disk/in the sqlite database, and is read from there on demand. That said, sqlite for blob storage is a bit inefficient, so the next version will use tightly packed structs on disk. I also managed to finetune an embedding down to 96d, so in the coming version the storage for the index is getting close to parity with the size of the text being indexed.

Here is a project with similar goals (I am the main author), but using late interaction semantic search: https://github.com/dropbox/witchcraft