frontsideair

I was able to confirm that it actually runs on ANE, I'm impressed.

> Apple locked it behind Siri. apfel sets it free

This doesn't feel truthful, it sounds like this tool is a hack that unlocks something. If I understand it correctly, it's using the same FoundationModels framework that powers Apple Intelligence, but for CLI and OpenAI compatible REST endpoint. Which is fine, just the marketing goes hard a bit.

> Runs on Neural Engine

Also unsure if this runs on ANE, when I tried Apple Intelligence I saw that it ran on the GPU (Metal).

Yeah, the initial experience with no colors doesn’t look great. I can implement this when I have some free time, if you feel like doing it please feel free to open a PR. Thanks!

14B Qwen was a good choice, but it became outdated a bit and seems like the new version of 4B surpassed it in benchmarks somehow.

It's a balancing game, how slow a token generation speed can you tolerate? Would you rather get an answer quick, or wait for a few seconds (or sometimes minutes) for reasoning?

For quick answers, Gemma 3 12B is still good. GPT-OSS 20B is pretty quick when reasoning is set to low, which usually doesn't think longer than one sentence. I haven't gotten much use out of Qwen3 4B Thinking (2507) but at least it's fast while reasoning.

Ollama adding a paid cloud version made me postpone this post for a few weeks at least. I don't object them to make money, but it was hard to recommend a tool for local usage and make the first instruction to go to settings and enable airplane mode.

Luckily llama.cpp has come a long way and was at a point that I could easily recommend as the open source option instead.

This is the command probably:

  sudo sysctl iogpu.wired_limit_mb=184320

Source: https://github.com/ggml-org/llama.cpp/discussions/15396

I'm interested in this, my impression was that the newer chips have unified memory and high memory bandwidth. Do you do inference on the CPU or the external GPU?

Good point, let me add a quick note.

Thank you, it was the integral part of the whole post!

According to the benchmarks, this one is improved in every one of them compared to the previous version, some better than 30B-A3B. Definitely worth a try, it’ll easily fit into memory and token generation speed will be pleasantly fast.

This is the first time I’m hearing about Nebula. How does it compare to Tailscale?

I saw from the comments that they provide smaller binaries, which may have worked on my Raspberry Pi. Maybe I’ll give it a try one day.

https://tailscale.com/kb/1207/small-tailscale

Thanks, this could’ve worked for my Raspberry Pi! I would’ve tried it if it was still in commission. Next time?

That’s exactly why I put the spend limit, and this was the first time to confirm that it actually works.

Exactly, I had a spend limit since I didn’t want to break the bank. It’s back up now.

Synthetic benchmarks should be retired. Build a real app, measure real impact. Unless your users insert 1000 rows, of course.

Other than that, I agree with the general premise of React becoming the new IBM, that's totally fair.

This looks like a treasure trove that I somehow missed! I'll give it a more thorough read later, but gave my star for now, thanks!

I'm really interested in `home-manager` honestly, and while I agree about the benefits of declarative package management, I find it easier to type commands at a terminal like `nix profile upgrade pkg` or `nix registry pin nixpkgs`, with the guarantee of rollbacks.

Since when I'm editing a text file to update a package, I have to look for the latest version separately and copy/paste it into my editor. If I somehow mess it up the file is broken, while no harm is done I still find this workflow a bit brittle.

If there are home-manager commands I missed to do this, I'd be eager to give it a try. That kind of workflow would remind me of running `npm upgrade pkg` and have it reflected on a `package.json` file.