xml
- LLM removes/disables/weakens tests (disallowing manipulation of tests is not really possible in Python since the language is too dynamic, so the entire execution has to be sandboxed, which makes timing more difficult)
- LLM mutates input, which might throw off some tests (for example, sorting an array where all values have been set to zero is easy), but this can easily be solved by copying the input to somewhere safe or regenerating it from a fixed random seed.
- LLM writes code that only passes the test cases and nothing else, often with a new special case inserted after every failed test. Randomizing everything seems to be a good defense, although it is not always easy to know beforehand what to randomize. Tensor shapes are obvious, but randomizing data distribution to prevent circumvention via precision downgrades is difficult.
And regarding "10. Baseline Kernel" from the article; I've had LLMs call __import__ or compile and obfuscate the code in order to circumvent tests. The proposed defense of static analysis is not quite sufficient here.
I can relate to all points mentioned in the article. They really do happen in practice, and many are also applicable to test-driven development with LLMs. Is there any benchmark to evaluate whether an agent solves a task "in the spirit of the prompt" instead of simply solving it to pass tests?
The fact that all big SoTA models use MoE is certainly a strong reason. They are more difficult to train, but the efficiency gains seem to be worth it.
> Every thing I know about them makes me believe they're a dead-end architecturally.
Something better will come around eventually, but I do not think that we need much change in architecture to achieve consumer-grade AI. Someone just has to come up with the right loss function for training, then one of the major research labs has to train a large model with it and we are set.
I just checked Google Scholar for a paper with a title like "Temporally Persistent Mixture of Experts" and could not find it yet, but the idea seems straightforward, so it will probably show up soon.
I also believe that it should eventually be possible to train a model with somewhat persistent mixture of experts, so you only have to load different experts every few tokens. This will enable streaming experts from NVMe SSDs, so you can run state of the art models at interactive speeds with very little VRAM as long as they fit on your disk.
I wonder if getting rid of those pop-ups would have slowed Stack Overflow's decline. The number of new posts has dropped by 98 % (198540 posts in 2020 down to 3097 in January: https://data.stackexchange.com/stackoverflow/query/1926661#g...)
There are of course power tools with obnoxious protections that make them difficult to use, but since we are dealing with software here, we are not bound by the laws of physics. I believe that we can create a better tool that is both powerful and easy to use.
Git is better than what came before, and it might be the best at what it does, but that does not mean that it is good.
- The interface is unintuitive.
- Jargon is everywhere.
- Feature discoverability is bad.
- Once something goes wrong, it is often more difficult to recover. If you're not familiar enough with Git to get yourself into that situation, then you certainly aren't familiar enough to get yourself out of it.
Many of those issues are due to git being a command line interface, but others (like no general undo and funny names) are simply due to bad design.
I think it is about time that we try again and build a better version control tool, but maybe git is just too entrenched.
> this code is not copyright protected, therefore you are not allowed to apply a MIT LICENSE to this project.
(Not sure if claiming copyright without having it has any legal consequences though.)
> Specifically, we collected new data created after January 2025, including: [...] new fiction on Archive of Our Own (Various, 2025),
https://huggingface.co/datasets/nyuuzyou/archiveofourown/dis...
https://github.com/zai-org/GLM-OCR
(Shameless plug: I also maintain a simplified version of GLM-OCR without dependency on the transformers library, which makes it much easier to install: https://github.com/99991/Simple-GLM-OCR/)
[1] https://github.com/99991/safeserialize
Side note: I think that a warning in the README about arbitrary code execution for deserialization of untrusted inputs would be nice.
Here is an evil example SVG for demonstration.
DON'T CLICK THIS LINK UNLESS YOU WANT TO RISK CRASHING YOUR BROWSER!
https://github.com/99991/SimpleTinyLlama
The new checkpoints did not seem much better and they changed the chat format for some reason, so I did not port the new checkpoints yet. Perhaps I'll get to it this weekend.
Here are a few ideas to make yours faster:
1. Use the browser's canvas element to compress JPEG or PNG images instead of WASM. The libraries which come with browsers can use processor-specific instructions, while WASM only targets the smallest common denominator of the most common architectures, so it will be slower.
2. Which resizing method are you using to find a fitting image size? For me, it worked well enough to downscale by some factor a few times until it fits, but if you want to get super close, you could use binary search.
For example, you could post one of those SVGs in every issue thread of a GitHub project if you wanted to mess with someone.
Not eligible for a bug bounty though since this issue has been known (but not fixed) for years.