I work on compilers. A friend of mine works on webapps. I've seen Cursor give him lots of useful code, but it's never been particularly useful on any of the code of mine that I've tried it on.

It seems very logical to me that there'd be orders of magnitude more training data for some domains than others, and that existing models' skill is not evenly distributed cross-domain.

This. Also across languages. For example, I suppose there is a lot more content in python and javascript than Apple script, for example. (And to be fair not a lot of the python suggestions I receive are actually mindblowing good)

i'm still patiently waiting for an easy way to point a model at some documentation, and make it actually use that.

My usecase is gdscript for godot games, and all the models i've tried so far use godot 2 stuff that's just not around anymore, even if you tell it to use godot 4 it gives way too much wrong output to be useful.

I wish i could just point it at the latest godot docs and have it give up to date answers. but seeing as that's still not a thing, i guess it's more complicated than i expect.

There's llms.txt [0], but it's not gaining much popularity.

My web framework of choice provides these [1], but they're not easily injected into the LLM context without much fuss. It would be a game changer if more LLM tools implemented them.

[0] https://llmstxt.org/ [1] https://svelte.dev/docs/llms

It's definitely a thing already. Look up "RAG" (Retrieval Augmented Generation). Most of the popular closed source companies offer RAG services via their APIs, and you can also do it with local llms using open-webui and probably many other local UIs.

Cursor can follow links

> I feel like I'm taking crazy pills when I read about others' experiences. Surely I am not alone?

You're not alone :-) I asked a very similar question about a month ago: https://news.ycombinator.com/item?id=42552653 and have continued researching since.

My takeaway was that autocomplete, boiler plate, and one-off scripts are the main use cases. To use an analogy, I think the code assistants are more like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim).

For me, only the one-off script (write-only code) use-case is useful. I've had the best results on this with Claude.

Emacs abbrevs/snippets (+ choice of language) virtually eliminate the boiler plate problem, so I don't have a use for assistants there.

For autocomplete, I find that LSP completion engines provide 95% of the value for 1% of the latency. Physically typing the code is a small % of my time/energy, so the value is more about getting the right names, argument order, and other fiddly details I may not remember exactly. But I find, that LSP-powered autocomplete and tooltips largely solve those challenges.

I think you make a very good point about your existing devenv. I recently turned off GitHub copilot after maybe 2 years of use — I didn’t realize how often I was using its completions over LSPs.

Quality of Life went up massively. LSPs and nvim-cmp have come a long way (although one of these days I’ll try blink.cmp)

> like an upgrade from handsaw to power tools and less like hiring a carpenter. (Which is not what the hype engine will claim).

I 100% agree with the not hiring a carpenter part but we need a better way to describe the improvement over just a handsaw. If you have domain knowledge, it can become an incredible design aid/partner. Here is a real world example as to how it is changing things for me.

I have a TreeTable component which I built 100% with LLM and when I need to update it, I just follow the instructions in this chat:

http://beta.gitsense.com/?chat=dd997ccd-5b37-4591-9200-b975f...

Right now, I am thinking about adding folders to organize chats, and here is the chat with DeepSeek for that feature:

http://beta.gitsense.com/?chat=3a94ce40-86f2-4e68-b5d7-88d33...

I'm thoroughly impressed as it suggested data structures and more for me to think about. And here I am asking it to review what was discussed to make the information easier to understand.

http://beta.gitsense.com/?chat=8c6bf5db-49a7-4511-990c-5e6ad...

All of this cost me less than a penny. I'm still waiting for my Anthropic API limit to reset and I'm going to ask Sonnet for feedback as well, and I figure that will cost me 5 cents.

I fully understand the not hiring a carpenter part, but I think what LLMs bring to the table is SO MUCH more than an upgrade to a power tool. If you know what you need and can clearly articulate it well enough, there really is no limit to what you can build with proper instructions, provided the solution is in its training data and you have a good enough BS detector.

> If you know what you need and can clearly articulate it well enough, there really is no limit to what you can build with proper instructions, provided the solution is in its training data and you have a good enough BS detector.

In other words: you must already know how to do what you are asking the LLM to do.

In other words: it may make sense if typing speed is your bottleneck and you are dealing with repetitive tasks that have well been solved many times (i.e., you want an advanced autocomplete).

This basically makes it useless for me. Typing speed is not a bottleneck, I automate or abstract away repetition, and I seek novel tasks that have not yet been well solved—or I just reuse those existing solutions (maybe even contributing to respective OSS projects).

The cases where something new is needed in areas that I don’t know well it completely failed me. NB: I never actually used it myself, I only gave into a suggestion by a friend (whom LLM reportedly helps) to use his LLM wrangling skills in a thorny case.

> In other words: you must already know how to do what you are asking the LLM to do.

Those that will benefit the most will be senior developers. They might not know the exact problem or language, but they should know enough to guide the LLM.

> In other words: it may make sense if typing speed is your bottleneck and you are dealing with repetitive tasks that have well been solved many times (i.e., you want an advanced autocomplete).

I definitely use a LLM as a typist and I love it. I've come to a point now where I mentally ask myself, "Will it take more time to do it myself or to explain it?" Another factor is cost, as you can rack up a bill pretty quickly with Claude Sonnet if you ask it to generate a lot of code.

But honestly, what I love about integrating LLM into my workflow is, I'm better able to capture and summarize my thought process. I've also found LLMs can better articulate my thoughts most of the time. If you know how to prompt a LLM, it almost feels like you are working with a knowledgeable colleague.

> I never actually used it myself, I only gave into a suggestion by a friend (whom LLM reportedly helps) to use his LLM wrangling skills in a thorny case.

LLMs are definitely not for everyone, but I personally cannot see myself coding without LLMs now. Just asking for variable name suggestions is pretty useful. Or describing something vague and having it properly articulate my thoughts is amazing. I think we like to believe what we do is rather unique, but I think a lot of things that we need to do have already been done. Whether it is in the training data is another thing, though.

> They might not know the exact problem or language, but they should know enough to guide the LLM.

I was in this exact situation. I worked with an unfamiliar area with a hardware SDK in C that I needed to rewrite for my runtime, or at least call its C functions from my runtime, or at least understand how the poorly written (but working) example SDK invocation works in C by commenting it. The LLMs failed to help with any of that, they produced code that was 1) incorrect (literally doing the opposite of what’s expected) and 2) full of obvious comments and missing implementetions (like “cleanup if needed” comment in the empty deinit function).

Later it turned out there is actually an SDK for my runtime, I just failed to find it at first, so the code the LLM could use or tell me about actually existed (just not very easy to find).

Those were two top LLMs as of December 2024. It left me unimpressed.

I don’t think I would be compelled to guide them, once I understood how the code works it is faster to just write it or read relevant reference.

My friend, who volunteered to waste those precious tokens to help with my project, does use chatbots a lot while coding, but he’s more of an intermediate than senior developer.

> Just asking for variable name suggestions is pretty useful.

I can’t see myself asking anyone, much less an LLM, for the name of a variable. I am known to ask about and/or look up, say, subject domain terminology that I then use when naming things, but to name things well you first need to have a full picture of what you are making. Our job is to have one…

> I wired one up

“One”? Wired up how? There is a huge difference between the best and worst. They aren’t fungible. Which one? How long ago? Did it even support FIM (fill in middle), or was it blindly guessing from the left side? Did the plugin even gather appropriate context from related files, or was it only looking at the current file?

If you try Copilot or Cursor today, you can experience what “the best” looks like, which gives you a benchmark to measure smaller, dumber models and plugins against. No, Copilot and Cursor are not available for emacs, as far as I know… but if you want to understand if a technology is useful, you don’t start with the worst version and judge from that. (Not saying emacs itself is the worst… just that without more context, my assumption is that whatever plugin you probably encountered was probably using a bottom tier model, and I doubt the plugin itself was helping that model do its best.)

There are some local code completion models that I think are perfectly fine, but I don’t know where you will draw the line on how good is good enough. If you can prove to yourself that the best models are good enough, then you can try out different local models and see if one of those works for you.

There is https://github.com/copilot-emacs/copilot.el that gets copilot to work in emacs via JS glue code and binaries provided by copilot.vim.

I hacked up a slim alternative localpilot.js layer that uses llama-server instead of the copilot API, so copilot.el can be used with local LLMs, but I find the copilot.el overlays kinda buggy... It'd probably be better to instead write a llamapilot.el for local LLMs from scratch for emacs.

Emacs has had multiple llm integration packages available for quite awhile (relative to the rise of llms). `gptel` supports multiple providers including anthropic, openai, ollama, etc.

https://github.com/karthink/gptel

there's avante.nvim

Yep - I don't get a ton of value out of autocompletions, but I get decent value from asking an LLM how they would approach more complex functions or features. I rarely get code back that I can copy/paste, but reading their output is something I can react to - whether it is good or bad, just having a starting point speeds up the design of new features vs. me burning time creating my first/worst draft. And that is the goal here, isn't it? To get some productivity gains?

So maybe it is just a difference in perspective? Even incorrect code and bad ideas can still be helpful. It is only useless if you expect them to hand you working code.

try cursor

> Is anyone actually getting value out of these models?

I've found incredible value in having LLMs help me write unit tests. The quality of the test code is far from perfect, but AI tooling - Claude Sonnet specifically - is good at coming up with reasonable unit test cases after I've written the code under test (sue me, TDD zealots). I probably have to fix 30% of the tests and expand the test cases, but I'd say it cuts the number if test code lines I author by more than 80%. This has decreased the friction so much, I've added Continuous Integration to small, years-old personal projects that had no tests before.

I've found lesser value with refactoring and adding code docs, but that's more of autocomplete++ using natural language rather than AST-derived code.

Honestly just disabled my TabNine plugin and have found that LSP server is good enough for 99% of what I do. I really don’t need hypothetical output suggested to me. I’m comfortable reading docs though so others may feel different.

i don't find value from the models that it makes economical sense to self-host. i do get value out of a llama 70b, for instance, though

I do not, but I suspect that's mainly because I far and away prefer statically typed languages with powerful LSP implementations.

Copilot, Ollama, and the others have all been strictly inferior to rust-analyzer. The suggested code is often just straight up invalid and takes just long enough to be annoying. Compare that to just typing '.'/'::' + a few characters to fuzzy-select what I'm looking for + enter.

ETA: Both did save me a few seconds here and there when obvious "repetition with a few tweaks each line" was involved, but to me that is not worth a monthly subscription or however much wall power my GPU was consuming.