Github Copilot seems to be the most effective code assistant currently. It seems to use many heuristics for figuring out relevant snippets to include in prompts, like computing Jaccard similarity of windows of the last 20 opened files. It also tries some tree-sitter cleverness for some languages, but when I snoop on the HTTP traffic it seems to almost always just give up and only include the rest of a file as context.

I have wondered whether a model fine-tuned on my own code would do much better, and be simpler. But perhaps building embeddings and searching them (like in the article you linked) would be superior.

Code assistants need to be super low latency though which maybe complicates things too.

I honestly think that if you could have all your private code indexed and accessible, this would be a game changer as it has way better context.

My open questions though are:

- does the tokenizer for the index need to be the same as the one used by the LLM?

- is running a local LLM feasible, or should I just assume prompts must sent over the network? This would affect latency and privacy a lot.

- is fine tuning helpful? Or necessary, even, to ensure the LLM only provides source code, and only in the target language?

Also possible when using the Vim copilot plugin (or its emacs port) which is pretty informative - you can see a lot of the extra tricks the VSCode implementation uses.

Does anybody know if this is already available on Hugging Face or somewhere else?

I'm using TypeScript, so my idea would be to finetune on all used packages, maybe even all of the documentation as well.

I've created a small app where I create a prompt for an eslint error and let ChatGPT come up with a git diff. I manually feed that back to my application, it eslints and compiles and if there are any errors, a new prompt is generated. This works quite well.

GPT-4 is not available for me in the OpenAI API, so it's quite cumbersome.

*The merged model contains the LoRA. Applying the LoRA over a "raw" llama model uses more VRAM and does not allow for full context in 24GB of VRAM.

I'm also super keen for the 32k API limit for OpenAI, that's going to be great.

It seems pretty expensive because you may be pasting a lot of context into each query. If you allow the user to have follow up queries and you want to retain context of the conversation that seems expensive too. But it does seem like it should give the best results for Q&A as long as question is directly answered in your data somewhere.

Ideally, we include documents that are relevant to the topics at hand and eliminate documents that aren't. Using prompt chaining, or an iterative process of labeling and extracting features, we may be able to increase the efficiency of the final prompt.

Obviously this increases costs, but so does repeated fine-tuning.

But, as far as I know, OpenAI will not help you fine-tune, they will not run a fine-tuned model for you, and they would probably prefer that you use their models over open models that can be fine tuned.

(None of this is to say that fine tuning is better. I’m just saying that OpenAI has a strong commercial bias.)

I like how the fine-tuning page talks about how fine tuning is supposedly better, though.

Although a lot of people are working on hallucination reduction, I think we're a long way off from that in the general case. So having the ability to point to a real piece of data, outside the model, is important for applications where accuracy matters.

https://community.openai.com/t/fine-tuning-myths-openai-docu...

Bottom line is that fine-tuning does not seem to be a feasible option for adding new knowledge to a model for question answering.

https://adapterhub.ml/blog/2022/03/adapter-transformers-v3-u...

https://arxiv.org/abs/2303.16199

https://youtu.be/9qq6HTr7Ocw

"OpenAI Q&A: Finetuning GPT-3 vs. Semantic Search - which to use, when, and why?"

Supabase Clippy was the first docs site to ship this experience to production as far as I can tell: https://supabase.com/blog/chatgpt-supabase-docs

I believe they called it "context injection" and I have been following suit in my own writing on the topic.

I am prototyping experiences like Supabase Clippy and am also very interested in fine-tuning for docs Q&A. But my main question is: what exactly would the fine-tuning inputs and outputs look like for docs Q&A?

Edit: For Q&A the question is the input and the answer is the desired output? Is that right?

A more general comment about fine-tuning for docs from my blog:

> AI is all about prediction. Given this temperature, this wind, this day of the year, what is the chance of rain? Temperature, wind, and date are your inputs. Chance of rain is your desired output. Now, try to apply this same type of thinking towards documentation. What are your inputs? What’s your output? The page title and code block could be your inputs. Whether or not the code builds could be your output. Or maybe the code block should be the output? This is why I keep saying that applying fine-tuning to docs is tricky. What are the inputs and outputs?

https://technicalwriting.tools/posts/ten-principles-response...

(I am an AI n00b and have not looked deeply into how fine-tuning works but it's high on my list to experiment with OpenAI's fine-tuning API. Please LMK if I am getting any fundamentals wrong.)

https://help.openai.com/en/articles/7127982-can-i-fine-tune-...

Maybe more correct to call it "CLI tool" rather than API but close enough

If you want exact detailed recall, then using a framework that provides search and recall (by embeddings or otherwise) is probably always going to beat fine tuning, but also remember, it doesn’t have to be either-or.

I mean, if you want a person to handle Q&A on a corpus, is it better for them to have studied it, or have direct access to the corpus with an appropriate index? The answer is clearly that its better if they’ve trained and have access to the corpus, and while LLMs aren’t the same as people, I think the answer for them is the same here.