> * Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research.

Did you find them to be correct?

I don't use Phind for coding, except occasionally, but I like it best for generalised tech search because each para has a reference and there's a list of references down the side -- often the references would really be sufficient for me on their own.

I've had one glaring error, I can't quite remember the details, but it switched the names/characteristics of two different processes (ie was exactly opposite in what it said); it was something to do with instruction caching and TLB, IIRC. I assumed you'd was a problem with the input corpus not allowing antonyms to be disambiguated.

Anyway, for me it's the best of the LLM tools I have access to and had mostly replaced search engine (Google, Dukgo) for my tech-related work.

I've only used chat.openai.com (free), bing chat, HuggingChat.

I don't think "correct" is the right word since these were open ended systems design type questions. There are many ways to accomplish the same task.

I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it.

In my experience, the sources were sufficiently relevant based on its responses. They were about as relevant as equivalent Google queries. Some tiny, tiny niggles, like I was explicit I wanted it to recommend approaches in Go and for one reference I recall related to distributed locking mechanisms it provided a reference to an implementation in Java. However, that is completely fine for me since the context was more about the locking on the database side and not really the implementation in a specific language.

And the sources actually existed? i.e. there weren't any made-up ones?

The sources are urls to the cited page (e.g. stackoverflow.com, pkg.go.dev). In the side-bar next to the answer is a more standard search-result style link list with pulled quotes from the pages (like a Google search).

I didn't click every single link (as I mentioned, the citations are copious) but the few I did follow went to relevant articles. I just went back and randomly clicked several more and they all went to pages that exist and mostly relate to the content of the answer. The inline citations seem a bit more on-topic compared to the side bar which does seem more like the links were lifted directly from a search engine.

To be fair there are some lower-quality blog-spammy kinda stuff - more or less the same kind of thing you would get out of Google. But compared to GPT-4, which provides no sources whatsoever, it is an advantage IMO.

mind providing some of the prompts you use to question them?

Here are the conversation logs:

https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599...

https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg

NOTE: there are a few more question/answer blocks in the phind conversation since I was testing out the follow up question feature.

Do you have custom instructions? Everyone needs to mention and post prompts else entirely antidotal

We support custom instructions at https://phind.com/profile.

I’m trying to get it to answer only in executable Python. I used the template with instructions I use for my system prompt on gpt4. And I tried using the additional context field for the same.

It gets to writing the expected code but it still wants to include formatted headings instead of commenting those out so the entire response is executable Python.

As a follow up I provided an example heading with the hash out front. It didn’t work.

Any ideas on how to get it to this? Fwiw, gpt4 often if ignores this request, but only about half the time. When it does it is typically a single block of explanatory text.

For that, I include prose detection and commenting as part of my post processing.

Also, I don’t see it easily, but do you have an API for this or is it intended to be run by the user?

Getting it to not output additional text is not something that it can do super well at the moment, unfortunately. We'll work on that.

My trick for this has been one-shot training + regex. I tell the model to produce executable code within triple backticks suffixed by a keyword, like:

```keyword // code ```

and then I just ignore anything outside of those blocks.

I did not have custom instructions for either assistant. You can see the full conversation logs which I posted as a reply to another comment.

The “give context” part has a lot to do with prompting well based on the model. To have a fair comparison there should be just code and see what that come up with

would you be able to share your prompt(s)?

Edit: they are already posted as comment in this thread.

This is a good point about GPT-4, it can intuit the "question behind the question" really well compared with other models. And it's been profoundly useful for me with the most random tasks I knew nothing about prior (like fixing a wall in my house), etc.

That's probably because OpenAi can train on the (succesfull) conversations we have with ChatGPT!