wayy

I came across this PR with many different systems kicking off AI reviews, comments, and changes. You can also see human devs chiming in. This is the future of how people will work, with more and more of the tasks handled by various AI. Fascinating.

It seemed to me the author was simply sharing his own lived experience, which happens to be a bit contrarian to the popular hype around LLMs. It may seem courageous for some but I can see a world where the author didn't think twice about writing down his thoughts in 15 minutes and publishing on his own personal site. Perhaps it comes naturally to people who have been around this industry longer

everybody loves building agents, nobody likes debugging them. agents hit the classic llm app lifecycle problem: at first it feels magical. it nails the first few tasks, doing things you didn’t even think were possible. you get excited, start pushing it further. you run it and then it fails on step 17, then 41, then step 9.

now you can’t reproduce it because it’s probabilistic. each step takes half a second, so you sit there for 10–20 minutes just waiting for a chance to see what went wrong

Thanks for pointing this out - we didn't consider how important stackoverflow comments could be. Will look into including as a source.

Ah we only put the option for detailed comment on negative feedback (try clicking the x and a form will pop up). Will also give that option for positive feedback in the future.

Thanks for trying it out!

We can definitely be clearer on our focus for programming related queries. We usually don't display code snippets for non-programming questions but definitely still tuning a couple things there.

We're not focused on simple factoid answers like population of cities because that's not where people get the most value.

AWS API is a bit tricky because it is a rather broad technology with SDKs in different languages and thus the search results for a question will return a mishmash of solutions which we then try to make sense of. If you share some sample queries you tried related to this, I'd be happy to look into them and improve our answers there.

Business model: We're currently just focused on building something developers want. Agree that ads and dev tools aren't the most synergetic.

The answer is based on the contents of the websites returned by the search engine + other some sources.

What is the correct/expected answer to that query?

We think the solution is simply having good sources and answer transparency. If you mouse over part of the answer, we try to show you the source of that sentence. Obviously this system is early and will improve over time, but if can easily check if an answer is from say, the FastAPI official documentation, then the false-confidence effect of these models become less of an issue.

You're right - there are so many good sources not often surfaced by a cursory web search like books and experts. Adding better sources is something we're improving on. A simple next step we're looking into is to expand sources to research papers e.g explicitly pulling from arxiv on some queries.

Interesting, we are trying to wrangle with the nondeterminism but sometimes it can't be helped as Bing itself can produce different results. Always actively working on the model though.

The citing sources feature can definitely be improved - right now it works at a sentence granularity and insists on finding the best source when sometimes not appropriate. Thanks for pointing all of this out

If you can't find the information yourself via a web search, it's going to be difficult for the model to find the answer too... for now ;)

You can leave feedback on individual search results via the check and X buttons below the answer. Leaving feedback e.g votes on each code snippet is on the roadmap.

Exactly, we use some information from traditional search engines as input to our model.

Thanks for trying it out - we're still quite early so the model isn't going to be perfect + we're focused on programming related queries at the moment. What is your use case? Are most of your searches related to history?

The answers are explicitly based on information found on the web instead of some knowledge found within the model parameters.

yep, it's super interesting to watch how cheap and plentiful compute starts to change the way we live

We're always actively exploring different prompting - a general question answering prompt most definitely won't work for all searches. Finding what people agree on and disagree about in e.g a reddit thread could be very useful.

The p50 time is around 2s end to end not under load. Bear with us as we scale our systems!

If you mouseover the answer paragraph, you'll see the relevant url / search result. You can click to go to that source.

Thanks for pointing this out - determining when code snippets are relevant is something we're actively working on.

Thank you so much for writing this up - this is extremely useful feedback. What you've described here is a use-case we're aware of called "recalling forgotten details," where "the developer knows what she’s doing, but doesn’t remember the specific syntax or naming. With these searches, a quick code example is what people were often after." [0][1]

Hello primarily does well for "how-to" questions at the moment; it's still early and we're working to improve results for the queries you've described.

[0] https://static.googleusercontent.com/media/research.google.c... [1] https://bootcamp.uxdesign.cc/the-hidden-insights-in-develope...

Thanks for the perspective - we're working on improving our functionality for troubleshooting.

Thanks for the feedback. I agree that the animations can get old if you're using the search every day. We'll look into simplifying.

Thanks for trying it out! We've gotten some solid feedback about the UI and will be working to improve it. Could you tell us a bit more about what you were trying to do and how it was confusing for complex queries?

We empirically find that most developer searches fall into a few categories consistent with this paper [0]. The author explains these categories nicely in her blog post [1]. It sounds like you're describing 4/6 of these categories: ad-hoc how-tos, understanding an API / recalling forgotten details, and troubleshooting.

We currently do well at answering "ad hoc how-to" questions and are working to improve our answers in the other categories. This could either look like augmenting our existing natural language answer, or building a separate view specializing in official documentation or errors.

[0] https://elisehe.in/i/googledrivendevelopment.pdf [1] https://bootcamp.uxdesign.cc/the-hidden-insights-in-develope...