News for hackers. Stuff that matters.

Joschkabraun

Joschkabraun on Hacker News

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use on Monday April 08, 2024 @02:19PM (#39972379)
What do you mean with being a bad model?

If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use on Monday April 08, 2024 @02:14PM (#39972325)
Ah yes. Have you tried out instructor [0] or Guidance [1]?

[0]: https://github.com/jxnl/instructor/

[1]: https://github.com/guidance-ai/guidance/tree/main

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use on Monday April 08, 2024 @02:08PM (#39972258)
Interesting. Do you have any benchmarks?
Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use on Monday April 08, 2024 @02:06PM (#39972238)
All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition
Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use on Monday April 08, 2024 @02:05PM (#39972229)
Have you tried the new beta tool use API? In the experiments I ran there were almost no issues parsing the function call response (similar to GPT-3.5-turbo & GPT-4 turbo)
Re:Open-source tools for LLM monitoring and observability on Thursday February 08, 2024 @04:38PM (#39308091)
https://docs.parea.ai/observability/logging_and_tracing isn't open-source for LLM monitoring but the evaluation metrics to assess LLM app quality are: https://github.com/parea-ai/parea-sdk-py
Re:Ask HN: How are you testing your LLM applications? on Thursday February 08, 2024 @04:33PM (#39308031)
Co-founder of Parea here, thanks for the mention!

We offer testing/evaluation in development ([1], [2]), and production ([3]). You can use pre-built evals ([4]) or create your own ([5]). Do logging ([6]) and go from trace to playground ([7]). For TypeScript & Python.

[1]: https://docs.parea.ai/evaluation/offline/experiments

[2]: https://docs.parea.ai/platform/test_hub/benchmarking

[3]: https://docs.parea.ai/evaluation/evals-in-trace

[4]: https://docs.parea.ai/evaluation/list_of_autoeval_functions

[5]: https://docs.parea.ai/evaluation/evaluation-functions/create

[6]: https://docs.parea.ai/observability/logging_and_tracing

[7]: https://docs.parea.ai/observability/open_trace_in_playground

Show HN: Reference-free evaluation of LLM-powered chatbots

Posted by Joschkabraun on Monday December 04, 2023 @12:02PM
from the show-hn dept.

https://github.com/parea-ai/parea-sdk-py/blob/main/parea/cookbook/tracing_and_evaluating_openai_endpoint.py

Hey HN!This an interactive demo with a *somewhat* helpful AI assistant. The goal is to demonstrate a good way to reference-free evaluate interactions between humans and AI assistants. Reference-free means that you do not provide a correct answer to a query. The used metric in this context is the goal success ratio, which measures how many queries a user needs to send to reach their goal.In the...

( Read More... | no comments | on HN )
Re:The first playground to experiment with the GPT-4 Vision API on Thursday November 09, 2023 @03:11PM (#38210502)
Happy to launch the first LLM/LMM prompt playground which allows experimenting with the GPT-4 Vision API! Curious to hear what you guys are thinking
Re:When Gradient Descent Is a Kernel Method on Saturday October 28, 2023 @06:30AM (#38048666)
Reminds me of "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine" by Pedro Domingos: https://arxiv.org/abs/2012.00152
Re:How has the ChatGPT model changed from March to June? on Thursday August 03, 2023 @01:17PM (#36988773)
Yes, GPT-4 is in general much more tuned to respect the system message.