raviisoccupied
Very true.
Show HN: Beval – Simple evaluations for your AI product
I have been working on a web app called Beval - Simple evaluations for your AI product.In my day to day as a Product Manager working in a team that ships AI products, I often found myself wanting to do 'quick and dirty' LLM-based evaluation on conversation transcripts and traces. I didn't need anything fancy, just 'did the agent answer the question', 'did the agent cover the 5 things it needed...
I feel like PMs coding unlocks a whole new category of work, mainly addressing the long tail of cool ideas/small optimisations that ordinarily would not be addressed. Time will tell how valuable these items are in the long term.
And I say this as a PM.
In my day to day as a Product Manager working in a team that ships AI products, I often found myself wanting to do 'quick and dirty' LLM based evaluation on conversation transcripts and traces. I found myself blocked by 'Gemini in Google Sheets', it was too slow and cumbersome, and it didn't handle eval changes well. And because I was exploring, it wasn't helpful to try and set up something more robust with the team.
To fix the problem I eventually learned to call the OpenAI API in python, but I really felt that I wanted a 'product' to help me and potentially help others.
So this weekend I built https://beval.space
Personally I find this kind of approach infantilizing. I don't think it's fair to have an expectation of yourself or of your colleagues to spell out in Google Calendar what you/they are spending their time doing throughout the day. People should look at gaps in people's diaries not as opportunities for new meetings, but as time someone is probably spending doing something important or interesting. And I say this as a PM.
Classify all the tasks with an impact / ease score from 0-5. Multiply the numbers together to understand the relative priority of each item.
It won't be perfect but it will a good place to start.
You also do this at a product level instead, and invest all your time onto one product to begin with. This might reduce your switching costs working between projects/products.
For now, I’m just sticking to using it for daily notes, but I feel there’s so much I’m missing.
UX feedback: When I scroll to the bottom of the page linked in the post, and click on the API/Pricing links in the footer, it fails to redirect me.
First Therapy Chatbot Trial Yields Mental Health Benefits
Mark Zuckerberg messages Facebook engineer
The Respell team is joining Salesforce
The Cognitive Style of PowerPoint [pdf]
Mexico is using an AI-powered app to prevent suicides
Send us a confidential tip using SecureDrop
A brief history of Notion's data catalog
A new mode in Halide that skips over standard iPhone image processing
That's interesting. If the use case is meant to be during one-off or exceptionally busy periods, do you anticipate team members having these on their desks the whole year round?
Scarlett Johansson's AI row has echoes of Silicon Valley's bad old days
This is interesting. I find it's very difficult to predict which links do and don't perform on Hacker News, resulting in quite a diverse (to me at least) series of stories. Thanks for the analysis.
NHS England published heavily redacted Palantir contract as festivities began
I know HN is can sometimes come across as 'management-phobic', and having been on that side of the fence there is a lot value that can be added by managers.
That said, I do wonder how many of these managers who have been asked to become ICs might actually be relieved. In an increasingly competitive economic environment, some could see this as a good thing.