News for hackers. Stuff that matters.

jellyberg

Re:Compare how Gemini Pro and GPT 3.5 answer the same questions on Thursday December 14, 2023 @05:29PM (#38648253)
Ah thank you, I missed this!
Re:Compare how Gemini Pro and GPT 3.5 answer the same questions on Thursday December 14, 2023 @02:31PM (#38646091)
There are four examples in the first category (imagination), there are five more categories with lots of other examples.

Currently, there's no API for Gemini Pro, so you need to use bard.google.com to ask it questions.

Re:Compare how GPT-2, 3, 3.5 and 4 answer the same questions on Tuesday October 31, 2023 @05:47PM (#38091865)
Yeah - worth noting that we use temperature=0 for reproducibility while ChatGPT I think uses t=0.7. We also prefix the prompt with few-shot examples of questions and answers with chain of thought examples to elicit the models' full capabilities.
Re:Compare how GPT-2, 3, 3.5 and 4 answer the same questions on Tuesday October 31, 2023 @04:24PM (#38090844)
Yeah it's hard to compare across models, interested in suggestions here.

We give all models a bunch of few-shot examples, which improves GPT-3 (davinci)'s question answering substantially. GPT-2 sometimes generates something that answers the question, sometimes it's just confused. Click "See full prompt" to see the few-shot examples that the models get.

Our goal was to exercise the full capabilities of each model.

Show HN: Fatebook – the fastest way to make and track predictions

https://fatebook.io/

I see forecasting as anti-bullshit technology:- It gives you truthseeking incentives- You communicate your uncertainty as a probability, which is way clearer (70% is better than “probably”)- You can aggregate forecasts to get wisdom of the crowd effects- You can build a track record, calibrate your uncertainty, and pay more attention to people who are consistently accurateI’m a fan of...

( Read More... | no comments | 2 points | on HN )
Re:Show HN: Fatebook – Superforecasting in your Slack on Tuesday June 13, 2023 @10:00AM (#36310347)
Some examples of different kinds of resolution criteria for forecasting questions:

Personal mechanisms

“If we choose this HR provider, will I think it was a good idea in two month’s time?”

“Each day I’ll write down whether I want to leave or stay in my job. After 2 months, will I have chosen ‘leave’ on >30 days?”

“Will volunteering abroad make me all-things-considered happier?”

“Will I judge that AI was a major topic of debate in the US election?”

Objective proxies

“Will our user satisfaction rating exceed 8.0/10?”

“Will Our World in Data report that >5% of global deaths are due to air pollution by 2030?”

“Will I still be discussing my fear of flying with my therapist in 2024?”

Experts

Consider only actually asking an expert on ¼ of the questions you generate, chosen randomly, to keep your accuracy incentive and get feedback on your predictions, but minimise the cost of consulting experts or mentors.

“Will my mentor agree that pivoting now was the right choice?”

“Will the rest of the team prefer this redesign to the current layout?”

“Will anyone on the animal advocacy forum share evidence that convinces me that abolitionist protests are net-beneficial for the movement?”

“On December 1st, will Marco, Dawn, and Tina all agree that the biosecurity bill passed without amendments that removed its teeth?”

Crowds

“Will >80% of my Twitter followers agree that I should keep the beard?”

“Will a Manifold market on this question trade at >90% YES in two weeks?”

“Will social media posts about our product be more positive than negative in the first month after launch?”

“If I survey 40 random Americans through Mechanical Turk, will our current favourite name be the most popular?”

Show HN: Fatebook – Superforecasting in your Slack

https://fatebook.io/for-slack

Hey HN! I’ve been building Fatebook for the past couple of months. It’s a slack bot to help your team make and track predictions, right where you work.I see forecasting as anti-bullshit technology:- It gives you truthseeking incentives- You communicate your uncertainty as a probability, which is way clearer (70% is better than “probably”)- You can aggregate forecasts to get wisdom of the crowd...

( Read More... | 1 comment | 3 points | on HN )