skorniienko

Hi, one fellow HN commenter suggested to use a credible list of sources in my AI fact-checker skills. I promised to test and publish the results.

Test failed, and what I've found is interesting - as of 47 tested news outlet sources - 16 blocked AI requests with 400 error. (Reuters, WSJ, The New York Times, The Guardian, etc)

Blocked content still arriving from news aggregators, but model always prefer the source over any citation. Sometimes, data from aggregators were even different to original post.

If you have experienced this and found a solution - I'd be thrilled to hear about it. As for now - it's a gap I acknowledge and cannot overcome.

Hi, ran the experiment with "credible sources". Results we unexpected but not less interesting. Few things worth noting: - same search query ran minutes apart return different sources, sometimes with only 1-2 overlaps. - 16 high-credibility news outlets are rejecting agent's access to a content. (like Reuters, NYT, Guardian), but all .gov, dedicated fact-checkers are working.

Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...

exactly, that is one of the use cases I see for these skills

What if you use OpenAI as a judge and check both responses? Not defending Gemini, just curious if it was that wrong, or Fable - that aggressive.

Added this as issue so I don't forget https://github.com/SerhiiKorniienko/bullshit-detector/issues...

Filed both issues from this: https://github.com/SerhiiKorniienko/bullshit-detector/issues... - for syndication https://github.com/SerhiiKorniienko/bullshit-detector/issues... - empirical sources

Not sure that I get your idea about short-form content. It already works with tiktok or yt shorts, especially if it has generated transcription/captions. Otherwise, skill will download video locally and ask your permission to run local models like whisper to transcribe video.

Checking before you publish - that's interesting usecase I haven't thought about. Nothing stops you to fact-check your own content and decide what to do with it. Point is a report itself not more video content from it

can't, at least with this design. Report itself takes minutes to proof-check. Realistic version is some kind of extension/plugin idea from earlier in a thread, but I'm not there yet

True, and it doesn't do logic. It decomposes content into claims and search for every claim citations. It catches a false premise, not a bad inference. Someone with valid reasoning from true facts to a wrong conclusion goes straight through.

Empirical sources - would be a first candidate to be implemented in a skill. And you are right about 10 credible sources are actually 2. It will look like 10 independent sources to it, real gap. Skill does prefer primary source over secondary. For example it'll prefer NASA article over blog post who cited NASA article.

I'll work on empirical sources improvement in a next releases.

And you are not alone, that is one of the purposes why I am building this

but have Claude actually fact-checked it or just provided an opinion?

Fantastic! Dying to hear the feedback. <3

thank you! comment "prompts" never works, right? Haven't seen single guide from it :D

Done, fixed 2 misleading lines in README. You can see example report in repo examples.

Thanks for that, btw! At least someone questioned it :)

why it can't be vibe-coded? Would you be more pleased if it would be poorly hand-coded and claimed "No AI used during development"? What's the point?

I don't mind that I've used AI to build this, and I'm open about it ;)

Done, fixed in v0.4.1. Reason why agent reads from a file - it can do it in chunks or pass the path to subagent and keep it out from own context. I've tested it on 3hrs long interview YT video - worked smoothly.

I'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something.

Will read reliable-sources page properly.

Love it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!

maybe some community-driven and curated source blacklist could be implemented. It thing it would be too much for you doing that alone :)

You are right, no excuses. Will have few grammar errors but I'll own them. Cheers!

At the moment - it's model's judgement upon search results for every claim from provided content. If search results returned inaccurate data, or were fabricated - it might affect a verdict for that claim, or highlight that it's unverifiable or contradictory.

But there is a room for improvement, what do you suggest? Have a Judge agent which will check results?

That's fair. Install is a one time real friction and extension would be much simpler.

But, there is a reason why it's not an extension. At least for now. Whole product isn't a UI or service, it's a set of skills for your AI agent of choice, which gets content from URL or whatever shared as a source, splits it into claims and does extensive web searches for every claim to compare it with statement from provided content.

On local models constraint isn't a model itself - but search. Model don't judge from it's trained memory, so even local model will need a backend for search, otherwise it can't provide a verdict.

There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.

Thank you! Would love to hear your results if you'll have a chance to try it

Fair, drafted posts with claude, getting all my comments flagged. Thought that something is wrong with my account. Writing myself from here