cyclecycle

yeah i work on this at https://groundedai.company/

we build a product called Veracity focused on citation and claim checking. legal is a major focus for us, also scientific and technical publishing, where we detect hallucinated citations in submitted works and all that jazz

planning to release something soon focused on the LLM harness layer

we've been working on the problem for years. gradually getting traction

all completely closed source atm, though we often revisit that discussion

i would contest a sibling comment - the solution is not RAG, at least not entirely. 1) because it doesn't necessarily involve 'generation' (e.g our grounding model doesn't depend on gen ai) and 2) RAG is just enriching the context. even if you can get the retrieval right (not a given) it doesn't guarentee that the LLM will faithfully or correctly use that context, or that all its claims are supported by sources. RAG is not the same as traceability and verification

Resonates exactly with my experience.

I'll add they also seem to make 'bad' judgement calls around which layer to implement something, often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.

They also duplicate code lots rather than finding and reusing existing.

Both of these patterns lead to degradation and time wasting in complex code bases. Part of the behavioural adaptation I've made in using these tools is heading these off as much as possible at prompt-time

My prompts are often verbose to several paragraphs

So far we basically just provide a very rule-based approach and try not use LLMs as much as possible. So we extract and parse the citations using various ML and rule-based approaches, and carry out a bunch of predetermined queries and do various fuzzy matching approaches on the metadata components, and have a bunch of rules around risk levels of things we should have found/matched based on what type of source it is, which venue we should have found it in, etc.

So there are absolutely a bunch of tasks that could be evaled/benchmarked, but "hallucination rate" isn't particularly applicable/interesting as a metric of how good the tool is

that said, we do use various LLMs (mostly local, fine-tuned, small, for things like NER/parsing/metadata comparison, etc.). and they can and do hallucinate, but we have very hard constraints on the validation, so any extraction results that don't match 1:1 back to the input text are discarded for example. so again, rather than hallucination risk we prefer hard constraints

This has become such a problem in scholarly publishing that we have a business that provides citation checking https://groundedai.company/ that we've been buidling for a couple of years now

Great to see this picked up.

Nick Morley from Grounded AI here (https://groundedai.company)

We ran the analysis in collaboration with Nature :)

Nick Morley from Grounded AI here (https://groundedai.company)

We collaborated with Nature here to study the extent of fake/frankenstein citations in scholarly literature (from top 5 publishers - Springer, Elsevier, Wiley, Sage, Taylor & Francis)

We're estimating hundreds of thousands of papers affected in 2025 with hallucinated citation issues

As part of the work we analysed 20k papers generated with ChatGPT API to figure out which citation errors are characteristic of gen AI use and use that classify the errors we saw in the wild.

The world's gone mad, publishing is in a nuts state, the training data is poisoned!

yeah this makes sense. we run a citation verification service and provide publishers with data of hey this citation could be fake etc. but we don't currently capture any "action" or "measured result" so i guess that's what we need to expand to next

Trying to think about our best chance of making a lasting contribution (therefore actually surviving?) as a startup.

I'm a big fan of focusing on verification tools ("Verification is all you need"?) but who knows.

I'm sure I'm missing something in my perspective. Here to expand my aperture

A classic case.

I work on Veracity https://groundedai.company/veracity/ which does citation checking for academic publishers. I see stuff like this all the time in paper submissions. Publishers are inundated

That's basically what we're doing with app.studyrecon.ai.

What we've found is that vector similarity is often not the final solution. It is still only a crude proxy for the true goal of 'informativeness' or 'usefulness' with relation to the user goal/query. Works okay, but we're definitely seeing a need for more rigorous LLM-postprocessing to enrich the results set.

Which, yes, the time adds up quick!

Here's my attempt at a simple explanation of transformers. I would love feedback on whether I've got it right and how I could improve it. Cheers

Some personal reflections on the direction of my work.

We're working on this at Grounded AI (https://www.groundedai.company/contact-us). We'd love to help you if we can. Feel free to contact me (email is on my profile page)

Really like the idea of trying to improve things that are so widely adopted and entrenched that most (me, at least) don't think to change.

We face the problem that people differ in what they think the features should be though. Ideally we would have a method to deduce what's the best symbolism for maximising input/output speed to human's minds. Some kind of scientific voodoo.

Beyond me what that might be.

I think its clear enough as is, and that the readme benefits from its current simplicity. Though for interest's sake this wouldn't hurt.

Nice work. I'd be interested to know how your customers are finding you. Have you invested much time into customer acquisition?

There are philosophical differences and practical differences. They are trying to build an AI scientist ground-up (love it). I would like to optimise for front-line use cases in a lean, user-driven manner. The common denominator technologically is knowledge extraction (text to subject-predicate-object triples), ontological mapping (e.g, these relationships express the same thing, these references are synonyms of this compound, etc.), and reasoning (comes free once you have information properly extracted and mapped with a schema in place thanks to knowledge graph implementations such as GraKn).

Practically speaking: Aristo takes questions in unstructured text, and answers in unstructured text. I'm interested in providing mechanistic queries and comprehensive, highly-structured result sets.

For a question such as, "what are the biological consequences of increasing the activity of molecule A", I want tabular and filterable results (where the number of rows depends on the volume of underlying data and the degrees of separation you carry the inference to). For this reason (alongside their current limitation to elementary science) I argue that Aristo is not currently a relevant resource for researchers and students looking to query and survey biomedical relationships.

The solution I'm aiming at takes a structured query and returns structured results. E.g, a query: [entity: molecule A, direction: increase] generates a list of direct and inferred consequences. It is more like a logic-driven search engine over structured information than it is a question answering system.