News for hackers. Stuff that matters.
Re:Consensus: Use AI to find insights in research papers by didibus on Thursday August 17, 2023 @03:44PM (#37166937)
I've been doing this with chatGPT directly. But it's cutoff data is 2021. It's pretty good though at telling you even the research name so you can validate.
Re:Consensus: Use AI to find insights in research papers by SamBam on Thursday August 17, 2023 @06:32PM (#37169104)
As someone who is starting a master's in science and will be looking at lots of research papers, I've been wondering what the best use of this could be.

If I have my own PDFs, I guess I could get ChatGPT to create summaries in some structured way, perhaps in a single file with citation:summary, and then send up that file with every question I ask?

Re:Consensus: Use AI to find insights in research papers by bee_rider on Thursday August 17, 2023 @11:04PM (#37170986)
FWIW is is pretty common to read the abstract, skim the intro, skip to the conclusion/results, and then only then read the rest of the paper (if you even need to at that point).

https://www.science.org/content/article/how-seriously-read-s...

So, this is pretty quick, you might not need a tool for it.

Re:Consensus: Use AI to find insights in research papers by SamBam on Thursday August 17, 2023 @11:13PM (#37171051)
Actually, what I was asking about was how to handle my storage and later retrieval of hundreds of papers. I don't mean just the database tool -- I currently use Zotero and make sure articles are tagged -- but I'm wondering if I can build a system for my articles where I can ask ChatGPT "are there any studies that show X causes Y" and have it use my articles directly, and not give me bullshit citations like it does 90% of the time now.

(To be clear, I'm pleasantly surprised when it does give me some real, useful citations of articles I didn't know about, but I'd like to get it's accuracy way up, at least for articles I have in my possession that I could feed it for context.)

Re:Consensus: Use AI to find insights in research papers by llmllmllm on Thursday August 17, 2023 @11:38PM (#37171169)
As part of my testing of https://flowch.ai (which we are developing), I've been uploading multiple PDFs (including scientific papers). I then ask a question and ask it to cite its sources aspart of the prompt and it'll give the PDFs it's using for the data / quotes it's using. Works well!

It's currently free to use :)

Re:Consensus: Use AI to find insights in research papers by chaxor on Friday August 18, 2023 @12:38AM (#37171455)
First you just extract all of the text from the PDFs. It's helpful to use arxiv instead, since it gives you latex. Then you can parse to separate content from style. Then store that in a a DuckDB database, with zstd compression since it's great on text. Then, just use some encoder model (decoder models are all the rage but they're not necessary here) to process all of these texts into Qdrant.

Then use your Qdrant database happily ever after, with whatever you like, such as Vicuna or Guanaco 30b GPTQ. Sprinkle your langcgain or whatever you like. Galpaca 30b may be appropriate for science text.

Re:Consensus: Use AI to find insights in research papers by jimmySixDOF on Friday August 18, 2023 @03:12AM (#37172269)
Litmap is a good surfacer of related papers if you have a starting seed to work from

https://www.litmaps.com/

Re:Consensus: Use AI to find insights in research papers by dragonwriter on Friday August 18, 2023 @12:45AM (#37171479)
> If I have my own PDFs, I guess I could get ChatGPT to create summaries in some structured way, perhaps in a single file with citation:summary, and then send up that file with every question I ask?

Extract the text, put it into a database (a vectordb would be the hot thing for with an LLM, is probably ideal, but ChatGPT does a pretty good job using Wikipedia as a “database” with a dirt simple ReAct pattern implementation, so you probably don’t need to be that fancy to get value out of it) and then use tooling to let ChatGPT use that database for questions.

Re:Consensus: Use AI to find insights in research papers by ant6n on Friday August 18, 2023 @04:52AM (#37172946)
Some article today (in spiegel) suggested these tools for pdf upload and various levels of support when extracting information from papers and writing new papers:

* https://jenni.ai/

* https://www.explainpaper.com/

* https://www.chatpdf.com/

* https://typeset.io/

Do any of you have any experience with these tools? Jenni ai seems interesting for doing research/thesis style work.

Re:Consensus: Use AI to find insights in research papers by Terretta on Friday August 18, 2023 @08:11AM (#37174571)
> jenni ai seems interesting for doing research/thesis style work.

“Join the Jenni influencer program and earn money for your posts. Earn up to $5,000 per post. We'll send you $1 for every 1,000 views your post receives. Minimum payout $20 (20k views).” — https://jenni.ai/influencer-program

Sounds very interesting to say it sounds interesting.

Re:Consensus: Use AI to find insights in research papers by ant6n on Friday August 18, 2023 @08:22AM (#37174662)
I think I would even be interested in what somebody has to say here about these services, even if they get paid for it.