Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by ocolegro on Wednesday June 26, 2024 @11:58AM
(#40801456)
All the pipelines are async, so for ingestion we have typically seen that R2R can saturate the vector db or embedding provider. We don't yet have backpressure so it is up to the client to rate limit.
Ingestion is pretty straightforward, you can call R2R directly or use the client-server interface to pass the html files in directly to the ingest_files endpoint (https://r2r-docs.sciphi.ai/api-reference/endpoint/ingest_fil...).
The data parsers are all fairly simple and easy to customize. Right now we use bs4 for handling HTML but have been considering other approaches.
What specific features around ingestion have you found lacking?
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by davedx on Wednesday June 26, 2024 @03:39PM
(#40803766)
Thanks, I’ll give it a try!
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by vintagedave on Wednesday June 26, 2024 @11:55AM
(#40801435)
I'd like to know this too. A quick: "take these docs as input, ingest and save, now sit there providing an API to get results" service guide.
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by ocolegro on Wednesday June 26, 2024 @12:13PM
(#40801592)
Take a look here - https://r2r-docs.sciphi.ai/quickstart#ingest-data and here https://r2r-docs.sciphi.ai/cookbooks/client-server#ingest-do...
Since multiple people have requested we are pushing a quick change to make this emphasized in the docs.
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by vintagedave on Wednesday June 26, 2024 @12:20PM
(#40801650)
Thankyou. My own comment giving a quickstart scenario was downvoted :( https://news.ycombinator.com/item?id=40801453 but I saw you kindly replied to it! Thankyou, I appreciate that.
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by namanyayg on Wednesday June 26, 2024 @11:24AM
(#40801111)
What do you want to do with the data after ingesting?
Re:Show HN: R2R V2 – A open source RAG engine with prod feat...
by shepardrtc on Wednesday June 26, 2024 @11:36AM
(#40801245)
LlamaIndex can ingest directories if you want to do bulk.
How do I do a bulk/batch ingest of say, 10k html documents into this system?