Launch HN: Captain (YC W26) – Automated RAG for Files
We also put up this demo site called “Ask PG’s Essays” which lets you ask/search the corpus of pg’s essays, to get a feel for how it works: https://pg.runcaptain.com. The RAG part of this took Captain about 3 minutes to set up.
Here are some sample prompts to get a feel for the experience:
“When do we do things that don't scale? When should we be more cautious?” https://pg.runcaptain.com/?q=When%20do%20we%20do%20things%20...
“Give me some advice, I'm fundraising” https://pg.runcaptain.com/?q=Give%20me%20some%20advice%2C%20...
“What are the biggest advantages of Lisp” https://pg.runcaptain.com/?q=what%20are%20the%20biggest%20ad...
A good production RAG pipeline takes substantial effort to build, especially for file workloads. You have to handle ETL or text extraction, chunking, embedding, storage, search, re-ranking, inference, and often compliance and observability – all while optimizing for latency and reliability. It’s a lot to manage. grep works well in some cases, but for agents, semantic search provides significantly higher performance. Cursor uses both and reports 6.5%–23.5% accuracy gains from vector search over grep (https://cursor.com/blog/semsearch).
We’ve spent the past four years scaling RAG pipelines for companies, and Edgar’s work at Purdue’s NLP lab directly informed our chunking techniques. In conversations with dozens of engineers, we repeatedly saw DIY pipelines produce inconsistent results, even after weeks of tuning. Many teams lacked clarity on which retrieval strategies best fit their data.
We realized that a system to provision storage and embeddings, handle indexing, and continuously update pipelines to reflect the latest search techniques could remove the need for every team to rebuild RAG themselves. That idea became Captain.
In practice, one API call indexes URLs, cloud storage buckets, directories, or individual files. Under the hood, we’re converting everything to Markdown. For this, we’ve had good results with Gemini 3 Pro for images, Reducto for complex documents, and Extend for basic OCR. For embedding models, ‘gemini-embedding-001’ performed reasonably well at first, but we later switched to the Contextualized Embeddings from ‘voyage-context-3’. It produced more relevant results than even the newer Voyage 4 models because its chunk embeddings are encoded with awareness of the surrounding document context. We then applied Voyage’s ‘rerank-2.5’ as second-stage re-ranking, reducing 50 initial chunks to a final top 15 (configurable in Captain’s API). Dense embeddings are just half the picture and full-text search with RRF complete our hybrid retrieval. In the Captain API, these techniques are exposed through a single /query endpoint. Access controls can be configured via metadata filters, and page number citations are returned automatically.
The stack is constantly changing but the Captain API creates a standard interface for this. You can try Captain, 1 month for free, and build your own pipelines at https://runcaptain.com. We’re looking for candid feedback, especially anything that can make it more useful, and look forward to your comments!
How do you handle more structured data like csv/xlsx/json? Would be cool if it were possible to auto-process links to markdown (e.g. youtube, podcast, arbitrary websites, etc) a la https://github.com/steipete/summarize (which can pull full text in addition to summarizing).
Love the auto-process markdown idea, we'll add it to our roadmap :D
:O
If you want to check out the Query API response example, here's a link: https://docs.runcaptain.com/api-reference/query/collection-v...
I also appreciate transparent pricing but I am not 100% sure the sense of scale of costs. It could be helpful to give some ballparks on things for each of the plans. I'm not sure exactly what i could get out of a plan. My guess, trying hard to figure it out, was if i had about 1,000 pages of new/updated content per month, I would pay $295/month for unlimited queries on top of it. Is that roughly correct?
Advanced and Basic do play a difference though. Advanced is for complex graphics or charts in the documents submitted. Basic is sufficient for most document workloads.
can you expand on that?
For larger enterprises that require governance and additional compliance, we've been relying on trusted partners to help establish a connection to Captain
I like how clean and compressed the info is
The most similar product I've seen is Vertex File Search. They're hosted inside of GCP which can fit nicely into existing cloud deployments. Captain indexes from more sources (like R2 for example) and anecdotally provides faster indexing.
Onyx, Sana, and Glean are closer to application-layer enterprise AI products. Their internal knowledge assistants can search across SaaS tools but the interface is more graphical and seats are purchased as end-user software.
Captain sits in between because it's an API-first retrieval system to fully-manage file workloads. This adds search capabilities to existing AI agents but the agents are managed by the developers, outside of Captain.
Kore.ai however is more of an agent platform. Their focus is building and orchestrating agent workflows (which can include document retrieval, but that's not their main focus).
How do you compare to kapa.ai ? I have tried them.. and on search quality, they are really impressive.
The cost is insane but its dumb simple
Look at Tobi vibe-coding QMD, he's not a full-time engineer and vibed that up and now it's used as the defacto RAG engine for OpenClaw.
I spent the last two days building this exact thing for our internal use.
Managed to get a full RAG pipeline integrated and running with all of our company documents in less than two days work.
Chunking, embedding and querying, connected to S3 and Google Drive, and running on our own hardware (and scaling on AWS too if needed).
If you know what Captain is, this is not an issue. I closed the browser tab at first, thinking "what the hell is this, I don't give a damn about shipping forecasts"
2. It seems like it tries to emit citations, but doesn't emit proper links and instead just wrote [filename].
> one of the most common pieces of advice Y Combinator gives to startups [153_do_things_that_dont_scale.pdf].
Yeah good catch on the demo. If this were a production deployment, the citations would be hyperlinked to object storage. Captain is just the index, so the real files would be wherever they were indexed from.
The problem that Captain really addresses comes when production pipelines need to run continuously over large file corpora with fast, incremental indexing, and reliable latency. The maintenance required in these situations is often quite significant.
Captain focuses specifically on making sure the retrieval layer can operate smoothly so folks don't have to scale & maintain the infrastructure themselves.
No disputation of the core idea, I think you are on the right track, but the pitch isn't compelling. People looking for these kinds of AI solutions tend to favor simplicity and ~80% is fine, because the overall perceived productivity improvement is 5-10x, with such wide error bars that the approximate gain is just not worth maximizing for right now.
You might be a few months-years early, or target people who have maxxed out because they cannot retrieve from their second brain effectively. Most folks I've talked to are just trying to keep up, optimization/efficiency is not on their radar.